redump_info tool WIP

[superg]

GitHub: https://github.com/superg/redump_info
Download: https://github.com/superg/redump_info/r … 0505pre.7z (Windows 64-bit release binaries)

Basically working on my PSX dumps and verification submissions I realized that I spend too much time on routine things like running psxt001z and edccchk, libcrypt mumbo jumbo, double check IsoBuster date, exe names and so on, if you dump PSX you know what I am talking about. So some time ago I started writing a batch tool which will simplify as much as it can to make my life easier. At this point I've been using it for a few weeks and I can say that the basic functionality is there and all is left is to make the command line more user friendly before it can be used by a wider audience.

So what it is and how that works:
Let's say you dumped a lot of PSX discs using DIC and now you have to come up with the submission information for every dump. If you used DICUI you have !submissionInfo.txt file you have to fill and that will be used to submit a new disc or a verification. Or you can just run the tool in the directory with your dumps (or one dump, doesn't matter), it recursively scans all the dumps and creates !submissionInfo_<GameName>.txt per game with all the information by analyzing the image files and getting as much information from redump DAT file. The only things that aren't filled are barcode and ring codes. It doesn't depend on DICUI at all and it's detached from DIC output as much as possible (*_disc.txt file is needed to get write offset and *.sub is needed for PAL libcrypt games). In fact it will generate a submission info file using just *.cue and *.bin file set.

Features:
1. redump DAT lookup, useful for verifications or just to identify some random dump, will fetch maximum information it can from the DAT
2. extract exe filename / serial / local date from SYSTEM.CNF, that is performed on the image file by traversing ISO9660 filesystem, it will not hang like psxt001z for the games where exe is named differently than retail game. No IsoBuster is needed as the date is coming directly from ISO directory record.
3. errors count calculation, that includes subheader warnings count checked by edccchk. Simply you don't have to run edccchk as you will get the final number to enter into redump
4. EDC is always determined properly, no hex viewing and psxt001z is needed.
5. Anti-modchip string search is performed on ISO9660 file level, that saves a lot of time because you don't need to dump with DIC "/am" option which saves one DIC full pass. Also because it's performed on a file level it's able to find more comparing to DIC - currently DIC is unable to find strings across sector boundary so we should have more anti-modchip games in redump than we currently have (I already found some)
6. libcrypt is detected by analyzing *.sub file, the implementation is so much simpler then psxt001z. Output format is as close to redump as I could do, SBI file is generated as well (all SubQ's with incorrect CRC).
7. PVD, DAT info and other stuff is done exactly like DICUI, I calculate all the checksums myself and it doesn't depend on DIC

All this is really a bigger framework than just this tool, I can mass process a lot of roms and get some statistics, like exe filenames, anti-mod games etc, anything I need. I will make the source available as soon as I clean up the things and figure out command line interface. If anybody wants to try it now, let me know and I can send the binary.

Basic usage:
redump_info submission --dat-file <path to redump DAT file> <path to process>

This is work in progress, I will add some proper examples a bit later.

[superg]

Disney Tarzan (Spain) autogenerated verification:

Code: Select all

Common Disc Info:
    Title: Disney Tarzan
    Foreign Title (Non-latin): (OPTIONAL)
    System: Sony PlayStation
    Media Type: CD-ROM
    Category: Games
    Region: Spain
    Languages: Klingon (CHANGE THIS)
    Disc Serial: SCES-01519

    Ringcode Information:
        Mastering Code (laser branded/etched): (REQUIRED, IF EXISTS)
        Mastering SID Code: (REQUIRED, IF EXISTS)
        Data-Side Mould SID Code: (REQUIRED, IF EXISTS)
        Label-Side Mould SID Code: (REQUIRED, IF EXISTS)
        Additional Mould: (REQUIRED, IF EXISTS)
        Toolstamp or Mastering Code (engraved/stamped): (REQUIRED, IF EXISTS)
    Barcode: (OPTIONAL)
    EXE/Build Date: 1999-10-06
    Error Count: 1
    Comments: (OPTIONAL)
    Contents: (OPTIONAL)

Version and Editions:
    Version: (REQUIRED, IF EXISTS)
    Edition/Release: Original (VERIFY THIS)

EDC:
    EDC: No

Extras:
    Primary Volume Descriptor (PVD):

0320 : 20 20 20 20 20 20 20 20  20 20 20 20 20 31 39 39                199
0330 : 39 31 30 30 36 31 37 30  38 33 31 38 36 00 30 30   9100617083186.00
0340 : 30 30 30 30 30 30 30 30  30 30 30 30 30 30 00 30   00000000000000.0
0350 : 30 30 30 30 30 30 30 30  30 30 30 30 30 30 30 00   000000000000000.
0360 : 30 30 30 30 30 30 30 30  30 30 30 30 30 30 30 30   0000000000000000
0370 : 00 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................

Copy Protection:
    Anti-modchip: Yes
SCES_015.19 @ 0x3ac88: EN
SCES_015.19 @ 0x3acdc: JP

    LibCrypt: Yes
14485    03:13:10    41 01 01 03 13 10 00 03 53 10 50 ec    SEC    8001 068d
14490    03:13:15    41 01 01 43 11 15 00 01 13 15 23 1e    MIN    8001 338d
14579    03:14:29    41 01 01 03 12 09 00 03 14 2d 04 73    FRM    8001 c831
14584    03:14:34    41 01 01 03 1a 34 00 03 04 34 e2 cf    SEC    8001 8e30
14899    03:18:49    41 01 01 03 1e 49 00 03 08 49 32 c5    SEC    8001 8e30
14904    03:18:54    41 01 01 01 16 54 00 43 18 54 d4 79    MIN    8001 fd4f
15056    03:20:56    41 01 01 03 18 57 00 03 20 d6 bc 27    FRM    8001 bbd8
15061    03:20:61    41 01 01 03 38 61 00 03 24 61 91 a9    SEC    8001 79cd
15312    03:24:12    41 01 01 03 02 12 00 03 20 12 49 43    SEC    8001 79cd
15317    03:24:17    41 01 01 03 22 07 00 03 24 1f 3a b1    FRM    8001 0553
15919    03:32:19    41 01 01 03 30 59 00 03 32 1b 2c c6    FRM    8001 b12b
15924    03:32:24    41 01 01 03 20 24 00 03 3a 24 e6 ac    SEC    8001 132c
16101    03:34:51    41 01 01 01 32 51 00 43 34 51 d7 a9    MIN    8001 fd4f
16106    03:34:56    41 01 01 03 33 56 00 03 b4 56 c0 9a    SEC    8001 de39
16167    03:35:42    41 01 01 03 32 42 00 03 b5 42 69 e2    SEC    8001 de39
16172    03:35:47    41 01 01 03 33 07 00 03 35 45 1a 10    FRM    8001 b12b
42432    09:25:57    41 01 01 09 23 53 00 09 25 77 21 03    FRM    8001 2d65
42437    09:25:62    41 01 01 0b 23 62 00 49 25 62 68 4c    MIN    8001 fd4f
42580    09:27:55    41 01 01 0d 25 55 00 29 27 55 ae 41    MIN    8001 c701
42585    09:27:60    41 01 01 09 25 61 00 09 27 e0 e7 0e    FRM    8001 bbd8
42813    09:30:63    41 01 01 0b 28 63 00 49 30 63 ed 18    MIN    8001 fd4f
42818    09:30:68    41 01 01 09 29 68 00 09 b0 68 b0 8c    SEC    8001 de39
43012    09:33:37    41 01 01 29 31 37 00 0d 33 37 6c 68    MIN    8001 3237
43017    09:33:42    41 01 01 09 31 4a 00 09 33 52 7c 8b    FRM    8001 901d
43354    09:38:04    41 01 01 01 36 04 00 19 38 04 9c df    MIN    8001 50cf
43359    09:38:09    41 01 01 09 36 0b 00 09 38 49 6c 08    FRM    8001 8c46
43963    09:46:13    41 01 01 09 44 1b 00 09 46 03 78 0d    FRM    8001 901d
43968    09:46:18    41 01 01 09 46 18 00 09 06 18 25 99    SEC    8001 068d
44159    09:48:59    41 01 01 09 44 59 00 09 08 59 6e 0a    SEC    8001 068d
44164    09:48:64    41 01 01 49 46 64 00 0b 48 64 a4 60    MIN    8001 338d
44312    09:50:62    41 01 01 09 08 62 00 09 52 62 03 5a    SEC    8001 8c73
44317    09:50:67    41 01 01 19 48 67 00 01 50 67 70 a8    MIN    8001 1edb

Tracks and Write Offsets:
    DAT:

<rom name="Disney Tarzan (Spain) (Track 1).bin" size="485916144" crc="b44f6cb7" md5="c9fe6e4620e0dfc2b303505985aba18f" sha1="b478cb578f04bed72809f3498e84997387779777" />
<rom name="Disney Tarzan (Spain) (Track 2).bin" size="32104800" crc="9b94ea54" md5="65aea234c174ee35fb574d981fe3fc4f" sha1="bd1fde88c2b79e3cf8821c969373460d51f3155a" />

    Cuesheet:

FILE "Disney Tarzan (Spain) (Track 1).bin" BINARY
  TRACK 01 MODE2/2352
    INDEX 01 00:00:00
FILE "Disney Tarzan (Spain) (Track 2).bin" BINARY
  TRACK 02 AUDIO
    INDEX 00 00:00:00
    INDEX 01 00:02:00

    Write Offset: -647

[Qubits]

Looks great.
I think you should also get in contact with the author of DIC and DICUI.
I mean it would be ideal if your enhancements in finding antimod games etc. got integrated into the established tools.

[superg]

Looks great.
I think you should also get in contact with the author of DIC and DICUI.
I mean it would be ideal if your enhancements in finding antimod games etc. got integrated into the established tools.

IMO that doesn't belong to DIC, DIC is already doing too much work like creating rom xml and calculating checksums. As of DICUI it already has some functionality but it's all implemented in a C#. Right not whenever time permits I am cleaning and reorganizing code and it will appear on github in a form of "disc image information utility" with a capability to generate redump submissionInfo if needed.

[fuzzball]

I would love to use this tool. Thank you.

[superg]

I would love to use this tool. Thank you.

The windows build is in the first post, inside there is runme.bat script which shows basic usage. I am working on it to make it a part of a broader disc information utility. I need another week-two until I'm done with the basic implementation. Let me know if you have any questions / suggestions.

[fuzzball]

wrong error count in verifying https://redump.info/disc/41435/

redump_helper

Code: Select all

Error Count: 1

edccchk

Code: Select all

edccchk - CD image EDC/ECC Checker
v1.26 (32-bit Windows, MinGW, large file support)
  Copyright (C) 2013 Natalia Portillo
  http://www.claunia.com/
  based on ecm v1.03
  Copyright (C) 2002-2011 Neill Corlett
Checking psx_arkanoid2k.bin...
Subheader copies differ in mode 2 form 1 sector at address: 12:54:37
Non-data sectors........ 0
Mode 0 sectors.......... 0
        with errors..... 0
Mode 1 sectors.......... 0
        with errors..... 0
Mode 2 form 1 sectors... 34284
        with errors..... 0
        with warnings... 1
Mode 2 form 2 sectors... 37304
        with errors..... 0
        with warnings... 0
Total sectors........... 71588
Total errors............ 0
Total warnings.......... 1
Total errors+warnings... 1
Done

dic (img_EccEdc.txt)

Code: Select all

[INFO] Number of sector(s) where EDC doesn't exist: 37304
[WARNING] Number of sector(s) where mode2 form1 subheader(0x10 - 0x17) isn't same: 1
    Sector: 57937, 
Total errors: 0
Total warnings: 1

*edit*
CDMage has reported an error.

[superg]

wrong error count in verifying https://redump.info/disc/41435/

redump_helper

Code: Select all

Error Count: 1

This disc has 1 subheader copy difference, this is warning in edccchk tool. Redump's error count field expects to use "Total errors+warnings" number from edccchk. This entry has an error in a DB, should be 1.
I just replicate the behaviour of edccchk but I don't call it errors + warnings, just errors.

I made some improvements in command line but have yet to publish sources and build new binaries, it's now a single executable file without libraries.
It's nice that somebody is using the tool though

[fuzzball]

Redump's error count field expects to use "Total errors+warnings" number from edccchk.

Oops, Is that so?
I'm sorry that I am so forgetful.

[TonyLizard]

Actually the number from the database is wrong. I've fixed it, now you should see 1 error instead of 0.