Merge program for CD and DVD images

[jamjam]

CDGroup merges related disc images to reduce combined file size.

This program takes a directory of images, processing them to remove redundancies to save space. It does this in four stages:

• Stage 1: Split CD images (2352 byte/sector images) into their component parts. Model non-data parts where possible (ecc, edc, etc)
• Stage 2: Group each data sector size to remove repeated aligned data (de-duplicate sectors)
• Stage 3: Diff grouped files to remove repeated non-aligned data
• Stage 4: Compress resulting files

Notes:

• Stage 1 takes advantage of redundancies in the CD format
• Stage 2 takes advantage of aligned redunduncies across related images
• Stage 3 takes advantage of non-aligned redunduncies across related images
• Stage 4 takes advantage of uncompressed data in the images
• Each stage acts on the output of the previous stage

Features:

• Reduces storage footprint by de-duplication of sectors (unique sectors are only stored once)
• CDs, DVDs and Blu-Rays can be merged (any .iso with 2048 bytes/sector, and CDs .bin with 2352 bytes/sector)
• Smart handling of 2352 byte/sector CD images (recognises audio, M0, M1, M2, M2F1, M2F2 sector types. Removes non-data portions of sectors such as ecc if it can be regenerated losslessly)
• A mixture of 2352 and 2048 byte/sector images can be merged together
• Identical input images should result in identical merged files
• Images represented within the merged files can be renamed without unmerging (names stored in a text file *.hsa)
• Supports merging combined image size of up to 4TiB
• External diff support (jojodiff)
• External compressor / decompressor support (7z)
• Settings chosen to try and balance size reduction with practical time / hardware constraints

Usage: Requires java to run. Can be used as supplied in three ways:

• Use jar files directly
  java -jar CDGroup.jar dir_to_merge
  java -jar unCDGroup.jar hsa_file
• Use bat files (windows)
  CDGroup.bat dir_to_merge
  unCDGroup.bat hsa_file
• Use sh files (linux)
  CDGroup.sh dir_to_merge
  unCDGroup.sh hsa_file

Note: v0.4.x marks a new format, different to that of v0.3.x.

CDGroup v0.4.2

[Sotho Tal Ker]

Too bad it is java

[jamjam]

The tools are jar files, or jar files wrapped in exes. Both require java to be installed.[/strikethrough]

In my opinion having .net dependencies or similar is much more annoying. Java is near ubiquitous. What is it you don't like about java?

[tossEAC]

Jamjam, 1 Quick question, if you have two near identical images, but both contain large dummy files or false LBA, its obvios its going to match those sectors, but will it create a rather large compressable file somewhere of all the matching dummy data or FLBA.

[Nexy]

I am not understanding what this is for, a better description please?

[jamjam]

@tossEAC
In a single pass, the data is split into pieces. Any identical pieces are stored only once, with another file listing all the places the piece is duplicated. Large zeroed files will be reduced to a single piece with a large list of locations.

If by false lba you mean the file system describes files beyond the size limit of the image, this will have no effect as the file system is not read.

@Nexy
If you have many related images, you can merge them with this to save space. Many regional variants of the same game for example would be a good candidate for merging.

[jamjam]

New version: v0.2.10

Differences from v0.2.1:

• Better memory usage
• Basic support to extract from zip and 7z files when unmerging (does not handle compressed files containing directories)
• Partial sector matching option without having to do multiple passes (not used by default)
• Bug fixes
• Better description

Note: Default behaviour has changed. There is now only one pass by default instead of three. If you want partial sector matching, it is recommended to use partial matching without multiple passes.

CDGroup v0.2.10

[jamjam]

New version, v0.3.1

• New file type, GSA (stores metadata of original images)
• GSN files now generic, GSA is the only place actual titles are stored
• Improved 7z extraction, extract direct to required location

Note: The new file type means the merged files are slightly different to those made with v0.2.x. The new format makes the gsn and def files generic. Now you only need to edit the gsa file (and any cues/gdis if they exist) for renaming images, and an extra hash check is performed for validation.

This is probably the final format of the merged files. I can't see many reasons to change it from here.

CDGroup v0.3.1

[tossEAC]

Resident Evil 4

---------------------------------------------------------------------

8 ngc raw isos = 10.8 GB

8 ngc pakkiso'd = 6.38 GB = 4.42 GB saved over raw iso

8 ngc raw CDGroup'd = 4.28 GB = 2.1 GB saved over pakkiso

8 ngc compress'd CDGroup'd = 3.07 GB = 1.21 GB saved over raw CDGroup

---------------------------------------------------------------------

Resident Evil 4 (Europe) (En,Fr,De,Es,It) (Disc 1).iso:1459978240:22DB91D40629DF75E63A019E0F9D72F5

Resident Evil 4 (USA) (Disc 2).iso:1459978240:2381ACD2199D6E7566932DF86901903D

Resident Evil 4 (Germany) (En,Fr,De,Es,It) (Disc 1).iso:1459978240:290D0B60E8B4D53A05C478D91A3CCDA2

Resident Evil 4 (Europe) (En,Fr,De,Es,It) (Disc 2).iso:1459978240:41A0F650DD5A80DC8D0C268037FF2320

Biohazard 4 (Japan) (Disc 2).iso:1459978240:499CBEC401C8D80D35B521B215E8D039

Biohazard 4 (Japan) (Disc 1).iso:1459978240:6B57D0AE0B872A50457276FB84D58CF2

Resident Evil 4 (Germany) (En,Fr,De,Es,It) (Disc 2).iso:1459978240:7B6C469694E9BB14EFFEC5D0017E08DD

Resident Evil 4 (USA) (Disc 1).iso:1459978240:CA749757E3B9D119F3FEB1F9F0F81BD7

[tossEAC]

deletefiles=false (set in ini).

but its still deleting them when i merge.