java -Xmx1000m -jar CDGroup.jar ... (to give the program 1000MB of ram to work with for example).
Note this is partly why multiple passes are not recommended, nor is an initial pass of anything other than 1. They eat ram when given a large enough file and / or small enough piece size. The standard settings should allow most people to merge most common file sizes (given 512MiB of ram it should easily handle full dual layer DVD sized images). Multiple passes will probably be removed. Initial pass may be kept in for experimentation.
How multiple passes works (and why you shouldn't use it)
The binary result from one pass is the input for the next. The only benefit (compared to changing the initial pass and keeping passes at 1) is decreased overhead (the size of the combined gsm files will be smaller). But multiple passes has disadvantages that far outweigh it:
- After each pass, sectors are more 'mixed' than they were before
- Maximum concurrent ram usage doubles wrt the biggest file after each pass
- The same data is being grouped again and again in a nested fashion
Taking 2048 as an example, for each successive pass the piece size is halved so the number of pieces the file is split into has doubled. The program ran out of memory at pass '256', where maximum concurrent ram usage would be roughly 8*25 = 200MiB per GiB of the maximum file size. Using passes instead of initialpass makes this worse, as a large file could have been made earlier, increasing concurrent ram usage further.
Freearc testing
As for your freearc testing, the only reason I can see for the result is that the repeated sectors are pulled out of where they are and placed in gs0. This may mess with a compressor as it could put unrelated sectors close together (although the sectors are ordered as naturally as possible within the gs0). Something else to consider is that as the images are small, freearc may have been able to see matching sectors from two images next to each other. It seems unlikely that the freearc performance would scale to bigger images (like full DVD images for example).
There are some things I can see to improve compressibility (or at least the chances of improved compressibility):
- Instead of pulling repeated sectors out to gs0, remove them from everywhere except the first location they are present
- Swap the name with the extension in the resulting files. I know freearc can order files any which way, but t7z orders them by extension, which is possibly the worst way to order them with the files named as they currently are (this would only affect merges with CDs with multiple sector types, which currently are compressed in the order 2048.gs0, 2324.gs0, 2048.gs1, 2324.gs1 ...)
- Store gsm file in a different manner (gsm is the overhead when merging). There's a way to store the gsm such that it is the same size and does the same thing, but is more compressible
Anything else that anyone thinks can improve compressibility let me know here or in pm.
