Merged Compression Tests

[themabus]

- no need for storing by ImageDiff when merging with repetition filter is much more conveniant
- with splitting you can achive amazing results with cost of convenience...

dunno, to me ImageDiff results look most appealing
it would depend on decompression speed difference, which one of two i'd consider the best

splitting in parts adds another layer the same as ImageDiff and both speed and compression is worse
otherwise it's just too slow, imho.

when i said years i mean single game from such merged 7z+ecm+flac (or tak)+ImageDiff set
should decompress in about 2-3 minutes or so
it's most frequent scenario, nobody really needs whole set decompressed at once,
and still ImageDiff would be faster than 10 minutes, i guess.
so, very roughly:
let it be 20 minutes overhead on game
let it be 100 such games (or decompressions user commits)
and they're shared by 1000 people
100/3 = 33 hours = 1.4 days
1.4 * 1000 = years
sure they would download faster, but download is done passively in background
when in contrast decompression takes full PC load and needs interaction from user

[Rocknroms]

Weeks ago I have suggested to combine winrar+quickpar, here I explain better:

1) One main uploader compress everything with winrar the same way you'd make with torrentzip. I mean one archive for every game.

2) Then he makes simple pars for all archives (only .par2) with quickpar.

3) Now he shares the pars with other testers to see how many blocks are needed. As par will search blocks, even if set in random mode, only some blocks for every archive should have to be fixed/parred.

4) Once you'll have the diff blocks you can share them and have equal rar archives on every pc.

5) This has to be done only once, unless when a game name will be changed.

Obviously everything I said has to be tested. I hope I have explained it well.

[Rocknroms]

...moreover.

If someone makes a little tool that set date like packiso and we choose a winrar version that people must use to archive (you don't have to share it, and telling people "use v4.00" is not illegal) probably every archive will have only a few kbytes or less to be fixed.

[themabus]

3) Now he shares the pars with other testers to see how many blocks are needed.

about equal to archive size, i guess, which would be huge

[Haldrie]

Doesn't 7zip have a greater compression over RAR and wasn't there a purpose for keeping the tracks in separate files even when torrenting? What this is suggesting is going to kill everything that has been worked on with packIso, especially with keeping the tracks separate so people can get only the tracks they need without having to download the whole image or even seed the tracks they currently have while downloading the ones they may not. Not to mention all the current torrents that have been made using packIso (which is being accepted by the torrenting community from what I can see).

I'll admit that the packIso format may not be the best compression but it is at least much better then the current alternative torrentzip and still offers the ability to keep tracks separate and keep the files the same for sharing purposes. The time it takes to unpack a packIso archive is also a big help unlike what is being suggested here where if I wanted to say play a game or make a patch I would have to wait at least 30 minutes just to decompress it which wouldn't be very helpful at all for me even if I were to only access them very rarely.

As far as setting the date, that is what the rmdtrash.exe file does in packIso. It sets the dates on the files themselves before they are compressed so that all of the 7zip archives have the same date for each files on everyone's computer. It also clears the archive bit set by the OS in the file system to further clear out unwanted data for compression.

Everyone seems to be so worked up over using clrmamepro that you have forgotten the purpose of the archives to begin with and although I can understand why you would want to use it the way you are going about this seems to be a bit far fetched as far as actual usability for everyone. I've already mentioned that adding packIso checksums is all we would have to do in order to get a dat for packIsoed files while still maintaining  our original compression. As long as there is someone that has a copy of the game that matches the database then they can make a packIso archive and post the checksums of the files and in the case of the data track the checksum of the ecm'd file in the 7zip archive should be taken instead of the archive itself so that file name changes won't effect anything.

The compression percentage is not something that we need to worry about attaining nor should we worry about whether or not the archives are compatible with any emulator or front-end application. The purpose of the archives is just to store them in a way that makes it easy for everyone to get and decompress for use however they see fit after they get the archive and right now packIso is the only thing I can see that really offers that at this time.

[themabus]

Doesn't 7zip have a greater compression over RAR

rar would compress audio better, so it makes sense if all tracks go to single archive.

i honestly don't think that ability of torrentzip to produce identical archives is either something to worry about.
how many people use that really? 20 maybe, maybe less.
PackIso isn't bad, but surely if better compression ratio/speed combination can be found, why not?
(for instance: replacing ape with tak would improve it a lot)
it's really what matters - fast download/decompression, not ability to join in and seed - pretty much useless, imho.
what really happens is: somebody uploads a file and rest of the people fetch it and seed, that's all.
nobody cares about torrentzip.
so would somebody recompress everything - it wouldn't be a big deal, imho.

[cHrI8l3]

dunno, to me ImageDiff results look most appealing
it would depend on decompression speed difference, which one of two i'd consider the best

creating Diffs and restoring original game from those takes a lot of time and user effort, even more than compressing/decompressing
as I proved you can merge images ultra fast with FreeArc LZMA (merging 5gb into 380mb in about 10 min - is that long ?? and decompressing everything to its original state in 5~8 min)

right now in present moment of time merging with low-memory decompression needs manual splitting, however ... I wrote to the author of FreeArc, he is open to suggestions and theres a pretty good chance that he will include splitting feature into one of next FA releases
+ add to that ECM filtering automatically done inside FreeArc
and imagine... in near future it will be possible to merge with even less than 256mb of RAM and with just a single command-line or few clicks inside GUI

if you would have audio tracks stored as .wav you could configure FA to compress those for you with Ape format and .bin with LZMA
...and if you would not want to convert to .wav you could create archive with 2-steps:
1. create + add all data tracks with LZMA method
2. add to previously created archive all audio with Ape codec
isn't that convenient ?

and dont forget, merging right now is only experimental
I'm only trying to show some possibilities, maybe someone will try those for saving HDD space only only remember to test data after compression if you deal with unstable alpha versions of soft

Doesn't 7zip have a greater compression over RAR and wasn't there a purpose for keeping the tracks in separate files even when torrenting?

7-Zip does much better with data than RAR
RAR does much better with audio than 7-Zip
APE does even better with audio than RAR, because it is dedicated audio algorithm
APE is very fast and works with low-memory
thats basically it...

and when it comes to audio... imho Ape is a good format, it is fast, well known and supported by many other software, yes you can get better ratio with TAK... but so what ? you can get even better (+ faster) with LA

[themabus]

creating Diffs and restoring original game from those takes a lot of time and user effort, even more than compressing/decompressing

it would be strange, if so
- when decoding with ImageDiff it would only insert certain pieces of data - it should be really fast
there would be only one ECM file (Reed-Solomon calculation) and decompression algorithm (most complex)
would need to process far less data
- when extracting whole set from single archive ther's ECM for every file and decompression algorithm would process more data

imho it doesn't really matter what compression speed is as long as it's sane, since it's done only once

and when it comes to audio... imho Ape is a good format, it is fast, well known and supported by many other software, yes you can get better ratio with TAK... but so what ? you can get even better (+ faster) with LA

well from what i read, it's anything but fast and isn't that much supported either
FLAC is faster (about 2..4 times) and more widely used but compression is slightly worse (2..3%)

compression:
http://flac.sourceforge.net/comparison_all_ratio.html
http://synthetic-soul.co.uk/comparison/lossless/
hardware/software support:
http://wiki.hydrogenaudio.org/index.php … comparison
http://flac.sourceforge.net/comparison.html
http://en.wikipedia.org/wiki/Comparison_of_audio_codecs

TAK is experimental - true, but it offers compression of APE at the speed of FLAC

LA is very-very slow

[themabus]

oh, i see - if you'd extract single file from archive
for ImageDiff you'd need to run it through ECM and ImageDiff
but for FreeArc only ECM (and then join files)
but still decompression should be notably longer for FreeArc since it's 5.5gb vs 700mb
so what's lost on ImageDiff should be regained on decompression

[cHrI8l3]

LA is very-very slow

as far as Ive been testing its slightly faster than ape (using maximum possible settings) need to check that again...

but for FreeArc only ECM (and then join files)

Ive been trying to explain that FreeArc will be soon able to UnECM (+ Join) data automatically after decompression - so the only thing user will need to do is click on "Decompress" to restore original .bin's, without further processing files...

but still decompression should be notably longer for FreeArc since it's 5.5gb vs 700mb

unpack 5.2gb (8 files) from 380mb FA archive ~7 minutes
unpack 650mb from 350mb 7z ~80 seconds
unpack 8 x 650mb from 2gb 7z ... 8*80 =
...times checked on external usb drive, with internal SATA should be better...