Merged Compression Tests

[themabus]

Ive been trying to explain that FreeArc will be soon able to UnECM (+ Join) data automatically after decompression - so the only thing user will need to do is click on "Decompress" to restore original .bin's, without further processing files...

ok, so it will be more convenient
but it's possible to make an frontend or a script for commandline applications - many do that now
there are quite a lot of programs (for cd recording, video/audio encoding/processing and so on)
that actually are only GUIs for command line *nix applications

unpack 5.2gb (8 files) from 380mb FA archive ~7 minutes
unpack 650mb from 350mb 7z ~80 seconds
unpack 8 x 650mb from 2gb 7z ... 8*80 =

but for your example it's 2nd case, right?
there wouldn't be 2gb archive, it would be 350mb.
so ImageDiff would have 5 minutes to complete in, and actually it shouldn't be much slower than joining files.
(it basically is joining of files)

edit:
so anyway such merged set (7z+ecm+ImageDiff+tak) seems very good idea to me
from example above it's 8 times compression improvement over PackIso
it would be rare of course to have so many versions merged
but generally it would still save a lot of time and space
(even single title decompression from this set on average shouldn't be slower than PackIso, i think,
because of speed gain from TAK and would TAK implement support for RAW audio data
sox.exe could be eliminated, as it's used now to add/remove RIFF header,
which means every audio track is basically copied after extraction,
which is about the same as doing ImageDiff)
kudos to cHrI8l3 for that

[cHrI8l3]

but for your example it's 2nd case, right?
there wouldn't be 2gb archive, it would be 350mb.
so ImageDiff would have 5 minutes to complete in, and actually it shouldn't be much slower than joining files.

I dont know if understanded your point correctly... example on which I based those decompression times is based on compression WITHOUT using ImageDiff (config "Arc 6" from base topic)

this is what that 380mb archive contains inside:


and those ~2gb 7z is total of 8 packiso archives ( 8*350mb )

ps. in a few days ill do merging test on game with audio tracks that may be interesting...

[themabus]

unpack 650mb from 350mb 7z ~80 seconds

is this FreeArc?

ok, so i guess, it's how fast single version from 378mb FreeArc would extract
could you test extraction of .ecm + one .imageDiff from 357mb .7z and 344mb .nz, please then
unecm would be the same, and ImagePatch would be slightly slower than joining, i guess

edit:
noooo, wait
ummm it says 7z

that's what i thought:
1st: FreeArc - prettyslow
2nd: 7z + ImageDiff - fast

i mean when you have files decoded from ImageDiff .7z you'd still beat FA by about 4-5 minutes
plus compression is better

[Sotho Tal Ker]

Haldrie mentioned it is easier to get just one track instead of all, when something is compressed with packiso.
But if you need a single audio track it is easier to get the whole archive than figuring out how to properly turn the ape file into the bin. Decompressing into wav is easy, but removing the RIFF header requires knowledge of the sox.exe commands.

[themabus]

current stats:

Code: Select all

"Sony PlayStation (2484) (2009-04-05 15-16-43).dat"
Records in DB total: 2484
Records with Audio : 774
------- Size -------
Total: 1298980571256
Data : 1169375041272
Audio: 129605529984

"Sega CD - Mega-CD (95) (2009-02-23 19-41-59).dat"
Records in DB total: 95
Records with Audio : 94
------- Size -------
Total: 43296314544
Data : 25283397888
Audio: 18012916656

audio makes only 10% of PSX data actually,
though it's off a little, since lately i submitted about 100 CDs without CDDA by selection
and others might be doing similar since CDDA dumping is such a drag
but anyway, for older consoles it's far more significant
so i guess extraction from PackIso might be slightly faster than from merged set on average for PSX

edit:
PackIso is locked @7z 4.53 which is unfortunate

Code: Select all

4.54 beta      2007-09-04
-------------------------
- Decompression speed was increased.

4.58 beta      2008-05-05
-------------------------
- Some speed optimizations.

and it uses default settings, hence producing less compression,
but as i understand it compression mode for 7z does not significantly influence decompression speed
so i don't know - it's difficult to tell without measuring

edit:
i mean extraction by single title
whole merged set should decode a lot faster
but it's unlikely scenario, imho

[cHrI8l3]

themabus, you have no idea about how those methods really work do you
no worries, lets start with imagediff, which if you used at least once you would know that its slower than simple decompression

unecm "Final Fantasy IX (E)(Disc 1 of 4)[SLES-02965].bin.ecm" ~75 sec
imadiff "Final Fantasy IX (E)(Disc 1 of 4)[SLES-02965].bin" into version "(F)(Disc 1 of 4)[SLES-02966]" ~5 min

imagediff is patching in 3 basic steps
1. calc md5 (hdd speed) to validate integrity of foreign image
2. create sector map of foreign image (also hdd speed)
3. rebuild image while comparing imagediff's sector map to sector map of foreign image (and this one is "slightly" 2-3x slower than just reading whole image)

it reads whole original image at least 3 times!

and now compare it to extracting ALL 8 files to their original form (without any patching) in 8 (eight) minutes from FreeArc archive (+ unecm on 8 files which is even slower than decompression = 8 *75 = ~10 minutes )

FreeArc is ultra fast because of its method:
- first decompress 380mb archive into repetition filters dictionary + non-common data (in this step you dont have 5gb un-merged set, only ~700mb dictionary + ~50mb of data)
- next repetition filter is working in a way which you though imagediff should work - rebuild files with common parts which are stored in dictionary and are loaded into RAM for fast access (and here is done the magic of un-merging into 5gb)
- rebuilded data is stored in cache and then written onto hdd

and as I already wrote from the beginning "no need for storing by ImageDiff when merging with repetition filter is much more conveniant"

[Sotho Tal Ker]

7zip 4.57 is the latest version that produces the same output as the 4.53beta included with packiso.
All later versions have a little bit changed.

[themabus]

unecm "Final Fantasy IX (E)(Disc 1 of 4)[SLES-02965].bin.ecm" ~75 sec
imadiff "Final Fantasy IX (E)(Disc 1 of 4)[SLES-02965].bin" into version "(F)(Disc 1 of 4)[SLES-02966]" ~5 min

are you sure it's 5 minutes?
i have whole PCE set on ImageDiff - it's always a matter of seconds on my machine
those are small ImageDiffs, however - rarely larger than 10 megabytes

md5 on 600mb file (fsum.exe) completes in ~20 seconds

if ImagePatch is slow it's implementation fault then, as in theory, it should be faster than unecm
ecm has a map of sectors where to recreate ecc
all program like ImagePatch needs, imho, is map of sectors to insert and those sectors themselves
it's practically 'copy /b file1+file2 file3'

as you said yourself FreeArc use symmetric compression algorithm which 7z does not
i don't think it's ultrafast - it can never be faster than 7z
it can compress better or be more convinient
but same files should extract faster from 7z

yes i'm theorizing
i'm out of space now and basically can not check anything so i have to trust you

so time results you provided earlier

unpack 650mb from 350mb 7z ~80 seconds

what are they from?

i mean if it's single file (or two) extracted from 7z containing one .ecm and ImageDiffs,
whole archive should decompress only little slower as it is solid most likely

so what we have is .ecm and .ImageDiffs in 2 minutes
vs one ecm (or several) in 7 minutes

now if ImagePatch is bottleneck here it should be possible to replace it with less complex patcher and it should zap

or i don't understand something?

- rebuilded data is stored in cache and then written onto hdd

ok, maybe you think author will implement ecm and joining of files in FreeArc and everything will be done in RAM (5gb)
and then written to HDD so time on HDD access will be saved
but it's overkill imho - 5gb of RAM is not sane and probably same could be done then with ramdisk

i mean it's all good to have all those features in one program but decompression algorithm itself is way slower
so that's where it ends, imho

edit:
i don't mean that 7z is ultimate archiver btw.
if ther's one faster at same or better ratio (like TAK vs APE & FLAC) - by all means
i just don't think that FreeArc is the case unfortunately

[cHrI8l3]

as you said yourself FreeArc use symmetric compression algorithm which 7z does not

I never said that! quite opposite when you look at the config table you will see that fast, low-memory method from FreeArc use this same algorithm as 7z - LZMA !! and FA has faster implementation of LZMA than 7z
LZMA is assymetric
symmetric compressors are used in NanoZip

ImageDiff uses 2 methods for creating patches, Ive been using patches created with "best diff type" option and "slow match" for smallest patches

unpack 650mb from 350mb 7z ~80 seconds

unpack 1 versions of FF IX (in .ecm format) from 1 archive (7z) in 80 seconds (no ImageDiffs here!!)

but it's overkill imho - 5gb of RAM is not sane and probably same could be done then with ramdisk

i mean it's all good to have all those features in one program but decompression algorithm itself is way slower
so that's where it ends, imho

man you should educate yourself or at least play some with those algorithms before make such statements

I know my english is not perfect, but I already wrote all informations which are needed to understand the whole idea
decompression tests might by confusing and the whole diffs, because I did not intended to make those in the first place
I will think about it and maybe add those unpacking times to table with results , perhaps that will be easier to understand ;/

[themabus]

ok, i'll clean up hdd, fetch those images and test for myself
asap

symmetric compressors are used in NanoZip

i thought it's the same program, like mode or something
so ther's 2?
what about records with FreeArc+NanoZip, then? are they compressed twice, like zip+arj?

edit:
i'm basing my statments on what you write
i've asked you about only comparable value in that table - whether it's slower, and you said it is
that 'fast' part you added later
and after that you wrote that FA extract in 7 minutes,
...about the same as compression value given ~10 (7 is about 10 - 7 approximates to 10) = symmetrical
would you have wroten  - FA extracts in 1 minute, i'd think it's fast (as 7z extracts in 2),
but you wrote - in seven

so what would be decompression speed difference of exactly same files 7z vs FA?
have you tested?

edit:

i don't mean that 7z is ultimate archiver btw.
if ther's one faster at same or better ratio (like TAK vs APE & FLAC) - by all means

so if FreeArc does that - it's great
if you can talk author into integrating ecm then - it's even better