Xbox1 random padding compression

[velocity37]

Any one know of any actuall programs that can split a 100mb or above into seperate 1 byte files numerically ordered

I really wouldn't suggest you do this. You would make that file 100MB file take up anywhere from 200GB to TBs. Explorer freezes up with tens of thousands, I'd be surprised if the OS didn't die in the tens of millions.

As r09 & amarok said, it is pretty much impossible to cut down random data.

I get what your idea is, to group all the same bytes together and compress them by describing them as a contiguous sequence. This isn't far from a real concept (RLE), but it doesn't work for bytes that aren't already grouped together, since it would take more than one byte to store the relative positions.

This is a byte:
o

If this byte was in a 7GB file, I'd have to name it something like this:
7421251637

This filename alone is 10 bytes (5 in hex).

There is a common way to store information about a large amount of data in a small amount, the checksum we all know and love. The checksum, however, can't be used to regenerate the data. A CRC32 is four bytes in length, so it can have ~4.3 billion values. If we had just one more file than its max, we'd 100% for sure run into an instance where two unique files had the same checksum (known as a collision). In fact, there are programs that can do this intentionally.

[tossEAC]

Thanks for sharing that nfo velocity37, you know more than me.

Im not that technically minded, but I am a fighter, and don't like giving in or loosing. So I persist, even if I am banging my head against a brick wall most of the time.

Just looking at that pic above, when I saw it 1st time, It made me feel sick. Why? Because looking at it, their is a hidden pattern. RANDOMNESS

If you think about it, they are using the same bytes just in a random order. I read somehere the gamecube garbage, is cipher something or other. In other words completely random.

In an ideal world we should make some small sized random file, using say every hex value, and have it shift randomly, like a blob of mercury, and at the same time have like another program like par2 that scans the random shifting file untill it finds a block then copies the block and carries on shifitng and finding more blocks.

FACT: (not much of one), but say quickpar's block size could be set to one byte. You can set it to 384,000 and below.

But thats stil 384 thousand times bigger than we want. The smallest size it goes on very small size files is 4 bytes


Take any 4 consequetive bytes "F3 28 A9 F7" and they only appear once. Take the next 4 consequetive bytes "4D 1B 95 1E" and they only appear once. And it would probably go on like that throughout the whole iso.

So theirs a clue in their pointing to the obvious it's as random as can be, but going back to my earlier point we need to make a small constantly shifting random file that kind of acts like a safe cracker.

If only par done 1 block par's, maybe that could be hacked, and it might do it as it may be able to but they left some of its possiblities out of the program for various reasons. Maybe on big files with 1 byte as its size, it would lock up syatems probably and thats why they left that option out!

Keep thinking we need to crack this, rather than forget it.

By the way I have tried converting a small samle about 4000 characters of the random data, to binary, it came out a lot bigger in binary, which I thought might compress to less, but it ended up bigger, and I have tried PNG

.

Would this work, converting hex to say base36?

F328A9F78C67ED651971BF74DCE989D5 - HEXADECIMAL

F328A9F78C67F0000000000000000000 - BASE 16

EE8OQPJ6BFKKWCGWGGOOOG0K0 - BASE 36

3631212476743063760000000000000000000000000 - OCTAL

[jamjam]

If the data is truly random, then by necessity no compressor will compress it. If you want to 'compress' it, the only hope there is is to find out how the random padding was generated, and emulate it if possible (severe longshot). One way I can think of where this might be possible is if they used a pseudo-random number generator (the entire sequence is determined by an initial 'seed' number), then brute force it. Maybe if microsoft has developed a random number generator you might have somewhere to start from, but anyway, good luck with that.

I have a few questions, just out of curiosity:
1/ How much padding would an average game have, does the amount vary from game to game?
2/ Would 2 copies of the same game have the same padding?
3/ Is the padding common to different games (maybe a particular developer has their own)?
4/ Is the entire padding for a game contiguous?
5/ Is the padding referenced in any way (maybe a hash check to ensure it's there), or is it properly just padding?

If padding is the same across many games, you've got the simplest answer. Distribute the games with the random padding zeroed, and distribute the padding seperately as a patch (which works for all the games with that padding). Couldn't use imageDiff, but a simple patch format could be made.

tossEAC, sorry man but the safe-cracker idea seems dubious. What would this constantly shifting random file do exactly? There's nothing to say that 4 consecutive bytes at location X can't be identical to 4 consecutive bytes at location Y, all we know about the data is that it is random. Hex is base 16, so there is something wrong with how you've converted the numbers, but what are you trying to do with converting between bases? Computers store data in base 2, there is no way around that. When you say that the binary ended up bigger, I'm guessing you are saving the codes in a text file? Each written character of binary then consumes 8 bits to describe (or 16 depending on format), instead of the intended 1 (and written hex takes up 8 or 16 bits instead of 4). Does that make sense?

[tossEAC]

This is all I've got time for now..

1/ How much padding would an average game have, does the amount vary from game to game?
2/ Would 2 copies of the same game have the same padding?
3/ Is the padding common to different games (maybe a particular developer has their own)?
4/ Is the entire padding for a game contiguous?
5/ Is the padding referenced in any way (maybe a hash check to ensure it's there), or is it properly just padding?

A1/ The smaller the actuall game data the more padding, so its exactly the same as the gamecube.
A2/ Yep I believe they do, two dumps that are equal have the same padding.
A3/ I haven't checked, as to check would be to make a 1:1 dump and the very difficulty wipe the game data, that could be something I might try though.
A4/ Dunno?
A5/ Dunno?


If padding is the same across many games, you've got the simplest answer.  Definately put I'm pretty sure its as random for each game, just as each individual game seems to have random padding from start to finish.

I did examine a section from the begggining of a padded part of the iso. I split of 999 parts 2048 bytes. I tried compressing using ape and they all came out bigger except two. I think it was probably those two half blank sectors, that contained some words of intrest.

IT SAID SOMTHING LIKE "MICROSOFT*XBOX*DVD" SHOULD HAVE MADE A NOTE OF IT. their was a bit more writing, I get the feeling it has something to do with the padding but it was just a header type bit of text like at the start of a dreamcast.bin file. But it was maybe 50 sectors from the start of the file.

You mods if you havent already seen this, go to the lba past the end of the video section, then sector view in isobuster, about 20-30 sectors and you'll see the MICROSOFT*XBOX*DVD text.

I have decided to give up finding better ways of compressing the random data.

The best compression I have found for the random padding is WinRAR store m0.

That only leaves one feasable option that would make us all jump for joy, would be to be able to generate it from scratch.

Wipe the padding first so it compresses then work out how to generate the exact data that was wiped, and hopefully the person in the process of doing this will spot what the random code is generated by.

[r09]

The "MICROSOFT*XBOX*DVD" thing is a header that marks the start of the file system, if I'm not mistaken. It's like the "CD001" header in ISO9660 compliant CDs.

[Ghazy]

nice thread and good thoughts. tossEAC's idea is great and sounds utopic, but this is the key.
jamjam suggestion is also good, it would be so awesome if someone could find out the algorithm for random padding.

i just doubt that both things are possible. i like this discussion. maybe someone has an even better suggestion. let's see.

in my opinion it's better to skip those data. who knows if they're useless because of copy protection?

[jamjam]

I've compared the compressed filesizes of some games released on ps2 and xbox, the difference is astounding. I can see why this is such an issue, and hope that someone can pull a miracle out of the bag. I guess larger games end up having a smaller filesize, because there's a lower % of random padding?

[iR0b0t]

I guess larger games end up having a smaller filesize

How is that meant?

[Jackal]

I'm not really in the mood to check all posts, but I just wanted to let you guys know that the mastering tool for Xbox discs is in the SDK, so maybe the algoritm for the padding can somehow be traced there.

[jamjam]

I guess larger games end up having a smaller filesize

How is that meant?

By that I mean that for games with more game data (as opposed to random padding), a higher % of the 7433027584 bytes an xbox image takes up will have a chance of being compressed well. I hope that makes sense.