Relation to tosec.org

[Eidolon]

There is no such thing as "intelligent checksums". There has to be a standard of how reference checksums are calculated before you can disregard the data offset. We prefer to take both the read and the write offset into account when determining the reference, allowing audio tracks (when saved using the standard) to have identical checksums across different regions/games and not just to look at the data integrity the way you are planning to do.

Yes, there has to be a standard. We propose the following one. Only do the checksum on the following audio data:
- starting at the first non-zero byte in the audio data section after the data track
- ending at the last non-zero byte in the audio data section before the end of the file

The positioning of this audio data block is different in the BIN files, depending on the audio offset of the drive. But, the checksum of the audio data block will be the same regardless of that audio offset.

In combination with the checksum of the data block, and the CUE sheet which lists the tracks, this allows for 100% verification of good (I would even call them perfect according to Redbook standard) SegaCD game dumps.

It makes me wonder if the benefit of speed will really be that great, because even a minor small scratch on any of your cd's will give you problems dumping and verifying them the 'fast' way.

But why is that? Doesn't that concern your own ripping method as well? If there really is a bad scratch in the audio data section of the disc, there is NO WAY to reproduce the original bytes in these particular sectors.

In our method, we simply rule that out by stating to rip a CD twice. If the resulting BIN files have no difference, this is proof that the drive had no problems in reading out the audio data at all and can do it reliably over and over again.
A badly scratched CD (of which I have one to confirm that) will always produce different results from the drive internal error correction mechanism, producing a different BIN file with every dump.

Sooner or later you will propably end up using EAC after all. Anyway, good luck with your projects.

The problem of EAC is that it can't cope with data tracks or extract them directly, and in the end you will end up with one file per track.
I love the simplicity of having one BIN file per CD, and that is difficult to achieve.

I would like to give that "PerfectRip" program a try though - I didn't find it on this site, and google results were inconclusive.

As for GoodGen, I like No-Intro's dat better, because it's more accurate. Then of course I'm not even talking about MAME and how they want to preserve the Sega Genesis roms (splitting the data into separate files exactly like they are stored on the actual rom chips). Most people also consider this 'pointless' while others don't (see the resemblance?).

The whole life is a constant stream of defining goals, weighing methods to achieve them, and adjusting goals to fit those methods...

[F1ReB4LL]

I would like to give that "PerfectRip" program a try though - I didn't find it on this site, and google results were inconclusive.

PerfectRip doesn't work on all the drives, IIRC only on _some_ Plextors...

[themabus]

Yes, there has to be a standard. We propose the following one. Only do the checksum on the following audio data:
- starting at the first non-zero byte in the audio data section after the data track
- ending at the last non-zero byte in the audio data section before the end of the file

The positioning of this audio data block is different in the BIN files, depending on the audio offset of the drive. But, the checksum of the audio data block will be the same regardless of that audio offset.

often it won't, you'd miss audio at the start of the firs track or the end of the last. e.g. [T-76044] Winning Post (J) has audio data till the very last byte of last track. if drive shifts that data to outside even by 1 sample crc will differ. and even if there is silence at the both ends of audio data it does not always compensate offset, you're right - drive offsets are pretty small most of the time but cd write offset can be up to several sectors large positive or negarive, so if you're not compensating it, in half of cases it will add with drive offset and produce even larger error. for your described method to work audio data should be enclosed in silence (both start and end) of maximum cd+drive offset sum but it's not.
for example: Lunar [T-45014] (cdoffset=+2072)
@theend (7104 zero bytes)
((cdoffset+driveoffset)*4)-slilence =>  8288+4x(driveoffset)-7104  => driveoffset=-296 samples <|larger than that and you cut off audio
@thebeginning (no silence)
((cdoffset+driveoffset)*4)+slilence => ((2072+driveoffset)*4)+0 => driveoffset=-2072 <|smalller than that and audio run into data track
only drives having offsets within this range would get similar audio segment crcs

[gigadeath]

Consquently, I've begun working on the GoodSegaCD and GoodSaturn projects on the Inn, hoping that this slightly easier method will be adapted by the Sega retrogaming community as a new defacto standard (similar to the GoodGen stuff).

Looking forward to hearing your thoughts on this!

Oh god that's the last thing we need... ANOTHER format? Moreover, another less accurate format? If you don't want to waste time just wait 'till there's click-it-once program. It's not like "oh cds are going to rot". I have CDs from 1985 that sound better than anything in SegaCD games.

And you forgot to mention the biggest difference between GoodGen/No-Intro and your project, they contain byte-perfect dumps too in first place. Every dump in GoodGen/No-Intro databases with missing/wrong bytes is tagged "hack","bad","overdump", "fixed" etc. for a reason, so you can't compare this "GoodSegaCD" to them, at all.

You're about to start a project that in a few years will be deemed full of bad dumps, think about it. It's not that you have to go that way just because Kega's author thinks obsolete shit like CDRWin works better (sorry if I sound rude)...

[gigadeath]

Another thing, even if you don't have the time to rip CDs the byte-perfect way, maybe someone in your community would have no problem doing it, IMO it's a way to force them to settle for an inferior dumping method, without letting them choose. You could tell your community members that there are alternatives like PSXDB who strive for a greater level of accuracy. After all cart dumps users are accustomed to perfect dumps, why would you force your community to renounce to them?

Finding offsets and dumping CDs with EAC takes me the same time of opening and dumping carts... 5-10 minutes of waiting haven't been a problem for the last 10 years of worldwide cart dumping, why should they suddenly become an unbearable wait now?

And it's not an ease of emulation matter either, PSXDB rips works perfectly under Daemon Tools and Kega

[Jackal]

I agree that there's no real point in switching to a poor man's way of dumping cd's just because the other one presumably takes a bit longer.. but I do like the 'smart checksumming' idea that you came up with (checking the data integrity of a cd or image on-the-fly by comparing blocks). However I agree with themabus that without read+write offset correction you will soon run into problems where the start and the end blocks of the audio will have missing data, thus making crc comparison impossible. Also, if I recall correctly, the usual cue/bin tools out there (not sure if this includes cdrwin) don't append the gaps correctly and skip the track02 pregap just like old EAC versions are doing, so the chance of missing samples will be even bigger.

As for scratches and not being able to dump audio correctly: EAC's error detection and rereading mechanism is pretty decent and helped me through a lot of scratched cd's that would be impossible to dump correctly with conventional tools like cdrwin. Also, you forgot that C2 can also be used to detect errors (PerfectRip uses C2 to check for corruption).

[chungy]

Some discs don't actually require purposely-altered EDC/ECC data, and just use normally constructed ones. In such cases (PC games usually), you can generate a 2352 byte/sector "dump" from a standard 2048 byte/sector image (I don't know of much that can do this... besides using a CD drive emulator like Daemon Tools or CDemu then using the dumping apps) that is identical to the normally-dumped method. In certain cases, this will never be possible (eg, PSX).

And the creators of the ISO 9660 standard had no concern in this matter. It's a filesystem and it wasn't their duty to dictate how CDs store information in the low-level manner, that was Philips' job (see ECMA 130 for details on the physical layout of CDs and the low-level 2352-byte sectors). You can also store ext2, FAT, UDF, any other filesystem on a CD, and it's of no concern to those filesystem authors what the low level CD format is.

(Technically speaking, redump.org DVD images are all wrong since they don't include raw DVD sectors, which is far more difficult to access and not all DVD drives do it in the same manner (essentially every vendor has their own proprietary commands); who's to say that non-chipped PS2s actually check data that's not in the 2048-byte user data area of a DVD sector?)

[Jackal]

Dear lord, as someone who's spent a decent amount of time archiving TOSEC dumps, this information is very disturbing.

If you diff'd a TOSEC ISO dump and a Redump dump, any idea how different the two files would be?  If the difference is very small then it should be possible for someone with a full set of both dumps to create patch files to convert individual TOSEC dumps to Redump dumps.  This way, TOSEC dumps could be merged to more accurate Redump dumps over time.

It's amazing that it is this complex to create 1:1 backups of CDs.  It's just astounding.  Did the original creators of the ISO 9660 standard never see this coming?  From a high level perspective, CDs just store bits- just read all the bits off of it! 

In short all data-only dumps like 3DO are easy to convert if you for instance mount the cuesheet in daemon tools and then raw extract.

The problem with adjusting the audio is that you'll have to know the write offset of the original disc. With systems like SNK Neo-Geo CD this is often pretty easy to detect from the image itself, because the audio data always tends to starts at a certain position in the track (for instance if the audio data would start 2 samples after the start it was safe to conclude that the write offset was +2, which is also a common PSX write offset).

Other discs with larger write offsets have a pretty big chance of having missing samples at the start or the end of the audio. When I helped a TOSEC guy to convert the SNK set to the Redump format once we came across several discs that needed audio redumped because there were samples cut off at the start or the end. I think other systems like Saturn will have similar issues. Of course it should be possible to create a patch, but it would only make sense just to convert TOSEC dumps to Redump ones for collecting purposes (and for saving bandwidth).

If a TOSEC dumper should care about the accuracy of his dumps, he could always come here and redump them using the better method. We will never convert/steal any dumps from other projects. I know that some of the TOSEC dumpers are aware of the differences, but they don't seem to care enough about them to start redumping. Maddog from TOSEC once gave me the same explanation as Eidolon did a couple posts before: who really cares about 1/75th of a second?.. I think this thread makes clear that we are the only project atm that DOES care.

[Jackal]

(Technically speaking, redump.org DVD images are all wrong since they don't include raw DVD sectors, which is far more difficult to access and not all DVD drives do it in the same manner (essentially every vendor has their own proprietary commands); who's to say that non-chipped PS2s actually check data that's not in the 2048-byte user data area of a DVD sector?)

You're right, RAW reading DVD's IS possible, but it's very difficult to accomplish: http://x226.org/?p=17 . I think the PS2 reads the DNAS ID in a special way (it should be in the user data area when extracting but it's not). Anyway, there's no point in including the DNAS ID either, because it can be injected after and the images can't be verified when it's included (I think).

If anyone has some info on extracting DVD's 2064 bytes/sector using custom firmware, and what the advantages are, plz let us know


Offtopic:

ps. after some google'ing I came across this thread: http://assemblergames.com/forums/showth … p?p=253548
I have a Datel PS2 cd here that has unreadable sectors in the same region, maybe it's possible to extract them in d8 and create a bootable copy

edit: I managed to extract the sectors and get the same patterns as the other guy, but according to this thread http://club.cdfreaks.com/f52/how-datel- … on-147005/ this data has no real purpose after all

[Eidolon]

Well, thanks for all the additional clarifications.

I now understand that it is really a problem that there is a chance to miss some audio data of the last track, and I think this is a flaw in Steve Snake's proposed method. It may work fine with 99% of all dumps, but would still produce errors in that 1% you gave examples of.

However, I still have a bad feeling about dumping RAW data (2352) from data tracks of systems (Sega CD, Saturn) which DEFINITELY do not make use of these additional 304 byte of information. What is the point? Those bytes are used for an "internal" (from the system's point of view) error correction algorithm to be able to fix sector errors. The user data (from the system's point of view) is just 2048 byte.

By ripping the RAW data from those data tracks and using that as the basis for the checksumming, you actually INCREASE the chance of producing unreliable dumps, because a 2352 byte block has less error correction than a 2048 byte block (spoken from the CD drive's perspective).

There is no such arguing in the audio section of CDs, because here the 2352 bytes definitely constitute the user data.