a brief summary on PCE

[themabus]

i've checked so far 51 PC Engine CD, that's close to 10% from all there is, and i thought, this might be a good moment to summarize this information, since i guess 10% can be sufficient, to outline main characteristics of the whole set, and if something is wrong, it's still not to hard to go back at this point and redo and correct mistakes.

layout:
[s]only rule, i think, is that cd starts with an audio track. so far 2nd track always was data, but in necstasy, i think, i saw a CDs where this is not true either but still audio would always preceed data.[/s] this 1st track is the same most of the time or often contains same audio data with a different offset, but there are quite a lot exceptions.
because of this audio track, the one, that preceeds data, it's a bit more complicated to dump PCE CDs.

tracks:
when offset is corrected, audio starts most of the time with an hex offset 0x0 from LBA: 1st track, and 0x4000: rest. except some rare exceptions audio tracks don't have any empty sectors at the end. only padding zeroes to fill till the end of last sector.
data tracks are mode 1 always. and except only a handful cases, where post-gap for last data track was messed up, there wasn't anything unusual at all.

subcode:
i did 3 checks on subcode (PMA only):
r-w channels!=0x00
q-mode=2 and q-data!=0x00
index>1
nothing. it's only some random bit's set in one or more channels now and then. and most cancels out on several rereads. i don't think deliberately altered data can look that way. it's nothing like NeoGeo subcode, where you can clearly see large blocks of similar changes follow each other. though there were some PCE releases post y2k, i think, maybe those are different.

gaps:
this is, what makes this system difficult to dump currently, partly because of the way how EAC calculates them.

theory (ECMA-130):

Code: Select all

20 Track structure of the Information Area
The Information area shall contain Information Tracks in
-  the Lead-in Area;
-  the User Data Area;
-  the Lead-out Area.
The Lead-in Area shall contain only one Information Track called Lead-in Track. The Lead-out Area contains only
one Information Track called Lead-out Track.
The user data shall be recorded in the Information Tracks in the User Data Area. All Information Tracks containing
digital data shall be structured in Sectors.
For the purpose of linking Information Tracks in the Information Area, these tracks may have:
a) Pause : A part of an Information Track on which only control information but no user data is recorded.
b) Pre-gap : A first part of a Digital Data Track not containing user data and encoded as a Pause. It is divided
into two intervals:
- first interval: at least 75 Sections (at least 1 s) coded as the preceding track, i.e. the Control field
(see 22.3.1) of the q-channel (see 22.3) and, in case of a preceding Digital Data Track, the setting
of the Sector Mode byte are identical with those of the previous Information Track;
- second interval: at least 150 Sections (at least 2 s) in which the Control field of the q-channel and
the setting of the Sector Mode byte are identical with those of the part of the track where user
data is recorded. In this interval of the Pre-gap the data is structured in Sectors.
c) Post-gap : A last part of a Digital Data Track, not containing user data, and structured in Sectors. It has the
length of at least 150 Sections (at least 2 s). The setting of the Control field of the q-channel and
the setting of the Sector Mode byte are identical with those of the part of the track where the user
data is recorded.
20.1 Lead-in Area
The Lead-in Track is either a Digital Data Track or an Audio Track. If it is a Digital Data Track, it shall be
structured in Sectors and end with a Post-gap. If it is an Audio Track, it shall be according to IEC 908.
20.2 User Data Area
The Information Tracks in the User Data Area shall be either Digital Data Tracks only or Digital Data Tracks and
Audio Tracks. The following rules apply to the tracks in the User Data Area:
-  If the first Information Track is a Digital Data Track, it shall start with a Pause of 150 Sections (2 s) and shall
be coded as the second interval of a Pre-gap.
-  A Digital Data Track, not being the first track in the User Data Area, shall begin with a Pre-gap if the
preceding track is a Digital Data Track with a different Sector mode or if it is an Audio Track.
-  A Digital Data Track shall end with a Post-gap if the following track is an Audio Track. This rule applies also
to the last Digital Data Track in the User Data Area, which is followed by the Lead-out Track.
20.3 Lead-out Area
The Lead-out Track is either a Digital Data Track or an Audio Track. If it is a Digital Data Track, it shall be
structured in Sectors, without Pre-gap. If the Lead-out Track is an Audio Track, it shall be according to IEC 908.

reality:

Code: Select all

Serial          |Ring                           |Gap Layout             |Empty M1 sectors in    |Audio Offset   |Notes
                |                               |A->D;D->A;A->A         |D->A Transition Area   |track=1st;     |
                |                               |                       |track<last;track=last  |track>1st      |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

KOCD3003        HRKO30721-1FAAT                 3.00;2.00;0.00                0;0                0;0..44628      last sector for the last data track = audio silence (0x00): gets junked with offset
KSCD2003        TTTTFKS31008-1 SAC1             3.00;2.00;0.00                338;191            4000;4000
MWCD2003        FFTTTMW30214-1 S2U2             3.00;2.00;0.00                150                0;4000
NAPR1013        TFLTFNA10131211-6 QBZ2          3.00;2.00;0.00                150;3              0;4000
NAPR1022        TFTTFNA10221105-9 RAG4          3.00;2.00;0.00                150;3              0;4000          Track 08 is dummy
NAPR1024        LLLFTNA10240828-4 R8Y4          3.00;2.00;0.00                246;99             0;4000
NAPR1026        TLFTT NA260301-1 U3R7           3.00;2.00;0.00                150;3              0;4800/5000
NAPR1029        TFFTTNA10290629-4 S6C3          3.00;2.00;0.00                150;3              0;4000
NAPR1034        FTTLLNA10340224-1 T3V3          3.00;2.00;0.00                150;3              0;4000
NAPR1035        LFLFT NA350601-1 U6W2           3.00;2.00;0.00                150;3              0;4800
NAPR1036        TFFFT NA360601-1 T6T1           3.00;2.00;0.00                150;3              4000;4000
NAPR1043        TLLTF NA430804-2 U856           3.00;2.00;0.00                150;1              4800;4800
NBCD3004        FTLTF NB41101-1 TBU7            3.00;2.00;0.00                150;3              0;4000
NSCD0003        TLFTFNS30121-10 R1S3            3.00;2.00;0.00                150                0;4000
NSCD2017        LLTFFNS170426-3 T6Z5            3.00;2.00;0.00                150                4000;4000
NXCD9001        TTFLNX11218-5TTFL ZCU2          3.00;2.00;0.00                150;3              0;4000
RHCD4008        TFFFT RH80205-1 U2T7            3.00;2.00;0.00                150                0;4000
RHCD4009        FHHLH RH91108-1 UAC5            3.00;2.00;0.00                150                4800;4800
RSCD4006        RSCD4006 HRRS61125-2FAAT        3.00;2.00;0.00                0                  0;4000
TICD4002        TICD4002 HRTI20307-2FAAT        3.00;2.00;0.00                150                4000;4000
TJCD1015        TLTTLTJ150212-4 R2U2            3.00;2.00;0.00                150,~450;3         0;4000
TJCD2029        TLFLTTJ290729-6 S7Y5            3.00;2.00;0.00                156,~450;9         0;4000
TJCD2032        FFTFFTJ321105-2 SBW1            3.00;2.00;0.00                178;150            2C000;4000
TJCD9003        TLLLTJ30205-10 Q2S1             3.00;2.00;0.00                152;5              0;4000
TPCD1003        TTLFLTP31210-1 S1X7             3.00;2.00;0.00                181                8000;4000

- clean (EAC data pre-gaps: 225) ------------------------------------------------------ 25 = 49% [TOTAL = 49%] ----------------------------------------------------------------------

HCD3051         HCD3051-4-1108-R2F V            2.74;2.00;0.00                150;1              0;4800
HMCD2002        HMCD2002-4-0113-R1F V           2.74;2.00;0.00                164;15             2C000;4000
KSCD1001        KSCD1001-5-0121-R2M V           2.74;2.00;0.00                150;1              0;0/4000
MWCD2001        MWCD2001-100525-R1P V           2.74;2.00;0.00                174;25             0;4000          pregap for Track 10 is the same as for data: 2.74
MWCD2002        MWCD2002-1-0213-R1F V           2.74;2.00;0.00                150                0;4000
SXCD2001        SXCD2001-2-0804-R2P V           2.74;2.00;0.00                150                4000;4000
TJCD1016        TJCD1016-3-0208-R1K V           2.74;2.00;0.00                150                2C000;4000
TJCD1018        TJCD1018-7-0910-R1M V           2.74;2.00;0.00                150                0;4000

- clean (EAC data pre-gaps: 224) ------------------------------------------------------ 08 = 16% [TOTAL = 65%] ----------------------------------------------------------------------

KOCD2002        KOCD2002-2-1124-R1P V           2.74;2.00;0.00                150;1              0;4000          *1: Track 13, 14
KSCD1002        TFLLFKS21017-9 RAY5             3.00;2.00;0.00                169;22             4000;4000       *1: Track 17
NAPR1039        TFTLT NA390221-1 U2Y7           3.00;2.00;0.00                150;3              4000;4000       gets -0.01 for all, from subcode it's 3.00/2.00 plus some sections set as pause @ the start of the next track (A->D), AFRAME messed up
NAPR1051        HTLTT NA511026-4 VAW8           3.00;2.00;0.00                150                2CEB0;56B8/0    *4: Track 02; all audio tracks except 1st exceed LBA
NSCD9001        NSCD9001-4-1110-R1D             3.00;2.00;0.00                150                0;4000          *2: Track 02; 1.74 track 03 AFRAME?; *3: 06,12,14,15,16
NXCD1005        TLFTFNX51114-6 RBB1             3.00;2.00;0.00                150;3              2C000;4000      gets -0.01 for all, from subcode it's 3.00/2.00 plus some sections set as pause @ the start of the next track (A->D), AFRAME messed up
PVCD0001        PVCD0001-4-0129-R1D V           3.00;2.00;0.00                150                0;4000          *2 Track 02,03; *3: Multiple
PVCD1003        PVCD1003-3-0116-R1K V           2.74;2.00;0.00                150;1              0;4000          *1 Track 07
PVCD3010        PVCD3010-1-0907-R1C V           2.74;2.00;0.00                150;1              0;4800          *1 Track 09
RHCD1001        RHCD1001-7-0615-R1F V           2.74;2.00;0.00                150;1              0;4000          *1 Track 16, 57
TJCD0007        TJCD0007-5-0219-R1D V           3.00;2.00;0.00                1238               2C000;4000      desync/*1
TJCD2024        FFLLTTJ240128-6 S1Y5            3.00;2.00;0.00                152,~450;5         0;4000          *4 Track 80
TJCD9002        TJCD9002-4-0816-R1D             3.00;2.00;0.00                150                2C000;4000/7000 Track 02 - 2.73 in EAC - *2

- minor artifacts, that resolve to 225 or 224 / 150 / 0 ------------------------------- 13 = 25% [TOTAL = 90%] ----------------------------------------------------------------------

HCD9009         HCD9009-7-1112-R1D              3.06;2.01;0.00                150                0;4000          *2: Track 02; *3: 10 audio tracks
NBCD2002        HRNB20505-4FACT                 2.00;2.00;0.00                150;?              0;4000          some (10?) Mode 0 sectors at the end of last data track
NSCD0002        NSCD0002-9-0423-R1D V           3.00;2.01;?.??                150                0;4000          A->D is detected as 2.73 by EAC, ther's some repeated AFRAME, 18.02; 05.25; 00.01; 55.57; 07.10; 29.25; 10.32
TJCD1019        HTJ191003-6FABT                 2.00;2.00;0.00                150,~450;?         0;4000          some (11?) Mode 0 sectors at the end of last data track
TJCD3033        TJCD3033-2-0201-R1P V           2.74;2.02;0.00                150                0;4000          Track 02 - Mode2 next section after gap +0.01; *2: Track 03; many are 0.01 - no clue

- problematic layout ------------------------------------------------------------------ 05 = 10% [TOTAL = 100%] ---------------------------------------------------------------------

*1 last pre-gap section in subcode = Mode 2; EAC change gap +0.01
*2 A-Frame repeats @ the last gap sections; EAC change gap -0.01
*3 Desync between TOC & Subcode, 2 first sections of track are 2 last of the gap; EAC change gap +0.01
*4 last section before pre-gap in subcode = Mode 2; EAC change gap +0.01

colum 'Gap Layout' list the sizes of Audio->Data (Pre-gap), Data->Audio (Post-gap) and Audio->Audio gaps and records are grouped by this characteristic.
about a half of CDs had a perfect gap layout as seen by EAC.
16% had 224 sectors pre-gap returned by EAC and indeed 224 sectors were set in subcode.
25% had some minor differences that would resolve to one of two schemes above.
10% - 5 CDs had an unique gap layout.

besides specific changes in subcode ECMA also states, that the minimum size of pre-gap is 225 sectors: 75 empty audio sectors followed by 150 empty mode 1 (in this case). and those CDs, those, that have 224 sectors marked in subcode indeed do have 225 empty secors. in fact every tested CD have (HCD9009 contains 81 audio sector+150 data; 81-75=6 hence pre-gap boost by 6 frames; NBCD2002 and TJCD1019 have 75, but they're unmarked in subcode).
also all 224 sector CDs has a ring serial of the same pattern and MWCD2001 had also a pause between two audio tracks set to 2.74 in subchannels but there were 225 empty audio sectors between those tracks. all this leads to believe, that there was something wrong with the machine that made those discs, so it would mark 1 sector less in subcode for 3 second gaps ...well, or something. mainly for this reason i asked some of mods and we agreed to go with 3seconds gap for all of them (>90%).
sometimes EAC return 0.01 or less often 0.02 gaps between audio tracks. it's mainly because of MODE=2 type sections in subcode. when ther's a pause between audio tracks on PCE, there also was so far the same amount of empty audio sectors between them: MWCD2001, NBCD2002. however this is untrue for 0.01 second gaps: there was no correlation between those gaps and silence.
also for data post-gaps there always were 150 empty audio sectors, even though sometimes EAC would return little more or less.
so all deviations, that did not exceed 2 frames mapped to 3/2/0 gaps. (also did TJCD3033. it's listed on exceptions because EAC would return a lot of 0.01 audio gaps and i have no clue why...)

so well, what i'm trying to say, ther's not really that many special CDs at the end. i guess it's mostly possible to just dump everything as 3/2/0 without even checking subcode, as long as gaps do not differ by more than 2 frames. but there are few things to remember:
1.pre-gaps for data tracks consist of two parts: audio and data, [s]both should be generated[/s] (it's hard to read, but always was empty sectors of appropriate type - nothing more so far).
update:
if your drive supports d8, please try this program: http://vigi.dremora.com/cdtoimg.rar
it will read whole cd and then you can extract data tracks with pre-gap and then unscramble and it should be ok. (please remember not to scramble audio part of pre-gap or trailing silence at the end of last data track if there is such)
maybe PerfectRip works as well, though maybe it doesn't - please try by yourself. generally when you have cue sheet and all tracks and you open this .cue with cdmage, only audio part of gaps (and trailing silence) should show up as errors on scan. if it does work, maybe it will solve those problems with incorrect gap sizes, so that would be great.
if your drive does not support d8, you can try to fake toc with an audio cd and then extract data track as sector range with IsoBuster or Cdrwin. it's described here. if escape eject doesn't work, eg. resets toc, it's possible to take drive apart and remove cd by lifting cover but please do not do this with expensive/new drive.
the reason why i do not recommend generating this data any more is because it's just more difficult - you have to check every data pre-gap anyway (it's not always 150 data sectors, though very, very rare - there can be exceptions)
2.can't dump audio track if it's followed with data and gap detected between them is larger than 3s - ther's no spare empty sectors, so audio data will be cut off. if this same track isn't preceded by another data track it's possible to reset gap in such case, if it is, you'll have to dump with and without gap and combine both files.
3.if data post-gap is detected different from 2 seconds, last expected data sector should be checked. this sector should be last valid data sector in data track. if it's audio silence or junk or next sector is still valid data - it's not right, then
4.when last track is data, it sometimes can have empty mode 0 sectors (some drives won't read and you may get errors) or audio silence ([s]only had this once[/s]) at the end. if it's audio silence it can get junked just like the gaps and should be replaced by clean audio sectors.

this way all data should be there and all junk left out and if something changes, about those gaps maybe, it's still possible to just shift them around - without redumping.
but if you think ther's something wrong, please, let's discuss it, ok?

[themabus]

rewrote some programs (commandline), that i used, for more general purpose. they're nothing special but help me a lot:
sExtract.exe - extract a segment of file
unScramble.exe - yes, unscrambele
reMove.exe - move a part of file to another, to remove gaps from datatracks, resize audio gaps [updated @20080322]
http://www.mediafire.com/?q1mbksntoje
mode1 sector generator still not done

[Jackal]

Hi, what does unscramble do? does it do the same as Truong's descramble_cdda tool?

Also, I was wondering if you could help us by writing a small tool/script to process scrambled gdrom dumps?.. Dremora is always busy (he's the only other coder that I know of), so maybe you can help us (such a tool could simplify the gdrom dumping process drastically) .

The input would be a scrambled 45000-549150 dump file. I guess the tool/script would have to begin by asking for the combined offset value (or maybe the tool can just correct it automatically.. after all, the write offset value is already obtained when dumping track2 so the offset isn't really used elsewhere). By entering this value, the tool can correct the offset (because all the data is in audio mode, the offset is only at the beginning of the file) and begin descrambling the first data track. When that process is ready and there is still data left, it will use the TOC (or a .sub file if you want it really fancy) from the data track to cut the audio tracks to the proper sizes. Then finally the last data track is left and is also unscrambled.

There are 2 problems that I can think of so far: I don't know if the first scrambled sectors always contain all the data (maybe if the combined offset is negative some of the data is moved to sector 44999 or earlier). Also, I don't know how the TOC should be read (gdlister by yursoft does this, but there's no source available of it afaik).

[themabus]

hi, Jackal.

yes, basically the same, except it's not automated in any way. 
i used it to experiment with that scrambled data inbetween tracks.

thanks, i'll try to.

[themabus]

at the end of TOC ther's 3 UInt32 values
does anyone know what first two mean?
i thought those are LBA for single-density area tracks, but it doesn't look so.

[Jackal]

I can't find any documentation on it.. anyway, here's the TOC (0100-02FF) of Mortal Kombat Gold.. maybe it is of some use: http://www.speedyshare.com/432639791.html

And thanks for trying!

[iR0b0t]

I think it's a PCE topic?

[themabus]

conspiracy

ok, i'm done (or so i think )
http://www.mediafire.com/?mznzwfynnun
(also you'll need ElbyECC.dll
it comes with CloneCD or just can be downloaded somewhere
mine is v4.0.0.0; crc32: 0d00ed6a)

it is small and simple
i've left those two TOC fields out for now and some other things too
so it could be fairly easy to confuse
in such case, when something strange occurs, it should bail out then
this will probably happen a lot, since i could not really test it myself.
so i don't think it will screw up anything, it should just stop.

syntax is 'ice.exe FileName StartLBAUsedToDumpData'

all gaps are assumed 2sec, so it's better to verify them manually at the start & end of each file.
also i'm not sure if shown offset is correct, i've forgotten quite a lot about those things.
and it could be, that last sector of last data track is missing.
so it would be better to try it out on some known gd-s at first.

and please let me know what could be just different.

edit:
from only gd-rom i could test it looks like gap preceding last data track is 3sec long:
75audio/150data, a lot like PCE (from sector content - i haven't checked subcode)
in database currently ther's either 2 or 0 sec gaps at this position.
this program will go for 2 and will freak out if ther's less than 150 data sectors there.

edit:
those two fields hold first/last track no., alike to real TOC.
what's strange: i've dumped my GD completely
and program seems to work fine - cutting everything as intended.
so total size of tracks should always stay constant: 1,185,760,800
since first/last LBA appears to be fixed: 45000; 549149
but in db every game has different size, hmm

[Jackal]

Wow! I just tested with ElbyECC.dll 3.1.0.0 and Vigilante 8: Second Offense and it works perfect (all checksums match db)! You're the man.. I've added it to the guide (hopefully all data is included on discs with negative offset). The only thing that's incorrect is the combined offset (but it's not really important I guess):

Combined offset (samples)      : -23813975

from only gd-rom i could test it looks like gap preceding last data track is 3sec long:
75audio/150data, a lot like PCE (from sector content - i haven't checked subcode)
in database currently ther's either 2 or 0 sec gaps at this position.
this program will go for 2 and will freak out if ther's less than 150 data sectors there.

I don't know about this.. perhaps you could check the subchannels.. it's propably just a postgap and not really part of the last data track.

but in db every game has different size, hmm

Don't forget that the total size in db also includes track 1 and 2

ps. I added the elbyecc.dll to the download in the dumping guide.. I hope that doesn't get us into trouble?

[themabus]

thanks!

Combined offset (samples)      : -23813975

yeah, it does looks wrong

I don't know about this.. perhaps you could check the subchannels.. it's propably just a postgap and not really part of the last data track.

ok. i got ill in last days, but tomorrow i'll check it.
the way i understand it - it's always about 3 seconds but in case when audio follows data only 2 are set as pause,
if data follows audio - everything is. but i'll check on that.
also i've made a mistake and currently gd with only one hd track won't work.
and i want to include TOC as cue sheet comments and add ability to set gap size with parameter.
so i'll fix all that and upload new version tomorrow.

Don't forget that the total size in db also includes track 1 and 2

oh i did forgot that

ps. I added the elbyecc.dll to the download in the dumping guide.. I hope that doesn't get us into trouble?

i hope not. it's available for download on mayny sites after all.
but in any case i only use it for ecc checking and i think i had code of my own for this somewhere.
i'll try to find it and so maybe this dll will not be needed anymore.