[DONE] Redump integration in Dolphin?

[JosJuice]

Recently, I've been working on improving Dolphin's ability to tell users whether their dumps are good or not. (https://github.com/dolphin-emu/dolphin/pull/7922, if you're interested.) What really would be nice to have is the ability to check whether the user's hash matches the hash in the Redump database, but I'm not sure what the best way to do that would be.

What would be ideal is if Dolphin had a way to map game ID+revision+disc number to a hash, either using a local copy of a database or by querying redump.org through some kind of API. (In case you're not familiar with GameCube/Wii games, the game ID can be obtained from the Serial field in the Redump database.) The intended way of doing something like this seems to be to use the downloadable dat files, but they don't contain game IDs, so all Dolphin would be able to do is check whether the user's hash exists in the database. What this means is that if the user's hash isn't in the database, Dolphin can't tell whether this is because the user has a bad dump or because the user has a dump of a disc that isn't in the database yet. So I would like to avoid using a database that doesn't contain game IDs if possible.

The only currently available alternative I can see that would be able to do what I want is sending HTTP requests to redump.org like a normal user browsing the website and then parsing the results. This works, but having to parse the HTML is annoying and can lead to things breaking if the website changes the structure of its HTML. Does this sound like a good or bad idea, both in terms of whether the HTML structure can be expected to stay the same and whether redump.org is fine with the extra load? (If this is implemented, Dolphin would only connect to redump.org if the user specifically presses the verify button for a game, so I hope the load won't be too high.)

Or is there a possibility of Redump releasing dat files with additional fields (serial, version, etc...) or making a web API available?

[Hiccup]

A problem with this - the serial field is for the disc label serial, not the disc image header serial. It usually matches, but doesn't always. Maybe redump should start cataloging the header serials?

[JosJuice]

A problem with this - the serial field is for the disc label serial, not the disc image header serial. It usually matches, but doesn't always. Maybe redump should start cataloging the header serials?

I'm not aware of any discs like that other than https://redump.info/disc/10000/ (which seems like a somewhat suspicious dump since everything beyond the first four bytes apparently is identical to https://redump.info/disc/7249/). Do you have any examples, other than that one?

[RibShark]

A redump.org API in general would be absolutely great for emulators, ROM Managers, and many other uses. Please implement one!

[Landcross]

I agree that any form of API would be a great, great addition to Redump.

Though, I can imagine there are also drawbacks to a Redump API. It would probably increase the server load (imagine all thousands of Dolphin users querying the API), especially when more rom managers/emulators/launchers will use it. And it would increase the 'risk' of someone scraping Redump and 'stealing' the data. Though, it's of course a point of discussion whether the data is really a property of Redump

Or, maybe not an online API, but an 'offline' one that generates a json/xml file for offline use. Where a user would select which data (maybe even from a limited selection, e.g. only hashes, titles, editions and serials) he/she wants and then Redump generates a file. It would at the very least be much less of a strain on the server.

[iR0b0t]

is there a possibility of Redump releasing dat files with additional fields (serial, version)

If you want to have it, I can make it happen.

What will you do about discs which don't have a serial or version, though?
https://redump.info/discs/system/gc/sort/serial/?page=1

[JosJuice]

is there a possibility of Redump releasing dat files with additional fields (serial, version)

If you want to have it, I can make it happen.

That would be very nice. For Dolphin's purposes, the serial and version are the only things that really are needed (in addition to the hashes, of course). The disc number would also be convenient to have, but it can be determined based on the serial.

What will you do about discs which don't have a serial or version, though?
https://redump.info/discs/system/gc/sort/serial/?page=1

I don't think there's anything we can do with those unlicensed discs other than checking whether the hash is in the database or not. A lot of those discs even share the name that is displayed in the GC IPL, so there really isn't anything good to go on other than hashes.

[iR0b0t]

Apparently we don't store the version of gamecube and wii discs anymore. I am kind of "delaying" the deletion task for that matter because i don't know yet if we still need it or not.

@JosJuice
Whould the serial tag still be enough for the integration purpose?

[JosJuice]

Apparently we don't store the version of gamecube and wii discs anymore. I am kind of "delaying" the deletion task for that matter because i don't know yet if we still need it or not.

The version is being stored, just in a different format. Old 1.01 is equivalent to new Rev 1, old 1.02 is equivalent to new Rev 2, and so on. (But it seems like 1.00 is being replaced with just nothing? Not sure why, but it's not a problem for me, since I can simply treat an empty field as meaning Rev 0.) I'll just need to add an exception for Korean GameCube games with the country code E, since it seems like they now start counting the revision from 0 rather than 48.

@JosJuice
Whould the serial tag still be enough for the integration purpose?

Serial + revision would be enough.

EDIT: Actually, a small correction: Due to the exception with Korean GameCube games mentioned above, I would also need information about whether games are Korean or not. But this information is already present in the current datfile if you check whether the game name has (Korea) at the end, so it shouldn't be a problem. It would have been convenient if those revisions still were being counted from 48, though.

[johnsanc]

Why not just store the header information as a new field and parse the first few bytes to put in a new field in the dat? That should contain all the info Jos needs AND it adds additional info the database that isnt there for Nintendo systems that many other systems do have.

This can be scraped easily and provided in bulk.

There is A LOT of good parseable information in nintendo disc headers that could be useful:

GameCube: https://www.gc-forever.com/yagcd/chap13.html#sec13.1
Wii: https://wiibrew.org/wiki/Wii_Disc

I believe Jos would only need the highlighted text portion in a new field in the GC/Wii dats: