Researching of a .dat archive with a .mnf file.

[stalja]

For whatever reason the quickbms script does not extract the zosft file from esodata files. I managed to find the file in eso0002.dat but it contains only a small portion of filenames (something like 30k as compared to over 250k present). It also seems that the quickbms script does not extract some additional files for whatever reason. I might be wrong though.

[stalja]

That bms script I posted several days ago will not properly unpack everything. Turns out it breaks when some of the empty archives are being called and eso0000.dat has files that are neither zlib nor snappy ( or something in the original bms script is not quite correct). Currently, I am skipping files that cause breaks in the script manually, but any help from the more experienced members would be welcome.

[LennardF1989]

Heya,

Never been active on this forum a lot, always lurking and messing around by myself. Let my extraction program run for a good half an hour yesterday , it would go through all archives, decompress the information, and check the header against a local file. I had to skip eso0000 and everything beyond eso0106, because there's stuff in there which I suspect is double compressed (so first Zlib then Snappy or other way around, because those files have multiple entries in the second block of the MNF with different "flags").

There aren't a lot of special files in there though: .xml, .lua, .dds, .gr2, some soundbanks, some audio files, a few custom files and thats about it.

Besides a single ZOSFT in all the scanned eso packages I also encountered a ANFT, which I suspect is some sort of internal filetable. Haven't been able to find another ZOSFT, but it could be its hidden away in side the packages I skipped.

However, as someone in this topic said before: It's not unlikely some files are only used interally.

[LennardF1989]

Attached is a rar with logs for all archives.

It's safe to say that Block 3 in the MNFs contains all entries into the DAT files as all bytes are consumed when going through them (except 8 bytes in archive 4 for some reason). The MNF for ESO also contains references to archive 0 which don't exist, so either its residual or it's used for whatever more is in the MNF (eg. the unknown block).

All archives beyond 203 are simply empty, they only have the first 14 bytes of the DAT header. However, the MNF contains over 50k entries into these files, so I guess the content is supposed to be there, but it's stripped/disabled for the beta.

Other than that, as said in my previous post, not a lot of interesting things going on in the files.

With these logs, you/I can check which files are in the ZOSFT to give them a name and which are most likely for internal use only.

EDIT: Logs.rar now also contains list of all filenames found in the ZOSFT, sorted by filename and extension. As you can see, a lot of the files you encounter in the other logs are not mentioned in the ZOSFT.

[stalja]

I came to the same conclusions working with the bms scripts. There is one more thing I'd like to add though - file id is not unique. There are some files that have the same ID but are from diferent archives. So, for example, file with id 9 can be found in eso0000.dat and eso000x.dat and so on. I did not manage to find the correlation between archive_id, file_id and zosft so that I can determine which file from which archive is connected to which filename.

[ehnoah]

While follow this, it looks like ESO use a strong method to save their files.

[Droolie]

After the latest update, it seems that as LennardF1989 said, eso0000.dat is compressed differently, or the MNF records are just wrong for that file. I haven't got the time to analyze it properly, so is anyone interested?

I'm currently using this adapted script to extract the audio and the ZOSFT only (since I'm only interested in those). You can easily adapt it (by using my older scripts) to extract all the files. It only extracts files in archives with 0 < archive number < 204. You can remove those restrictions to be able to extract files from MNFs other than eso.mnf.

[LennardF1989]

I purposely waited for the next ESO weekend, because it seems it's not a mature format yet

[Cagli]

I've been lurking this post for a while, and I must say great work!
I'm only interested in the .dds files and have been succesfully getting plenty of those in the eso.mnf/eso00xxx.dat files.

But I don't seem to get the same result for the game.mnf one.
The topic started with the game.mnf file and shifted to the eso.mnf file.
Does anyone have a bms for the game.mnf one? I'm still pretty much a n00b when i comes to figuring it out.
I don't care about the file names, as long as the dds files are exported.
TERAB1T uploaded an exported one, but it seems to be missing a lot of files (perhaps due to not getting the right filenames)
Anyone that can help?

[Ferugre]

None of the QuickBMS scripts seem to work, they end up with xmalloc errors. I think incorrect values are being used to extract files. Anyone have working scripts still?

[Lord Vaako]

Extracting main game archives is the first step but what about rev eng data/files inside?
Was any of you able to find structure of any, non simple, files inside?
There are maps (gfx & names), localized text (en/fr/de) containing quests data (titles, descriptions, objectives), lore books texts, npc names, items, names, etc. and even some ids, textures, sounds, ui scripts, etc.
But I'm still missing "connections" between those data - some kind of internal database, i.e. I know quest ids, names and descriptions but I'm missing quest level and category, or connection between quest and its objectives.
Or I'm not able to find connection between map gfx and map id - I know that for most of the zones its simple (just one map and gfx), but multi level zones (i.e the wailing prison) are the problem.

Have any of you found such kind of data?

[stalja]

None of that has been publicly reverse engineered. No one has yet delved into decrypting eso0000.dat and eso0001.dat, which contain a lot of files not in zosft files and are compressed with an unknown method (not zlib, not snappy). Looking at some websites out there I can see that probably noone reverese engineered the client and they are all sniffing packets or using in-game addons for data collection. I guess there is not enough interest by the pros to go deep into how zenimax encrypted their files.

[stalja]

I was snooping around eso0000 and offzip-ed it. Bunch of files with header

Code: Select all

fa fa eb eb

or

Code: Select all

FB FB EC EC

One of these have multiple zlibed blocks, while other seems to be and index of sorts. When I offzipped the one with zlib blocks I got thousands of new files. Some of them contain some strings and I believe this has something to do with map names and their IDs (possible map tiles?) and possible quest/NPC names (possible map locations?). I also found something strange which I am not able to decipher on my own yet. There are files that apparently have some sort of encrypted or compressed strings. From my crude analysis the files are composed of a header followed by the number of strings contained, followed by 8 byte (Id?), followed by 00, followed by str_length, followed by string, followed by 00.

Attached is one of those files. Does anyone recognize what this is compressed/encrypted with?

[matreco]

Shame, would be nice to see these models out.

[Bungie]

The Strings are encoded with somethink like Base64.

Here the first three links, decoded with Base64 and converted to hex.

Code: Select all

0413787f1451300d0a2f59
047809043108273978044b496969391d64103d0f297b3f5d7a2f3e06480e480120035f01021820082a601055121a254405493040362f3f185e7662682224743d130c6f481b6c48010201064808475d234f
045d78636018050234674470471f471e5919076448123f241a6044474038115903043b443d07112e3c641f5c6040401e66611805581f3a386b