Researching of a .dat archive with a .mnf file.

[Lugnut360]

Do you unpack the files from the .dat, then use a zlib decompression on them or do you somehow run zlib decompression on the entire eso0000.dat?

I'm very new to computer programming/datamining and would appreciate any explanation of how you write the extraction script for this.

When I open up eso0000.dat in hex editor, I see:
Hex: ASCII in parenthesis:
50 45 53 32 (PES2)
01 00 00 00 (....)
00 00 0E 00 (....)
00 00 78 9C (x..}) - zlib file header

I'm not sure how you write a BMS script from this. I assume something like:

idstring "PES2"
get UNK1
get FILES long
for i = 0 < FILES
getdstring NAME (????)
Not sure how to find OFFSET, ZSIZE, SIZE

Thanks TERAB1T for the unpacked file!

[Droolie]

So does anyone know yet what's in the 21 unknown bytes at the beginning and the data after the 3rd zlib block of eso.mnf? Or what data the first two blocks contain? I'm pretty sure they'd have to at least keep references to the filetables there...
And I'm sure you have this already by now, but for those who don't, this QuickBMS script unpacks the 3 zlibbed blocks we know of from a *.mnf file:
*snip - you can find the script a few posts after this*
I would've used a loop but I'm planning to expand the script to get the data from the blocks too, that's why I always added the file number.

EDIT -> Block 2 seems to contain data about the files as well: for every file just one field of 8 bytes, little endian. I have no idea what block 1 is.

[Ekey]

The third block from MNF is the file table.

Code: Select all

0x4 - szFileSize
0x4 - szFileZSize
0x4 - szUnknown01
0x4 - szOffset
0x1 - szComType(0 - Not Compressed, 1 - Zlib, 2 - Snappy)
0x1 - szArchiveNum
0x2 - szUnknown02

Hash Function: Initialize hash by 0xA8396u

http://rdf3x.googlecode.com/svn/trunk/i ... l/Hash.cpp
http://rdf3x.googlecode.com/svn/trunk/i ... l/Hash.hpp

[Droolie]

Exactly, but it might not be linked with the filenames. What if block 1 or 2 (or the extra data after block 3) contains that link and block 3 just contains sizes and offsets?

[Ekey]

In 3 block - 0x4 - szUnknown01 - it's seems filename hash (see my previous post)

[Droolie]

Yes, that will help a lot once we have found the complete list of filenames, but we still need to find that, right?

By the way, these are the 15 unknown bytes after the MES2 header:

Code: Select all

             unknown1  num_dat_archives       EOF_offset           dummy         unknown2
FILENAME         [02 00] [XX] 00 00 00 00 [XX XX XX XX] 00 03 [00 00 00] 04 [XX XX XX XX]
eso.mnf:         [02 00] [D0] 00 00 00 00 [7B FB 4F 00] 00 03 [00 00 00] 04 [00 08 0C 01]
esoaudioen.mnf:  [02 00] [04] 00 00 00 00 [2B 09 1E 00] 00 03 [00 00 00] 04 [00 04 06 01]
game.mnf:        [02 00] [01] 00 00 00 00 [3A A2 00 00] 00 03 [00 00 00] 04 [00 00 10 19]

unknown1: This always has to be 2 for some reason. The files don't open otherwise.
num_dat_archives: number of DAT files
EOF_offset (little endian): After reading these 4 bytes, add EOF_offset to current offset to reach the end of the third block. This is EOF except for eso.mnf which has extra data.
unknown2 (big endian): This is 4120*(2^n)+1, with n ranging from 0 to 2. It seems n is also the number of the DAT archive containing the ZOSFT, but maybe this is just a coincidence.

EDIT -> I'm in the process of decompiling eso.exe. It's not going that well as expected but I fixed a few things in the description above.

[Droolie]

Bumping this topic.
About a month and a half has passed. I got stuck in the decompilation process.
Has anybody made any progress?

Here are my scripts so far. My code is a bit messy but then again, QuickBMS isn't that clean as a language.

• eso_mnf_with_id.bms: extracts all files listed in the MNF file with a unique identification found in block 1.
• eso_mnf_blocksonly.bms: extracts MNF zlibed blocks
• eso_zosft_blocks.bms: extract ZOSFT zlibed blocks
• eso_zosft_extract_names.bms: create a batch script to rename files extracted with eso_mnf_with_id.bms. The names aren't linked correctly, and I currently have no idea how I would fix that.

TERAB1T, it would be great if you could share how you linked the ZOSFT filenames to the files.

[TERAB1T]

About game.mnf filenames:

There are three zlib-ed blocks in .mnf file. As you know, the last one contains filesizes, offsets etc (20 bytes for each file). And the second one contains indexes for some of filenames from zosft-file: 8 bytes for each file. Look at this screenshot:



If 'Unknown1 = 0' and 'Unknown3 = 128 (decimal)' index (Unknown0) will be valid.
And 'Unknown1 = 1' and 'Unknown3 = 128' is filetable (zosft).

Use valid indexes and link them with files at the end of filetable (after 9 zlibed blocks).

Also I figured out, that if you remove all the "00 00 00 00" parts from the first block in .mnf, you'll get unique ID for each file in archives (4 bytes for each file).

[Droolie]

Interesting - there are only 1768 valid indices in the MNF, although there are 1772 records in the ZOSFT file. It's not so hard to combine those indices with the ZOSFT without checking their validity though: they return in ZOSFT block 2.3 (that is, the third zlib block of the second big block, so it's the 6th zlib block), along with offsets for the corresponding filenames.
The indices are also unique in game.mnf, while they aren't in eso.mnf... quite strange indeed. I think that when combined with another number (which isn't necessary for game.mnf), they might be unique.

I had also found out that there are unique IDs for the files in the first MNF block (check the ZOSFT script in my previous post, it used them), and these IDs appear again in a few of the ZOSFT blocks, but they seem to be in the wrong order. The indices in block 2 seem to be the only link. I called that index file_number in my research notes which I made available here: http://pastebin.com/rsgh1irr

I adapted my scripts so the filenames of game.zosft are correctly applied.

• eso_mnf_with_filenumber.bms: This extracts the files with file_number as filename. Some filetypes automatically receive the correct extension.
• eso_zosft_extract_names_by_filenumber.bms: This creates a batch script to rename the files extracted with the previous script based on the names in the ZOSFT file.

[Luriam]

Hey guys, anyone succeeded at extracting the sounds ? eso0000.dat through eso0207.dat should contain sfx. And esoaudioen0000.dat through esoaudioen0003.dat got to be the container for the voices.

I have attached a sample of the voice archive HERE. (1,09 gb)
And also the esoaudioen.mnf, just in case.

And the possible sfx HERE.

Anyone gets anything, let me know. Thanks!

[Droolie]

Extracting the audio is easy, you can do it with my QuickBMS scripts to extract everything using the MNF files. You'd better just extract all the archives in one go though, because the files the MNF file points to is divided over all of those DAT files.
Psst! If you know how to use Google, you can also find the soundtrack which I ripped and posted online some time ago...

[stalja]

I believe eso.mnf header has changed. Looking at this

             unknown1  num_dat_archives       EOF_offset           dummy         unknown2
FILENAME         [02 00] [XX] 00 00 00 00 [XX XX XX XX] 00 03 [00 00 00] 04 [XX XX XX XX]
eso.mnf:         [02 00] [D0] 00 00 00 00 [7B FB 4F 00] 00 03 [00 00 00] 04 [00 08 0C 01]
esoaudioen.mnf:  [02 00] [04] 00 00 00 00 [2B 09 1E 00] 00 03 [00 00 00] 04 [00 04 06 01]
game.mnf:        [02 00] [01] 00 00 00 00 [3A A2 00 00] 00 03 [00 00 00] 04 [00 00 10 19]

I do not think bytes are in this order anymore. (there are extra bytes now and some are not all 00)

[stalja]

It seems that that unknown block of data at the end of eso.mnf you mentioned previously is now inserted after the first 22 bytes. I'll try and create a functioning bms script to compensate for this, but I'm a noob so ...

[stalja]

For the latest build things seem to have changed in eso.mnf. Block of dat that used to be after the third zlib block is now moved to 22nd byte. I modified Droolie's quickbms script to compensate for this change. I might have broken something else, but I did manage to extract a lot of DDS textures and gr2 models this way. WAVs seem to be broken (won't play in media players), but it might be something else.

There does not seem to be a ZOSFT file extracted with this script from eso*.dat files so I could not use the 2nd Droolie script to connect filenames with files. Any help is appreciated.

[fxfire]

Thanks for all the great work so far. Does anyone have a hint what type of packer/obfuscator was used for the current stress test beta exo.exe ?