Marvel Ultimate Alliance 2 .DAT

[Acewell]

nice to see you getting the hang of it!
the face has holes because it is only 1 of 4 submeshes
search for 00 00 01 00 02 00 to jump to the face indices start address of each submesh

sequence assumes there is no room for other data within the vertex stride
therefore the other data like normals and UVs etc must be in their own block
Hex2obj doesn't deal with normals so we don't worry about that.
a float is 4 bytes, if XYZ position are stored in floats then that is 12 bytes
if the vertex stride is 12 bytes, there is no room left for other data
and must be elsewhere in the file.

i go by this rule
if FVF is 12 and stored as Floats you can use sequence
if FVF is 6 and stored as ShortAll you can use sequence
if FVF is 6 and stored as HF_all (half-float) you can use sequence

i can't recoginze half-floats and shorts by looking at the numbers, but i do know what 2 bytes looks like


there is a table with offsets and lengths in the 00000000.dat file starting at 0x9a0

little-endian byte order
0x9a0 - 3 bytes - length of vertex block for 1st submesh
0x9a8 - 3 bytes - start address of vertex block for 1st submesh
0x9b0 - 3 bytes - length of face indices for 1st submesh (divide by 2 to get num faces)
0x9b8 - 3 bytes - start address of face indices for 1st submesh
0x9c0 - 3 bytes - length of vertex block for 2nd submesh
etc

those lengths and addresses (except face indices lengths) actually end with either 10 or 02
10 represents a length
02 represents an address

[IAmTheClayman]

nice to see you getting the hang of it!
the face has holes because it is only 1 of 4 submeshes
search for 00 00 01 00 02 00 to jump to the face indices start address of each submesh

sequence assumes there is no room for other data within the vertex stride
therefore the other data like normals and UVs etc must be in their own block
Hex2obj doesn't deal with normals so we don't worry about that.
a float is 4 bytes, if XYZ position are stored in floats then that is 12 bytes
if the vertex stride is 12 bytes, there is no room left for other data
and must be elsewhere in the file.

i go by this rule
if FVF is 12 and stored as Floats you can use sequence
if FVF is 6 and stored as ShortAll you can use sequence
if FVF is 6 and stored as HF_all (half-float) you can use sequence

i can't recoginze half-floats and shorts by looking at the numbers, but i do know what 2 bytes looks like


there is a table with offsets and lengths in the 00000000.dat file starting at 0x9a0

little-endian byte order
0x9a0 - 3 bytes - length of vertex block for 1st submesh
0x9a8 - 3 bytes - start address of vertex block for 1st submesh
0x9b0 - 3 bytes - length of face indices for 1st submesh (divide by 2 to get num faces)
0x9b8 - 3 bytes - start address of face indices for 1st submesh
0x9c0 - 3 bytes - length of vertex block for 2nd submesh
etc

those lengths and addresses (except face indices lengths) actually end with either 10 or 02
10 represents a length
02 represents an address

You evil genius! I don't know how you realized that the data structure would include information on the submesh blocks, but this is golden information (I also don't know why a submesh would be divided right though the middle of Black Panther's face, but whatever lol.) Thank you so so much for this

Looking a the data now that I understand it, I realize that you NEED to convert to .dat files as opposed to using the .igz. The reason is that everything is the .igz is offset in such a way that the start addresses located in that data block are wrong, but work for the smaller, non-offset .dat files. So looking at 00...00.dat start at 0x09A0 we can see that we should find:

a vertex block of length 0x030510 (197,904) beginning at address 0x10
a list of face indices of length 0x64c8 / 2 (12,900) beginning at address 0x030520 (aka the one already talked about in an earlier post)
a vertex block of length 0x867c (34,428) beginning at address 0x0369f0
a list of face indices of length 0x12a8 / 2 (2,388) beginning at address 0x03f06c

and so on. However these addresses only line up if you look at file 00...03.dat. If you're using 1101.igz the list of faces you'd expect to start at 0x030520 actually starts at 0x036c34. If it feels like this is a pedantic explanation it's half to help any other people new to these concepts (the selfless reason) and half to make sure I fully understand what's going on (the selfish reason.)

And sure enough here's that missing bit of face, sitting pretty in the second block:

BlackPantherHexToObjResult2.png

[Acewell]

right, those table offsets make sense when the igz is split into dat files, but if i was to make a script to
automate importing i would leave the igz file intact and just use simple addition to make up the difference.

[IAmTheClayman]

sequence assumes there is no room for other data within the vertex stride
therefore the other data like normals and UVs etc must be in their own block
Hex2obj doesn't deal with normals so we don't worry about that.
a float is 4 bytes, if XYZ position are stored in floats then that is 12 bytes
if the vertex stride is 12 bytes, there is no room left for other data
and must be elsewhere in the file.

i go by this rule
if FVF is 12 and stored as Floats you can use sequence
if FVF is 6 and stored as ShortAll you can use sequence
if FVF is 6 and stored as HF_all (half-float) you can use sequence

This is the one part I'm still perplexed by though. I get that a 3d point takes up 12 bytes of space, but how did you get to the conclusion that the appropriate block size was 76 bytes? And what is the significance of the UV pos value?

Also RE saying that you could get the appropriate offset via addition: can you assume that every .igz file will have the same offset? Is the header padded to a defined length?

[Acewell]

.. but how did you get to the conclusion that the appropriate block size was 76 bytes?

here is where i explained how i do this before
viewtopic.php?p=120284#p120284

Also RE saying that you could get the appropriate offset via addition: can you assume that every .igz file will have the same offset? Is the header padded to a defined length?

yeah there should be a common structure among similar game files, the trick is finding it!

0x40 - offset to 16 bytes before first submesh vertex block
add this offset to every offset in the table to make up the difference

getting to the table looks pretty easy
using 1101.igz (BlackPanther) as example

seek absolute to 0x82c
read integer 0x6f8
seek relative length of that integer - 4 to 0xf24
read integer 0x1c0
seek relative length of that integer - 4 to 0x10e4
read integer 0x84
seek relative length of that integer - 4 to 0x1168
read integer 0x30
seek relative length of that integer - 4 to 0x1198
tablelength = read integer
numSubmesh = tablelength - 20 / 32
seek relative 0x04 to 0x11a0

and you are now at the table data where you can loop through the offsets+lengths for each submesh

[IAmTheClayman]

Thank again AceWell, checked out the thread you linked to and was able to set up my hex editor to see why the FVF block would be 76 bytes. As for UV pos, HexToOBJ tutorial recommends either 12 or 24 when using floats, and if you use 12 you end up with negative values, which my professor always said was a big no-no for UV mapping (except in very rare instances.)

Was able to crank out the Iron Man mesh this morning much faster than I did the Black Panther mesh now that I know what to look for:

IronManModel.png

Link to the .igz here: https://dl.dropboxusercontent.com/u/41414718/1501.igz

[shakotay2]

As for UV pos, HexToOBJ tutorial recommends either 12 or 24 when using floats, and if you use 12 you end up with negative values,

If I hate something then it's uncorrect or incomplete "quotes".
(always use the last version of hex2obj!)

From the actual tutorial it reads:

For simple floating point formats UVpos is 12 or 24. Or try other multiples of 4.

It also states:

Getting the UV position is the harder part. Mostly it is trial 'n error.

So for floats and an FVFsize>20 I'd recommend trying a uvpos= FVFsize - 8.

[IAmTheClayman]

Lol Shakotay, didn't mean to offend. Was attempting to quote the tutorial from memory since I wasn't at my computer.

But perhaps you'd be the best person to ask: is there a way to automate hex2obj so that, for example, if we know there are x number of submeshes we can write up a script to enter the appropriate setting and output each mesh without typing in values manually?

Also a more general question: now that I have a decent enough understanding of just extracting the mesh, what would be the best way to get the skeleton information? Is that something that can be done through Noesis?

[shakotay2]

Lol Shakotay, didn't mean to offend.

I didn't feel so, and I know hex2obj has some weakness, but I'm up to clear things where it has not.

is there a way to automate hex2obj so that, for example, if we know there are x number of submeshes we can write up a script to enter the appropriate setting and output each mesh without typing in values manually?

This is what The Make_H2O project was good for (viewtopic.php?f=29&t=12756).
You need to be a programmer, though, and it's not very popular.

So I decided to put more efforts on maxscript from time to time.

what would be the best way to get the skeleton information? Is that something that can be done through Noesis?

Noesis is one of the best tools I know when it comes to skeleton/animation.
But you need to feed it with plugins (dll or python script). And plugins must be written in most cases.

I'm on my way to create a universal skeleton extractor (but I'm lazy, slow and dumb, so it will take more time ).

Feel free to upload skeleton data and we could have a look at together.

[IAmTheClayman]

Haven't had time to translate it from the hex code to something more easily readable, but the skeleton data is included in the .igz files I included in my earlier posts. Shouldn't be too hard to locate, just look for references to joints near the start of the file

[Acewell]

https://mega.nz/#!CgAAVQiD!nc6Ttc-nEtCI ... Mi8xTs9tHs

i made a Noesis python script to open most of those igz samples

*script removed see this post*
viewtopic.php?p=121800#p121800

out of 194 files these are the only ones that won't open so i will call it a success for now
i know there was a change in the stride for a couple but i can't find the flag yet
you can just use Hex2obj for these
0002.igz - vertex stride is 44
12201.igz - either corrupt or just a skeleton
17001.igz - either corrupt or just a skeleton
21001.igz - either corrupt or just a skeleton
21101.igz - either corrupt or just a skeleton
22401.igz - vertex stride is 80
7003.igz - vertex stride is 44
8802.igz - vertex stride is 44
gambit_deck.igz - doesn't seem to be a model
grenade.igz - doesn't seem to be a model

[Teancum]

Nice work AceWell! Any idea whether your script below can be reversed? I'm curious as to whether I can reswizzle and just inject a modified texture back in as a chunk. I know Noesis can't do that, but if I have the hex offset and size I can reinject, I just have to get it back to the appropriate format.

here is version 2 of the igz texture script

tex_MarvelUltimateAlliance2_igz.zip

it turned out to be a different 8 byte hash representing the 3 formats used
supports dxt1, dxt5, ati2
this one should work for all of the igz textures, let me know if not

[Acewell]

i suppose you're asking for an export script which i'm sure Noesis should have no problem
with but i have zero knowledge of that kind of stuff, i''m not a programmer and if it
requires anything other than python there is not much chance i can do it.

[Rutabaga]

Thanks a lot AceWell! works awesome! god bless you

Can you make a new script for textures please? here a samples:
https://mega.nz/#!3shDVQCJ!XyYsaJjteIkJ ... i3SZVaTpLU
Thanks!

[Acewell]

Can you make a new script for textures please? here a samples:
https://mega.nz/#!3shDVQCJ!XyYsaJjteIkJ ... i3SZVaTpLU

here is the new script for these igz texture samples

*script removed see this post*
viewtopic.php?p=121800#p121800

supports dxt1, dxt5, and ati2

you can't have this script and the model script in the python folder at the
same time or else you will get errors when trying to open any igz texture.
this is because they both look for the same magic and Noesis will think you are
trying to open a model when you are trying to open a texture. i need to add a
second type check to the model script so they won't conflict, but until then
you will have to move one out temporarily