Dead or Alive series formats and tools

[b0ny]

hi everyone!

may you please give me a helping hand with this: viewtopic.php?p=59977#p59977 ?

[Rayved]

I need help with Crimpxp. I modified a mesh and re-exported it as a 3ds file but when I use crimpxp to inject the new one into the xpr, nothing really happens. I click GO and I get a popup that says "ruby crimp.rp blablabla" and when I open the xpr file with greed explorer, the model looks just the same. I'm running the app in Windows XP SP 3 mode, the GO button wouldn't appear otherwise (on windows 7).

Is there any other app that can reinject a 3ds file into a xpr, or is there some step I'm missing?

[b0ny]

Download the DOA3(U,X)Viewer
just for fun tried to add to the doa3viewer compatibility with doax and doau.

don't know why but the doau and doax objects are mirrored horizontally.

[mariokart64n]

do you have format specs on DOAU?

EDIT
found an old doc; http://www.viewdocsonline.com/document/wpe6kq

its not very detailed, only reconfirmed what I had just figured out in the past hour.

do you have any additional information?

[b0ny]

do you have format specs on DOAU?

EDIT
found an old doc; http://www.viewdocsonline.com/document/wpe6kq

its not very detailed, only reconfirmed what I had just figured out in the past hour.

do you have any additional information?

doau format is not very different from the doa3 xpr. "DOAXExporter.exe" is building a "debug.txt", which you can use to understand the difference.

imao format specs for the "*.cat" files is match more interesting because it contains such things like cloth, bones, boobs, texture animation, speech animation...

[ganjul]

Hey mariokart, I wanted to write an open source program for the DOAX xprs, maybe if you have some info on DOAU I'll support that format too.

The document is a bit confusing.
First it doesn't mention some obvious stuff like endianness.

After the xpr header, there is are the 3 types of “files”
First there is a magic number
0x80000000 for a model
0x00040001 for a texture
0x00800001 for a vertex buffer

Does this mean after the header there can be 3 types of files and a magic number determines which type it is?

OK, fine. A bit lower:

Following the model header is an array of dwords the count is in the model header “numobj”

Okay, what are these dwords for? He could at least meantion it's unknown.

So the 3 object offsets are
38 00 00 00
D2 0C 00 00
D6 11 00 00
These offsets are from the beginning of the file
So looking at offset 38 we see the first object block

Starting from this it doesn't make sense anymore. What object offset, what "beginning of the file"?

Sounds like you can make sense of all this, could you explain what he means?

[mariokart64n]

OK, I've written a format spec based on what I've been able to figure out in the past day.

please help fill in the holes if possible.

The models are stored into a xpr file, which from what I recall. someone mentioned that XPR was a microsoft file container for the xbox. I don’t know if this is true, but I’ve seen the xbox360 SDK, and it uses xpr too, only its a bit different. but the xbox360 xpr is used for storing textures.. and you can also embed a “user” file

so keeping that in mind the XPR is just a container holding the various “components” that the game reassembles during rendering. More commonly, Textures, and whatever else the dev stuffs in there relating to that texture resource.

Although I’m still not sure why they store part of the model in a separate file, since they could have stacked everything within the XPR.. but whatever, I’ll break down my findings so far

XPR FORMAT
the format uses chunk ids, meaning you keep reading an ID, and keep reading the data until a terminator is met. usually 0xFFFF
the structure is pretty simple, keep reading the chunks into memory, and use them to unpack the “Data Buffer” the data buffer always seems to be aligned to 2048bytes [0x0800]. the byte 0xAD is used to pad any between the start of the file and the start of the buffer. Then padding is used again from the end of the buffer to the end of the file.
Its best to read the chunks all into memory, then unpack once you’ve hit the 0xFFFF terminator.

*Note: Data Stored as Little Endian
*Note: there are different vertex types used, some storing vertex weights, some not.
*Note: Face Data seem to be Stored as Triangle Strips, separated by material
**Note:
a 0xFF terminator is seen at the end of the file, after you read the
texture and model entries. A 0xFF terminator is also used within the MDL Chunk.
Also a 0x00 terminator is seen at the start of the file, after the MDL chunk.
So keep that in mind, when re-writting a new XPR.


Model Format
After you read the model table, it’ll jump you to another table, before finally retrieving the geometry data in the buffer. But its a bit confusing because they do this weird filetree thing where chunks are inside chunks which are inside other chunks.



LONG: ChunkID

ChunkID = 0x[58505230] // “XPR0” Header, the first chunk [12bytes]
{
LONG: File Size
LONG: Data Buffer Offset
}

ChunkID = 0x[00000080] // Jumps to texture and mesh tables.
{
LONG: Offset, jump your cursor from this spot, continue to loop chunks until 0xFF is met
then return cursor to where you you left off reading the 0x[00000080] ChunkID
And then continue your loop until the terminator 0x00000000 is met.
)



ChunkID = 0x[01000400] // Texture Entry [32bytes per entry]
{
LONG: Data Offset, Jump From Start Data Buffer. Offset given in XPR0 header
LONG: Unknown: usually 0x00
LONG: Texture Details, data is stored in bits, dimensions etc.. haven’t broken it down yet
LONG: Unknown: usually 0x00
SHORT: Unknown: usually 0x01
SHORT: Unknown: usually 0x83
LONG: Data Size
LONG: Unknown: usually 0x00
}
ChunkID - 0x[01008000] // Model Entry [12bytes per entry]
// This Table is kinda useless, but just points you to the vertices used in the buffer
{
Long: Data Offset, from start of data buffer
LONG: Unknown: usually 0x00
}

ChunkID = 0x[290F6106] // Unknown
{
LONG: Unknown: usually 0x00
}

ChunkID = 0x[4D444C00] // “MDL ”
{
LONG: Object Count
LONG: Texture Count
LONG: Unknown: usually 0x00
LONG: Object Count, count duplicated ????
{ // Loop = Object Count
LONG: Object Offset, Jump from start of file
}
}

ChunkID = 0x[4F424A00] // “OBJ ”
{
LONG: Vertex Type ???
LONG: Unknown: usually 0x00 | 0x04
LONG: Unknown, may be a count??
LONG: Unknown, may be a count??
FLOAT[32]: Unknown, may be position coordinate
FLOAT[32]: Unknown, may be position coordinate
FLOAT[32]: Unknown, may be position coordinate
LONG: Vertex Count
LONG: Vertex Offset, points to data in data buffer, jump from start of file
LONG: Face Count
LONG: Face Offset, jump from start of file
BYTES[112]: Reserved Space??? data within range are all 0x00
{
LOOP: Loop until subID is 0xFFFFFF
{
LONG: SubChunkID, ** these IDs seem to denote different material properties
0x00000080 | 0x01000080 | 0x02000080 | 0x05000080
{
LONG: SubChunkSize
LONG: Unknown, Always 0x00
LONG: Unknown, Always 0x08
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Unknown
LONG: Number of Faces used by this material??? need to add 2?
LONG: Unknown, always 0x02
}
}
}

ChunkID = 0x[FFFFFFFF] // Stop Loop!
ChunkID = 0x[00000000] // Stop Loop!


****Anything above is stuff I’ve figured out my own by visual inspection of the files via a hexeditor. I still don’t understand the material chunks which are the key to the mesh import.

And theres alot of values there I need to test ingame to discover there function. but I’ll update once I solve this.



-mariokart64n

EDIT
I poked the material bytes and ran alot of tests in game.. none of them do anything lol

the only values that do anything are like the first three, they control transparency. and the last three control the material element need to draw the mesh.

I'm wondering if the materials are what the cat file does?? if so then why did they bother wasting all that space storing material info they're obviously not accessing

[ganjul]

Thanks mariokart.

LONG: ChunkID

ChunkID = 0x[58505230] // “XPR0” Header, the first chunk [12bytes]
...

You might want to put an "if" before ChunkID = something, although I'm probably one of the few who didn't get it straight away.

FLOAT[32]: Unknown, may be position coordinate
FLOAT[32]: Unknown, may be position coordinate
FLOAT[32]: Unknown, may be position coordinate

Can be easily checked by simply creating the vertices in max. You can easily tell random junk from something real even only given vertices in a 3d view. Or you could wait until I get something done in Blender.

EDIT: Wait, are we working on DOAU xprs or DOAX? DOAX xprs don't seem to have the header chunks you mention.
Also did you mean big endian?

EDIT2: I'm starting to get somewhere.

Code: Select all

# -*- coding: utf-8 -*-
import struct
class globals:
	pass

file = open('ayaA0.xpr', 'rb')

# the chunk ids seem big endian to me

# The file consists of smaller chunks, which start with a chunk id.
# Each chunk type stores different data and is different Just go
# through all chunks until you reach the end.

# functions for each chunk type
def header():
	print '\nHeader chunk\n----------------'
	#LONG: File Size
	globals.filesize = struct.unpack('<l', file.read(4))[0]
	print 'filesize:', globals.filesize
	#LONG: Data Buffer Offset
	globals.databufferoffset = struct.unpack('<l', file.read(4))[0]
	print 'databufferoffset:', globals.databufferoffset

def textureAndMeshTable():
	print '\ntextureAndMeshTable\n----------------'
	
	# LONG: Offset, jump your cursor from this spot, continue to loop chunks until 0xFF is met
	# then return cursor to where you left off reading the 0x00000080 ChunkID
	# And then continue your loop until the terminator 0x00000000 is met.
	offset = struct.unpack('<l', file.read(4))[0]
	print 'offset:', offset
	
def textureEntry():
	print '\ntextureEntry\n----------------'
	# LONG: Data Offset, Jump From Start Data Buffer. Offset given in XPR0 header
	dataoffset = struct.unpack('<l', file.read(4))[0]
	print 'dataoffset:', dataoffset
	# LONG: Unknown: usually 0x00
	file.read(4)
	# LONG: Texture Details, data is stored in bits, dimensions etc.. haven’t broken it down yet
	file.read(4)
	# LONG: Unknown: usually 0x00
	file.read(4)
	# SHORT: Unknown: usually 0x01
	file.read(2)
	# SHORT: Unknown: usually 0x83
	file.read(2)
	# LONG: Data Size
	datasize = struct.unpack('<l', file.read(4))[0]
	print 'datasize:', datasize
	# LONG: Unknown: usually 0x00
	file.read(4)
	
def modelEntry():
	# This Table is kinda useless, but just points you to the vertices used in the buffer
	print '\nmodelEntry\n----------------'
	# LONG: Data Offset, from start of data buffer
	dataoffset = struct.unpack('<l', file.read(4))[0]
	print 'dataoffset:', dataoffset
	# LONG: Unknown: usually 0x00
	file.read(4)
	
def unknown():
	print '\nUnknown\n----------------'
	# LONG: Unknown: usually 0x00
	file.read(4)
	
def MDL():
	print '\nMDL\n----------------'
	# LONG: Object Count
	objcount = struct.unpack('<l', file.read(4))[0]
	print 'objcount:', objcount
	# LONG: Texture Count
	texcount = struct.unpack('<l', file.read(4))[0]
	print 'texcount:', texcount
	# LONG: Unknown: usually 0x00
	file.read(4)
	# LONG: Object Count, count duplicated ????
	file.read(4)
	
	objoffsets = []
	print 'For each object:'
	for i in range(objcount):
		# LONG: Object Offset, Jump from start of file
		objoffset = struct.unpack('<l', file.read(4))[0]
		print '	objoffset:', objoffset
		objoffsets.append(objoffset)
	
def OBJ():
	print '\nOBJ\n----------------'
	# LONG: Vertex Type ???
	vertextype = struct.unpack('<l', file.read(4))[0]
	print 'vertextype:', vertextype
	# LONG: Unknown: usually 0x00 | 0x04
	file.read(4)
	# LONG: Unknown, may be a count??
	file.read(4)
	# LONG: Unknown, may be a count??
	file.read(4)
	# (FLOAT[32]) * 3: Unknown, may be position coordinate
	posx, posy, posx = struct.unpack('<3f', file.read(12))
	print 'position:', posx, posy, posx
	# LONG: Vertex Count
	vertexcount = struct.unpack('<l', file.read(4))[0]
	print 'vertexcount:', vertexcount
	# LONG: Vertex Offset, points to data in data buffer, jump from start of file
	vertexoffset = struct.unpack('<l', file.read(4))[0]
	print 'vertexoffset:', vertexoffset
	# LONG: Face Count
	facecount = struct.unpack('<l', file.read(4))[0]
	print 'facecount:', facecount
	# LONG: Face Offset, jump from start of file
	faceoffset = struct.unpack('<l', file.read(4))[0]
	print 'faceoffset:', faceoffset
	# BYTES[112]: Reserved Space??? data within range are all 0x00
	file.read(112)
	
	subids = [0x00000080, 0x01000080, 0x02000080, 0x05000080]
	
	while True: # Loop until subID is 0xFFFFFF
		# LONG: SubChunkID, ** these IDs seem to denote different material properties
		subchunkid = struct.unpack('<l', file.read(4))[0]
		print 'Subchunk id:', subchunkid
		
		if subchunkid == 0xFFFFFF: break
		
		# LONG: SubChunkSize
		subchunksize = struct.unpack('<l', file.read(4))[0]
		print '	subchunksize:', subchunksize
		# LONG: Unknown, Always 0x00
		file.read(4)
		# LONG: Unknown, Always 0x08
		file.read(4)
		# (LONG: Unknown) * 30
		file.read(4*30)
		# LONG: Number of Faces used by this material??? need to add 2?
		unknown = struct.unpack('<l', file.read(4))[0]
		# LONG: Unknown, always 0x02
		file.read(4)

# chunk ids and the functions to run
chunkids = {0x58505230: header,
	0x00000080: textureAndMeshTable,
	0x01000400: textureEntry,
	0x01008000: modelEntry,
	0x290F6106: unknown,
	0x4D444C00: MDL,
	0x4F424A00: OBJ}

# start looping through the chunks
while True:
	chunkid = struct.unpack('>l', file.read(4))[0]
	
	if chunkid in (0xFFFFFFFF, 0x00000000):
		break
	else:
		chunkids[chunkid]()

I think it doesn't parse till the end because I don't quite understand what you mean should be done with offsets.

# LONG: Offset, jump your cursor from this spot, continue to loop chunks until 0xFF is met
# then return cursor to where you left off reading the 0x00000080 ChunkID
# And then continue your loop until the terminator 0x00000000 is met.

Do you mean jump to offset, counting from current position, and continue reading chunks as usual before 0xFF is met and then come back to this position? But why do you mention to "continue your loop until the terminator 0x00000000 is met." here? Shouldn't we have been doing it all along?

[mariokart64n]

yeah that ID x80 00 00 00 gives you a jump relative to that address. but don't follow the jump yet. keep reading until you reach 0 to terminate your loop

then to evaluate the table that the ID x80 00 00 00 points you to. you just loop until -1 is reached.

I need to re-script, to include DOA3 and DOAX. the findings are based on DOAU.

I got meshes imported now, the thing I missed was where the face count is, that LONG before it is the face position to read from.
but some xprs are giving me errors, or bad mesh imports. I saw mention of vertex types so I need to investigate that aswell.

I'll post my script once its all coded, I need help figuring out the positioning info.. I know where it is, but they've formatted it weird.

they supply 4 float32, I know the first 3 are xyz. either the 4th is a vector or its a radian??? but neither seem to work as such

[ganjul]

Sorry man, but your wording confuses me at some places, I'll try to reread it again.

As for the 4th float value... if you are really sure it's really a float used for transforms, then I've only came across rotations stored in 3 floats, as radians or heading, pitch and roll (0:360) values. if it's a single value, I guess you could threat it as a vector and use the 3 position values as the origin. Technically this is enough to store both angle and scale. What problems are you having now? Are the 3 positions not enough?

And again, the chunk ids you posted seem to be in big endian.

EDIT: Nope, still not getting it. I also don't get what these mesh and texture "tables" are. All I get is they are after the chunks but before the data buffer. Also, are you going to post info on how the vertices and texture data is actually stored?

[mariokart64n]

I haven't figured out the tables yet actually...

anyway the 5th long in the XPR is the address to the tables. to make this not soo confusing, your going to read that LONG and add 0x14 to the address and jump from the starting of the file

next the table entries all start with a SHORT 0x0001 then followed by a SubID: 0x0004 = Texture Entry | 0x0080 = Vertex Entry

the vertex table is really useless since theres nothing we can do with raw vertices until we read the primitive chunks "OBJ "
whats important is the texture table, cause it contains the address and size of a single texture. we can also get the DXT type of the DDS texture and its dimensions

I haven't looked at any of the tables being there not critical to the mesh importation. the main problem right now is the meshes not positioning


another thing is I'm comparing XPRs from all of tecmos known games at the time.
- DOA3
- DOAX
- DOA1U
- DOA2U
- NG

the NG xpr contains the same 0x00000080 chunk id, so I'm actually wondering is that ID is the apart of the XPR filing system..

I'm trying to come up with a way to import all five games

[ganjul]

Well good luck to you.

However, I think you are terrible at writing file specs jk, it's probably just me. However, this is not the first file specification I'm reading, so If you'll have time please consider rewriting the file specs like the BRRES file specs from the smash bros wiki. it's really confusing right now. I really hope you'll at least consider it so I can write a similar tool for Blender.

[mariokart64n]

alot of my format details are based on my own assumptions, so you shouldn't feel confined by my format specs.
its best if you solve the format on your own. that way a consensus can be made on the specification of the format.

I posted what I know just to give others an idea of my own thoughts.
but since I'm not confident in what I posted, I won't be creating a wiki because its probably all wrong lol
-sorry

Hopefully someone can help me solve the positioning problem, because thats something that really needs to be solved before heading onto the secondary file ".CAT"


EDIT
Here is an example of some texture entries

Code: Select all

00000000   01 00 04 00 00 00 00 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 40 00 00 00 00 00 00   ............*.q.......ƒ..@......
00000020   01 00 04 00 00 44 00 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 84 00 00 00 00 00 00   .....D......*.q.......ƒ..„......
00000040   01 00 04 00 00 88 00 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 C8 00 00 00 00 00 00   .....ˆ......*.q.......ƒ..È......
00000060   01 00 04 00 00 CC 00 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 0C 01 00 00 00 00 00   .....Ì......*.q.......ƒ.........
00000080   01 00 04 00 00 10 01 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 50 01 00 00 00 00 00   ............*.q.......ƒ..P......
000000A0   01 00 04 00 00 54 01 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 94 01 00 00 00 00 00   .....T......*.q.......ƒ..”......
000000C0   01 00 04 00 00 98 01 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 D8 01 00 00 00 00 00   .....˜......*.q.......ƒ..Ø......
000000E0   01 00 04 00 00 DC 01 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 1C 02 00 00 00 00 00   .....Ü......*.q.......ƒ.........
00000100   01 00 04 00 00 20 02 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 60 02 00 00 00 00 00   ..... ......*.q.......ƒ..`......
00000120   01 00 04 00 00 64 02 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 A4 02 00 00 00 00 00   .....d......*.q.......ƒ..¤......
00000140   01 00 04 00 00 A8 02 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 E8 02 00 00 00 00 00   .....¨......*.q.......ƒ..è......
00000160   01 00 04 00 00 EC 02 00  00 00 00 00 2A 0B 71 07  00 00 00 00 01 00 83 00  00 2C 03 00 00 00 00 00   .....ì......*.q.......ƒ..,......

0x0100 is the ChunkID, then 0x0400 is the subID
if the subID is 0x0400 its a texture, if its 0x0080 its a mesh entry.. but notice theres another entry above, which is 0x8300... I believe this is just a general offset entry

anyway there some offsets there, which relate to the position and size of the listed texture. then in there you got this "7107" thats actually the compression and dimensions
all I did was take x=0x71 ((x-1)/0x10) to get 0x07
so I get Height=0x07 and Width=0x07
then to get the actual dimension in pixels, you just take 2 to the power of 7 to get the dimension


next here is an example of the mesh or vertex buffer table

Code: Select all

00000000   01 00 80 00 00 00 00 00  00 00 00 00   ..€.........
0000000C   01 00 80 00 00 6B 00 00  00 00 00 00   ..€..k......
00000018   01 00 80 00 00 C4 01 00  00 00 00 00   ..€..Ä......
00000024   01 00 80 00 40 E2 01 00  00 00 00 00   ..€.@â......
00000030   01 00 80 00 80 00 02 00  00 00 00 00   ..€.€.......
0000003C   01 00 80 00 40 EA 02 00  00 00 00 00   ..€.@ê......
00000048   01 00 80 00 78 12 03 00  00 00 00 00   ..€.x.......
00000054   01 00 80 00 58 13 03 00  00 00 00 00   ..€.X.......
00000060   01 00 80 00 D0 22 03 00  00 00 00 00   ..€.Ð"......
0000006C   01 00 80 00 2C 4B 03 00  00 00 00 00   ..€.,K......
00000078   01 00 80 00 EC 4B 03 00  00 00 00 00   ..€.ìK......
00000084   01 00 80 00 88 5B 03 00  00 00 00 00   ..€.ˆ[......
00000090   01 00 80 00 C8 F2 03 00  00 00 00 00   ..€.Èò......

it gives offsets to the start of every vertex set used by each object.. but no sizes

[ganjul]

I posted what I know just to give others an idea of my own thoughts.

I know, I just mean you could repost what you "think" you know. Right now some things are missing. Seems like you forgot to post some info which you forgot only you know, and there are places which are just confusing (the wording).

I don't have the required knowledge to reverse engineer formats. I've only written tools using existing format specifications created by others.

[mariokart64n]

lol
for me its the opposite, I have no education with programming. but have been looking around in a hex editor for a couple of years.. just as a hobby