Page 15 of 135

Re: Dead or Alive series formats and tools

Posted: Sat Jan 14, 2012 1:29 pm
by mariokart64n
gjinka wrote:There seems to be only 1 type of OBJ chunk documented. You guys say there are multiple and they store different things, you also say they can have multiple vertex/index buffers. How do you know how much? I think it would be simpler to just document the different types of the OBJ chunk.

I was asking these to b0ny but he suggested to wait until the format will be documented in the wiki, I kind of doubt it will, you guys just moved to other formats, which is great, but it would be better if the formats were documented on the way, people could create scripts for other modelling packages and stuff. I don't think it's too much to ask.
to my knowledge there is only one type of obj block.

between DOAU and DOA3 the material block is extended. that's the only difference.

the vertex type flag doesn't mean its a all new obj block, nothing changes except the stride of a vertex definition.

again I've only dealt with one type, anywho :P here's a sniplet from my maxscript, you can see that when I encounter a new vertex type I just skip more lol

Code: Select all

vx=readfloat fstream
vy=readfloat fstream
vz=readfloat fstream
nx=readfloat fstream
ny=readfloat fstream
nz=readfloat fstream
tu=readfloat fstream
tv=readfloat fstream
if vtype==1 do fseek fstream 4 #seek_cur
if vtype==2 do fseek fstream 8 #seek_cur
if vtype==3 do fseek fstream 12 #seek_cur
you could even use the vertex type as a actual stride size..

something like;
stride = ((4*vtype)+32)
^,^

Re: Dead or Alive series formats and tools

Posted: Sat Jan 14, 2012 1:40 pm
by b0ny
i had a look in some scripts and wrote this import script.
io_import_xpr.zip
it will load only the vertices from the first or the second vertex buffer for each OBJ:

Code: Select all

    file = open(realpath, 'rb')    # Universal read
    print('Importing %s' % realpath)
    
    try:
        dwXprMagic, dwXPRSize, dwHeaderSize, dwChunkID = struct.unpack("<4sLLL", file.read(16))
    except:
        print("Error parsing file header!")
        file.close()
        return
        
    # Handle XPR
    if dwXprMagic == b'XPR0':#is "xpr"
    		if dwChunkID == 0x80000000:
    		    print("mdl ID")#is first chunk a mdl one
    		    dwMDLSize, dwMDLMagic, dwNumObj, dwNumTxt, dwUnk0, dwNumIVBuf = struct.unpack("<1L4sLLLL", file.read(24))
    		    if dwMDLMagic == b'MDL\0':#is magic "mdl"
    		        print("mdl magic. objects %d"% dwNumObj)
    		        objects = []
    		        for object in range(dwNumObj): objects.append(struct.unpack("<1L", file.read(4))[0])
    # Handle OBJ
    		        for object in range(len(objects)):
    		            file.seek(objects[object])
    		            dwOBJMagic = struct.unpack("<4s", file.read(4))[0]
    		            if dwOBJMagic == b'OBJ\0': 
    		                print("%x obj magic"% (objects[object]))
    		                verts = []
    		                faces = []
    		                indices = []
    		                normals = []
    		                diffuse = []
    		                uv = []
    		              #load first vertex buffer
    		                file.seek(28, 1)
    		                dwNumVertices, dwVBufOffset, dwNumIndices, dwIBufOffset = struct.unpack("<LLLL", file.read(16))
    		                buffer = 0
    		              #if first vertex buffer not found load the second one
    		                if dwNumVertices == 0:
    		                    dwNumVertices, dwVBufOffset, dwNumIndices, dwIBufOffset = struct.unpack("<LLLL", file.read(16))
    		                    buffer = 1
    		              #go to buffer offset and read the fvf
    		                file.seek(dwIBufOffset)
    		                for i in range(dwNumIndices): indices.append(struct.unpack("<1H", file.read(2))[0])#H - usigned short
    		                for i in range(dwNumIndices - 2): faces.append((indices[i], indices[i + 1], indices[i + 2]))
    		                file.seek(dwVBufOffset)
    		                scale = 3
    		                for v in range(dwNumVertices):
    		                    vx, vy, vz = struct.unpack("<fff", file.read(12))
    		                    verts.append( (scale*vx, -scale*vz, scale*vy) )
    		                    if buffer == 0:
    		                        normals.append(struct.unpack("<3f", file.read(12)))
    		                    else:
    		                        diffuse.append(struct.unpack("<f", file.read(4))[0])
    		                    uv.append(struct.unpack("<2f", file.read(8)))
    		              #create object from verts and faces
    		                name = "vtf%d"% (object + 100)
    		                me = bpy.data.meshes.new(name)
    		                me.from_pydata(verts, [], faces)
    		                me.update()
    		              #add new obj to scene
    		                scn = bpy.context.scene
    		                ob = bpy.data.objects.new(name, me)
    		                scn.objects.link(ob)
    		                scn.objects.active = ob
    		                #go to edit mode and fix the mesh
    		                bpy.ops.object.mode_set(mode='EDIT')
    		                bpy.ops.mesh.select_all(action='SELECT')
    		                bpy.ops.mesh.normals_make_consistent(inside=False)
    		                bpy.ops.object.editmode_toggle()
yet have no idea how to handle anything else like material, textures, weight, normals or uv :(
i think now i'll look in the cat file for the "obj types"(head, body,...), and in the skeleton for coordinates and will place the body-parts to their places

Re: Dead or Alive series formats and tools

Posted: Sat Jan 14, 2012 1:47 pm
by mariokart64n
not everything is listed in the first block.

the first block contains a bunch of node linking info.. which we believed positioned the various objects to the skeleton.

however not everything is listed or assigned.

some things are revisited in other blocks. its complicated :\

Re: Dead or Alive series formats and tools

Posted: Sat Jan 14, 2012 4:41 pm
by b0ny
mariokart64n wrote:not everything is listed in the first block.

the first block contains a bunch of node linking info.. which we believed positioned the various objects to the skeleton.

however not everything is listed or assigned.

some things are revisited in other blocks. its complicated :\
the point zero in the objects coordinates is used as a pivot point and as a bone connection, and is also shared by the joint attached to this body-part.

in the cat file we find the the info about which OBJs are shadow shapes, low poly, high poly and joints; that allows as to assemble almost all objs to a model(by moving them to the points listed in the skeleton file).

the morphs can be loaded as time keyframes to make an animation. and i'm wondering if the "apa" magic block contains the lips speech info. and the physics can be imported as a hierarchy and interpreted by some python script for the blender game engine

i'm very impressed by this gta script:http://www.gtagarage.com/mods/show.php?id=9798
download it and look in it's manual - i think the doa script should be something like that. all non model info is available as an editable hierarchy. a script like this is like a fully-functional editor that will allow as to make complex skins from the scratch, without using an existing model as a base.

[Edit]

in the skeleton files(used by the cat file), the bones coordinates are relative to the parent bones, i've made a hierarchy list to calculate the real coordinates. but this hierarchy list doesn't contain branches for all bones

Code: Select all

2-ass
|_6-lheap
| |_7-lknee
|   |_3-lfoot
|
|_c-rheap
| |_d-rknee
|   |_9-rfoot
|
|_0-chest
  |_1-head
  |   
  |_8-lshoulder
  | |_4-lelbow
  |   |_5-lpalm
  |   
  |_e-rshoulder
    |_a-relbow
      |_b-rpalm
i added some lines to the script and it now looks for "skeletons" folder, and for cat file in the same directory as the xpr file we want to import, and use the information from these files to setup the high poly model with the joints.
xpr script.7z
this model was setup automatically by the script:
Image

Re: Dead or Alive series formats and tools

Posted: Sun Jan 15, 2012 6:08 pm
by mariokart64n
can you get DOAX skeletons? I can send you the XBE

Re: Dead or Alive series formats and tools

Posted: Sun Jan 15, 2012 7:41 pm
by b0ny
mariokart64n wrote:can you get DOAX skeletons? I can send you the XBE
send me the executable, i'll have a look...

you did a good job in researching "apv" morphing format. can you please research also the "apa" morphing format in doau cat files, and the "LNK" morphing format in doa3 cat files. i'd like to try to convert the doa3 morphing to doau but don't have enough experience and knowledge to do this myself.

Re: Dead or Alive series formats and tools

Posted: Sun Jan 15, 2012 8:18 pm
by mariokart64n
apa is morph data? thats strange that there would be anything outside of the face, hands and chest

Re: Dead or Alive series formats and tools

Posted: Sun Jan 15, 2012 9:12 pm
by gjinka
Oh, a Blender script, thats enough for me.

btw b0ny, do you still have questions about Blender Python? I'm still learning the new version of Blender, but maybe I could help.

Re: Dead or Alive series formats and tools

Posted: Mon Jan 16, 2012 12:59 am
by mariokart64n
the APA controls which morph target does what i think. and its hard to break it down, but there are basically 3 listings

1.) Hands
2.) Eyes(Blinking)
3.) Mouth

breasts morphs don't seem to be listed here. I deleted this whole bloc and they still bounced <_< soo...

anyways I think? this is the structure... but I don't know about the indinces, or the IDs given???..
Either way it is going to be hard to map out this section and what all the IDs actually mean?

Image

above you should see an image breaking the bloc into sections, I'll now attempt to describe each section

61 70 61 00
File Id "APA "

03 00 00 00
File Count, always 3??

2C 05 00 00 4C 05 00 00 74 05 00 00
3 Offsets that jump from the start of cat bloc1 to each APA subHeader


1.) Hands
00 00
01 01
0A 04
0D 05
06 08
03 0B

Index #1 [MorphID,TargetID]

01 02 02 03
05 02 06 07
08 02 09 0A
0B 02 0C 0D
FF 00 00 00

SubIndex #1 From second Table?? larger then count, donno what this is

00 00 00 00 08 05 00 00
04 00 00 00 08 05 00 00

SubTable From SubHeader #1 [count,offset]
offset from start of cat bloc1, jumps to another index

06 00 00 00 FC 04 00 00
02 00 00 00 1C 05 00 00

SubHeader #1 [Count,Offset]
following the two offsets they bring us to Index #1 and Table #1 from earlier in this bloc

0E 00 00 00
SubHeader #1 ID?



2.) Eyes(Blinking)
00 00
01 01
02 04
65 02
66 03
00 00

Index #2 [MorphID,TargetID]

05 00 00 00 40 05 00 00
00 00 00 00 00 00 00 00

SubHeader #2 [Count,Offset]
the count and offset are 0 for the 2nd entry, thus the second table is not seen above

05 00 00 00
SubHeader #2 ID ??



3.) Mouth
00 00
01 00
02 01
03 01
04 03
06 02
05 07
07 06
65 04
66 05

Index # 3 [MorphID,TargetID]

0A 00 00 00 60 05 00 00
00 00 00 00 00 00 00 00

SubHeader #3 [Count,Offset]
the count and offset are 0 for the 2nd entry, thus the second table is not seen above

05 00 00 00
SubHeader #3 ID ? ?



EDIT1
I've been looking at the first morph listing (Hands)
seems the model contains more morph targets then are actually used ingame
I'll list as I find them out, these might not even be consistent?
Anyway changing the ID changes both hands.. so I'm not sure why the hands have 2 different lists ??


Hand IDs (Partial)
0x00 - Neutral
Image

0x01 - Fist
Image
0x02
0x03 - Cliffed Fingers
Image
0x04
0x05
0x06 - Fingers Out
Image
0x06
0x07
0x08
0x09
0x0A - Middle Finger Arched
Image
0x0B
0x0C
0x0D - Finger Gun
Image
0x0E



Eye IDs (Partial)
0x00 - Neutral
Image

0x01 - Eyes Closed
Image

0x02 - Wink
Image

0x65 - Eyes Closed (Sad)
Image

0x66 - Eyes Open (Sad)
Image


Mouth IDs (Partial)
**note index from list did not matchup with my dump.
example I have 12 face morphs total, 6 are eyes morphs and the other 6 are mouth morphs
my morph index is obviously 0 - 12, however the game doesn't index from 0 - 12, it indexes from 0 - 6
..the eye and mouth morphs for the face are in two different sets. I'm unsure how to seperate the face morphs into these 2 categories :(

>>>index resets to 0

0x00 / 0x01 - Neutral / Mouth Closed
Image

0x02 / 0x03 - Mouth Open
Image

0x04 - LowerLip Down
Image

0x05 - Smile
Image

0x06 - Teeth Clenched
Image

0x07 - Mouth Oval
Image





Re: Dead or Alive series formats and tools

Posted: Tue Jan 17, 2012 2:28 pm
by b0ny
doax skeletons have exactly the same format as doau skeletons(both games use the same engine),
but in doau the doax bones are not used(are null).
i added these null bones to the doa3 skeleton and now it's also in doau format(can be inserted in doau cat file).

here is the archive:
skeletons.7z
both doau and doax games contain the same model - "prarie dog". it's great that i can now compare the differences between doau and doax format to make a converter from doax to doau :)
gjinka wrote:Oh, a Blender script, thats enough for me.

btw b0ny, do you still have questions about Blender Python? I'm still learning the new version of Blender, but maybe I could help.
yes, i have a lot of questions on python importing functions. in this script i only imported the coordinates from the vertex buffers, but i'd like also to now how to import the materials, uvmapping, normals and textures :!:

Re: Dead or Alive series formats and tools

Posted: Tue Jan 17, 2012 4:21 pm
by gjinka
1. You make textures like this:

Code: Select all

image = bpy.data.images.new(name, width, height, alpha = 1)
image.pixels = pixels
# save in blend
image.pack(as_png = True)
	
texture = bpy.data.textures.new(name, type = 'IMAGE')
texture.image = image
texture.use_alpha = True
Where "pixels" is a python list of rgbargbargbargba... color values. The color values have to be a float for some reason, so if you have an int in the range (0,255), divide it by 255.

2. Make materials like this:

Code: Select all

material = bpy.data.materials.new('materialname')
# ambient
material.mirror_color = ambient[:-1]
material.ambient = ambient[3]
# diffuse
material.diffuse_color = diffuse[:-1]
material.alpha = diffuse[3]
# specular
material.specular_color = specular[:-1]
# emissive
material.emit = emissive[3]
# shininess
material.specular_hardness = hardness
	
# assign textures to materials
mtex = material.texture_slots.add()
mtex.texture = textures[index] # textures is a list of texture objects you created above
mtex.texture_coords = 'UV'
3. As for assigning the materials to faces, do this:

Code: Select all

for material in materials:
    mesh.materials.append(material)

Code: Select all

for face in mesh.faces[fromindex: toindex]:
    face.material_index = yourindex
4. For UVs:

Code: Select all

uvtex = mesh.uv_textures.new()
uvtex.name = 'DefaultUVMap'
for face in len(mesh.faces):
	uvtex.data[face].uv1 = uvs[face * 3]
	uvtex.data[face].uv2 = uvs[face * 3 + 1]
	uvtex.data[face].uv3 = uvs[face * 3 + 2]
	# 3 if you have triangles
where "uvs" is a python list of uv values stored one after another.

5. For normals:

Code: Select all

index = 0
for vertex in mesh.vertices:
	vertex.normal = normals[vindex]
	index += 1
6. For vertex colors:

Code: Select all

mesh.vertex_colors.new(name = 'VertexColor')
index = 0
for face in mesh.vertex_colors[0].data:
	face.color1 = colors[index]
	face.color2 = colors[index + 1]
	face.color3 = colors[index + 2]
	index += 3
7. For armatures and bones:

Code: Select all

bpy.ops.object.add(type='ARMATURE', enter_editmode=True)
object = bpy.context.object
object.name = 'ArmatureObj'
armature = object.data
armature.name = 'Armature'

Code: Select all

for i in range(your_bone_count):
    bone = armature.edit_bones.new('yourname')
    bone.tail = mathutils.Vector([0,0,1])
    # is it stored as matrices?
    #bone.transform(matrix)
	
# go to Object mode
bpy.ops.object.mode_set(mode='OBJECT')
That's as much as know. If you'll learn how to assign bone indexes and bone weights, let me know.

Re: Dead or Alive series formats and tools

Posted: Wed Jan 18, 2012 5:26 am
by mariokart64n
yes! finally I understand the DOA3 material block. MAN! O_O


I had structures detailing the doa3 material section before hand. but they called the section a subset?? and didnt give any info on how to read it or what it did..
Then I used the DOA3 Exporter, and it dumps a debug.txt file, which describes all the sections.

basically instead if the materials just giving you a offset to the face buffer and a count. in doa3 they detail EVERY strip.
its like there giving a new index for each degenerate face I think... but thats a total waste of space. they should have just used triangle list.

whatever, finally DOA3, DOAX, and DOAU are cracked once again! now to start working on importing data from the cat

Re: Dead or Alive series formats and tools

Posted: Wed Jan 18, 2012 2:18 pm
by gjinka
Oh nice. I havent had enough time to study the whole blender script, but two things I noticed bony:
1) you can actually store a list of length 0 or 1 in python. Why wouldn't you be able to?

Code: Select all

list.append([]) # add an empty list to a list
# OR
a = []
list.append(a)

list.append([1]) # add a list with 1 member to a list
# OR
a = [1]
list.append(a)
2) instead of writing "BBBBB" for the struct module, you can just write "5B". Even "2B3B" would work.

Re: Dead or Alive series formats and tools

Posted: Wed Jan 18, 2012 7:53 pm
by b0ny
gjinka wrote:Oh nice. I havent had enough time to study the whole blender script, but two things I noticed bony:
1) you can actually store a list of length 0 or 1 in python. Why wouldn't you be able to?

Code: Select all

list.append([]) # add an empty list to a list
# OR
a = []
list.append(a)

list.append([1]) # add a list with 1 member to a list
# OR
a = [1]
list.append(a)
2) instead of writing "BBBBB" for the struct module, you can just write "5B". Even "2B3B" would work.
i spent a lot of time with a "integer has no length" error, till i found this page http://en.wikibooks.org/wiki/How_to_Thi ... Solution_1(CH 9 - Solution 1) that says that you can't access lists inside a list if it contains lists of length 1(and maybe 0):

Code: Select all

mylist.append((1,2,3))
mylist.append((98,32))
mylist.append((3))
#because of the third "append", you can't do this:
len(mylist[1]) #"integer has no length" error
#or this 
myvar = mylist[1][1] #"can't subscript an integer" error (or something like that)
#it's like when you can't do like this:
myvar = struct.unpack('<L', file.read(4))
#you should do that instead
myvar = struct.unpack('<L', file.read(4))[0]


[edit]

i looked in the mot files. a move is like a bunch of offsets that point to some "keyframe" data. i have no idea about the actual animation data but i found that one move have exactly 44 offsets there is what every offset means('3.' - means that there are three offsets):
mot struct(size44):
3.side/ud/fb body
3.fb/rot/side-angle ass
3.fb/rot/side-angle chest
3.fb/rot/side-angle head
3.side/ud/fb back palm pos
3.global back palm rotation
2.? (shoulder+elbow rotation)
3.side/ud/fb forw palm pos
3.global forw palm rotation
2.? (shoulder+elbow rotation)
3.side/ud/fb back foot pos
3.global back foot rotation
2.? (knee+heap rotation)
3.side/ud/fb forw foot pos
3.global forw foot rotation
2.? (knee+heap rotation)
doax animation is different it have 56 'bones' (offsets) and some additional data in the offsets block. possible the animation stream is also different

Re: Dead or Alive series formats and tools

Posted: Wed Jan 18, 2012 9:12 pm
by gjinka
It works for me and I believe it should for you too, unless there is some math paradox I've never heard of.
So what are you doing? Can you post an exact code which crashes with the error message?

Also not that len() is meant for lists/tuples and dictionaries, which can have multiple elements. len() just tells the number of elements. An integer has "one element", its value, so len() on it is pointless.
If you want to use an integer for a while loop, do this instead:

Code: Select all

for i in range(intgervalue):
    # do something

Code: Select all

#it's like when you can't do like this:
myvar = struct.unpack('<L', file.read(4))
#you should do that instead
myvar = struct.unpack('<L', file.read(4))[0]
That's because struct.unpack() returns a list, even if it has one value, so you need to do [0] to assign the first value instead of the whole list.

Also, did my Blender code help in any way?