Havok binary files

[SergeantJoe]

I made this for dark souls hkx files, maybe its applicable:

https://github.com/Danilodum/dark_souls_hkx/releases

Place hkx converter "SSFADF.exe" in root of hkx animation directory, double-click, it will recursively search for all animations (*.hkx except Skeleton.hkx and Skeleton-out.hkx)

For hkx files with this header (which I was referring to in my last post):

Code: Select all

Offset 0  1  2  3  4  5  6  7   8  9  A  B  C  D  E  F
00000 57 E0 E0 57 10 C0 C0 10  00 00 00 00 0B 00 00 00 WààW.ÀÀ.........
00010 08 01 00 01 03 00 00 00  02 00 00 00 00 00 00 00 ................
00020 00 00 00 00 4B 00 00 00  68 6B 5F 32 30 31 33 2E ....K...hk_2013.
00030 32 2E 30 2D 72 31 00 FF  00 00 00 00 FF FF FF FF 2.0-r1.ÿ....ÿÿÿÿ

SSFADF.exe says "bad file"

Hot damn, this could finally be it!

Files from the original game are dated 2011. If you cut out all the data before WaaW, and change 2011 to 2010, hkxcmd can now process them! Example (.bin is the original file, .hkx is the 2011->2010, -out.hkx is the processed file)

SSFADF recognizes them, but does not think they are animations for some reason. I also can't seem to load it in the standalone tool.

[Snaz]

I made this for dark souls hkx files, maybe its applicable:

https://github.com/Danilodum/dark_souls_hkx/releases

Place hkx converter "SSFADF.exe" in root of hkx animation directory, double-click, it will recursively search for all animations (*.hkx except Skeleton.hkx and Skeleton-out.hkx)

For hkx files with this header (which I was referring to in my last post):

Code: Select all

Offset 0  1  2  3  4  5  6  7   8  9  A  B  C  D  E  F
00000 57 E0 E0 57 10 C0 C0 10  00 00 00 00 0B 00 00 00 WààW.ÀÀ.........
00010 08 01 00 01 03 00 00 00  02 00 00 00 00 00 00 00 ................
00020 00 00 00 00 4B 00 00 00  68 6B 5F 32 30 31 33 2E ....K...hk_2013.
00030 32 2E 30 2D 72 31 00 FF  00 00 00 00 FF FF FF FF 2.0-r1.ÿ....ÿÿÿÿ

SSFADF.exe says "bad file"

Hot damn, this could finally be it!

Files from the original game are dated 2011. If you cut out all the data before WaaW, and change 2011 to 2010, hkxcmd can now process them! Example (.bin is the original file, .hkx is the 2011->2010, -out.hkx is the processed file)

SSFADF recognizes them, but does not think they are animations for some reason. I also can't seem to load it in the standalone tool.

Both hkxcmd and ssfadf can process up to 2012.2 hkx files, also, I don't believe ssfadf works well on xml

[SergeantJoe]

Nope, hkxcmd only supports 2010. It will take a 2011 file without a word, but then doesn't write anything.

I changed the version to 2010, converted the files with hkxcmd, then shoved the converted data into the structure used by Dark Souls files, and it WORKS!





Somewhat. The skeleton is 100% perfect, but the anim is all funky. It's like none of the bones move except the core one.

EDIT: I even managed to get the root motion! You can clearly see the base animation, but it's just not moving all the bones for some reason.

[Snaz]

You're right havok goes up to 2010. Maybe this could help https://github.com/Highflex/havok2fbx/releases

havok2fbx reads up to 2012

[SergeantJoe]

havok2fbx only works with 2014.1.0, it even specifically says so. But, all it takes is a quick rename from 2010 to 2014, and it works!

However, once again the skeleton is perfect but the anim is borked, in the exact same manner. What the heck is going on here?


An interesting point, is that if I removes bones from the transformTrackToBoneIndices array in hkaAnimationBinding those bones will not be affected by the animation, leaving them in their normal pose.
So, I'm fairly sure the format conversion is all correct, it must be the animation data itself that is doing that. Could it have gotten messed up during the binary->xml conversion?

I tried changing the header data and got SDAFS.exe to accept the binary file, but it crashes. Probably because the file has an hkaDefaultAnimatedReferenceFrame and lacks an hkaAnimationContainer and hkRootLevelContainer.
Do you think it would be possible to upload the source to the exe as well instead of just the dll? Maybe I can modify the program to only look for the anim data and nothing else.


Or could it be something else that's locking them at zero? I did notice the DS skeleton had the lockTranslation values all set to false, but changing that doesn't do anything.

[Snaz]

All I did was mix hkxcmd and havok2fbx a bit so I could get to the animation key data. Here you go:

https://github.com/Danilodum/dark_souls_ssfdaf

You'll need this too https://www.dropbox.com/s/uxfu2bkm907gx ... r1.7z?dl=0 , but this still won't do you much good, best would be to get the sdk havok2fbx uses and use that one to write something like ssfdaf, try asking highflex for the sdk

[SergeantJoe]

I switched hkaAnimationBinding blendHint from NORMAL to ADDITIVE just for giggles, and the animation is all there!




The hkx file structure is correct, but it's like all the bones are being rotated 90 degrees in the wrong direction. Fuck! I tried messing with the skeleton direction and animation binding, but no dice. I don't know what else to try at this point.


However it's worth noting, in the Havok SDK and Content Tools there are programs called AssetCc. Those can convert from binary to XML, and XML back to binary.
Unfortunately it gives the same exact result as a file converted with hkxcmd.

I do suspect the problem here is version mismatching.
All the files are 2011.2 while hkxcmd is 2010.2, the AssetCc I have from Snaz's SDK is 2012.2, and my content tools are 2014.1. Really wish there was a full download page somewhere.

[SergeantJoe]

You think I'm just going to give up when we're the closest we've been in four years? HA!

A complete breakdown of AssetCc:

Code: Select all

AssetCc1.exe --strip bin.hkx xml.hkx
AssetCc1.exe --strip --rules8011 xml.hkx bin.hkx

Conversion back and forth between binary/xml!

-The difference between Cc1 and Cc2 is that 1 keeps the original version number, 2 converts it to the number the exe it was built with.
-Doing --rules4101 instead of --rules8011 gives a slightly different binary file, although I'm not quite sure what the significance of the differences are.
-The --strip removes all the extra junk at the start of the file.

Now we can convert a file to xml, change a value, and convert it back to binary and see what's the difference.


It's just the thing is, everything seems to be working just fine already. The 2010 Skyrim anims work 100% perfectly with this tool, why are these all messed up?

I don't think it's the version number, I've tried converting the files to 2012 and 2013 binary with AssetCc2, then converting it to xml, but same thing.

And I still need samples of 2012 files from A:I. Maybe those will have the same issue.

[SergeantJoe]

-Doing --rules4101 instead of --rules8011 gives a slightly different binary file, although I'm not quite sure what the significance of the differences are.

According to the Havok Standalone Tool the Rules are platform versions:




So I suppose it's best to stick with 4101.

[SergeantJoe]

Still no luck with game->Max, so I tried the other way around.

To replace animations, you simply swap data starting with the "WaaW" and ending with "+ yyyyyy".





I have no idea what the heck those two header numbers do but they're clearly a memory size pointer or something.
If they're too large it crashes so I set them both the the minimum possible.

Works perfectly, at least for swapping animations that are already ingame.




Although I wonder if it would be possible to create 100% custom anims?
In the Havok Content Tools for Max the Prune modifier removes all the extra junk, leaving a file that is almost identical to the original.

Theoretically it would be possible to use AssetCC to convert the xml files to binary, then stick them ingame with the above method.

[Loomy]

Hey Lads, my sincere gratitude for all your continued efforts & brilliant progress so far ! And a special thx to the original/current author of hkxPack java client (sgtJoe ?).

I hope it's not too Off-Topic, but humbly requesting some brief advice or nudge in right direction WRT manual parsing of hkBinaryPacked Files, specifically BigEndian (PS3)

Also, like to share (a work in progress) 010 Template for Parsing hk-2011-01-r1 (classname/type/data - 48/48/48) - (was bundled with first or2nd VISION / anarchy release - still cant find it anywhere, so have had to make due with newer/older Libs without much success).



Currently stuck on the Vlookup, cant quite wrap my head around it, ive also included a sample file, if anyone can find the time to check it out. Would love a brief explanation, just fetching first one or 2 types/names, already have the 6 offsets from Names, and another offset in types/data, how to tie it all up.

- ( On a sidenote Are ClassID's different from Hk version to version, or is it just the Sig ?
- While PS3 flag in Hdr makes little difference, BigEndian flag will always cause problems with Deserialization (even if manually editing to 0 - and even removing Platform/Endianness checks in Source) - obv. needing BitSwap
-> so can one maybe parse this on a PPC /BE System (Linux Box/or Debug PS3 cache dump) .or would i need to manually parse,swapping uints/floats, so on
- or even as last resort, parse the entire thing manually (c# /c) and then reconstruct/instantiate the hkObjects one by one.

Would love any advice or your own experiences, would it be better for me to build a manual parser - as ive had nothing but grief with deserialization/unpack, though i havnt yet tried the SDAC/headerswap.

Keep up the great work, hope to make some contributions in near future, once i figure out my a$$ from my elbow with it all

Here is the 010 (Unfinished) Template / Structure Definition:- To use with ur own hkFiles, Change Fseek on Line12 to 0 or to the start address of hk WaaW header. Will update as & when i make progress

Download/View @ Pastebin
http://pastebin.com/HerN0A2U


Here is my working sample hkxPacked File (containerfile - hkHeader starts @ offset 54324 / 0xD434)

https://mega.nz/#!rsxWkZoI!nHT0ogG_oYY2 ... rhOtqLl6Jw

[HunterAP]

Hate to bump on this topic myself, but I was looking into Dark Soul III HKX files, which have the following header:


The weird thing about these files is that at lines 00000080 and some others are just the single character repeating, which I have not seen in any other HKX files.

Also, havok2fbx does not work due to the HKX files being 64bit rather than 32bit.
I tried editing the revision numbers and most of the binary information to match the 2012 r1 and 2010 r2 binaries, but to no avail.

Interestingly, I did find this program called hkxpack which does unpack the HKX files, but unfortunately it unpacks them into an XML which neither Noesis with the damnhavok plugin nor hkxcmd can read, and opening it up it can be clearly seen that typical HKX->XML conversions do not look the same.

Looking further into the hkxpack tool, I see that it outputs the HKX files as Tag-XML, where the importers I use expect regular XML, and hkxcmd is not converting them to regular xml or any other format.

EDIT: I tried editing the 64bit skeleton from DSIII and matched the format/structure as best as I can, but no matter what I do, even with getting the format 100% the same, it can't be converted to any other format through hkxcmd nor the Havok tools.

Does anyone know what can be done about these?

[shakotay2]

Does anyone know what can be done about these?

if it's about the skeleton only it can be extracted manually from the hkx file from my experience:

DarkSoulsIII_skel.jpg

save code as smd file:

Code: Select all

version 1
nodes
0 "Master" -1
1 "Lower_Root" 0
2 "Pelvis" 1
3 "bag_B_L" 2
4 "bag_b_R" 2
5 "bag_L" 2
6 "bag_R" 2
7 "L_Thigh" 2
8 "L_Calf" 7
9 "L_Foot" 8
10 "L_Toe0" 9
11 "L_Toe0Nub" 10
12 "LCalfTwist" 7
13 "L_Wepon_Case" 2
14 "R_Thigh" 2
15 "R_Calf" 14
16 "R_Foot" 15
17 "R_Toe0" 16
18 "R_Toe0Nub" 17
19 "RCalfTwist" 14
20 "R_Wepon_Case" 2
21 "Upper_Root" 0
22 "Spine" 21
23 "Spine1" 22
24 "B_Wepon_Case" 23
25 "B_Wepon_Case0_L" 23
26 "B_Wepon_Case0_R" 23
27 "L_Arm_Root" 23
28 "L_Clavicle" 27
29 "L_UpperArm" 28
30 "L_Forearm" 29
31 "L_Hand" 30
32 "L_Finger0" 31
33 "L_Finger01" 32
34 "L_Finger02" 33
35 "L_Finger0Nub" 34
36 "L_Finger1" 31
37 "L_Finger11" 36
38 "L_Finger12" 37
39 "L_Finger1Nub" 38
40 "L_Finger2" 31
41 "L_Finger21" 40
42 "L_Finger22" 41
43 "L_Finger2Nub" 42
44 "L_Finger3" 31
45 "L_Finger31" 44
46 "L_Finger32" 45
47 "L_Finger3Nub" 46
48 "L_Finger4" 31
49 "L_Finger41" 48
50 "L_Finger42" 49
51 "L_Finger4Nub" 50
52 "L_Shield" 31
53 "L_Weapon" 31
54 "L_ForeTwist" 29
55 "L_Shoulderpad" 29
56 "LUpArmTwist" 28
57 "Neck" 23
58 "Head" 57
59 "HeadNub" 58
60 "R_Arm_Root" 23
61 "R_Clavicle" 60
62 "R_UpperArm" 61
63 "R_Forearm" 62
64 "R_Hand" 63
65 "R_Finger0" 64
66 "R_Finger01" 65
67 "R_Finger02" 66
68 "R_Finger0Nub" 67
69 "R_Finger1" 64
70 "R_Finger11" 69
71 "R_Finger12" 70
72 "R_Finger1Nub" 71
73 "R_Finger2" 64
74 "R_Finger21" 73
75 "R_Finger22" 74
76 "R_Finger2Nub" 75
77 "R_Finger3" 64
78 "R_Finger31" 77
79 "R_Finger32" 78
80 "R_Finger3Nub" 79
81 "R_Finger4" 64
82 "R_Finger41" 81
83 "R_Finger42" 82
84 "R_Finger4Nub" 83
85 "R_Shield" 64
86 "R_Weapon" 64
87 "R_Weapon_to_left" 64
88 "R_ForeTwist" 62
89 "R_Shoulderpad" 62
90 "RUpArmTwist" 61
end
skeleton
time 0
0 0.000000 0.970140 -0.054000 0.000000 1.570796 0.000000
1 0.000000 0.000000 -0.000000 0.000000 -1.570796 0.000000
2 0.000000 0.000000 0.000000 3.141590 0.000000 1.570798
3 0.102369 0.067501 -0.118728 3.141593 -0.349069 1.570798
4 0.102369 -0.067498 -0.118728 -3.141590 0.349063 1.570798
5 0.102559 0.172900 0.055072 -3.141591 -0.174536 1.570798
6 0.102560 -0.172895 0.055071 -3.141590 0.174530 1.570798
7 -0.000000 0.103349 0.000000 -0.000001 -0.019934 -3.141591
8 0.443364 -0.000000 0.000000 -0.000000 0.189795 0.000000
9 0.423298 0.000000 -0.000000 0.000000 -0.169516 -0.000000
10 0.110321 0.000000 0.134743 3.141416 -1.570451 -3.141416
11 0.078623 0.000000 0.000000 0.000000 -0.000000 -3.141593
12 0.443364 -0.000000 0.000000 0.000000 0.189795 0.000000
13 0.035116 0.185465 0.005854 -1.623160 -0.697336 -3.141589
14 0.000000 -0.103349 -0.000000 -0.000001 -0.019934 -3.141591
15 0.443364 -0.000000 -0.000000 -0.000000 0.189797 0.000000
16 0.423298 0.000000 -0.000000 0.000000 -0.169518 -0.000000
17 0.110321 0.000000 0.134743 3.141416 -1.570451 -3.141416
18 0.078623 0.000000 0.000000 -0.000000 -0.000000 0.000000
19 0.443364 -0.000000 -0.000000 0.000000 0.189797 0.000000
20 0.035116 -0.185000 0.005853 -1.518440 -0.697336 -3.141589
21 -0.000199 0.089070 -0.000000 0.000000 -1.570796 0.000000
22 0.000000 0.027248 -0.000053 -3.141591 0.000000 1.570796
23 0.137738 0.000000 -0.000150 -0.000000 -0.000000 -0.000000
24 -0.002311 -0.000289 -0.158794 -0.000668 -0.000433 0.994837
25 -0.002311 0.097063 -0.158794 -3.141439 -0.000000 1.570796
26 -0.002311 -0.097060 -0.158794 -3.141439 -0.000000 1.570796
27 0.192637 0.054731 -0.035426 3.141591 0.000000 1.570796
28 -0.000000 -0.026113 -0.000000 0.000798 -0.000003 -0.000000
29 0.116357 0.000000 0.000000 -0.000796 -0.023052 0.000019
30 0.282076 0.000000 0.000000 0.000000 0.113538 0.000000
31 0.231722 0.000000 -0.000000 1.570797 -0.090485 -0.000000
32 0.022472 -0.020140 0.007940 -1.365101 -0.381454 -0.561813
33 0.042931 -0.000000 -0.000000 -0.000000 -0.026295 0.000000
34 0.024978 0.000000 0.000000 0.000000 0.000000 -0.000000
35 0.028448 -0.000000 -0.000000 -0.000000 0.000000 -0.000000
36 0.076584 -0.022206 -0.015239 0.000796 -0.000000 -0.000000
37 0.038286 0.000000 0.000000 0.000000 -0.000000 -0.000000
38 0.020783 0.000000 0.000000 0.000000 -0.000000 0.000000
39 0.026549 0.000000 0.000000 0.000000 -0.000000 0.000000
40 0.078816 -0.001225 -0.015238 0.000796 -0.000000 -0.000000
41 0.041986 0.000000 0.000000 0.000000 -0.000000 -0.000000
42 0.024718 -0.000000 -0.000000 0.000000 -0.000000 0.000000
43 0.031424 0.000000 0.000000 0.000000 -0.000000 0.000000
44 0.076947 0.019304 -0.015238 0.000796 -0.000000 -0.000000
45 0.039014 -0.000000 0.000000 0.000000 -0.000000 -0.000000
46 0.021675 0.000000 0.000000 0.000000 -0.000000 0.000000
47 0.027197 0.000000 0.000000 0.000000 -0.000000 0.000000
48 0.074308 0.038334 -0.015237 0.000796 -0.000000 -0.000000
49 0.032427 -0.000000 0.000000 0.000000 -0.000000 -0.000000
50 0.017567 0.000000 0.000000 0.000000 -0.000000 0.000000
51 0.023240 0.000000 0.000000 0.000000 -0.000000 0.000000
52 0.059999 0.000000 0.012112 -0.000001 -0.261800 0.000000
53 0.059999 0.000000 0.012112 3.141592 -0.000000 0.000000
54 0.282076 0.000000 0.000000 0.000000 0.113538 -0.000000
55 0.000002 0.000002 0.000000 -0.000000 0.023053 -0.000000
56 0.116357 0.000000 0.000000 -0.196404 -0.023052 0.000019
57 0.239683 -0.000000 -0.033470 0.000000 0.000000 0.000000
58 0.067741 -0.000000 -0.000000 -0.000000 0.000002 -0.000000
59 0.181918 -0.000000 0.000000 0.000000 -0.000000 0.000000
60 0.192637 -0.054729 -0.035426 3.141591 0.000000 1.570796
61 -0.000001 -0.026113 -0.000000 -0.000798 0.000003 -3.141593
62 0.116357 0.000000 -0.000000 0.000796 -0.023052 -0.000019
63 0.282076 0.000000 0.000000 0.000000 0.113538 0.000000
64 0.231723 0.000000 -0.000000 -1.570797 -0.090485 0.000000
65 0.022473 0.020140 0.007940 1.365101 -0.381454 0.561813
66 0.042931 -0.000000 -0.000000 0.000000 -0.026295 0.000000
67 0.024978 0.000000 0.000000 0.000000 0.000000 0.000000
68 0.028448 -0.000000 0.000000 -0.000000 -0.000000 -3.141593
69 0.076585 0.022206 -0.015239 -0.000796 -0.000000 -0.000000
70 0.038286 -0.000000 0.000000 0.000000 -0.000000 0.000000
71 0.020783 -0.000000 0.000000 -0.000000 -0.000000 -0.000000
72 0.026549 -0.000000 0.000000 -0.000000 0.000000 3.141593
73 0.078817 0.001225 -0.015238 -0.000796 -0.000000 -0.000000
74 0.041986 0.000000 -0.000000 0.000000 -0.000000 0.000000
75 0.024718 0.000000 -0.000000 -0.000000 -0.000000 -0.000000
76 0.031424 -0.000000 0.000000 -0.000000 0.000000 3.141593
77 0.076948 -0.019303 -0.015237 -0.000796 -0.000000 -0.000000
78 0.039014 -0.000000 0.000000 0.000000 -0.000000 0.000000
79 0.021675 -0.000000 0.000000 -0.000000 -0.000000 -0.000000
80 0.027197 0.000000 -0.000000 -0.000000 0.000000 3.141593
81 0.074309 -0.038334 -0.015237 -0.000796 -0.000000 -0.000000
82 0.032427 0.000000 -0.000000 0.000000 -0.000000 0.000000
83 0.017567 -0.000000 0.000000 -0.000000 -0.000000 -0.000000
84 0.023240 0.000000 0.000000 -0.000000 0.000000 3.141593
85 0.060000 -0.000001 0.012110 0.000001 -0.261800 -0.000000
86 0.060000 -0.000001 0.012110 0.000001 -0.000000 -0.000000
87 0.060000 -0.000001 0.012110 0.000001 -0.000000 -0.000000
88 0.282076 0.000000 0.000000 -0.000000 0.113538 0.000000
89 -0.000000 0.000000 -0.000000 0.000000 -0.023050 -3.141592
90 0.116357 0.000000 0.000000 0.196404 -0.023052 -0.000019
end

Some dumbo dev added additional 16 zeroes after R_Weapon_to_left so you've to care not to overview R_ForeTwist, R_Shoulderpad and RUpArmTwist when scanning for bonenames programmatically.

btw: I've always been asking myself how anyone could ever have the idea to hide such simple things in such a shitty format like hkx? (Hope it will be burried by MS in the deepest sea to be found, hahaha...)

[HunterAP]

Strange, whatever skeleton file you were looking at has only 91 bones, whereas the one I'm looking at (c0000/hkx/skeleton.hkx) has 110 bones. You can look at the file from this link.

How did you manage to convert from the HKX bone coordinates and rotations to SMD format?
If I could find a way to batch the process, I could get full DSIII animations out.

[volfin]

How did you manage to convert from the HKX bone coordinates and rotations to SMD format?
If I could find a way to batch the process, I could get full DSIII animations out.

This is a project I started last year for Alien:Isolation, it's a HKX converter that outputs SMD skeleton. It works with 2012.2.0 and 2010.2.0, I had intended to take it all the way with support for many versions, and animation output, but I simply moved on to other projects. It's written in C#, a lot of the file structure is defined at the top of Program.cs, and maybe will help people, so I'm making the code public.

https://bitbucket.org/Volfin/hkx2smd/overview