Ninja Gaiden 2 x360

[MrAdults]

Well slap my ass! Thanks for mentioning that chrrox, or I probably wouldn't have even bothered looking for a freely available xex decryption tool. I'd just assumed such things would be crushed by Microsoft immediately.

Well, after decrypting NG2's xex, sure enough, there's the full file table. Seems like NG2's animations are "mtn" files, most likely very similar or identical to DOAX2's "mot" files. Unfortunately, though, this does not solve the mystery of either format regarding bone weights.

Also, chrrox, I still haven't gotten around to investigating those Fist of the North Star files. I'll have a look once I've either solved or given up on proper skeletal/bone index data in these Tecmo models.

[revelation]

The list of indices at the end of each joint are indeed the children. There a few gmd files for NG2 that have more then 3 children, a couple have had 5 in some of the files i have seen. Unfortunately as with most of the other formats i have worked, as i have mentioned my code base is not in a state that easily allows me to visually verify things out of the hex editor. But i have stepped through those indices and the parent value as i have mapped them match and seem to verify the parent child relationship. i will see if i can get my code u pand running to render properly and see what happens. Do you have the DOA file you mentioned earlier so that i could possibly compare aginst what i am seeing within the NG2 files? Otherwise i will see what i can do in the meantime, heh. i will probably go ahead and atleast output the vertex declarations for the files i am investigating and start mapping what indices are being reported for the vertices as well.

[Andersen]

The contents of this post was deleted because of possible forum rules violation.

[MrAdults]

Well, I think there is some legit skeletal data in NG2, but none in most (maybe all) of the DOAX2 girls. They use morph targets and seem to be animated by unique "influence lists" in a pretty fascinating way, although this is purely a guess based on what I was able to make out with only a hex editor and some screwing around with bits to try to decode the motion stream. NG2 models probably use that extensively as well, though I didn't actually look at NG2's motion data.

But since it's its own unique flavor of data that I in reality have no clue whatsoever about, I'm just gonna throw in the towel on this one and release source for my toolset module right here. Primarily for other programmers that happen upon this in the future and go "man, why didn't they post specs or source?!", since I know Surveyor and revelation already have most of this stuff documented privately. As usual, it's just ripped right out of the rest of my toolset and is in my sloppy-as-hell hammered-out-overnight style, so it's only enough to get the proper specs for anyone who wants to do their own project with this data.

[Surveyor]

Hi,

Would anyone know or can figure out, how to get the ng2 stuff out from the dlc packs?

The fiend hayabusa you posted here contains 2 gmd files + some other data, the texture extractor works on both models but not the model converter as they changed the data base adress to something else. I'll have a deeper look, meanwhile, you can use a hex editor and get the 2 gmd's and apply the texture extractor on them.

Unfortunately, though, this does not solve the mystery of either format regarding bone weights.

It'll probably help if you have the ng2 sigma (ps3) models, from what I saw, ps3 files seem clearer, unfortunately I deleted them when I dropped the reasearch on this format, You can ask Chrrox if he still has them.
It's also worth mentioning that all doax2 body models share the same structure in terms of vertex buffer and index buffer alignement:

In a post on page 6, I showed a rigged model of a robot enemy, I've chosen this model because it's a robot and robots have rigid limbs so there's only one bone per vertex. skin weights turned out to be correct, and I used the joints in the "Hie" chunk to get this pose which leads to the conclusion that these "joints" actually play a role in animation. But when sikn weights are applied to soft bodies (hayabusa or sonia) they don't map correctly to the joints (for example left thumb bone influences right leg vertices) so there's probably a bone group chunk somewhere to be discovered.
All this still doesn't explain how animation is being performed, for example the hands on ng2 characters: they have 1 joint and still move beautifully unlike in doax2 where the girls look like if they had plastic hands and there's a very limited set of hand poses (when they push against sand, the hand pose looks horribly unrealistic) IMO, Tecmo developers used morphing on Doax2 then switched to bone animation in Ng2.

In an interview with the developers (present in the Softimage website - showcasing how incredible softimage is), they talked about how they used motion capture to get face animation and the pic below shows plenty of face bones which can't be found in the model files, one way to make sure skin weights are indeed used (I hardly imagine a next gen engine relying on morphing) is to print these weights as shown in the pic below.

I've been through many japanese made games and one thought always comes into my mind: the only thing japanese developers excel at is art, their programming is a total mess, don't you agree ?

[MrAdults]

Yeah, displaying weights like that might help. Assuming you have the bone index data on the verts in NG2, to know how to color according to bone reference. As mentioned, they don't exist in the DOAX2 models, but I really haven't looked for bone indices in the NG2 models at all. If you think there is traditional bone data, I guess you must have spotted them. But something bone-like surely exists in DOAX2, still (something to utilize those "indexless" weights), and it's got to be buried in the motion data if it's anywhere. So if NG2 uses traditional bones, it's pretty likely it uses bones AND whatever DOAX2 is using.

Actually, I should say DOAX2 *does* seem to use "bones", the bones we see in the Hie chunk. But what's missing are the bone indexes, which leads me to believe those bones are not used in a traditional manner. If they really animated the bodies normally with vertex morphs, there would be a whoooole shitload of data. No way that's in the motion files, cause you can see quite plainly in most sections they are not compressed (and never mind that, the cost of all those morph frames in memory would be horrifying, with their vertex counts).

Those motion files have pretty clearly defined offset tables and sections, and some of the data streams are easily-readable, but I haven't actually tried parsing the sections out and examining each. A lot of the data in there could easily be a "reference list" for the weightless verts, if you want to interpret it that way. But man, that just seems so ridiculous, to store those arrays in your motion data, that I don't even buy into the theory enough to try it out. Then again, if it's not there, it's nowhere. What a mess indeed.

I did notice that in the DOAX2 girls, for example, the whole lower torso does seem to only have a couple different locations it seems to want to be weighted to. In fact, I can map out those weights to a couple bones from the Hie chunk, and it does look perfect (can bend it nice and fluid in Max with the Hie skeleton). But I haven't seen references to those bones anywhere in the actual XPR2... But it really makes the silly "only 4 bones per draw" limitation quite believable for the DOAX2 models. Quite possible the mot data just contains that obscure "which 4 bones to use for which surface" connection inside of it.

I'm quite curious if revelation has seen anything I didn't in Tina's body XPR.

Team Ninja does seem to have done some weird crap in here, and from what I do know for sure of the DOAX2 models, there sure is a hell of a lot of vertex memory going to waste. But, at least in NG2's case, it's hard to argue with the final performance+visuals they managed to pull out.

[mariokart64n]

my theory through observing the game play, was that they based the mechanics off the old arcade version of DOA1. meaning your character has no skinning, just animating nodes which a series of blocks are connected to.

so in the new DOA, they used the same principle. not uncommmon really, they dice the models into pieces and make everything a static mesh attached to the animation nodes. then on only certain required meshes, need for skinning. they applied some sort of vertex blending, to deform that particular mesh.

like example

[HAND(Static)] - [Wrist(Skinned)] - [Forearm(Static)] - [Elbow(Skinned)] - [UpperArm(Static)] - [Shoulder(Skinned)]


when I asked my friend if he could find anything in the CAT files for DOAu for xbox1. all he found where specious vertex lists
your write these points into a 3d editor and its pretty clear what they were. they just appear to be the edge vertices where each Skinned mesh joins to each static mesh. so basically they are attaching or welding each skinned piece to every static piece. but again looking at the game play, no mesh welding for smoothing is preformed.. many of times you can see those mesh parts separate O.O

so thats where my theory comes to. DOA never used skinning at all. just static meshes, a vertex welding or attachment list held the skinned parts to each static part. although the skinned meshes are not really skinned, because I believe the engine uses some auto rigging method, to blend the transitions together when they deform. not sure why they wanted to avoid this?

anyway these are my thoughts being a gamer of DOA, since I first played it on PS1.

thanks for reading;
-mariokart64n

[MrAdults]

Hi Mario_Kart (haven't I seen you post on my web site before?),

I had suspect something like this too, except it's not so much "welding data" as it is just a very small batch of "bones" as seen in the Hie data of this format (in DOAX2's case, dunno about previous titles).

You definitely can't see any "seams" whatsoever in DOAX2 and the girls all bend very nicely. But I believe the bones we see combined with vertex morphs and just a few bones per surface (since the meshes are split up so much) would be more than enough for those results.

It's a bizarrely primitive method, though, if it is really what they've done. Rare for a game made this generation, or even the last! Although supposedly Team Ninja's programmers are the best in the world at what they do, at least according to Itagaki, so who am I to question?

Edit: Oh yeah, Surveyor, I took a look at that DLC too, but I noticed there are big chunks of crap spread throughout all the files! It exists in some of the textures too (you can see the corrupted areas if you try to take the data as-is, or at least I can). Must be an oddity of the "LIVE" container, or could it just be a bad rip?

[Surveyor]

Although supposedly Team Ninja's programmers are the best in the world at what they do at least according to Itagaki

And who the hell is Itagaki?? Well there's something very unique in this ng2 engine (comapred with the current gen engines) is that instead of going bumpy on low res models (John Carmack's bump map everything approach) this engine uses high res models and very little bump maps, getting the same performance/visuals as a regular engine, but here artists don't have to go through the process of downscaling their models and are free to unleash their artistic rage (and thus are artistically better games)

[MrAdults]

Although supposedly Team Ninja's programmers are the best in the world at what they do at least according to Itagaki

And who the hell is Itagaki?? Well there's something very unique in this ng2 engine (comapred with the current gen engines) is that instead of going bumpy on low res models (John Carmack's bump map everything approach) this engine uses high res models and very little bump maps, getting the same performance/visuals as a regular engine, but here artists don't have to go through the process of downscaling their models and are free to unleash their artistic rage (and thus are artistically better games)

Kind of...although really, the raw triangle counts on Ryu for example are only around 35k-40k - it fools you, because there is a separate "shell" mesh in the same model file! You can delete a whole layer from the model and it looks exactly the same. I don't know if they chose to use this method to render a separate specular layer or what, although that would be even more silly if so. It definitely is not a LOD, because the geometry is identical.

So compared to this, games like FF13 use similar triangle counts (25k-45k, usually), but with fully normal-mapped models, which clearly came from many-millions-of-triangles, hand-sculpted models. In this regard, I think Team Ninja didn't do a great job. But their art does still look great, in the game, and it runs blazingly fast. I think, in the end, this is all a render guy should wish for, so they still did their job well enough.

[Tosyk]

Hi to all. Can we convert the model already with bones and weights?

[undead]

nop

[revelation]

Well,
After all that not sure how many people are still even looking into this format, heh.

Not too much new, although i have had the chance to find a verify a few things. As such i think i have found the joint indices necessary for the DOA skinning data. As mentioned it does appear to usually max at around 4 indices, if they are present at all.

Code: Select all

struct DATA_HEADER
{
	// 0x00
	union
	{
		uint64 tagValue      <format = hex>;
		char   tagString[8];
	} dataTag;
	
	uint16 unknown0x08[2]  <format = hex>;
	uint32 headerSize      <format = hex>;
	// 0x10
	uint32 dataSize        <format = hex>;
	uint32 tableCount; // total offset array count
	uint32 dataCount; // number of valid entries in table
	uint32 unknown0x1C;
	// 0x20
	uint32 tableOffset     <format = hex>;
	uint32 unknown0x24[3];
	
	local int64 filePos = FTell();
	FSeek(startof(this) + tableOffset);
	uint32 dataOffsets[tableCount] <format = hex>;
	FSeek(filePos);
};

struct OBJECT_INFO_DATA
{
	// 0x00
	uint32 index;
	uint32 unknown0x04;
	uint32 unknown0x08;
	uint32 unknown0x0C;
	// 0x10
	uint32 unknown0x10[4];
	// 0x20
	float3x4 unknown0x20;
};

struct OBJECT_INFO
{
	struct DATA_HEADER header;
	
	// 0x30
	uint16 unknown0x30;
	uint16 unknown0x32;
	uint32 unknownIndex; // since Geo, Info, and Joint counts usually match, i am assuming just a local copy of this structures index
	uint32 unknown0x38;
	uint32 jointCount;
	
	struct OBJECT_INFO_DATA unknownData;
	
	if (jointCount > 0)
	{
		uint16 jointIndices[jointCount];
	}
	
	if (header.tableCount > 0 && header.dataCount > 0)
	{
		struct
		{
			local int64 dataOffset = 0;
			local uint32 dataNum = 0;
			for (dataNum = 0; dataNum < header.tableCount; ++dataNum)
			{
				if (header.dataOffsets[dataNum] != 0)
				{
					local int64 filePos = FTell();
					dataOffset = (startof(parentof(this)) + header.dataOffsets[dataNum]);
					FSeek(dataOffset);
					struct OBJECT_INFO_DATA objectData;
					FSeek(filePos);
				}
			}
		} childData;
	}
}

Sorry for the syntax, as this is only part of a larger template for 010 Editor. If not obvious these structures correspond to the objInfoMeat_s and objInfoHdr_s structures from the code posted by MrAdults. The only real important thing here is the jointCount value at about 0x3C bytes into the ObjInfo data and the associated list of indices after the initial data structures floats.

Hopefully those that have a more readily available rendering framework for these formats will be able to test and verify and/or disprove this information. Hopefully this will work out for the DOA information.

Unfortunately, so far i have not found too many files that have this information present when it come to NG2 data. Since the joint indices are present in the actual vertex data, unlike DOA, i am assuming a similar list, initial joint and count, or other sort of indicator is present somewhere to specify which joints are actually used for each mesh. Still a good bit of unknowns as you can tell from some of the information above.

i really need to make some time to get some real coding in, heh.

Hope it helps.

[undead]

first of all thanks for the tools

could someone tell me how to solve the smooth problem
I'm new at this and I could not solve

[MrAdults]

Nice find, revelation!
http://img89.imageshack.us/i/tinapose.jpg/

There's a bit of hierarchical dysfunction going on (some chains are flattened, surely compensated for in motion data), but all the weights work nicely with the bones they're associated with.

I can't believe there ended up being a 4-bones-per-draw limit after all, but it looks like we've finally got evidence to demonstrate such.

Oh yeah, and those indices are actually into the *mesh* list, then you use the index of the bone which is pointed at by the mesh.

first of all thanks for the tools

could someone tell me how to solve the smooth problem
I'm new at this and I could not solve

Surveyor knows how to decode normals now, so it'll probably be fixed whenever he gets around to updating his tool. Otherwise you'll have to smooth out normals manually, but auto-smoothing probably won't look perfect on many models.