Page 1 of 2

rare (xbox) - grabbed by the ghoulies, conker

Posted: Sat Jan 03, 2015 3:35 pm
by WRS
i'm exploring the 3d models used by rare, starting with grabbed by the ghoulies 8)

models are named 'aid_model_ghoulies_' but reference other resources in a bnl container - such as textures and pools of floats

and the model files are horrible:

Code: Select all

header:
footer offset (absolute)
count

footer (for header->count)
chunk type
chunk offset (absolute)

chunks:
varying structures  - and there are ~20 chunk types

chunk ids
0: main model definition
1: ??
2: initial pose ?? 
5: ??
7: texture headers
9: flags ??

the main model info is further split into chunks. named this time, but also varying structures :(
i've attached the format (010 binary template) which parses most of the structures, but i'm stuck with describing the 3d data.

dumping one of the float pools shows it is definitely the vertex points, and some of the ushort pools in the model data->chunk 0->ndPushBuffer data are the triangles:

Image
(i think i'm using the wrong triangle mode here)

can anyone help with the skeleton? :)

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Sat Jan 03, 2015 3:36 pm
by WRS
other references

dxt texture data
Image

render:
Image

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Sat Jan 03, 2015 4:32 pm
by JayK
I might jump on this, thanks for the sample of the format

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Sat Jan 03, 2015 10:44 pm
by JayK
I think it's using some technique but I'm not sure, I can get this when shifting the array and building a second mesh, and then once more shifting the array and building yet another mesh, there's a lot of things wrong like the extruding faces, it's definitely not right, but it might help find the right path, hmm I'll look into it more

Image

edit: (or reading it as a quad and then doing a shift of 2 in the face array has the same effect)

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Sun Jan 04, 2015 12:48 am
by WRS
thanks for your update :)

i haven't made a lot of progress on the data pool.
the 26 textures total 251,392 bytes, and start at offset 358336 - which actually exceeds the data sample - so i cut it short...

before that are several pools of data, and i found some references to them after the ndSkeleton chunk is read

add this to the ND_CHUNK struct:

Code: Select all

            } skele_info <bgcolor=0x22ddff>;

            uint un1, un2;
            uint res_hdrs; // incorrect count? 
            uint pool[7];

            struct res_head
            {
                uint a, b, c, d;
                uint offset, size;
                Printf("%u, %u\n", d, offset);
            } res[res_hdrs +1] <optimize=false>;

this highlights several pools -
0, 39240 - vertex data - (3270 * x+y+z floats)
39240, 26160 - odd rounded floating point numbers (24, 33, 30) - no idea what this is
65400, 26160 - another float pool. values >= 0.0f and <= 1.0f so perhaps uvs
91560, 39240
130800, 26160
0, 0
0, 0
0, 0
156960, 13080 - entire table of -1 bytes...
i had limited success with triangle_strip with the first ndPushBuffer pool

Image

this doesn't work for other pools though

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Sun Jan 04, 2015 4:26 pm
by JayK
Ok so using the technique I was using (reading in as quads and then shifting the array by four)

I firstly read every push buffer into a face array, I noticed that at some point in the array it looks like its starting over [00 00, 00 00, 00 01] so I stopped reading it up to there. I only build the mesh straight from the array after its populated (not one push buffer at a time)

Still not quite there, after building there are still stray faces, so I think there must be parts at which I have stop reading the face array, and then start again (separate objects?) But here's the outcome.

Once reading in the two objects (array, then array shifted by 2) lot of stray faces, flipped normals
Image


After cleaning up the stray faces and unifying the normals for the two objects
Image

Then after combining the two objects and manually flipping a lot(but not all, became too tedious :p) of the face normals
Image

I think we're on the right track, but still need to look into it more.

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Thu Jan 22, 2015 12:43 am
by WRS
bump in hope this catches some attention :ninja:

i can pm model samples from gbtg

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Tue Feb 10, 2015 6:25 pm
by Szkaradek123

Code: Select all

About rare models:

For each model we have 2 files:
1.file info - with name "aid_model_ghoulies_actor_" + model name
2.file data - with name model name + "_resource_pool"

and about 1.
this file has sections which are joint together.
 - each section have header:	offset to section name int32+64
								chunk int16
								unk int16
								offset1 int32
								offset2 int32
								offset3 int32
								offset4 to next section int32+64 - if 0 than stop make sections joint
								unk int32
								unk int32
								offset5 int32
								counter int32
	example:							
		section with chunk 21 joints with section 18	
		section with chunk 18 joints with 11 and 20	
		section with chunk 17 joints with 21	
		section with chunk 12 joints with 17

and about 2.
	- access to this data has only section 19 - ndVertexBuffer
		go to offset5+64 in 1.file
		stream count=counter
		we have stream types:
			2316 - vertex position
			....
			....
			....
			
about faces:
	- access to this data has section 20 and if offset5 is eqal 1 than we have tristrips else we have quads.
about meshes:
	- access to this data have sections 19 and 23
		if section is 23 than all next section 19 joint in one mesh 

Updated 2015-02-11
Here is importer for textured models from Grabbed_By_The_Ghoulies [XBOX] .
It requires Blender249 and Python26.
How use:
1.unpack importer
2.run Blender249.blend
3.in Blender Text Window press alt+p and select:
- bnl - for unpacking archive
- info (files with "aid_model" in name)- for import rigged, static or backrounds meshes with textures (skinweights too, but bad bone map).
Importer is not perfect and requires a lot of work to get all models.

for testing: https://mega.co.nz/#!z9VCTTLR!p0Ps-IK4s ... K3iVe1vRBo

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Tue Feb 10, 2015 10:29 pm
by WRS
i cannot thank you enough Szkaradek123 - this is fantastic - thank you so much :)

edit

this is amazing work!!

Image


the textures are even nicer:
Image

----------------


Image

i want to say that my template didn't really come close to what you archived. thanks again.

  • the model data for grabbed by the ghoulies is stored in *.bnl files - i'm working on a tool to automate this.
  • for conker - the archive files are stored in *.caff files. i've reversed the checksum hash but the algorithm is still unknown

    sources: https://gist.github.com/x1nixmzeng/c55c248c384bf1c9df2a

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Sat Jun 20, 2015 10:41 pm
by WRS
'rare replay' has just been announced for the xbox one and contains all their old games!!

http://www.xbox.com/en-US/games/rare-replay

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Sun Aug 23, 2015 3:25 pm
by SubDrag
I am interested in this! Is there any comprehensive documentation on the format, what you guys figured out, or just the python importer? This is a fantastic tool. I wonder what leftovers are in here...

Also any way to automate ripping them all?

Too bad test_dialogs.bnl doesn't load.

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Wed Sep 02, 2015 10:09 pm
by WRS
SubDrag wrote:I am interested in this! Is there any comprehensive documentation on the format, what you guys figured out, or just the python importer? This is a fantastic tool. I wonder what leftovers are in here...

Also any way to automate ripping them all?

Too bad test_dialogs.bnl doesn't load.
luckily test_dialogs does not contain any meshes (:

you could write a script for blender to automatically convert them - although most of the textures need fixing up manually anyway (some uvs are broken in places)

there are plenty of leftovers though. if you're interested i have written tools for most of the file formats (exception animations......)

Image

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Sun Sep 06, 2015 3:13 pm
by SubDrag
Very interested in this, can you post your findings/tools! I'd love to help, or at least, see them :)
there are plenty of leftovers though. if you're interested i have written tools for most of the file formats (exception animations......)

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Fri Sep 11, 2015 11:58 pm
by WRS
SubDrag wrote:Very interested in this, can you post your findings/tools! I'd love to help, or at least, see them :)
k. for starters here are the known filenames i know of: https://gist.github.com/x1nixmzeng/1195c80d2c551fae5b7d

rough has implementation:

Code: Select all

unsigned int gbtgHash(const char* str)
{
  unsigned int hash = 0;

  // Skip "aid_" prefix
  str += 4;
  for (; *str; ++str)
  {
    hash = 16 * hash + (*str & 0xDF);

    unsigned int hiBit = hash & 0xF0000000;

    if (hiBit) {
      hash ^= hiBit | (hiBit >> 24);
    }
  }

  return hash;
}
the hash is pretty terrible with lots of collisions :ninja:

Re: rare (xbox) - grabbed by the ghoulies, conker

Posted: Sat Sep 12, 2015 1:07 am
by SubDrag
That's certainly a useful mapping. Do you have a simple text ripper, and a list of some findings? They seem to for some reason, have 00s in there between characters. Not sure if more special than that, probably some special characters in there. And do we know format of sound and movies?