Important information: this site is currently scheduled to go offline indefinitely by end of the year.

Extracting simple models

Read or post any tutorial related to file format analysis for modding purposes.
Post Reply
User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 4286
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 1147 times
Been thanked: 2242 times

Extracting simple models

Post by shakotay2 »

Get 3 Zips from 3 posts starting here:
shakotay2 wrote: Sun Mar 07, 2021 3:29 pm
o noobs' hex to obj conversion (poor knowledge required)
Image
It should allow u to extract 3D models with simple formats.
Means such games like Mafia2 or SleepingDogs not to be supported.

Before using hex2obj on a difficult format try out some of the dozens of simple samples on Xentax
to understand HOW it's going. Searching for "simple" over all of my posts, you'll find something like this:
shakotay2 wrote: Sun May 23, 2021 12:15 pm
Link to hex2obj (zip parts 2 and 3 in adjacent posts):
shakotay2 wrote: Sun Mar 07, 2021 3:29 pm
- preliminary -

Ok, the truth is - there are two things you should understand:

the hexadecimal numeral system (base 16) using the "numbers" 0..9, A..F
So 0x123 is 291 decimal.
(To avoid confusion with other numbering systems the hex numbers are marked with 0x or an 'h'.)

Secondly you should be able to use a hex editor.

If you don't have these two skills leave now, and come back as soon as you gained them.

(If you don't have a clue which hex editor to use: TinyHexer or HexEdit by A.W. Phillips should do the trick.)
See list here and comments in the following posts:
jumpjack wrote: Mon May 04, 2020 10:00 am
--------------------------------------

As I said you don't need to learn "basics" but you'll have to understand
"the structures" to use hex2obj.

o So here is a short tutorial on it's use.

You don't need to know the technical terms - I just use them habitual.

Simple 3D models consist of vertices and faces.

hex2obj needs the following infos to perform it's task:
the startaddress of the vertices and the one of the faceindices.

(Some experience is required to find them but it's not too hard.)

Secondly we need the vertices count and the faceindices count.

I will explain this with a model example soon but now for the structures -

the two most common are:
a) "sequential"

Code: Select all

v x,y,z
v x,y,z
v x,y,z
v x,y,z
...
vn x,y,z
vn x,y,z
vn x,y,z
vn x,y,z
...
vt x,y
vt x,y
vt x,y
vt x,y
...
(v: vertex position, vn: normals, vt: uv)
With a sequential model you only need to provide the above mentioned counts.

b) "blocked"

Code: Select all

v  x,y,z
vn x,y,z
vt x,y

v  x,y,z
vn x,y,z
vt x,y
...
With a blocked model you need to choose the vertex block size (FVF size), for example 32.
(And 'yes': the order might be v, vt, vn for example)

If you don't understand anything you'll have to do a trial 'n error (No, no I really don't recommend this. :eek: )

Or you calculate the FVF size: each coordinate (x, y or z) consists of 4 bytes.
So the above two blocks have a blocksize of 32 (8x4) each.

It might be a little bit more difficult if there are DWORDs contained in the blocks.
But anyway the most common blocksizes are 32, 40, 52. Sometimes 24, 44, 48 whatever.


o the v,vn,vt patterns to search for

well, to identify what is v, vn or vt is the tricky part. This is for little endian
(for big endian the patterns are in reverse order of course).

v: xx xx xx nn with nn= 42..44 (these are rough values only, can also be 3F, 40)
vn: value of each component 0.0 .. 1.0 (1.0 = 00 00 80 3F)
square sum of components is 1.0 (autocalculated by hex2obj)
vt: value of each component 0.0 .. 1.0

It's important to know that half floats use two bytes only.

------------------------------------------------------
o Now lets start analysing our first model
------------------------------------------------------
(We don't care for normals because they can be autogenerated in most 3D tools.)
We leave the "LitE" button as it is (little endian data).

The face indices are very easy to find since parts of the indices block look like a scrambled alphabet.

If we scroll through hw_bu_000.bm in a hex editor we'll find this:

Image

To find the correct ending of the list is a very important step.
Here it's rather easy (0x4845) because it's bound by a string size (09 000000)
and the corresponding string.

address mapping of the tutorial example:
  • 0x56 ..0x15C1 vertices
  • 0x15C2..0x2B2D normals
  • 0x2B2E length DWord and string "Texture"
  • 0x2B39..0x3980 uvs, length: 0xE48
Rest of the tutorial to be found in the zip. Because you need to start hex2obj.exe
to get full use of the tutorial.

---------- version 0.24e3, "end of project, more or less" ----------
Since uploadmb is down atm (as of 7th of March, 2021) here's a temporary hosting of version 0.24e3:
shakotay2 wrote: Sun Mar 07, 2021 3:29 pm
Medium font version (for laptops with high DPI screens for example):
will be added to above link as soon as someones needs it (nobody requested it so far)

---------- version 0.24d, autogenerated FIs ---------- (.exe only, as a fallback)
http://forum.xentax.com/viewtopic.php?f ... ed#p119417

Since there a still some downloads after all the years I feel obliged to tell you guys once again:
hex2obj is NOT a one-click-solver tool - it was designed to help people to help themselves.

If you don't have a basic understanding of 3D model formats you surely will get better results using a 3D ripper (for PC).

With this said I think it's time to make a check list for noobs concerning the analyzing of 3D models:

Code: Select all

1) is the data compressed?
Try offzip if your unsure (works for zip compressed data only) or search for a decompressing tool.
(There's also Quickbms scripts for many archive formats.)

2) is it model data, texture data or animation data?
Use TextureFinder for textures. 
Some 3D models may contain both: model data and texture(s)

3) Before using hex2obj on a difficult format try out some of the dozens of simple samples on Xentax.

4) Then try out to get a point cloud  from your specific model as a first step. 

Look for continuous data sections with a size of 1 kB at least (0x400).

Yeah, what the heck does that mean: "continuous data"?

Hard to explain - in fact you'll need experience for that.
If there's gaps with more than 10 zero bytes for example I'd say it's not continuous data.

While float data is easy to recognize (with some experience) it's
rather hard to find vertex data in short/Word format.
---------- These outdated versions should be used as a fallback only! ------------
---------- >>> version 0.24c <<< ----------
(zippyshare link, blocked in some countries now, in mine, too :eek: )
hex2obj_0.24c.zip
- uv preview
- allows face indices > 65535 (for non-tristripped meshes)

-----------------------------------------------------------------

known bugs:
v0.22: for rare cases when there is a '.' in a folder name:
........ model names without extension require a .mdl to be added before loading the model
v0.23c - f intermediate versions with bug, don't use!
v0.24a uvs preview (float/VB)

reported bugs:
v0.21: wrong clipping for HF uvs
........ kao issue
v0.24: libgcc_s_dw2-1.dll missing on W7, W8 (due to CodeBlocks/MinGW update)

fixes
Fixed a bug (prior to v 0.24e3) with triStrips not resetting on parameter change

todo list
project finished , more or less
(or call it "abandoned" since there are better tools from Bigchillghost and Lazov)

HTH,
shak-otay
Last edited by shakotay2 on Tue Sep 05, 2023 8:39 am, edited 67 times in total.
Tuts: a) Bigchillghost, viewtopic.php?f=29&t=17889
b) Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
"Quoting the whole thing. Would u ever stop this nonsense?"
User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 4286
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 1147 times
Been thanked: 2242 times

Re: Extracting simple models

Post by shakotay2 »

FAQs

Q: how to find the start addresses of vertices and face indices (FIs)?
A: as for FIs, search for a "scrambled alphabet" but in general it's like this:
I don't know where vertices end and where faces start. The only way is make an assumption, check it. If its wrong, make another one, and check again. With time and experience, you will usually be able to make it right from the first try. (daemon1)
Q: when I press the 'mesh' button it gives me an exception.
A: A problem with the external mesh_viewer app. It expects a proper obj.
But hex2obj will not always provide a proper one.
It's some kind of trial 'n error tool, you know?

(With some knowledge you could have a look at test.obj to see what's wrong.)

Q: when I press the 'mesh' button nothing happens.
A: have a look at the output of the console window. If it's like this

"Reading test.obj... Error reading file [test.obj].
Usage: D:\CodeBlocks\Hex2obj\mesh_viewer.exe [-grab] infile..."

then see previous answer.

Q: how to calculate the vertex count of a (sub) mesh?
A: hex2obj doesn't need to search for the vertex count because
it's using the max face index value as vertex count.

That's the fundamental idea behind the project.


Q: what about the '99' UVpos in VB mode?
A: means: no UVs used. So with a VBsize of 12 and HF_no
test.obj should contain same data as in seq mode.

Q: why didn't you write this tool 5 years ago?
A: I r-e-a-l-l-y don't know. (Would have spared me tons of lifetime. :D )

Q: why the f... is Hex2obj's GUI messed up with Win7?
Image

A: happens, when you use a bigger font size in the Win7 GUI.
Since this seems to be rare I won't upload a separate exe version.
(Feel free to request one if required.)


links

DIZZEL
http://forum.xentax.com/viewtopic.php?f ... bj#p104615

more detailed hints on use of hex2obj:

another short tutorial:
http://forum.xentax.com/viewtopic.php?f ... ed#p102267

when you're stuck with the tutorial ('tut' button in hex2obj)
shakotay2 wrote: Sun Jul 31, 2016 5:12 am
answered some detailed questions:
shakotay2 wrote: Mon Sep 12, 2016 7:11 pm
three 3D formats examples:

Kinetec Star Wars
http://forum.xentax.com/viewtopic.php?f=16&t=11436

LittleBigPlanet 3 .mol
http://forum.xentax.com/viewtopic.php?f ... bj#p104214

LithTech model
http://forum.xentax.com/viewtopic.php?f ... bj#p103497
-------------------------------------------
submeshes of simple models
Flintstones Bedrock Bowling .AMW
http://forum.xentax.com/viewtopic.php?f ... 91#p103091

preview uvs feature:
Elite Dangerous
http://forum.xentax.com/viewtopic.php?f ... j&start=15

absolute/relative face indices:
Ride (by Milestone) .BIN format
http://forum.xentax.com/viewtopic.php?f ... bj#p105798

--------------------------------
hex2obj/Make_H2O
Wipeout HD/Fury
http://forum.xentax.com/viewtopic.php?f ... bj#p106658

Shining Resonance .MLX
http://forum.xentax.com/viewtopic.php?f ... j&start=30



old links

NeverDead http://forum.xentax.com/viewtopic.php?f=16&t=10960, post Thu Nov 14, 2013 12:06 pm
young justice legacy http://forum.xentax.com/viewtopic.php?f=16&t=11014, post Fri Dec 06, 2013 9:55 am
MDK 2 http://forum.xentax.com/viewtopic.php?f ... 803#p90803, post Thu Dec 12, 2013 12:51 pm
Last edited by shakotay2 on Mon May 24, 2021 10:13 pm, edited 19 times in total.
Tuts: a) Bigchillghost, viewtopic.php?f=29&t=17889
b) Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
"Quoting the whole thing. Would u ever stop this nonsense?"
User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 4286
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 1147 times
Been thanked: 2242 times

Re: Extracting simple models

Post by shakotay2 »

A model from this thread: viewtopic.php?f=10&t=8368&p=89143#p89143

Image

updated exe (big endian untested) in start post.
Tuts: a) Bigchillghost, viewtopic.php?f=29&t=17889
b) Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
"Quoting the whole thing. Would u ever stop this nonsense?"
User avatar
shekofte
mega-veteran
mega-veteran
Posts: 221
Joined: Sun Jan 18, 2009 1:45 pm
Location: Sagittarius
Has thanked: 301 times
Been thanked: 20 times
Contact:

Re: Extracting simple models

Post by shekofte »

nice !
in near future i hope you extend these tutorials to more advanced levels of a typical modern 3d model like normals , texture coordinates ...
ILLUSORY VIDEO GAMES ARE MORE IMPORTANT THAN PRECIOUS INTERNATIONAL FOOTBALLs
User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 4286
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 1147 times
Been thanked: 2242 times

Re: Extracting simple models

Post by shakotay2 »

Normals are of subordinate meaning imho because you can autocreate them in nearly every 3D tool - as I wrote.

texture coords (uvs) are covered.

But your right: the models in this thread viewtopic.php?f=16&t=10841&p=89008#p89008 use a vertex block where the uv data is supposed to be half floats.
Image

Code: Select all

Offset  0  1  2  3  4  5  6  7   8  9  A  B  C  D  E  F
0A80   BF 54 71 82 42 87 D5 A8  3F B1 95 57 25 10 83 12
0A90   80 49 18 65 02 00 00 00  FF FF 00 00 00 00 00 00
Where the first 12 bytes are the x,y,z position and the following 8 bytes might be 4 half floats
vtx1,vty1, vtx2,vty2?

edit: interpreting bytes 12..15 as 2 words, first divided by 32768.0, 2nd by 65536.0 you'll get
vt 0.289551 0.511993
vt 0.501404 0.542847
vt 0.073792 0.800491
vt 0.251343 0.551270
vt 0.289551 0.511993
vt 0.438599 0.538651
vt 0.838684 0.787979
...
From the values that looks pretty nice but isn't. This is some garbage but not an uv map.
So finding the the uvs might drive you carzy.

Automated analysation will always be limited.
This is a tool for noobs to reduce the threshold.
In the end there's no other choice than using your brain on analyzing of more complex models.
Last edited by shakotay2 on Fri Nov 01, 2013 1:20 pm, edited 2 times in total.
Tuts: a) Bigchillghost, viewtopic.php?f=29&t=17889
b) Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
"Quoting the whole thing. Would u ever stop this nonsense?"
luxox18
mega-veteran
mega-veteran
Posts: 176
Joined: Fri Jul 29, 2011 9:18 pm
Has thanked: 54 times
Been thanked: 46 times

Re: Extracting simple models

Post by luxox18 »

Is possible in the future load the entire model and not only submeshes? (for example, the model from dante's inferno)
User avatar
Bastien
advanced
Posts: 70
Joined: Sun Apr 15, 2012 1:08 am
Has thanked: 27 times
Been thanked: 13 times

Re: Extracting simple models

Post by Bastien »

Sounds like a pretty cool tool to also test unknown formats!
will test it later
cheers
User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 4286
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 1147 times
Been thanked: 2242 times

Re: Extracting simple models

Post by shakotay2 »

luxox18 wrote:Is possible in the future load the entire model and not only submeshes? (for example, the model from dante's inferno)
It's planned to introduce a param's file where to store the data of submeshes (vertex start addresses, vertex counts). But this will not release you from searching for the submeshes manually before.

Building the entire model from this param file's data should be possible.

edit: version 0.1 (view start post) can handle half floats and triangle strips.
Load/save of a H2O (param) file is possible.
Composing of submeshes to an entire model not implemented because of
possible "65535 indices border" crossing (DWORD required).

Although it's not really difficult, it has low priority for me.
This is the reason: atm you'll have to search manually for every single submesh (SM) of the model.
Often there are patterns by which you can identify the start addresses of the SMs.

But you need to know these patterns which are different with every game.
(And for some models there aren't such patterns.)
I can't think of a feature how to get them automatically.

Once you know the pattern you can find the SMs by using your hex editors search function.

You can search for 00 00 01 00 02 00 (00 00 00 01 00 02 for big endian) as an identifier for the start of WORD face indices lists but that does apply to few games only.
Last edited by shakotay2 on Sat Nov 16, 2013 7:25 am, edited 1 time in total.
Tuts: a) Bigchillghost, viewtopic.php?f=29&t=17889
b) Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
"Quoting the whole thing. Would u ever stop this nonsense?"
User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 4286
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 1147 times
Been thanked: 2242 times

Re: Extracting simple models

Post by shakotay2 »

Handling triangle strips with version 0.1
With the model c_pnzgren_body1.ps3mesh from this thread viewtopic.php?f=16&t=10906 you'll get the head by simply loading
the model and the pnzgren.h2o (param) file.

You'll have to toggle the 'std' button to 'FFFF' (separator) bevor you click the 'mesh' button.

Getting further submeshes goes as follows:
well, we find another scrambled alphabet at 0x12620. The list ends at 0x1308D.
So there are 0xA6E bytes or 2670/2= 1335 face indices.

Entering these values and clicking 'go1' reveals a vertex count of 755 at the lower left listboxes' end.

The start address of the vertices block is easily to be found: whole vertex list is 755*VBsize= 18120 dec.
Subtracting 0x46C8 from 0x12260 gives 0xDF58 as the vertices start address.
File SaveAs mesh et voila: Image (with normals drawn)

(the status of the 'std/FFFF' button is not saved into a H2O file so far)

edit: for the vertices start of the body mesh you'll have to stick to 0x1BB88.
(There's some 'blurring' between calculated (2032) and factual (2044) vertices count, hehehe.)
Last edited by shakotay2 on Fri Nov 01, 2013 11:08 am, edited 1 time in total.
Tuts: a) Bigchillghost, viewtopic.php?f=29&t=17889
b) Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
"Quoting the whole thing. Would u ever stop this nonsense?"
User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 4286
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 1147 times
Been thanked: 2242 times

Re: Extracting simple models

Post by shakotay2 »

version 0.2 is out. Can handle vertices/UV data as WORDs (unsigned 16 bit values).

(As always I hope I didn't mess up too many things.)

The use of the 'cut'-button might be required for "Word_all" models:
Image Image

("Word_all" simply means that there are 3 WORDs for the vertex pos and 2 WORDs for the UV data.)

edit:
with the fixed version 0.2 I got a rage bmodel:
Image

but .bmodel is not always triangle strips as you can see:
Image

edit2: ships always being my preferred models. Here's locr00_spielerschiff_s_0.nvx2 from Drakensang, ROT:
Image
UV coords as words (looks like a divisor of 8000 being rather better than 65536).

edit3: space-ships:
viewtopic.php?f=16&t=11436
Last edited by shakotay2 on Mon Apr 21, 2014 7:40 am, edited 7 times in total.
Tuts: a) Bigchillghost, viewtopic.php?f=29&t=17889
b) Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
"Quoting the whole thing. Would u ever stop this nonsense?"
User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 4286
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 1147 times
Been thanked: 2242 times

Re: Extracting simple models

Post by shakotay2 »

Version 0.2x introducing "mixed mode" (for advanced users only!)

From the tut you know there are two main ways how the model data is organized:
"sequential", that means there are all vertices (v) then all normals (vn) to follow
then all uvs (vt). (May also be: all v, all vt, all vn.)

Another possibility is the "blocked" mode.
Means you have a Vertex Block with a size of let's say 32 bytes.
First block contains first vertex, first normal, first uv.
2nd block contains 2nd vertex, 2nd nomal, 2nd uv.
And so on.
(That's an example only! There may be other data in the blocks, such as DWords.)

Now we introduce the "mixed" mode where vertices and uvs don't share the same block.
---------------------------------

You may have a vertex block with a size of 16 bytes (3 floats vertex, 1 float unknown),
then the 2nd block and so on.

After the last vertex block the uv blocks start.
May have a size of 12 bytes (2 floats UV, 1 unknown DWORD)
Then the 2nd uv block follows and so on.

I call it "mixed" mode because you can look upon it as a blocked AND as a sequential mode, too.
"Sequential" because there are all vertices, then all uvs.
But vertices and uvs are in blocks, too. Not with each other but together with other data.


As an example you might take the model from this thread:
viewtopic.php?f=16&t=11056&p=91650#p91650
AcheronGolem_weapon.JPG
----------


advice: the address and the size of the UV block are
NOT saved into the H2O file so far.
You do not have the required permissions to view the files attached to this post.
Last edited by shakotay2 on Mon Jan 20, 2014 1:52 pm, edited 1 time in total.
Tuts: a) Bigchillghost, viewtopic.php?f=29&t=17889
b) Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
"Quoting the whole thing. Would u ever stop this nonsense?"
luxox18
mega-veteran
mega-veteran
Posts: 176
Joined: Fri Jul 29, 2011 9:18 pm
Has thanked: 54 times
Been thanked: 46 times

Re: Extracting simple models

Post by luxox18 »

With hex2obj 0.2x now I can obtain UV from Dante's inferno models

Image

thanks shakotay2
Doronetty
advanced
Posts: 54
Joined: Thu Jun 03, 2010 11:05 am
Has thanked: 27 times
Been thanked: 3 times

Re: Extracting simple models

Post by Doronetty »

shakotay2, the links are dead :( Please re-upload your tools!
Acewell
VIP member
VIP member
Posts: 1330
Joined: Wed Nov 05, 2008 12:16 pm
Has thanked: 2710 times
Been thanked: 884 times

Re: Extracting simple models

Post by Acewell »

I have the file and could upload it as last resort, but this really should be added to the Tools blog.
User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 4286
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 1147 times
Been thanked: 2242 times

Re: Extracting simple models

Post by shakotay2 »

thx - updated start post

--------------------------
(This is a text backup only for maybe future quotations.)

Workaround for formats which use shorts for the vertices and half floats for the uvs:

save the mesh in two ways, first as ShortAll then as HF_UV (or HF_all).
In the first obj file do a replacement of 'vt' by '#vt'.
In the (HF_UV) obj file do a replacement of 'v ' by '#v '. (<<< Don't forget the space here.)

Then copy all vts (without the face indices) into the (ShortAll) obj.

(Yep, this is ugly but should work.)
Last edited by shakotay2 on Sun Apr 24, 2016 1:00 pm, edited 3 times in total.
Tuts: a) Bigchillghost, viewtopic.php?f=29&t=17889
b) Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
"Quoting the whole thing. Would u ever stop this nonsense?"
Post Reply