JX3 HD Remake 剑网3 (.dat files)

[Bigchillghost]

because the BBox calculated from the unsigned vertex data in hex2obj doesn't match with the real values so far. Need to look what's going wrong there:
# BBox 0.000000 255.468750, 0.000000 255.468750, 0.000000 255.468750

Why does it need to match with the calculated bounding box? I think they're using unsigned shorts to map with the floats within the BBox so that they won't lose too much precision. So we just need to divide the bounding box space into 65535 segments on each axis to get their corresponding scale units.

[shakotay2]

yeah, my bad, I thought the BBox auto calculation went wrong, but seems it doesn't

65535? depends on your divider when creating the mesh - I just need to divide 5376.0376 by 255.469 for the x-scaling for example.

well, seems to fit:

scale 21,6.2,16.png

[petventh18]

Is possible, but generally that splitting results in random submeshes (SMs). It's just kind a luck that the static models seem to have roundabout 500 faces clusters here, imho.
So for characters it's more sensefull to insert those lines manually, as there's aren't so many SMs, head, upper body, lower body for example.

I see, thank you...
I'm just wondering, is it possible for you to add some code to the Make_obj-JX3.exe tool to make it so the output .obj file had the same name as the .mesh file?
And if it's not too much trouble then maybe add batch feature to the tool too? Cuz this game had too many files and going through each .mesh at a time is really tedious and tiring to find the player models. Sorry for nagging so much... Just ignored my babbling if it's too much trouble =)

EDITED:
I tested a few of the meshes using Make_obj-JX3.exe tool and these 3 samples are the one that cannot be imported into 3D Software after forming the obj file.
https://drive.google.com/open?id=1FoZn5 ... t1Qogf4E5y

[shakotay2]

I'm just wondering, is it possible for you to add some code to the Make_obj-JX3.exe tool to make it so the output .obj file had the same name as the .mesh file?
And if it's not too much trouble then maybe add batch feature to the tool too? Cuz this game had too many files

yeah, I was thinking about this since a long time. The batch feature is simple for console apps, but a GUI app needs an auto exit feature like SendMessage (hwnd, WM_CLOSE, 0L, 0L) ; which I used; but not sure about the side effects.

But didn't you plan to create a Noesis script?

EDITED:
I tested a few of the meshes using Make_obj-JX3.exe tool and these 3 samples are the one that cannot be imported into 3D Software after forming the obj file.

yeah, I see. Would require to use the addresses from the "magic table" at 0x90 where I only take the counts from so far. (Auto calculation of addresses doesn't seem to work for all files.)

[petventh18]

But didn't you plan to create a Noesis script?

I've only started learning C# language a year ago (so still super noob when come to program languages)... And when I tried python script along with some example plugin from other games and some tutorial on Youtube... it's basically a nightmare for noobie such as myself since every script seem to different from one another when it come to section like

Code: Select all

def noepyLoadModel(data, mdlList):

or

class mesh(object):

or

Code: Select all

class mesh(SanaeObject):

... There's so many variation that I can't do it with my current skill... I guess I'll have to start learning python first before attempting again...

yeah, I see. Would require to use the addresses from the "magic table" at 0x90 where I only take the counts from so far. (Auto calculation of addresses doesn't seem to work for all files.)

Since can't change the offset from the tool, I guess will have to manually using Hex2Obj...

[shakotay2]

it's basically a nightmare for noobie such as myself since every script seem to different from one another when it come to section like

I have a deja vue
Yea, I know that feeling; here's some simple script for beginners with some Noesis/python knowledge:
viewtopic.php?f=16&t=9534&p=141865&hili ... an#p141865
(ah, see, there's some math with the face indices buffer, feel free to ask, why, that's a matter of the format, not of python)
(Noesis is a powerful tool, there's absolutely no doubt, but the first scripting steps, are not so nice. )

Since can't change the offset from the tool, I guess will have to manually using Hex2Obj...

That's a good start, imho.
But for hundreds of *.mesh files you might prefer this one (cmd line tool, GUI pops up and exits):

Make_obj-JX3_cmdline.zip

tested with 3 files only

• you need the dlls from the previous zip
• you need to adjust the "path to the exe" in the .cmd file.
• blanks in *.mesh pathname not supported!
• start the cmd file in the folder where your *.mesh files reside

btw: you might drag 'n drop a single *.mesh onto the exe, too

[petventh18]

...here's some simple script for beginners with some Noesis/python knowledge:
viewtopic.php?f=16&t=9534&p=141865&hili ... an#p141865
(ah, see, there's some math with the face indices buffer, feel free to ask, why, that's a matter of the format, not of python)....
But for hundreds of *.mesh files you might prefer this one (cmd line tool, GUI pops up and exits):

Thanks a lot for the info and for the tool update... it'll save me lots of time to sort through those models...

You mentioned that the UVs is 2 bytes off... so restart the tool and type in the new value on the editbox... then load the same mesh again, correct? Because it's hard to find the correct texture for the correct model since they're name differently... would be so nice if the name of texture match well like how Umodel arrange them... or even like how Ninja Ripper did it...
I wonder how the developer even keep track on all their models/texture with all these kind of names... or is it that the extract tool automatically rename them?

[Bigchillghost]

I wonder how the developer even keep track on all their models/texture with all these kind of names... or is it that the extract tool automatically rename them?

There do seem to be some text recording the asset names but not every package outputs them, and some of these info is in json format which makes it so annoying/difficult to read. Yep you're correct the tool names them automatically using the orders of their chunks.

[petventh18]

There do seem to be some text recording the asset names but not every package outputs them, and some of these info is in json format which makes it so annoying/difficult to read. Yep you're correct the tool names them automatically using the orders of their chunks.

Ah I c... thanks for clearing that up... I opened the json file on Notepad++ and only some .srt are recorded in it...
ex: "srtFile": "data\\source\\maps_source\\树入库临时保存\\s_雪竹019c4_st.srt"
But the output file will be something like 0000XXXX.SRT

And the .SRT files itself does have some texture name recorded like
(SP_wanhuashupi001H.tga SP_wanhuashupi001H_Nor.tga SP_wanhuashupi001 S_wanhuazitengluo002.tga S_wanhuazitengluo002_Nor.tga Atlas S_wanhuazitengluo002_Billboards.tga S_wanhuazitengluo002_Billboards_Nor.tga...)
So based on that we can see some diffuse and normal texture that are corresponding to the wanhuashupi001.mesh...

But because everything was renamed, it will take a lot of guessing game to match the correct texture into the model (assuming that the UV is correct after turning the mesh to obj)...

If it's not too much trouble then may I ask if it's possible to modify the extraction tool to not rename the file based on the chunk but leave it as is? (Doesn't matter if it's Chinese characters)... Some folder had over 6000 files, so trying to guess their texture based on the current name is quite literary a nightmare.
That or maybe had a .log file that keep track of the original name to the current exacted name?
And again, if it's too much of a hassle then forget about it... I'm already very thankful that you created the tool as it is.

[Bigchillghost]

assuming that the UV is correct after turning the mesh to obj

Well, about that, have you checked if they're mapping?

If it's not too much trouble then may I ask if it's possible to modify the extraction tool to not rename the file based on the chunk but leave it as is?

Maybe. That would require more research on the organization of the chunks, which can take some time. I'll see what I can do with it. But don't expect too much yet.

Edit: I can say that it's impossible to read filenames now because:
- The assets are linked by different objects in the json files, usually more than one times for each asset, and you can't locate the asset names with the MD5 hashes coz they don't even esixt in the json;
- There do seem to be a filelist in a certain json file, and there're some texture names in a binary chunk(using the ext .def in the following version), but their amounts and extensions don't match with the actual outputs, nor their sums.

Doesn't matter if it's Chinese characters...

I was wondering if the names will output correctly if they contain any Chinese. I hope I don't have to deal with the character encoding stuffs.

Some folder had over 6000 files, so trying to guess their texture based on the current name is quite literary a nightmare.

Ah, I can imagine that.

That or maybe had a .log file that keep track of the original name to the current exacted name?

Sounds good, but not necessary, nor possible. Because the tool didn't read any filenames at all. It just decompresses each chunk and assigns the index of the loop to the string buffer as the output filename (in hex). The simplest solution is to save every chunk with a decimal index name so that you can sort them with your Windows explorer. It still needs your guessing work though. But it's an alternative. Better than nothing.

Edit: Now you can have it:

JX3HDUnpacker.rar

Changes:
- Changed naming patterns from hex to decimal index;
- No text/json files will be merged;
- Improve detection of text/json data (first 4 bytes only).
- The files with the signature "\x08\x00\x00\x00" will be separated from the TBD files, using the extension ".8";
- The binary chunk(s) containing texture names will use the extension ".def".

[shakotay2]

...here's some simple script for beginners with some Noesis/python knowledge:
http://forum.xentax.com/viewtopic.php?f ... an#p141865
(ah, see, there's some math with the face indices buffer, feel free to ask, why, that's a matter of the format, not of python)....
But for hundreds of *.mesh files you might prefer this one (cmd line tool, GUI pops up and exits):

Thanks a lot for the info and for the tool update... it'll save me lots of time to sort through those models...

I wasn't sure if it would work for 100s of models...

You mentioned that the UVs is 2 bytes off... so restart the tool and type in the new value on the editbox... then load the same mesh again, correct?

Forget about that; that's for the old version with auto calculation of addresses; sometimes it gave incorrect values (2 bytes off for example) for the uvs' start.

The cmdline version gets all values from the magic table at 0x90 (counts and addresses) now. But it "relies" on a 12 bytes offset in the uvs' block (3 floats to be skipped before the uv data).

btw: this format is a very good start to be used with a Noesis script (you don't need the face indices buffer math in my previous linked sample).
The codelines for getting the face indices should look like this (untested):
faceBuff = bs.readBytes(faceCount*3*4)
rapi.rpgCommitTriangles(faceBuff, noesis.RPGEODATA_UINT, faceCount*3, noesis.RPGEO_TRIANGLE, 1)
UINT because of DWord face indices
and you don't need this weird
b = struct.pack('H' * len(idxBuf), *idxBuf) for this format

here's a working Noesis script, tested with 00000161.mesh:

Code: Select all

from inc_noesis import *
import noesis
import rapi

def registerNoesisTypes():
    handle = noesis.register("JX3", ".mesh")
    noesis.setHandlerTypeCheck(handle, noepyCheckType)
    noesis.setHandlerLoadModel(handle, noepyLoadModel)
    return 1

def noepyCheckType(data):
    bs = NoeBitStream(data)
    bs.seek(0x54, NOESEEK_ABS)
    idstring = bs.readUInt()
    if idstring != 0x4D455348:
        print("not a JX3 file!")
        return 0
    return 1

def noepyLoadModel(data, mdlList):    
    ctx = rapi.rpgCreateContext()
    bs = NoeBitStream(data)
    #rapi.rpgClearBufferBinds()
    bs.seek(0x90, NOESEEK_ABS)
    vertsCount = bs.readUInt()
    faceCount = bs.readUInt()
    bs.seek(0xA8, NOESEEK_ABS)
    uvAddr = bs.readUInt()
    uvAddr += 12
    bs.seek(0xB4, NOESEEK_ABS)
    FIaddr = bs.readUInt()

    bs.seek(0x12C, NOESEEK_ABS)
    VertBuf = bs.readBytes(vertsCount * 6)
    bs.seek(uvAddr, NOESEEK_ABS)
    uv1Buf = bs.readBytes(vertsCount * 6)
    bs.seek(FIaddr, NOESEEK_ABS)
    idxBuf = bs.readBytes(faceCount*3*4)
    rapi.rpgBindPositionBufferOfs(VertBuf, noesis.RPGEODATA_USHORT, 6, 0)
    rapi.rpgBindUV1Buffer(uv1Buf, noesis.RPGEODATA_USHORT, 6)
    rapi.rpgCommitTriangles(idxBuf, noesis.RPGEODATA_UINT, faceCount*3, noesis.RPGEO_TRIANGLE, 1)

    mdl = rapi.rpgConstructModel()
    mdlList.append(mdl)

    return 1

(Maybe I should have used some NOESEEK_REL constants but I'm too lazy to calculate the required offsets. )

edit: attaching above mentioned sample, in case it's not available any more:

00000161.zip

Also I didn't care for the normals buffer, as always.
addr can be found in the magic table; refer to the struct that Bigchillghost posted.

[Bigchillghost]

Now the problem is to get correct normal vectors:

I can't believe that they actually packed the normals in this way. You need to perform the following operations to get the correct normals:

Code: Select all

for (int i = 0; i < Vcount; i++)
{
	dblNM[i].x = charNM[i].x / 127.0;
	dblNM[i].y = charNM[i].y / 127.0;
	dblNM[i].z = charNM[i].z / 127.0;
	if (dblNM[i].x < 0.0)
		dblNM[i].x = -dblNM[i].x - 1.0;
	if (dblNM[i].y < 0.0)
		dblNM[i].y = -dblNM[i].y - 1.0;
	if (dblNM[i].z < 0.0)
		dblNM[i].z = -dblNM[i].z - 1.0;
	dblNM[i].x = -dblNM[i].x;
	dblNM[i].y = -dblNM[i].y;
	dblNM[i].z = -dblNM[i].z;
}

Test result:

normals.jpg

Is that a common case?

[petventh18]

@shakotay2
Thanks for the script... It make it a lot easier for noobie me Although I couldn't get the CMD version of the tool to work at all. I place the prev folder with some .mesh inside of it. But it just won't do anything when click or drag a mesh file to it. I made sure to rename the path inside the .cmd file as well...

But it's very hard matching the texture and model so I decided to buy an Xoyo account and tried my luck with Ninja Ripper... good thing it worked...



So if anyone don't wanna sort through the mesh manually then try Ninja Ripper... the down side is that you can only rip the model you see and you'll have fking nightmare trying to register for a game account without a chinese phone number. And the face and hair position don't match properly so you have to manually trying to put it together.

[Bigchillghost]

Probably this is pointless but here's the truth: since I've updated my binary FBX builder to support face materials and other features,
this format had been the perfect sample for testing purpose. Wouldn't it be a waste if it'd never got released?

For example here's one of the material group of the entire mesh from 000005E0.mesh (7 in total):


Note that the UVs endoced as WORD are a bit off the correct position which I have no clue of why.
I made a little correction for this model only so it's not guaranteed to work for all UVs.
For float UVs you don't have worry about it though.

JX3Convertor.zip

Just run the tool without prams to see the usage.

[usabdt]

Probably this is pointless but here's the truth: since I've updated my binary FBX builder to support face materials and other features,
this format had been the perfect sample for testing purpose. Wouldn't it be a waste if it'd never got released?

For example here's one of the material group of the entire mesh from 000005E0.mesh (7 in total):


Note that the UVs endoced as WORD are a bit off the correct position which I have no clue of why.
I made a little correction for this model only so it's not guaranteed to work for all UVs.
For float UVs you don't have worry about it though.

JX3Convertor.zip

Just run the tool without prams to see the usage.

Why can't I open?