Page 1 of 7

JX3 HD Remake 剑网3 (.dat files)

Posted: Fri Sep 28, 2018 12:26 am
by petventh18
EDITED (10/3/2018):
Just put this out there so if anyone can write a Noesis/Max plugin to import/export this game then here's some mode samples 3MB (https://drive.google.com/open?id=1hrUJM ... tPD7NvzE9M)...
Here's the full folder with everything extracted 743MB ==> https://drive.google.com/open?id=16XYbB ... TMxPmfxd7G
----------------------------------------------------------------------------------------------------------
Hi, I know this game has multiple versions in the past... but this JX3 HD Remake look really awesome and I'm just posted here cuz I checked the other posts and I don't think it's been cover yet (or sorry if I missed it).

This game has a total of 74.5GB of .DAT files and .IDX files (mostly .dat files... I don't know if these dat files can just be imported into Noesis/Max/Blender by writing some script on it or maybe it need to be extracted from the dat files first). So if anyone can help extract these DAT files, then you have my thanks and anyone who is interested in the future.

Sample file 202MB https://drive.google.com/open?id=1PD7W4 ... L-15LbQkWH

If need more sample, just let me know... I'll upload them ASAP.
Here's some Screenshots
ImageImageImageImage
http://image.9game.cn/2018/3/2/19669711.jpg
http://jx3.xoyo.com/uploadfile/2018/030 ... 524187.jpg
http://jx3.xoyo.com/uploadfile/2018/030 ... 519705.jpg
-----------------------------------------------------------
Game Site: http://jx3.xoyo.com/index/
MMOByte Overview: https://www.youtube.com/watch?v=MIv731kIFu4

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
All the necessary tool(s) from this thread is here. https://drive.google.com/open?id=18UGXT ... cWytrWQFuP

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Fri Sep 28, 2018 1:39 pm
by Bigchillghost
You can use this BMS script to unpack the chunks as a first step:
datUnpack.rar
Just select the dat file as input and leave the rest to the script.

No time to dig into the format. Sorry about that.

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Fri Sep 28, 2018 4:55 pm
by zaramot
Looks pretty cool, could someone share model samples after extraction?

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Fri Sep 28, 2018 5:03 pm
by shakotay2
I didn't find face indices in the orgUncompressed folder at a first glance.
Here's some file types in the orgCompressed folder:
JX3-data.jpg

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Fri Sep 28, 2018 5:24 pm
by Bigchillghost
The raw data size of chunk 0 is around 110 mb while it's almost the equal size after decompression. Maybe there's another layer of compression. Chunks with smaller size seem OK though, like in chunk_1 there're some texture name strings and material info.
Edit: Never mind. It's just a mp4 file.

BTW, data of some chunks in the orgUncompressed folder seem to be in big-endian.

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Mon Oct 01, 2018 7:05 pm
by petventh18
Hi thanks a lot guys... I've used the datUnpack.bms script and got a lot of chunks .dat files from it. There's like 250+ folders with their respective .dat files from the game. So it's gonna take a while to find where the models are.
I'll upload some samples to here and see if anyone can make sense of anything.
https://drive.google.com/open?id=1647lR ... iTcAQOMVPn
It would be easier to identify the models through the textures but since there's like 300 files to extract from, I'll upload it once I got any lead.
In the mean time, here's some additional samples (not extracted)
https://drive.google.com/open?id=14Oafz ... YdgQNsexPk
https://drive.google.com/open?id=1kNlFI ... KN8Q_6nmTF
https://drive.google.com/open?id=1pNsT2 ... LZDFWJNHiB
https://drive.google.com/open?id=1YbOpx ... zXw7tcZZPa

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Mon Oct 01, 2018 9:23 pm
by petventh18
Bigchillghost wrote:You can use this BMS script to unpack the chunks as a first step:
datUnpack.rar
Just select the dat file as input and leave the rest to the script.

No time to dig into the format. Sorry about that.
Hi Bigchillghost, thanks for the script...
I kept running into these issues running out of memory so I can't extract all these .dat files... I had 32GB RAM on my machine so I kinda doubt that's the main issue.
Sample https://drive.google.com/open?id=1cV4Xz ... c0prvQ3Cgb
Image

And sometime it show the message
"Not enough storage is available to process this command." but I don't remember which file it was so I can't provide you a sample.

I've used both the normal quickbms.exe and quickbms_4gb.exe... They pretty much the same... So I guess some of the header got messed up in these files or something.

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Tue Oct 02, 2018 12:00 pm
by Bigchillghost
petventh18 wrote: In the mean time, here's some additional samples (not extracted)
The script works fine with these files(tested on the first sample). You just need to wait patiently before you witness any output info.
petventh18 wrote:Hi Bigchillghost, thanks for the script...
I kept running into these issues running out of memory so I can't extract all these .dat files... I had 32GB RAM on my machine so I kinda doubt that's the main issue.
I modified the script to export correct file extensions this morning, but when I was testing it, that issue occurs to me on the same file that my script handled correctly twice before. I guess it might be some bugs of QuickBMS. So I decided to make a standalone tool for unpacking the assets. It's roughly tested on several samples(the one in the first post and the one from the first url) but it's still a beta.

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Tue Oct 02, 2018 12:14 pm
by Bigchillghost
Here's a x86 release of the tool:
JX3HDUnpacker.7z
Place them in the dir of the dat/idx files and double-click the .bat to run the program in batch command.
It'll log all text info into a new corresponding file(.txt or .json) within 8 megabytes for convenience of reading, unless current text chunk has already exceed that size.

Note: Files end with ".TBD" means their extensions are to be determined.

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Tue Oct 02, 2018 7:53 pm
by petventh18
Bigchillghost wrote:Here's a x86 release of the tool:
Place them in the dir of the dat/idx files and double-click the .bat to run the program in batch command.
It'll log all text info into one corresponding file(.txt or .json) for convenience of reading.
Oh thanks so much, work like a charm... I tested and worked on most files... Here's the files that had a model textures and some .mesh files along with some subtitle scripts too...
60MB ===> https://drive.google.com/open?id=1FcQyb ... LvWG2SNMFH
743MB ==> https://drive.google.com/open?id=16XYbB ... TMxPmfxd7G
Image
...with this maybe someone can help write a Noesis/Max script to import/export them.

There's still some files that came out empty such as this screenshot... but since it can already extracted the textures and mesh files from some other .dat files, I guess these files not necessarily contain any more models (I hope)... I guess the only thing I might miss is if these are animation files or something.
Image
Here's a sample of the file that can't be extracted.
596MB ==> https://drive.google.com/open?id=12i5GJ ... 8n2Pn3IWIe

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Wed Oct 03, 2018 5:17 am
by Bigchillghost
petventh18 wrote:There's still some files that came out empty such as this screenshot... but since it can already extracted the textures and mesh files from some other .dat files, I guess these files not necessarily contain any more models (I hope)... I guess the only thing I might miss is if these are animation files or something.
Here's a sample of the file that can't be extracted.
Well it's just an issue resulting from the illegal extensions. I've improved the code for determining them, especially more accurately for the mesh files so you might need to run the lastest version on all packages again. Sorry about that.

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Wed Oct 03, 2018 10:31 am
by Bigchillghost
petventh18 wrote:Here's the files that had a model textures and some .mesh files along with some subtitle scripts too...
743MB ==> https://drive.google.com/open?id=16XYbB ... TMxPmfxd7G
Mesh from 000003B3.mesh:
000003B3.mesh.jpg
And the one from 000005E0.mesh:
Image
Image

It seems that the model needs proper scaling to get the perfect mesh.

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Wed Oct 03, 2018 11:22 am
by shakotay2
Bigchillghost wrote:It seems that the model needs proper scaling to get the perfect mesh.
I don't even know how this can work: floats for the mesh don't fit into an FVF of 6?

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Wed Oct 03, 2018 11:36 am
by Bigchillghost
shakotay2 wrote:I don't even know how this can work: floats for the mesh don't fit into an FVF of 6?
You mean the 6 floats before the vertex buffer?

Re: JX3 HD Remake 剑网3 (.dat files)

Posted: Wed Oct 03, 2018 12:11 pm
by shakotay2
FVFsize= 6; but I see, it's ShortAll (instead of WordUV, which you seem to have used for uvs' displaying only.)

My confusion also aroused from the fact that there seems to be 6 floats values (bounding box) before the mesh using shorts.