2022 - Tom Clancy's Rainbow Six: Siege Asset Extraction Tools

[Tushkan]

I have seen files in your Dropbox. They are not clear for understanding, and I didn't find forge archive structure analyzed.

Did you remember when Ubi changed .forge archives? Because I'm working on a season operation in old version game (about Mar - April in 2018).

Afair they changed compressor somewhere between operation health and outbreak event. So you should be good. Of the most recent changes they modified depgraphbin structures, but it's not a very serious change (they seem to have split uint64 into [int32, int16, uint8, uint8]) and does not affect any actual data parsing. The other one is now I find string IDs for many basic structs like textures, meshes and havoks. I'm not sure wether those are reanny new, but I don't remember seeing them before. If your forge chunks are compressed with zstandard, then you're 'up to date'. You can actually send me some small forge file to test out if it unpacks.

And my dropbox is basically out of date. It's about year old, when lots of things were undiscovered and sources where messy. It's actually messy even now because it's a mix of jyputer notebooks, .md notes, test py files and actual core code. But at least core is more consistent and documented now.

I guess I should release this stuff on github in it's current state so people at least can get some info on structures and types... But zaramot's Noesis plugin looks alot more promising and comfortable. ) My stuff is designed to work from cmd or scripts and it's intended do gather extensive amount of data in one go. Say if you want a character, you should be able to run a search for some string like "Caveira" and grab it's mesh and from mesh grab it's asset struct and dump all that is needed at once in one folder (all meshes, textures etc.). I even found shaders per model, like if it would interest anyone xD...

[Custard]

Of the most recent changes they modified depgraphbin structures, but it's not a very serious change (they seem to have split uint64 into [int32, int16, uint8, uint8]) and does not affect any actual data parsing. The other one is now I find string IDs for many basic structs like textures, meshes and havoks. I'm not sure wether those are reanny new, but I don't remember seeing them before.

Maybe some changes are happen with the game currently undergoing Vulkan API implementation? Which may forebode more changes as well.

Sure, I'm mostly sharing stuff which I did or doing, with rare examples when it's just for my personal use only or not finished for end-user, not user friendly xD So, of course with siege I could post everything I did later on, so anyone could take a look and continue the process or just take it as it is.

I'm sure we all really appreciate your work on this zaramot, and for a game you're not highly motivated for. So you are able to extract meshes with UVs and weights, and textures so far. Did you plan to do further development for Siege? It sound like if this is combined with asset identification it'd serve many purposes already.

[comword]

If your forge chunks are compressed with zstandard, then you're 'up to date'. You can actually send me some small forge file to test out if it unpacks.

Here is the link, have a try. https://www.dropbox.com/sh/sii6e34kymrk ... DBY3Qv2TUa
I believe these files are main parts of Outbreak, Operation Chimera as I viewed change log on SteamDB.
I found all my forge files start with a magic string "scimitar", and I found a BMS script can extract it to files. Files name are hashed.
Some extracted file started with 34 AA FB 57 99 FA 14 10 02 00 03 00 ... Have you seen this pattern before?

[Tushkan]

Some extracted file started with 34 AA FB 57 99 FA 14 10 02 00 03 00 ... Have you seen this pattern before?

Yes. [34 AA FB 57 99 FA 14 10] is a Container_ID. It denotes file It can hold uncompressed data (meta) and compressed (actual file split into compressed chunks). you can see those all over forges. They generally go in groups: one container for uncompressed metadata and emmideatly after that goes another with compressed file. It should go as:

Code: Select all

uint64 Container_ID
uint16 = 2 // unknown
uint16 = 3 // unknown
uint8 = 0 // unknown
uint16 varying // unknown
uint32 num_chunks // how many data chunks are in this container
[chunk_size, chunk_size, ...]
[chunk, chunk, ...]

each chunk_size is a struct of:

Code: Select all

uint32 uncompressed_size
uint32 compressed_size (actual chunk size that you need to read)

each chunk is:

Code: Select all

uint32 hash // at least it looks like some hash
bytes [compressed_size] //actual compressed data

The files that start emmideately with this header, are most likely depgraphbins. Those hold data about links between different entities. When you unzompress and concatenate all chunks (with zstandard), you will get a binary file, that starts with uint8 = 2 and then its a buncho if structs of type:

Code: Select all

uint64 parent_uid
uint64 child_uid
uint64 child_type

In newer versions child_type now seems to be split into [int32, uint16, uint8, uint8]

[Tushkan]

well. Good news is your assets are inline with my tools. But there are no string hints so I'll have to continue at least some work on locating asset packs. Plus some types of meshes are still badly parsed. This is only a fraction of assets, there are lot more, but they are compressed with 0d24 bytes per vertex and they get teared apart on import.

[zaramot]

Tushkan, could so send me in PM of somewhere those compressed meshes? I haven't encountered this mesh type, since I have only few siege .forges, it's interesting.

[Tushkan]

https://www.dropbox.com/sh/b2cuse4hp90y ... qTfja?dl=0

There are the 7z archive (an old one) and the fury*.zip, (from outbreak). They contain unpacked mesh files. Site-packages folder contains the code to parse them. It's just a fraction of r6s module, but it should be enough to parse unpacked files. Just add an empty __init__.py into r6s folder, I forgot to do it myself. Usage:

Code: Select all

from r6s.mesh import Mesh
mesh = Mesh.parse('path\\to.file')  # parses header data
data = mesh.getmesh()  # simple struct with 3 buffers: vetrs, tris, uvs

Bad_samples.txt contains names of files from fury archive, that are parsed badly. Those aren't all, just a fraction I've seen so far.

There are also folders with x## names, those should also contain samples of meshes with different compression, but I'm not sure, I saved those a year ago.

[zaramot]

Tushkan, could you please make a screenshot of your version of imported 12538.mesh? I imported 12538.mesh from datapc64_merged_fury_bnk_mesh archive, that's how it looks like for me. I tried others from list of bad files, they are basically the same - some boxes and one I guess small floor piece. Or any other mesh, maybe bigger which looks not correct for you? I want to compare the outputs.

[Tushkan]

This is how 12528 looks in my case. There is a 1x1x1 cube for reference on the left. Strangely enough most objects of same compression type look just fine. But some get imported like that. Also I have hard time detecting lod bounds some times for some models everything looks fine, but for others two lods get imported simultaneously and, because they use same vertices, second lod messes up the first one. Also sometimes isnands count (number of separate shells per one lod) doesn't seem to reflect actual number of shells. They can differ +\-1 and that is causing that type of lod merging as far as I see. Can you show your algo on parsing those meshes? Or describe your working process on how you interpret data and search for buffers bounds?

[Tushkan]

Hey waaaaait a minute! There are ascii strings in it! I've never seen those before. My code wasn't suited to deal with them. So it reads data in all the wrong ways.

Edit:
Daaaamn, I'm an idiot. I checked the binary that actually was a python's pickle file. That's why it had strings. I'll go revisit the binaries. My question still applies then. How did you unpack the data for vertices? That's the most frustrating thing for me now. My algo works on most objects except for such rare exceptions.

[Tushkan]

Finally...
I was expecting a struct of type

Code: Select all

[
	[pos, pos, pos,...],
	[uv,uv,uv,...],
	[norm,norm,norm,...],
]

But it turned out to be

Code: Select all

[
	[pos, uv, norm,..],
	[pos, uv, norm,..],
	[pos, uv, norm,..],
]

I was expecting that uint32 @ 0x2c controls those two types, but I guess I was wrong. I'll have to find another parameter, that is correlating with data ordering.
Our proportions seem to differ though, that's strange...

[zaramot]

Looks pretty good already, I'm doing it a bit in a different way, I guess, since I have an experience with similar example before, not only with ubisoft games but with lost via domus (not exact but still) and with ghost recon: phantoms. It's hard, for me personally to decide which version is more correct - because of this simple box mesh, if it was some sort of vehicle, building - it would be easier to see which variant is correct!

[Tushkan]

Well, I use

Code: Select all

x, y, z, s = int16, int16, int16, int16
norm = 0x7f ff
s = s/norm
return x*s, y*s, z*s

This works well on other models and yields humanoid models to be ~1.7-1.8 units tall, which is an average human height.
But now I need to find how format regulates between buffers and struct sequences for vertex data...

[MaZTeR]

It seems you people are getting results, but is there a chance the tools can be made public at some point? Also, is that Noesis texture script ready?

[Tushkan]

As I said previously, I am planning to share my tools. Only problem is they are going to require some skills from a person using it. It's python based and it's tailored towards scripted execution i.e. there is no gui.