XeNTaX

I'm working on an Unity asset extractor by myself

Nice. I'm keeping my fingers crossed ^^

barracuda wrote:Which Unity engine version was this tool written for? I tried to load asset files for 2.0, 3.3, 3.4, 3.5 and 4.1, but none of them are read correctly.

I'm working on an Unity asset extractor by myself and I identified at least three completely different asset file formats. So far, it can read files for 3.5 and 4.1. I wonder if there has been any research on the older formats?

I don't mean to be rude, but an asset extractor is pretty much pointless.

First of all, depending on the version and game platform, you'll have different data structures for meshes, materials, textures or other assets.
Second, all assets are linked together via their indices from the preload table (which I assume you're using to get offsets and sizes for "files"). Furthermore, assets can be linked across multiple .asset files too.

By extracting assets as individual files, you lose all these pieces of information, making it hard to import 3D meshes and impossible to link them to materials, transformation matrices and scene hierarchy.
In order to import assets properly, you must keep them as they are in .asset files.

I have been working for some time on a maxscript for importing unity games, which you can download here. For now it's fully compatible with Web and iOS games made with Unity 3.3 and 4.0.1 (possibly some older versions too).
I'm looking to test and add compatibility for other versions and platforms. Would you mind sending me some of those games you tested?

Chipicao wrote: I don't mean to be rude, but an asset extractor is pretty much pointless.

First of all, depending on the version and game platform, you'll have different data structures for meshes, materials, textures or other assets.

I'm aware of that. But while the surrounding asset file format is indeed pretty variable, the raw serialized object data seems to be very similar across versions. And asset files for Unity 3.5 even contain information about their own data structure, which makes it pretty simple to deserialize them.

Chipicao wrote: Second, all assets are linked together via their indices from the preload table (which I assume you're using to get offsets and sizes for "files"). Furthermore, assets can be linked across multiple .asset files too.

By extracting assets as individual files, you lose all these pieces of information, making it hard to import 3D meshes and impossible to link them to materials, transformation matrices and scene hierarchy.
In order to import assets properly, you must keep them as they are in .asset files.

My tool currently supports Texture2D, AudioClip, Font and Shader, which store their data in single unity objects with no dependencies. It also has a mode to export the raw serialized object data.

A SkinnedMeshRenderer, for example, is indeed much more complex. But I'm currently focused on the content that is relatively easy to extract or inaccessible from within the Unity editor.

Chipicao wrote: I have been working for some time on a maxscript for importing unity games, which you can download here. For now it's fully compatible with Web and iOS games made with Unity 3.3 and 4.0.1 (possibly some older versions too).

I already tried to view it before, but the thread link doesn't seem to work as a guest.

Chipicao wrote: I'm looking to test and add compatibility for other versions and platforms. Would you mind sending me some of those games you tested?

Well, I found some free UnityWeb games with Google.

2.x.x:
http://www.idancerecords.com/games/MoonDragon.unity3d

3.3.x/3.4.x:
http://unity3d.com/gallery/demos/live-demos

3.5.x:
http://85playgames.eval.hwcdn.net/games ... un.unity3d
http://85playgames.eval.hwcdn.net/games ... ay.unity3d
https://w8f8a7t9.ssl.hwcdn.net/dragons/ ... 32.unity3d

4.1.x:
http://media.jumpstart.com/JSW3DUnity/JSMain.unity3d
http://media.mathblaster.com/MBUnity/MBMain.unity3d

These files are just LZMA containers with some metadata in the header, btw. They can be extracted with Unity 3D Obfuscator, my tool will support that as well.

barracuda wrote: I'm aware of that. But while the surrounding asset file format is indeed pretty variable, the raw serialized object data seems to be very similar across versions. And asset files for Unity 3.5 even contain information about their own data structure, which makes it pretty simple to deserialize them.

Sorry, but that's not true at all. On the few versions I've tested, I have seen differences in the raw data for the following asset types: Mesh, Texture2D, Renderer, SkinnedMeshRenderer. There are probably others too.
You may not have noticed it because from what I see you're only testing Web games. Try an iOS or PC game and you'll see what I mean.

barracuda wrote:My tool currently supports Texture2D, AudioClip, Font and Shader, which store their data in single unity objects with no dependencies. It also has a mode to export the raw serialized object data.

A SkinnedMeshRenderer, for example, is indeed much more complex. But I'm currently focused on the content that is relatively easy to extract or inaccessible from within the Unity editor.

What you say is true, but that's pretty much as far as you're gonna get regarding independent assets.
Let me give you an example on how a 3D object would be properly loaded:

                    / Transform -> parent-Transform -> parent-GameObject
GameObject - MeshFilter -> Mesh
                    \ Renderer -> Material -> Textures

If you're planning to extract only textures/sounds then it doesn't matter. But otherwise you'll be losing any means of linking assets together.

barracuda wrote:I already tried to view it before, but the thread link doesn't seem to work as a guest.

Yeah you just have to register. My tool also extracts 12 types of textures, including PVR

barracuda wrote:Well, I found some free UnityWeb games with Google.

2.x.x:
http://www.idancerecords.com/games/MoonDragon.unity3d

3.3.x/3.4.x:
http://unity3d.com/gallery/demos/live-demos

3.5.x:
http://85playgames.eval.hwcdn.net/games ... un.unity3d
http://85playgames.eval.hwcdn.net/games ... ay.unity3d
https://w8f8a7t9.ssl.hwcdn.net/dragons/ ... 32.unity3d

4.1.x:
http://media.jumpstart.com/JSW3DUnity/JSMain.unity3d
http://media.mathblaster.com/MBUnity/MBMain.unity3d

Thanks, I'm gonna have a look. They all look like Web games, which would explain why they don't work with this Explorer.
P.S. You don't need that obfuscator tool, use aluigi's unity3d extraction script

Chipicao wrote:Sorry, but that's not true at all. On the few versions I've tested, I have seen differences in the raw data for the following asset types: Mesh, Texture2D, Renderer, SkinnedMeshRenderer. There are probably others too.
You may not have noticed it because from what I see you're only testing Web games. Try an iOS or PC game and you'll see what I mean.

Well yes, I mostly tested some web games. But I also tested two PC standalone games without problems. Even if there are differences, it doesn't have to mean that they can't be figured out, right? 100% support for all Unity games that have ever been developed is a highly unrealistic target anyway.

But I'd like to test a mobile Unity game... do you have an example? Preferably not from official app stores.

Chipicao wrote:Yeah you just have to register. My tool also extracts 12 types of textures, including PVR

Ah, mine only supports 7 right now. These DDS color bitmasks for exotic RGB formats are a bit tricky.
Do you maybe also know how to fix the vertical flip and convert AG to RGB normal maps?

Chipicao wrote:Thanks, I'm gonna have a look. They all look like Web games, which would explain why they don't work with this Explorer.
P.S. You don't need that obfuscator tool, use aluigi's unity3d extraction script

That script could've saved me some time to figure out the archive format, but it somehow slipped through my Google research.

barracuda wrote:Even if there are differences, it doesn't have to mean that they can't be figured out, right? 100% support for all Unity games that have ever been developed is a highly unrealistic target anyway.

It's not about figuring out the differences, it's about knowing when there are differences. And the key to that are the file version and file "type", both of which would be discarded after extraction.
100% support for all Unity games is not unrealistic, it's my goal Ok, maybe except xbox and PS3. Those use big endian data and would be better off with a separate script.

barracuda wrote:But I'd like to test a mobile Unity game... do you have an example? Preferably not from official app stores.

Fast & Furious 6: The game would be one example. Or CSR Racing and Drift Mania. You can find them at your local "pirate shop"
But here's a free app from Opel promoting one of their cars: https://mega.co.nz/#!2F9VRByY!HPu2GjBsj ... qmp9IxQlmM

barracuda wrote:These DDS color bitmasks for exotic RGB formats are a bit tricky.
Do you maybe also know how to fix the vertical flip and convert AG to RGB normal maps?

Yeah, I've been through hell figuring out which texture type is ARGB, which is RGBA or BGRA and so on. But in the end I didn't use custom bitmasks because for some reason the nvidia plugin for PS ignores them. So for the sake of compatibility I just switch image bytes instead
There's nothing to fix, they are supposed to be flipped and they look as they should once applied to the models.

What do you mean by "AG normal maps", Alpha-Green? Maybe you just have the wrong channel order...

Chipicao wrote:It's not about figuring out the differences, it's about knowing when there are differences. And the key to that are the file version and file "type", both of which would be discarded after extraction.
100% support for all Unity games is not unrealistic, it's my goal Ok, maybe except xbox and PS3. Those use big endian data and would be better off with a separate script.

Well, I'll know when there are differences, because my program will start throwing exceptions. And it's written in Java, so the endianness isn't a big issue in my case. It can easily switch between both during runtime without manual byte swapping.

Chipicao wrote:Fast & Furious 6: The game would be one example. Or CSR Racing and Drift Mania. You can find them at your local "pirate shop"
But here's a free app from Opel promoting one of their cars: https://mega.co.nz/#!2F9VRByY!HPu2GjBsj ... qmp9IxQlmM

Thanks! It mostly seems to work, just the PVR textures aren't supported yet. Hell, I don't even know how to read or convert these and the Wikipedia page about the format is a stub.

Chipicao wrote: Yeah, I've been through hell figuring out which texture type is ARGB, which is RGBA or BGRA and so on. But in the end I didn't use custom bitmasks because for some reason the nvidia plugin for PS ignores them. So for the sake of compatibility I just switch image bytes instead

Oh, so I'm not the only one with that problem.

The NVIDIA Texture Tools 2 can read the DDS files correctly, unlike the Photoshop plugin or ImageMagick. Really weird, since both the tools and the PS plugin are made by Nvidia...

Chipicao wrote: There's nothing to fix, they are supposed to be flipped and they look as they should once applied to the models.

Oh, ok then. I was a bit confused by the Unity editor, which flips the textures in the preview as well.

Chipicao wrote:What do you mean by "AG normal maps", Alpha-Green? Maybe you just have the wrong channel order...

No, the color channels are fine. Unity 3 and higher encodes the normals into alpha and green for DXT compressed textures. Red and blue are simply low-quality copies of green for better compression, so the texture appears to be grayscale even though it's not a bump map. It's explained in detail here.

Oh, and do you maybe have a mirror for your maxscript? I know, I could simply register on the forum, but I'm not a big fan of throwaway accounts...

barracuda wrote:Well, I'll know when there are differences, because my program will start throwing exceptions.

Yes but that's still not my point. It doesn't matter if your extractor knows about the differences, because whoever looks at the extracted data won't know. I'm talking about differences in the raw data structure (see below)

barracuda wrote:Thanks! It mostly seems to work, just the PVR textures aren't supported yet. Hell, I don't even know how to read or convert these and the Wikipedia page about the format is a stub.

Here's all you need to know about the PVR format. http://www.imgtec.com/powervr/insider/d ... ternal.pdf

I can see why it mostly works. The Web versions you've been working on are very similar to the iOS formats I linked. But have a look at these Web assets: http://www.mediafire.com/download/a684a ... 378/tc.rar
The version is not listed, but I think it's 3.3.0. You're gonna notice some differences in the position of the preload table, and also the data structure of Textures, Renderers, SkinnedMeshRenderers and meshes.

barracuda wrote:Oh, ok then. I was a bit confused by the Unity editor, which flips the textures in the preview as well.

I guess that's just an option, Nvidia's PS plugin has it as well. Most games I've worked with have flipped texture coordinates and "regular" textures are flipped in the shaders. So I'm guessing unity takes it a step forward by flipping textures from the get go.

barracuda wrote:No, the color channels are fine. Unity 3 and higher encodes the normals into alpha and green for DXT compressed textures. Red and blue are simply low-quality copies of green for better compression, so the texture appears to be grayscale even though it's not a bump map. It's explained in detail here..

Ah, so it's the old trick used in Doom 3. If that's the case, then the solution is indeed moving AG image data in RG channels, and discarding the rest. However, you would have to know when the texture is a normal map and when it isn't. And the only way to do that is to go through material slots.

After inspecting some of the games you linked, I realize a universal and automatic script might be impossible. I'll have to do some more tests to see if I need to differentiate it by platform, and I';; post a link when I\m done.

Chipicao wrote: I can see why it mostly works. The Web versions you've been working on are very similar to the iOS formats I linked. But have a look at these Web assets: http://www.mediafire.com/download/a684a ... 378/tc.rar
The version is not listed, but I think it's 3.3.0. You're gonna notice some differences in the position of the preload table, and also the data structure of Textures, Renderers, SkinnedMeshRenderers and meshes.

Seems to be the Scion car builder. It uses version 2.6.1f3 according to the .unity3d file. And yes, my tool can't read these older 2.x and 3.3 formats yet.

Indeed it is

Assets explorer works ok but errors out on some larger assets files.

Aluigi's BMS script apparently doesn't work on what it identifies as "Unity3d Raw" files, only works on "Unity3d Web"

anyone know if the BMS script will be updated to open these "Raw" files?

Do you have a sample for a Unity3d Raw file? Then I could add support for it for my tool, which will be released in a few weeks.

barracuda wrote:Do you have a sample for a Unity3d Raw file? Then I could add support for it for my tool, which will be released in a few weeks.

use your username to open the file

UnityRaw is essentially the same as UnityWeb, just without the LZMA compression. Here's the fixed BMS script:

hmmm, something wrong here. tried the new script on some of the Unity3d Raw files kicked out by the earlier script. It kicked out a single file with a wacky name
CAB-62d024b8560c18343a45fc4dff7f5562 and no file extension.