Nice. I'm keeping my fingers crossed ^^I'm working on an Unity asset extractor by myself
Important information: this site is currently scheduled to go offline indefinitely by end of the year.
Unity Assets Explorer [v 1.2]
- ikskoks
- Moderator
- Posts: 1666
- Joined: Thu Jul 26, 2012 5:06 pm
- Location: Poland, Łódź
- Has thanked: 647 times
- Been thanked: 431 times
- Contact:
Re: Unity Assets Explorer [v 1.2]
- Chipicao
- ultra-veteran
- Posts: 476
- Joined: Thu Feb 03, 2011 11:18 am
- Has thanked: 42 times
- Been thanked: 305 times
- Contact:
Re: Unity Assets Explorer [v 1.2]
I don't mean to be rude, but an asset extractor is pretty much pointless.barracuda wrote:Which Unity engine version was this tool written for? I tried to load asset files for 2.0, 3.3, 3.4, 3.5 and 4.1, but none of them are read correctly.
I'm working on an Unity asset extractor by myself and I identified at least three completely different asset file formats. So far, it can read files for 3.5 and 4.1. I wonder if there has been any research on the older formats?
First of all, depending on the version and game platform, you'll have different data structures for meshes, materials, textures or other assets.
Second, all assets are linked together via their indices from the preload table (which I assume you're using to get offsets and sizes for "files"). Furthermore, assets can be linked across multiple .asset files too.
By extracting assets as individual files, you lose all these pieces of information, making it hard to import 3D meshes and impossible to link them to materials, transformation matrices and scene hierarchy.
In order to import assets properly, you must keep them as they are in .asset files.
I have been working for some time on a maxscript for importing unity games, which you can download here. For now it's fully compatible with Web and iOS games made with Unity 3.3 and 4.0.1 (possibly some older versions too).
I'm looking to test and add compatibility for other versions and platforms. Would you mind sending me some of those games you tested?
Please post any requests or issues with my tools in the appropriate topics.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.
-
- beginner
- Posts: 30
- Joined: Sat Nov 24, 2012 2:15 pm
- Has thanked: 3 times
- Been thanked: 27 times
Re: Unity Assets Explorer [v 1.2]
I'm aware of that. But while the surrounding asset file format is indeed pretty variable, the raw serialized object data seems to be very similar across versions. And asset files for Unity 3.5 even contain information about their own data structure, which makes it pretty simple to deserialize them.Chipicao wrote: I don't mean to be rude, but an asset extractor is pretty much pointless.
First of all, depending on the version and game platform, you'll have different data structures for meshes, materials, textures or other assets.
My tool currently supports Texture2D, AudioClip, Font and Shader, which store their data in single unity objects with no dependencies. It also has a mode to export the raw serialized object data.Chipicao wrote: Second, all assets are linked together via their indices from the preload table (which I assume you're using to get offsets and sizes for "files"). Furthermore, assets can be linked across multiple .asset files too.
By extracting assets as individual files, you lose all these pieces of information, making it hard to import 3D meshes and impossible to link them to materials, transformation matrices and scene hierarchy.
In order to import assets properly, you must keep them as they are in .asset files.
A SkinnedMeshRenderer, for example, is indeed much more complex. But I'm currently focused on the content that is relatively easy to extract or inaccessible from within the Unity editor.
I already tried to view it before, but the thread link doesn't seem to work as a guest.Chipicao wrote: I have been working for some time on a maxscript for importing unity games, which you can download here. For now it's fully compatible with Web and iOS games made with Unity 3.3 and 4.0.1 (possibly some older versions too).
Well, I found some free UnityWeb games with Google.Chipicao wrote: I'm looking to test and add compatibility for other versions and platforms. Would you mind sending me some of those games you tested?
2.x.x:
http://www.idancerecords.com/games/MoonDragon.unity3d
3.3.x/3.4.x:
http://unity3d.com/gallery/demos/live-demos
3.5.x:
http://85playgames.eval.hwcdn.net/games ... un.unity3d
http://85playgames.eval.hwcdn.net/games ... ay.unity3d
https://w8f8a7t9.ssl.hwcdn.net/dragons/ ... 32.unity3d
4.1.x:
http://media.jumpstart.com/JSW3DUnity/JSMain.unity3d
http://media.mathblaster.com/MBUnity/MBMain.unity3d
These files are just LZMA containers with some metadata in the header, btw. They can be extracted with Unity 3D Obfuscator, my tool will support that as well.
- Chipicao
- ultra-veteran
- Posts: 476
- Joined: Thu Feb 03, 2011 11:18 am
- Has thanked: 42 times
- Been thanked: 305 times
- Contact:
Re: Unity Assets Explorer [v 1.2]
Sorry, but that's not true at all. On the few versions I've tested, I have seen differences in the raw data for the following asset types: Mesh, Texture2D, Renderer, SkinnedMeshRenderer. There are probably others too.barracuda wrote: I'm aware of that. But while the surrounding asset file format is indeed pretty variable, the raw serialized object data seems to be very similar across versions. And asset files for Unity 3.5 even contain information about their own data structure, which makes it pretty simple to deserialize them.
You may not have noticed it because from what I see you're only testing Web games. Try an iOS or PC game and you'll see what I mean.
What you say is true, but that's pretty much as far as you're gonna get regarding independent assets.barracuda wrote:My tool currently supports Texture2D, AudioClip, Font and Shader, which store their data in single unity objects with no dependencies. It also has a mode to export the raw serialized object data.
A SkinnedMeshRenderer, for example, is indeed much more complex. But I'm currently focused on the content that is relatively easy to extract or inaccessible from within the Unity editor.
Let me give you an example on how a 3D object would be properly loaded:
/ Transform -> parent-Transform -> parent-GameObject
GameObject - MeshFilter -> Mesh
\ Renderer -> Material -> Textures
If you're planning to extract only textures/sounds then it doesn't matter. But otherwise you'll be losing any means of linking assets together.
Yeah you just have to register. My tool also extracts 12 types of textures, including PVRbarracuda wrote:I already tried to view it before, but the thread link doesn't seem to work as a guest.
Thanks, I'm gonna have a look. They all look like Web games, which would explain why they don't work with this Explorer.barracuda wrote:Well, I found some free UnityWeb games with Google.
2.x.x:
http://www.idancerecords.com/games/MoonDragon.unity3d
3.3.x/3.4.x:
http://unity3d.com/gallery/demos/live-demos
3.5.x:
http://85playgames.eval.hwcdn.net/games ... un.unity3d
http://85playgames.eval.hwcdn.net/games ... ay.unity3d
https://w8f8a7t9.ssl.hwcdn.net/dragons/ ... 32.unity3d
4.1.x:
http://media.jumpstart.com/JSW3DUnity/JSMain.unity3d
http://media.mathblaster.com/MBUnity/MBMain.unity3d
P.S. You don't need that obfuscator tool, use aluigi's unity3d extraction script
Please post any requests or issues with my tools in the appropriate topics.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.
-
- beginner
- Posts: 30
- Joined: Sat Nov 24, 2012 2:15 pm
- Has thanked: 3 times
- Been thanked: 27 times
Re: Unity Assets Explorer [v 1.2]
Well yes, I mostly tested some web games. But I also tested two PC standalone games without problems. Even if there are differences, it doesn't have to mean that they can't be figured out, right? 100% support for all Unity games that have ever been developed is a highly unrealistic target anyway.Chipicao wrote:Sorry, but that's not true at all. On the few versions I've tested, I have seen differences in the raw data for the following asset types: Mesh, Texture2D, Renderer, SkinnedMeshRenderer. There are probably others too.
You may not have noticed it because from what I see you're only testing Web games. Try an iOS or PC game and you'll see what I mean.
But I'd like to test a mobile Unity game... do you have an example? Preferably not from official app stores.
Ah, mine only supports 7 right now. These DDS color bitmasks for exotic RGB formats are a bit tricky.Chipicao wrote:Yeah you just have to register. My tool also extracts 12 types of textures, including PVR
Do you maybe also know how to fix the vertical flip and convert AG to RGB normal maps?
That script could've saved me some time to figure out the archive format, but it somehow slipped through my Google research.Chipicao wrote:Thanks, I'm gonna have a look. They all look like Web games, which would explain why they don't work with this Explorer.
P.S. You don't need that obfuscator tool, use aluigi's unity3d extraction script
- Chipicao
- ultra-veteran
- Posts: 476
- Joined: Thu Feb 03, 2011 11:18 am
- Has thanked: 42 times
- Been thanked: 305 times
- Contact:
Re: Unity Assets Explorer [v 1.2]
It's not about figuring out the differences, it's about knowing when there are differences. And the key to that are the file version and file "type", both of which would be discarded after extraction.barracuda wrote:Even if there are differences, it doesn't have to mean that they can't be figured out, right? 100% support for all Unity games that have ever been developed is a highly unrealistic target anyway.
100% support for all Unity games is not unrealistic, it's my goal Ok, maybe except xbox and PS3. Those use big endian data and would be better off with a separate script.
Fast & Furious 6: The game would be one example. Or CSR Racing and Drift Mania. You can find them at your local "pirate shop"barracuda wrote:But I'd like to test a mobile Unity game... do you have an example? Preferably not from official app stores.
But here's a free app from Opel promoting one of their cars: https://mega.co.nz/#!2F9VRByY!HPu2GjBsj ... qmp9IxQlmM
Yeah, I've been through hell figuring out which texture type is ARGB, which is RGBA or BGRA and so on. But in the end I didn't use custom bitmasks because for some reason the nvidia plugin for PS ignores them. So for the sake of compatibility I just switch image bytes insteadbarracuda wrote:These DDS color bitmasks for exotic RGB formats are a bit tricky.
Do you maybe also know how to fix the vertical flip and convert AG to RGB normal maps?
There's nothing to fix, they are supposed to be flipped and they look as they should once applied to the models.
What do you mean by "AG normal maps", Alpha-Green? Maybe you just have the wrong channel order...
Please post any requests or issues with my tools in the appropriate topics.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.
-
- beginner
- Posts: 30
- Joined: Sat Nov 24, 2012 2:15 pm
- Has thanked: 3 times
- Been thanked: 27 times
Re: Unity Assets Explorer [v 1.2]
Well, I'll know when there are differences, because my program will start throwing exceptions. And it's written in Java, so the endianness isn't a big issue in my case. It can easily switch between both during runtime without manual byte swapping.Chipicao wrote:It's not about figuring out the differences, it's about knowing when there are differences. And the key to that are the file version and file "type", both of which would be discarded after extraction.
100% support for all Unity games is not unrealistic, it's my goal Ok, maybe except xbox and PS3. Those use big endian data and would be better off with a separate script.
Thanks! It mostly seems to work, just the PVR textures aren't supported yet. Hell, I don't even know how to read or convert these and the Wikipedia page about the format is a stub.Chipicao wrote:Fast & Furious 6: The game would be one example. Or CSR Racing and Drift Mania. You can find them at your local "pirate shop"
But here's a free app from Opel promoting one of their cars: https://mega.co.nz/#!2F9VRByY!HPu2GjBsj ... qmp9IxQlmM
Oh, so I'm not the only one with that problem.Chipicao wrote: Yeah, I've been through hell figuring out which texture type is ARGB, which is RGBA or BGRA and so on. But in the end I didn't use custom bitmasks because for some reason the nvidia plugin for PS ignores them. So for the sake of compatibility I just switch image bytes instead
The NVIDIA Texture Tools 2 can read the DDS files correctly, unlike the Photoshop plugin or ImageMagick. Really weird, since both the tools and the PS plugin are made by Nvidia...
Oh, ok then. I was a bit confused by the Unity editor, which flips the textures in the preview as well.Chipicao wrote: There's nothing to fix, they are supposed to be flipped and they look as they should once applied to the models.
No, the color channels are fine. Unity 3 and higher encodes the normals into alpha and green for DXT compressed textures. Red and blue are simply low-quality copies of green for better compression, so the texture appears to be grayscale even though it's not a bump map. It's explained in detail here.Chipicao wrote:What do you mean by "AG normal maps", Alpha-Green? Maybe you just have the wrong channel order...
Oh, and do you maybe have a mirror for your maxscript? I know, I could simply register on the forum, but I'm not a big fan of throwaway accounts...
- Chipicao
- ultra-veteran
- Posts: 476
- Joined: Thu Feb 03, 2011 11:18 am
- Has thanked: 42 times
- Been thanked: 305 times
- Contact:
Re: Unity Assets Explorer [v 1.2]
Yes but that's still not my point. It doesn't matter if your extractor knows about the differences, because whoever looks at the extracted data won't know. I'm talking about differences in the raw data structure (see below)barracuda wrote:Well, I'll know when there are differences, because my program will start throwing exceptions.
Here's all you need to know about the PVR format. http://www.imgtec.com/powervr/insider/d ... ternal.pdfbarracuda wrote:Thanks! It mostly seems to work, just the PVR textures aren't supported yet. Hell, I don't even know how to read or convert these and the Wikipedia page about the format is a stub.
I can see why it mostly works. The Web versions you've been working on are very similar to the iOS formats I linked. But have a look at these Web assets: http://www.mediafire.com/download/a684a ... 378/tc.rar
The version is not listed, but I think it's 3.3.0. You're gonna notice some differences in the position of the preload table, and also the data structure of Textures, Renderers, SkinnedMeshRenderers and meshes.
I guess that's just an option, Nvidia's PS plugin has it as well. Most games I've worked with have flipped texture coordinates and "regular" textures are flipped in the shaders. So I'm guessing unity takes it a step forward by flipping textures from the get go.barracuda wrote:Oh, ok then. I was a bit confused by the Unity editor, which flips the textures in the preview as well.
Ah, so it's the old trick used in Doom 3. If that's the case, then the solution is indeed moving AG image data in RG channels, and discarding the rest. However, you would have to know when the texture is a normal map and when it isn't. And the only way to do that is to go through material slots.barracuda wrote:No, the color channels are fine. Unity 3 and higher encodes the normals into alpha and green for DXT compressed textures. Red and blue are simply low-quality copies of green for better compression, so the texture appears to be grayscale even though it's not a bump map. It's explained in detail here..
After inspecting some of the games you linked, I realize a universal and automatic script might be impossible. I'll have to do some more tests to see if I need to differentiate it by platform, and I';; post a link when I\m done.
Please post any requests or issues with my tools in the appropriate topics.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.
-
- beginner
- Posts: 30
- Joined: Sat Nov 24, 2012 2:15 pm
- Has thanked: 3 times
- Been thanked: 27 times
Re: Unity Assets Explorer [v 1.2]
Seems to be the Scion car builder. It uses version 2.6.1f3 according to the .unity3d file. And yes, my tool can't read these older 2.x and 3.3 formats yet.Chipicao wrote: I can see why it mostly works. The Web versions you've been working on are very similar to the iOS formats I linked. But have a look at these Web assets: http://www.mediafire.com/download/a684a ... 378/tc.rar
The version is not listed, but I think it's 3.3.0. You're gonna notice some differences in the position of the preload table, and also the data structure of Textures, Renderers, SkinnedMeshRenderers and meshes.
- Chipicao
- ultra-veteran
- Posts: 476
- Joined: Thu Feb 03, 2011 11:18 am
- Has thanked: 42 times
- Been thanked: 305 times
- Contact:
Re: Unity Assets Explorer [v 1.2]
Please post any requests or issues with my tools in the appropriate topics.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.
-
- advanced
- Posts: 40
- Joined: Mon Jun 21, 2010 3:32 am
- Has thanked: 2 times
- Been thanked: 6 times
Re: Unity Assets Explorer [v 1.2]
Assets explorer works ok but errors out on some larger assets files.
Aluigi's BMS script apparently doesn't work on what it identifies as "Unity3d Raw" files, only works on "Unity3d Web"
anyone know if the BMS script will be updated to open these "Raw" files?
Aluigi's BMS script apparently doesn't work on what it identifies as "Unity3d Raw" files, only works on "Unity3d Web"
anyone know if the BMS script will be updated to open these "Raw" files?
-
- advanced
- Posts: 40
- Joined: Mon Jun 21, 2010 3:32 am
- Has thanked: 2 times
- Been thanked: 6 times
Re: Unity Assets Explorer [v 1.2]
use your username to open the filebarracuda wrote:Do you have a sample for a Unity3d Raw file? Then I could add support for it for my tool, which will be released in a few weeks.
-
- beginner
- Posts: 30
- Joined: Sat Nov 24, 2012 2:15 pm
- Has thanked: 3 times
- Been thanked: 27 times
Re: Unity Assets Explorer [v 1.2]
UnityRaw is essentially the same as UnityWeb, just without the LZMA compression. Here's the fixed BMS script:
Code: Select all
# Unity Web Player (unity3d files)
# script for QuickBMS http://quickbms.aluigi.org
endian big
idstring "UnityRaw"
get DUMMY long
get VER byte
get VER string
get VER string
get SIZE long
get DATAOFFSET long
goto DATAOFFSET
get NAME filename
string NAME += "_unpacked"
set PATH string NAME
get FILES long
for i = 0 < FILES
get NAME string
get OFFSET long
get SIZE long
set FNAME string PATH
string FNAME += /
string FNAME += NAME
math OFFSET += DATAOFFSET
log FNAME OFFSET SIZE
next i
-
- advanced
- Posts: 40
- Joined: Mon Jun 21, 2010 3:32 am
- Has thanked: 2 times
- Been thanked: 6 times
Re: Unity Assets Explorer [v 1.2]
hmmm, something wrong here. tried the new script on some of the Unity3d Raw files kicked out by the earlier script. It kicked out a single file with a wacky name
CAB-62d024b8560c18343a45fc4dff7f5562 and no file extension.
CAB-62d024b8560c18343a45fc4dff7f5562 and no file extension.