Star Wars - The Old Republic - Beta

[SWTOR fan]

The NodeViewer.exe was written in .NET, therefore you can easily get the original source code in a disassembler. Download here: viewtopic.php?p=63821#p63821
The main code is in the Hero.dll; the NodeViewer.exe is just doing the UI and handing over file contents to the Hero.dll.

But I don't know about any other public tool to read the GOM. Good luck trying to port that to PHP! It took me a while to understand the serialization. Maybe you can even include the original Hero.dll into PHP, that would save you a lot of work.

Unfortunately, I don't have enough time to document the DOM serialization, so I hope you can understand it just from the source code.
The NodeViewer was developed sometime during the late beta. I believe it originated from a user at BetaCake.net, but I do not know who really is behind it. Whoever did it receives a lot of credit for figuring the DOM out!

[jpreston84]

Ugh. Not only do I not know anything .NET (so its syntax, file setup, etc is foreign enough to me that I don't much know where to begin), but there are also apparently no comments in that source either, which means I'm probably out of luck.

Anyone out there willing to lend a hand dumping out the contents of client.gom to something that's actually usable? I don't really care whether the solution is PHP, C++/C#, .NET or anything else. A PHP class, or a CLI tool that just takes a client.gom input and outputs a bunch of text and other files with appropriate data, would be ideal to me!

(I'll keep working at it, but I must stress that it's highly unlikely I'll succeed at this.)

[jpreston84]

Well, I've gotten further than I'd thought. I've gotten through much of the client.gom, and into the ZLib sections in the bucket files, and decoded the ZLib compression. While I'm sure these are the key/value pairs, the actual format of these sections still escapes me.

[TheArk]

Thanks for posting all this stuff SWTOR and Vindis and whoever else has been involvt.


Was just wondering TORFAN if with the last filename list you posted there was any knowledge of where the Texture files were, we found all the new Czerka Stuff model wise but no luck on textures?

[Acewell]

..we found all the new Czerka Stuff model wise but no luck on textures

The models and their textures are in swtor_main_art_zed_1.tor, but most of the textures still don't have names yet.

[TheArk]

Alright, so then what's the process for giving them names, if there's any way I can help speed up the process I will.

[Acewell]

I don't know, the only ones finding file names are SWTOR fan, Rick and Vindis.

Vindis released a tool that helps find names but i can't get it to work.
viewtopic.php?p=83851#p83851

This is what Rick says about the filenames:
viewtopic.php?p=63422#p63422
http://www.burtleburtle.net/bob/c/lookup3.c

I'm sure a lot of people would contribute to finding names if those guys released their tools.

[jpreston84]

FYI -- Despite not being able to use the .NET source code that was made available (except for the list of data types, half of which I'd already identified), by using a hex editor and looking at data in NodeViewer, I've deciphered 95% or so of the Bucket file spec. I'll try to post that as soon as I can solve a little detail about the ZLib-compressed sections. (This is all somewhat changed from the data that's been posted here previously.)

SWTOR Fan,

Can you possibly enlighten me on where the labels are pulling from in the ZLib-compressed sections? In cases where the label for an entry is something like CC XX XX XX XX XX or CF XX XX XX XX XX XX XX XX, I am assuming these bytes represent an address, perhaps an offset, but I can't figure out the specifics of what it's an offset to. Can you help with that? Just a pointer in the right direction is all I think I need.

My one other issue at present is that the ZLib-compressed sections have a varying number of header bits in some cases, and I haven't been able to decipher the purpose of these bits. Any help would be appreciated.

[swtor miner]

Here's my latest hashlist: http://www.sendspace.com/file/nedoar
It has several thousand new files, including most of the files in the main_gfx_1.tor that's not in the Assets directory.

You guys might want to load up the *.gfx files extracted from the main_gfx_1.tor in a hex editor.
1. Change the header from CFX to CWS
2. Change the extension to .swf
3. Run them through a flash decompiler like trillix.

For those that don't know, the .gfx files are in Scaleform GFX format. Which is an embedded version of Flash, and you can decompile them with this method. Some very interesting stuff in the SCFF tagged files.

Expect updated versions of EasyMYP and Nodeviewer in the near future. Once I'm sure they don't have any major bugs.

edit: I guess this is the wrong thread for the Noesis plugin now that I think about it. I'll leave it here. But, I posted the information to: viewtopic.php?f=16&t=9703&p=88035

And, I know you guys will like this. Here's an updated version of Vindis's Noesis Plugin that now loads multiple meshes into one model: http://www.sendspace.com/file/1mchxl
1. Put it in the Noesis/plugins/python directory, and it will allow you to view .gr2 files.
2. Now put all the .gr2 files you want to view, and their associated textures (just the files ending in _d.dds) into a folder.
3. Create a file in that directory called model_list.txt, and put all but one of the names of those .gr2 files in that file. (Just the file name, one per line, no /'s)
4. Load up the one .gr2 you left off the list, and it will reassemble it for you.

I've left in/fixed support for the Specular Maps(_s.dds) and Normal Maps(_n.dds), and you can set Noesis to load these by enabling the loading non-diffuse textures in the Data Viewer. Unfortunately the Normal Maps seem to require tweaked lighting data to function properly, and loading of that data is not implemented yet.

[SWTOR fan]

Hi swtor_miner, it's nice to finally see you on XeNTaX!
Thank you very much for posting your hashes_filename.txt file! I found a few thousand new file names myself since I last published the hashes_filename.txt file, and will post an update on the current file name progress later.

I spent a lot of time with the BWA Material format recently. It was not too much fun working with the HLSL shader code, but I am happy that I am mostly done with it.

I can now confirm that the 8 unknown bytes for vertices with vertexSize>12 stand for the normals and tangents.
Basically, the values from -1 to +1 are encoded into a byte, so that 0x00 is -1, 0x7F is 0, and 0xFF is +1. You can convert a byte to the float value the following way:
float byteToFloat(byte byteValue) {
return byteValue / 127.5 + 1.0;
}
The eight bytes in the vertices are stored this way:
byte: normal.x
byte: normal.y
byte: normal.z
byte: always -1 or +1
byte: tangent.x
byte: tangent.y
byte: tangent.z
byte: always -1 or +1
Unfortunately, I always get an error when I try to edit the wiki, so I cannot update the .gr2 specification there.

For the standard materials that just have a DiffuseMap, RotationMap1 and GlossMap, the output color is calculated the following way:
- Use the RGB channels of the DiffuseMap as diffuse color
- Ambient color and emissive color are the same as the diffuse color.
- Use the red channel of the RotationMap1 as opacity
- Use the blue channel of the RotationMap1 as emissive luminance
- Use the green and alpha channel of the RotationMap1 as red and green channel of the bump map.
- Use the RGB channels of the GlossMap as the specular color, and the alpha channel as the shininess.

Also, my program can preview models with AnimatedUV and PaletteMap materials, but it is still looking different from the 3d models in-game.
Here is a preview of the /resources/art/static/area/all_all/item/tech/all_item_tech_city_computer_01.gr2 file with an AnimatedUV material (and yes, the texture is animated, but I cannot take an animated screenshot ):

[SWTOR fan]

Alright, I calculated the current progress of how many file names we have discovered, based on the current versions of the client.

Retailclient, 2.3.2b:
444,135 / 455,078 files named (97.60%), 10,943 file names missing

PTS, 2013-09-11:
452,313 / 468,471 files named (96.55%), 16,158 file names missing


This looks very promising. We are mostly missing some animation files (.jba/.mph/.amx) and the Cartel Market files (.dds).
I am still looking for the names of the Cartel Market icons, which are stored in the swtor_main_gfx_assets_1.tor archive.
Like I said, all the files are ordered alphabetically, and the Cartel Market icons are placed in between
/resources/gfx/loadingscreen/warzone_voidstar_overlay_text.dds and
/resources/gfx/placeables/galaxy_map/noise_1.swf.
Because of that, I am confident that the folder name starts with "mtx".

Also, I am quite certain that the Cartel Market icon file names can be found in field 0x40000040A41F2780 of the mtxStorefrontInfoPrototype. Some of those file names are:
generic_unlock, lifeday_fireworks_pack_x12, legacy_upsell, interface_upsell_color_match, interface_upsell_legacy_display, class.war.bis.mdps2.t2x2, heavy_chest_bh_rheal_6x1.

In the game, there are five different-sized images shown:
120x120: displayed on the front page as a small size
260x260: displayed in the Collections Interface as a small icon
260x400: displayed on the front page as a 1x3 sized icon
328x160: displayed when browsing through the different categories in the Cartel Market
400x400: displayed on the right side in the Cartel Market, as well as the large icon in the Collections interface

So it seems like there is a certain suffix to the file name to reference the current size of the image, like _a.dds, _b.dds, etc.
I have looked through the SCPT files because they contain the folder names for many other files, but unfortunately I haven't found the folder name for the Cartel Market there, and I do not have much motivation to look through them again.
I guess I will once more try out different variations of /resources/gfx/mtx/generic_unlock_1.dds...

If anyone encounters an error message that accidentally exposes the file name, or has any idea how a folder name might be called, feel free to tell me.

[swtor miner]

Hi swtor_miner, it's nice to finally see you on XeNTaX!
Thank you very much for posting your hashes_filename.txt file! I found a few thousand new file names myself since I last published the hashes_filename.txt file, and will post an update on the current file name progress later.

Hi SWTOR fan! I've been lurking for a while now, and I very much appreciate your posts.

As, someone not very well versed in the art of 3d modelling the Noesis plugin helped me a lot. Long term, a standalone modelviewer that reads directly from the game files seems like a better option than the Noesis plugin. But, it's functional for now, and I think I'll implement your fixes when I get some time.

I have a few standalone programs for searching extracted output for filenames. Which I'm in the process of updating for animations. I made some progress tonight, but was mainly looking at the latest PTS build. Scanned through it for more file names. Here's my latest hash: http://www.sendspace.com/file/muqtt2

A lot, and I mean a lot of the file names are in xml files, which makes it a breeze to write programs to find them.. I really need to write a program like yours to see how many are left, but that's not high on my priority list.

edit: On a hunch I checked "\resources\gfx\gfx_production\ui_hud.gfx", and it contained the format for the Cartel Market assets:

Code: Select all

"/resources/gfx/mtxstore/" + file + "_" + imgWidth +  "x" + imgHeight + ".dds"

Which I checked with: "/resources/gfx/mtxstore/lifeday_fireworks_pack_x12_120x120.dds" and it was found! Enjoy!

[SWTOR fan]

Wow, good job on finding the folder name for the MTX icons! Also, thank you very much for your lead about the .gfx files. I never imagined that they would contain so many helpful information. Just from looking through the ActionScript code, I found a lot of answers to things I had wondered about previously.

And finally, I figured out the PaletteMap materials. I can now display NPCs and items from the game with the correct garment hues.

For example, this is npp.location.flashpoint.shared.mandalorian_raiders.trash.mandalorian_tech_commando. The green and red colors do not come from the diffuse texture, but from the garment hues:


And here is an example of an NPC with visible skin (npp.location.flashpoint.shared.cademimu.flavor.dancer). The red color of the skirt and the yellow headdress and buttons once again come from the garment hues, not the diffuse texture. Note that the white pixels should in fact be transparent; that's just a bug with my 3D viewer.
There are a lot of threads on the official forums regarding revealing clothes items where the companions have gray skin, so it's nice to see that the developers were also having problems working with the skin.


Now I am only missing hair and eye colors, as well as adding the age/complexion/facepaint textures to the head slot.


The AMI system is so complicated that I do not want to spend too much time explaining it, but here are the most important parts:
The "derived" field in the .mat files explicitly describes the type of texture. The game uses this field to determine which shader should be used for the material.
Most of the normal models (walls, terminals, tables, ...) have "derived" set to "Uber", where the materials are parsed like I wrote in my previous post.
All of the body parts (boot/chest/leg/etc., but not eye/hair/head) have "derived" set to "Garment". The pixel shader for Garment materials then hues the pixel according to the palettes.
If the PaletteMaskMap is black, then the diffuse texture is displayed.
If the PaletteMaskMap is red, then the palette1 color mixed with the PaletteMap texture is displayed.
If the PaletteMaskMap is green, then the palette2 color mixed with the PaletteMap texture is displayed.
If the PaletteMaskMap is blue, then this pixel has a metallic specular color.
The alpha value of the PaletteMaskMap is not used.
Any other colors (cyan, purple etc.) are just combinations of these cases.

palette1 and palette2 are of the format "hue, specular, lightness, contrast" and they default to (0, 0.5, 0, 1).
The ipp and npp nodes usually reference two garment hues (appApperanceSlotHuePrimary, appApperanceSlotHueSecondary). Read the respective garment hue .xml file in /resources/art/dynamic/garmenthue/ and override the palette1/palette2 fields from the .mat file with the new hue/specular/lightness/contrast values from the .xml.
If you are interested in how the palette1/palette2 colors are mixed with the PaletteMap, I'll just upload the original HLSL shader code from the game: http://pastebin.com/sDRVvBYy You best start reading the code at the HuePixel() function, which is called by the Garment shader.
See also http://msdn.microsoft.com/en-us/library ... s.85).aspx if you are new to shaders.

[swtor miner]

Yeah, I've never even attempted anything remotely like 3d modeling/programming. But, I taught myself python. Which is kind of an odd language, and that you have to target a very small subset of the language for Noesis. I've mainly dabbled with c++/c#/java and was a major slacker on learning them. Now that I've got the basics I think I'm going to write a native C# version of it.

Mainly thanks to your posts, and others here. I taught myself to reconstruct some of the progress you've made. I still have to implement the vertexsize>12 normals and tangents, but I don't see that being too hard. I probably could've put together which channels were which by myself, but you really saved me some time. I even assumed, at first, that some of the color channels might be normalized or flipped, but it was really nice that they weren't. So, far I've got emissives and bump maps to work.



A laundry list of things I need to work on include making the glossmap work properly, loading bones/skeletons, animations, mutliUVExtractions, palatte map materials, scrolling UVs, mat/asset xml parsing for gear/npcs.

I've made a little bit of headway on the file names, but most of my free time has been spent playing or working on the modelviewer. It works on a surprising large number of models now, considering it's 12KB of code. http://www.sendspace.com/file/mjqh3u

If you have a decent hashlist most static models should load from the EasyMYP extracted output. You can put several .gr2 filenames in a "model_list.txt" file. Then open a .gr2 in that folder and it will load the models from "model_list.txt" as well. There are a lot of models that don't work with it right now. But, I think it's a huge improvement over the other one.

I'm downloading the fr/de localizations now to run my filename finder against. With just the English assets, I'm sitting at:

434,458/447,694 files 13,236 missing files 97% filenames found

[SWTOR fan]

Why oh why did they have to put all the space pvp stuff into 2.3.3...
I was hoping I had at least until 2.4 before I had to deal with that, but no such luck. I also wonder who at Bioware had the stupid idea of not placing those nodes under a nice folder name, but placing them in the main GOM directory instead.

Anyway, I now adjusted my program to automatically rename nodes from the spvp base classes so that they are placed into a separate folder and do not pollute the main directory.

For anyone who wants to do the same, here is a list of all the new spvp base classes that do not have a folder name:
4611686299377594007 - BlasterVisual
4611686299377594016 - BlasterConfig
4611686299409904000 - SCFF Damage FX Package Prototype
4611686299409904002 - Hull Damage FX Tier Prototype
4611686299446374010 - Shield Damage FX Prototype
4611686299499134001 - Damage FX Names Prototype
4611686299618324017 - Barrel Roll Camera Behavior
4611686348326037001 - A free flight component