Star Wars - The Old Republic - Beta

[swtor miner]

Latest Hashlist: http://www.sendspace.com/file/11zzgt

Current Stats for the latest 2.4PTS Build(Including FR/DE Locals, and main_gfx_1.tor):
456,811 of 468,479 Filenames known. 97.5% Completion
11,668 Missing Filenames.

Here's the breakdown of .tor files with incomplete file names:

Code: Select all

Tor File	Known Names	Total Names	Unknown	% Completion
localizations\swtor_de-de_global_1.tor	3847	3890	43	98.90%
localizations\swtor_de-de_zed_1.tor	454	617	163	73.60%
localizations\swtor_fr-fr_global_1.tor	3879	3922	43	98.90%
localizations\swtor_fr-fr_zed_1.tor	470	619	149	75.90%
publictest\retailclient\main_gfx_1.tor	254	282	28	90.10%
swtor_en-us_global_1.tor	4131	4139	8	99.80%
swtor_main_anim_creature_a_1.tor	9490	9888	398	96.00%
swtor_main_anim_creature_b_1.tor	9529	9930	401	96.00%
swtor_main_anim_creature_npc_1.tor	20588	21271	683	96.80%
swtor_main_anim_humanoid_bfab_1.tor	15058	15680	622	96.00%
swtor_main_anim_humanoid_bfns_1.tor	15054	15678	624	96.00%
swtor_main_anim_humanoid_bmaf_1.tor	15061	15782	721	95.40%
swtor_main_anim_humanoid_bmns_1.tor	15361	15890	529	96.70%
swtor_main_anim_misc_1.tor	4089	4222	133	96.80%
swtor_main_area_balmorra_1.tor	1563	1579	16	99.00%
swtor_main_area_belsavis_1.tor	2778	2793	15	99.50%
swtor_main_area_coruscant_1.tor	2159	2161	2	99.90%
swtor_main_area_dromund_kaas_1.tor	1624	1625	1	99.90%
swtor_main_area_epsilon_1.tor	154	161	7	95.70%
swtor_main_area_ilum_1.tor	691	713	22	96.90%
swtor_main_area_makeb_1.tor	3923	3970	47	98.80%
swtor_main_area_misc_1.tor	6461	6662	201	97.00%
swtor_main_area_nar_shaddaa_1.tor	2067	2117	50	97.60%
swtor_main_area_raid_1.tor	444	459	15	96.70%
swtor_main_area_tatooine_1.tor	2646	2648	2	99.90%
swtor_main_area_tython_1.tor	1656	1657	1	99.90%
swtor_main_areadat_epsilon_1.tor	644	786	142	81.90%
swtor_main_art_area_all_item_1.tor	9737	9759	22	99.80%
swtor_main_art_creature_a_1.tor	2471	2485	14	99.40%
swtor_main_art_creature_b_1.tor	1245	1255	10	99.20%
swtor_main_art_creature_npc_1.tor	1938	2006	68	96.60%
swtor_main_art_dynamic_cape_1.tor	6591	6931	340	95.10%
swtor_main_art_dynamic_chest_1.tor	5591	5888	297	95.00%
swtor_main_art_dynamic_chest_tight_1.tor	6311	6897	586	91.50%
swtor_main_art_dynamic_hand_1.tor	18920	19208	288	98.50%
swtor_main_art_dynamic_head_1.tor	24006	24534	528	97.80%
swtor_main_art_dynamic_lower_1.tor	33814	34490	676	98.00%
swtor_main_art_dynamic_mags_1.tor	973	978	5	99.50%
swtor_main_art_fx_1.tor	31376	32820	1444	95.60%
swtor_main_art_misc_1.tor	2484	2593	109	95.80%
swtor_main_art_vehicle_1.tor	2942	2949	7	99.80%
swtor_main_art_weapon_1.tor	5267	5277	10	99.80%
swtor_main_art_zed_1.tor	7313	7611	298	96.10%
swtor_main_bnk_audiodata_1.tor	77	78	1	98.70%
swtor_main_bnk_location_1.tor	60	61	1	98.40%
swtor_main_bnk_streamed_a_1.tor	416	427	11	97.40%
swtor_main_bnk_streamed_b_1.tor	453	474	21	95.60%
swtor_main_bnk_streamed_c_1.tor	292	297	5	98.30%
swtor_main_bnk_voc_1.tor	415	428	13	97.00%
swtor_main_gfx_assets_1.tor	60202	61966	1764	97.20%
swtor_main_systemgenerated_gom_1.tor	889	936	47	95.00%
swtor_main_zed_1.tor	1038	1067	29	97.30%

I agree, the space stuff being in the root of the GOM kind of blows.

[jpreston84]

Okay, so, question to those of you who have been finding a bunch of file names....

1) Can someone give me a basic walkthrough on finding file names, including...
i) A pointer to the .tor/.myp file spec. I'm assuming one exists, but can't seem to find it (I could just be blind, in which case, you're perfectly entitled to hit me over the head with a blunt object).
ii) A how-to on the hashing algorithm used to match the filenames.
iii) If you have it, a program that's better than EasyMYP for working with these files. It doesn't have to be perfect. I'd just really love to not have to write my own program for this right now.

2) Can someone please tell me if you've actually scanned the /resources/word/areas/*/area.dat files for filenames yet? I've been working with area.dat (and related files) a bit, and if you look inside there, in the [ASSETS] section, you'll notice that many of the entries are clearly file names. For instance...

4611686041729870061=\art\static\area\all_all\arch\imperial\all_arch_imp_space_door_med_closed.gr2

...is a filename which has already been discovered and placed in the file lists (technically stored under the /resources/art path, not just /art) . On the other hand, consider these...

4611686045449371975=\spn\location\coruscant\taxi\taxi_hub3_poi01_blacksun.spn_c
4611686045449371977=\spn\location\coruscant\taxi\taxi_hub1_poi02_senate.spn_c
4611686045449371983=\spn\location\coruscant\taxi\taxi_hub2_poi01_market.spn_c
4611686045449371985=\spn\location\coruscant\taxi\taxi_hub4_poi01_justicar.spn_c
4611686045449371987=\spn\location\coruscant\taxi\taxi_hub5_poi01_works.spn_c

...which to my knowledge, are NOT in the file lists so far. These particular files would seem, I guess, to be related to the spawning of taxi droids on the map (and are referenced in the individual room .dat files, with coordinates, etc).

There are over 115K entries in the area.dat files [ASSETS] sections, and so I'm wondering if some of these aren't the ones missing from the file lists. If it seems like this would be relevant, and someone wants to give me next steps for how to identify these files in the archives, or wants to just do the work themselves and provide us with an updated file list, that'd be swell.

[SWTOR fan]

Hi jpreston84!

You can find the MYP file format on https://code.google.com/p/easymyp/sourc ... YPInfo.txt.

For hashing filenames, I just included the HashCreator.dll and HashDictionary.dll libraries from EasyMYP since that saves you a lot of trouble of dealing with the hash function itself.
If you use .NET, you can copy & paste the following code:

Code: Select all

myDict = New nsHashDictionary.HashDictionary();
myCreator = New nsHashCreator.HashCreator(myDict, nsHasherFunctions.Hasher.HasherType.TOR);

to initialize the hasher. Then call the following function to save the primary/secondary hashes for the file name in ph and sh.

Code: Select all

uint ph;
uint sh;
myCreator.hashFilename("/resources/your_path_here", ph, sh);

Sorry, I do not have plans to share any of my tools.


Regarding the area.dat files, those are in fact one of the best sources to get file names. My program is automatically reading and testing them, so rest assured that the hash lists I publish contains those files. (although I may be behind one or two patches)

Back in the beta, the .spn_c files were in fact contained as .xml files under those paths, but they have since been removed. Instead, the game is now reading the spawners by their GOM node.
For example, if a file is given by
\spn\location\coruscant\taxi\taxi_hub3_poi01_blacksun.spn_c
, the game does in fact load the GOM node
spn.location.coruscant.taxi.taxi_hub3_poi01_blacksun

Hope I could answer your questions.

[jpreston84]

Thanks for the information! Clearly, you folks are way more on top of this stuff than I am.

As the tools that are available (including your hash lists) are sufficient, I think I'll avoid writing my own for the moment, particularly since, as mentioned, I'm a PHP junkie, which unfortunately means loading Hero.dll would be a problem for me (PHP has no good way to do this).

I have no doubt I'll have more questions later.

Again, thanks for the help so far.

[jpreston84]

Okay, I've got another question for whoever will still put up with me!

In the process of parsing the buckets (and client.gom), I'm able to parse only some of the labels, and am missing a few ancillary pieces of data that would be helpful to have. This is a bit long-winded, so please bear with me.

For example, if you consider node "world.areas.4611686018427650020.mapdata". The first node/section inside here has a label of 4611686042955270002, which can be looked up in an old version of client.gom to reveal the proper label of "mapDataContainerMapDataList". The label of the first sub-node in this list is simply the integer 1, which shows identically in NodeViewer. This is a class which contains data about a map page. The first entry in this class is clearly labeled with a value that, once looked up in client.gom, translates to "mapName". However, the following two values (of the Vector3 type) are labeled with 0x04 and 0x01, respectively, where they should be (according to NodeViewer) labeled with entries for "mapPageMinCoord" and "mapPageMaxCoord".

I've double and triple-checked the data in this section by hand, and there are definitely no bytes which I am not accounting for. I believe I'm understanding the format correctly. My current guess is that because we're inside a class structure, the labels of 0x04 and 0x01 are actually referencing something else, which ultimately references to the client.gom entries. However, thus far, I've been unable to determine exactly how this correlation is made. I really could use some help here.

Also, I'm interested to find out how NodeViewer is making the association for what type of class a node is. For example, the definition for the first list (mapDataContainerMapDataList) is as follows:

CF 40 00 00 05 B5 F9 C7 72 07 09 90 90

To my understanding this breaks down as follows:

CF 40 00 00 05 B5 F9 C7 72 - Reference to client.gom entry "mapDataContainerMapDataList"
07 - Data type 0x07 is "List".
09 - The list is a list of classes (because type 0x09 is Class)
90 90 - Number of entries (144) in the list, repeated once.

Note that I have no idea how NodeViewer figures out this is "list of Class mapData". Is it just looking inside the list and seeing what types of classes are there, and using that to generate the type column?

Then the definition for the first contained class follows:

01 1B 14

I understand that as:
01 - Label for this class, apparently just the integer 1.
09 - Not specified, but implicit because the list specifies the data type.
1B - Unknown
14 - Number of entries (20) in this class.

I'm assuming that 1B is how NodeViewer identifies this class as "Class mapData", but I'm not sure HOW that happens.


So, to recap:
1) When a class member has a label which is not directly a reference a client.gom entry, how do I look up the proper label?
2) How do I determine what type of class, a given class instance is?

[SWTOR fan]

For example, if you consider node "world.areas.4611686018427650020.mapdata". The first node/section inside here has a label of 4611686042955270002, which can be looked up in an old version of client.gom to reveal the proper label of "mapDataContainerMapDataList". The label of the first sub-node in this list is simply the integer 1, which shows identically in NodeViewer. This is a class which contains data about a map page. The first entry in this class is clearly labeled with a value that, once looked up in client.gom, translates to "mapName". However, the following two values (of the Vector3 type) are labeled with 0x04 and 0x01, respectively, where they should be (according to NodeViewer) labeled with entries for "mapPageMinCoord" and "mapPageMaxCoord".

It's much easier than you think. To save memory, the game only stores the full id of the first field. For all other fields, it just stores the difference to the previous field. So just create a variable that stores the id of the previous field, and add the number of the current field to get the full id.

In your example:
4611686020073980011 -> 4611686020073980011 = mapName
0x04 -> 4611686020073980011 + 4 = 4611686020073980015 = mapPageMinCoord
0x01 -> 4611686020073980015 + 1 = 4611686020073980016 = mapPageMaxCoord


From my understanding, the nodes from the bucket files just contain information of the form "List of Class". The "List of Class mapData" you're talking about is contained in the client.gom file I believe, but I have never looked too much into that.
There's not much of a point reading the client.gom since the names and descriptions have been removed.

[jpreston84]

Ah yes, I actually figured this out a bit earlier today. Sorry I didn't reply to alert you to that.

I'm still struggling a bit through the top of each section in the bucket files (the data before the label of the first entry in the section), but I am hopeful that I'll figure it out.

[jpreston84]

Greetings all!

So, I've been working on figuring out a lot of the GOM structure, but have been trying in vain to find certain nodes for several hours.

Consider, for example, a node like npc.location.tatooine.vendor.vendor_army_heavy_h1_g1_emp_poi00_mos_ila . In it, it contains a list of strings, for what I presume are sets of items which it has available for sale. These are as follows:

pkg.vendor.armor.heavy.level_24_25
pkg.vendor.armor.heavy.level_26_27
pkg.vendor.armor.heavy.level_28_29

Unfortunately, these nodes do not exist either within the prototypes or the buckets (as far as I can tell). Now, certain "packages" do exist. For example, pkg.profession_trainer.armstech_base contains a list of what appears to be all of the armstech schematics the vendors usually sell.

I've tried scanning through the GOM for many different things, with no success. Does anyone have an idea where the pkg.vendor.* nodes might have moved to?

[SWTOR fan]

Certain nodes are only contained in the server GOM, not in the client files, and they are requested by the client on a need-to-know basis. The vendor packages are an example of this; you won't find them in the client files.

[luke9511]

im working on updated my outdated files for swtor model extraction and conversion and when i go to open easymyp it just crashes is there a new version or is the latest hash list just to big for it to handle? im on win 7 x64

[swtor miner]

My latest Hashlist.

I'm showing the following stats including the main_gfx_1.tor, but not the french/german localizations.
451,706 named
458,471 total
6,765 unnamed
98.5% complete

A while back I disassembled the latest EasyMYP and made additions and recompiled it. You can find a version with my updates here.

It's got a few nagging bugs, like the about window and potentially a few others in functions I didn't often use. So, I didn't feel a pressing need to fix them. Here's my disassembled source. If you want to play with it.

When you do fix it, you can merge the code with the the original project. Or start a new repo for it somewhere. I've included a default .gitignore file if you want to get a git repo going with it.

Right now I'm working on the new GS Expansion data. With it, I'm helping to create a ship calculator for a relatively well known community site. If successful it might just be the first of many projects I'm working on to see the light of day.

[TRDaz]

Thanks for that update swtor miner! I can finally extract some files
One problem though, some of the models/textures dont extract, the main example i tried was Darth Malgus, i press extract, no files appear. But Satele's model works fine.. any idea why his model/textures wont extract? I havent tried anymore models yet.

EDIT: After extracting the whole .tor file it seems that Malgus' model is the only one that doesnt extract :/

[swtor miner]

Thanks for that update swtor miner! I can finally extract some files
One problem though, some of the models/textures dont extract, the main example i tried was Darth Malgus, i press extract, no files appear. But Satele's model works fine.. any idea why his model/textures wont extract? I havent tried anymore models yet.

EDIT: After extracting the whole .tor file it seems that Malgus' model is the only one that doesnt extract :/

I'll have to look into that when I get some time.

Here's an updated version of Nodeviewer for you guys to play with in the mean time.

And a link to my latest post where I discuss how to datamine some of the Galactic Starfighter information

[TRDaz]

Thank you, take your time

[swtor miner]

I didn't have any trouble extracting the textures for malgus's model.

They were in swtor_main_art_creature_a_1.tor - "\resources\art\dynamic\npc\unique_characters\texture\___psd\"