Star Wars - The Old Republic - Beta

[hazballs]

Thanks for the replies!

I edited my original message after posting it, SWTOR fan, but it seems you were already in the process of replying - it seems the 50% hash list DOES work on the current beta files. Whether or not all the files in the hash list are still accurate I don't know, but I managed to get a good few .gr2 files (which is the main thing I'm currently looking for, as I'd like to see some of the end game boss models).

However, as you have probably noticed in your own findings, in the "release" version of the client (Bioware has stated that people in the big weekend beta won't have to re-download the client for the live game) the gr2 files now have the GAWB "magic number" instead of )ÞlÀ and cannot be converted to .ms/.smd files using the GrannyReader'98 tool posted in the other thread.

But yes, as you say, a general viewer would be ideal, similar to the general file/model viewers released for other games.

[SWTOR fan]

ACB file specification (WEM file archive)
I found a new file type in the newest version of the beta files that is only contained in the localized archives. It has the file extension "*.acb", like "/resources/en-us/bnk2/cnv_location_alderaan_bronze_imperial_adjusting_history_caliman_thul.acb".
These archives seem to be an updated version of the BKHD/.bnk archives and do not have a file header. According to the strings in these archives, the audio files contained in them have the file extension .wem, but they actually are just normal WAVE files (with Wwise encoding, of course).

Metadata in .tor archives:
The WEM file archives do not have a file header but contain a 36-byte long metadata in the TOR archives:
two bytes (always 02 00): maybe version number / ID
two bytes (always 20 00): length of following metadata → 32 bytes
unknown 32 bytes, different for each file, maybe some CRC checksum or version information

WEM archive file format:
64-bit integer (8 bytes): number of file entries (eg. 06 00 00 00 00 00 00 00 → 6 files)
LOOP (for each filename) {
--- variable-length string (= filename), terminated by 00
--- 64-bit integer (8 bytes): file length (eg. 9D D7 00 00 00 00 00 00)
--- 64-bit integer (8 bytes): file offset (eg. 78 3E 00 00 00 00 00 00)
} END LOOP
LOOP (for each file) {
--- data[]: Wwise encoded .ogg file, with length and offset as given above (header: RIFF....WAVEfmt)
} END LOOP

Example filenames: (from swtor_XX-XX_area_corellia_1.tor)
cnv_location_corellia_ambient_hub1_green1_imperial_1_m.wem
cnv_location_corellia_ambient_cc_poi_barks_226_m.wem
cnv_location_corellia_ambient_cc_poi_barks_1149_m.wem

Edit: Just to clarify, BKHD/.bnk archives are still used inside swtor_main_bnk_[...].tor, but the difference to the WEM archives is that WEM archives contain multiple files with different filenames for each of them, while BKHD archives only contain one filename with different audio files inside. And the BKHD also contain some additional sections on top of the audio files.

Edit 2: Here's a BMS script for extracting these kinds of archives:

Code: Select all

get numberOfFiles longlong
for i = 0 < numberOfFiles
get fileName string
get fileSize longlong
get fileOffset longlong
log fileName fileOffset fileSize
next i

Edit 3: Thanks to Rick, I found out that these file archives have the extension "*.acb"!

[SWTOR fan]

I looked into the release notes for Wwise and found out that Wwise has changed the file extension of the audio files, that's why they are now called .wem:

2.1 All Streamed and Converted Files Have a New File Extension (WEM)
All streamed and converted files now have the .wem file extension (Wwise Encoded Media). Wwise does not generate anymore files with the ogg, xma or wav extensions. All streamed or converted files now have the .wem extension, whatever their encoding format is. The External Source files are also all converted to .wem files.
You may have to modify your tools or installers to manipulate wem files instead of ogg, xma or wav files.

(source: http://www.audiokinetic.com/download/do ... eNotes.pdf)

In any case, the audio files no longer work with the current version of the ww2ogg tool. For more information, see this topic.

Edit: hcs has now updated ww2ogg and it is working again!

[SWTOR fan]

Unknown string list file specification
There's another new file format contained in localized assets like swtor_de-de_global_1.tor, with the hashes 51378F9D_96EEED14D7CD0574.txt or CB7EAB01_96EEED14D7CD0574.txt, depending on the version of the beta files.
This file contains a list of strings for the abilities.
Thank you very much to Nomis who first discovered and analysed this file!

Each ability is described by four strings:
1. Name of the ability (eg. Bereich verlassen)
2. Repetition of the name (eg. Bereich verlassen)
3. Description of the ability (eg. Teleportiert dich aus dem aktuellen Flashpoint heraus.)
4. Repetition of the ability (eg. Teleportiert dich aus dem aktuellen Flashpoint heraus.)
Some strings have zero length and can therefore only be found in the metadata and not in the main body.

File format:
01 00 00 → header, maybe version number?
4 bytes: 32-bit integer, number of strings
LOOP (metadata foreach string) {
--- 4 bytes: 32-bit integer (0 = name of an ability, 1 = description of an ability)
--- unknown 4 bytes, maybe ID
--- unknown two bytes (41 01 if it is the first string, 46 01 if it is the repetition)
--- unknown 4 bytes, maybe ID
--- 4 bytes: string length
--- 4 bytes: string offset (absolute file position)
--- 4 bytes: string length
} END LOOP
LOOP (raw text foreach string) {
--- List of strings with the aforementioned offset and length
} END LOOP

As an example, here is the metadata for one ability, consisting of four strings:
00 00 00 00 C2 39 02 00 41 01 A3 01 40 40 11 00 00 00 53 99 0E 00 11 00 00 00 Bereich verlassen
00 00 00 00 C2 39 02 00 46 01 A3 01 40 40 11 00 00 00 64 99 0E 00 11 00 00 00 Bereich verlassen
01 00 00 00 C2 39 02 00 41 01 A3 01 00 40 36 00 00 00 75 99 0E 00 36 00 00 00 Teleportiert dich aus dem aktuellen Flashpoint heraus.
01 00 00 00 C2 39 02 00 46 01 A3 01 00 40 36 00 00 00 AB 99 0E 00 36 00 00 00 Teleportiert dich aus dem aktuellen Flashpoint heraus.

[stalja]

About the unknown string list file specification.

metadata for each string section
the first 4 bytes are the string id
next 4 bytes are the id of the xml they belong to (to put it plainly)

In your example of abilities - the first 4 bytes present id of first and second string in the XML that describes abilities. The second 4 bytes describe exactly that abilities xml that contains those strings.

the next 2 bytes are what you said - describing whether it is a repetition or not.
last 4 unknown bytes are a mystery to me. They are not unique so I guess they are some sort of flag that describe how to build the unique id of the string.

[Duhaccgzhn]

And scratch what I said earlier about the version files; the newest version files are password-protected ZIP files and cannot be read so easily.

I am new at this forum, but i have an idea on how to extract that zip files.
I have found that all .version, .bundle, .patchmanifest, .solidpkg and itself launcher.exe (except that this file is an exe) are actually same-type ecrypted zip-files with start as emty dll-file with certificate.
Laucher starts as a webserver emulator, which binds on unique port at localhost each time is launched. Then exe creates instance of webkit browser. But it is not important at this time. After find opened port i`ve searched for, i`ll navitage to http://127.0.0.1:8175/app/swtor.html (for example) in my browser.
.version files actually an hash-file on .tor files which launcher uses for patching and checking for errors.
I have started at disassemble laucher and see wheres password is located. But i am new at reverse-engineering, so i am dont have found it yet.
I am ask people who actualy good at reverse-engineering for finding info about password. So we can move forward.

[stalja]

I don't see a reason why you would want to unpack those files. That was implemented to prevent people from downloading the client if they do not have access to beta. Now that the client is readily available for download for everyone I see no reason to get into those files at all. If you want to reverse engineer the executable rather do it to find the piece of code that unpacks bucket files...

[Duhaccgzhn]

I know this. But i dont have broadband internet. I am doing my research to ensure that prevent laucher from downloading those big over 5 gb files again.. Sometimes launcher did not success patching game files and after rechecking goes to redownload. Well i am do not agree with their approach for patching. I want to myself patch those files. Well human are more smart than any limited algorithm programs and can expect all incorrect actions.

[Darjk]

I've been keeping a close eye on this thread as its saving me a lot of the ground work which I'm extremely appreciative of.
I'm currently in the process of writing an app to open and view Tor files on the fly specifically for SWTOR to assist in figuring out all the file formats, etc (very similar to my Shadowbane Cache viewer that I wrote). Then eventually down the track, I will be converting.. or adding the 'Model Viewer' part of the program to go along with my original WoW Model Viewer.

Once I have the base work done of opening, viewing, saving, and editing the files as well as a preview of each file, I'll release it for general use. Hopefully won't take too long but hard to determine as I work a lot of hours.

Thanks to all of you who have contributed!

[mvanderw]

Indeed. As part of my effort to grok the files contained in the buckets I've made a quick bucket explorer, available at http://holocron.so/buckets.

It seems to me the buckets are fairly interesting as they seem to contain most (all?) of the game's items and abilities as well as lots of other curious bits (templates, area info, etc). Anyone gotten further in decoding those files?

[stalja]

Indeed. As part of my effort to grok the files contained in the buckets I've made a quick bucket explorer, available at http://holocron.so/buckets.

It seems to me the buckets are fairly interesting as they seem to contain most (all?) of the game's items and abilities as well as lots of other curious bits (templates, area info, etc). Anyone gotten further in decoding those files?

Looking at the number of databases out there I'm fairly certain they have

[Rick]

If it has not been noted before, name hash algorithm is jenkins lookup3 (see hashlittle/hashlittle2).

[SWTOR fan]

Bucket file format
As I wrote earlier, I already have analysed files which are similar to the files in the bucket files, so I will just post what I have. I hope the format did not change too much since I analysed them.

Thank you very much for your bucket explorer, mvanderw! It is helping me a lot with the analysis. I hope you can improve the coloring with my information and we can improve the analysis of these files.

I recommend looking at the integer format specification for variable-sized integers (see page 1 of this topic). The compiled files make frequent use of them.

The general format of these files is like this:
0-...: varying number of header bytes (0 to at least 7 bytes)
One variable-sized integer, eg. 15, 2F or C9 01 59 (15 = mapnote, 59 = item) ??
Another variable-sized integer, eg. 06, 05 or C9 00 14
LOOP (for all nodes) {
--- ID of the current node as variable-sized integer (most files, including PROT) or as 4-byte integer (compiled MPN files)
--- Node type, one or two bytes.
--- More data, different for each node type
} END LOOP

Eg., one node type is the string type and is identified by the byte 06 (most files, including PROT) or 85 (compiled MPN files).
Example analysis for one string node:
CF 40 00 00 02 F5 E6 DC 5A (Ï@...õæÜZ) → node id, in this case: 4611686031142870000
06 (.) → node type #6 = string
25 (%) → length of string (37 bytes)
65 70 70 2E 73 74 61 74 65 2E 73 77 69 74 63 68 5F 77 65 61 70 6F 6E 5F 73 74 61 6E 63 65 2E 70 69 73 74 6F 6C
→ the actual string, in this case epp.state.switch_weapon_stance.pistol

In the following, you can find a list of the different node types. Some nodes reference other node ids, other nodes contain one string or a list of string, and even other nodes contain some kind of flags (e.g. 01=05, 02=DE, 03=AB, 04=01, ...). And of course there are many more node types where I do not know what they stand for.
But at least with this information one can write a better format analyser.
Once we have that done, we can try to make some sense of the data. It seems that the node id has something to do with what the node stands for. For example, in dialogue files, the node id classifies a node to say which audio should be spoken, what the parent node is, which animations/cameras to play etc.


01 (reference to a different node id)
Variable-length integer: Node id
One byte: 01
Variable-length integer: Another node id

02 (reference to a different node id)
Variable-length integer: Node id
One byte: 02
Variable-length integer: Another node id

In dialogue files, if the first node id is 0x2C, then the second id stands for the string id that is to be spoken.

03 (short node, end of section?)
Variable-length integer: Node id
One byte: 03
One byte: 01

04 02 (short node)
Variable-length integer: Node id
One byte: 04
One byte: 02

04 03 (short node)
Variable-length integer: Node id
One byte: 04
One byte: 03

04 02 02 (unknown)
Variable-length integer: Node id
Three bytes: 04 02 02
Four bytes: unknown
Three bytes: unknown

05 (short node)
Variable-length integer: Node id
One byte: 05
Two unknown bytes

06 (single string)
Variable-length integer: Node id
One byte: 06
Variable-length integer: Length of string
String with the aforementioned length

07 01 (multiple flags)
Variable-length integer: Node id
Two bytes: 07 01
Variable-length integer: Number of flags
Variable-length integer: Number of flags, same as above
LOOP {
--- Variable-length integer: Flag id
--- Variable-length integer: Flag value
} END LOOP

07 03 (short node)
Variable-length integer: Node id
Two bytes: 07 03

07 05 (multiple flags)
Variable-length integer: Node id
Two bytes: 07 05
Variable-length integer: Number of flags
Variable-length integer: Number of flags, same as above
LOOP {
--- Variable-length integer: Flag id
--- Variable-length integer: Flag value
} END LOOP

07 06 (multiple strings)
Variable-length integer: Node id
Two bytes: 07 06
Variable-length integer: Number of strings
Variable-length integer: Number of strings, same as above
LOOP {
--- Variable-length integer: String id
--- Variable-length integer: String length
--- String with aforementioned length
} END LOOP

07 09 (unknown)
Variable-length integer: Node id
Two bytes: 07 05
Two or five unknown bytes

08 01 09 (unknown length)
Variable-length integer: Node id
Three bytes: 08 01 09
Variable-length integer: Unknown length
Variable-length integer: Same as above

08 02 02 (multiple flags)
Variable-length integer: Node id
Three bytes: 08 02 02
Variable-length integer: Number of flags
Variable-length integer: Number of flags, same as above
LOOP {
--- Variable-length integer: Flag id
--- Variable-length integer: Flag value
} END LOOP

08 02 06 (multiple strings)
Variable-length integer: Node id
Three bytes: 08 02 06
Variable-length integer: Number of strings
Variable-length integer: Number of strings, same as above
LOOP {
--- Variable-length integer: String id
--- Variable-length integer: String length
--- String with aforementioned length
} END LOOP

08 02 07 (beginning of multiple string nodes)
Variable-length integer: Node id
Three bytes: 08 02 07
Variable-length integer: Number of string nodes following this node
Variable-length integer: Same as above

08 02 08 (beginning of multiple string nodes)
Variable-length integer: Node id
Three bytes: 08 02 08
Variable-length integer: Number of string nodes following this node
Variable-length integer: Same as above

08 02 09 (unknown length)
Variable-length integer: Node id
Three bytes: 08 02 09
Variable-length integer: Unknown length
Variable-length integer: Unknown length, same as above

08 05 08 (multiple string sections)
Variable-length integer: Node id
Three bytes: 08 05 08
Variable-length integer: Number of string sections
Variable-length integer: Number of string sections, same as above
LOOP (for each section) {
--- Variable-length integer: Section number
--- Two bytes: XX XX
--- Variable-length integer: Number of strings per section
--- Variable-length integer: Number of strings per section, same as above
--- LOOP (for each string in this section) {
--- --- Variable-length integer: String id
--- --- Variable-length integer: String length
--- --- String with aforementioned length, sometimes length = 0
--- } END LOOP
} END LOOP

08 05 (short node)
Variable-length integer: Node id
Two bytes: 08 05

08 06 03 (multiple strings)
Variable-length integer: Node id
Three bytes: 08 06 03
Variable-length integer: Number of strings
Variable-length integer: Number of strings, same as above
LOOP (for each string) {
--- Variable-length integer: String id
--- Variable-length integer: String length
--- String with aforementioned length, sometimes length = 0
--- Sometimes: one zero byte (00)
} END LOOP

08 06 08 (multiple strings)
Variable-length integer: Node id
Three bytes: 08 06 08
Variable-length integer: Number of strings
Variable-length integer: Number of strings, same as above
LOOP (for each string) {
--- Variable-length integer: String id
--- Variable-length integer: String length
--- String with aforementioned length, sometimes length = 0
} END LOOP
Four bytes: 06 03 01 01
Variable-length integer: String id
Variable-length integer: String length
String with aforementioned length, sometimes length = 0
One byte: 01

09, 39 or 3B (beginning of a section)
Variable-length integer: section number
One byte: 09, 39 or 3B (on dialogue files: 39=audio string, 09=metadata, 3B=unknown)
One unknown byte (on dialogue files: 05=metadata, 15=NPC speaking, 16=PC speaking, 16/17=unknown)

-----------------------------------------

Following 08 02 07 or 08 02 08 are not regular string sections; instead, they have the following format:

Following 08 02 07:
Variable-length integer: Node id
One byte: 06
Variable-length integer: Number of strings
Variable-length integer: Number of strings, same as above
LOOP (for each string) {
--- Variable-length integer: String id
--- Variable-length integer: String length
--- String with aforementioned length, sometimes length = 0
} END LOOP

Following 08 02 08:
Variable-length integer: Node id
One byte: 06
One byte: 07
Variable-length integer: Number of strings
Variable-length integer: Number of strings, same as above
LOOP (for each string) {
--- Variable-length integer: String id
--- Variable-length integer: String length
--- String with aforementioned length, sometimes length = 0
--- One unknown byte: 02 (maybe flag mode??)
--- Variable-length integer: Number of flags
--- Variable-length integer: Number of flags, same as above
--- LOOP (for each flag) {
--- --- Flag number
--- --- Flag value
--- } END LOOP
} END LOOP

[Rick]

Is anyone building a file list for the retailclient besides myself? (sitting at 35% complete)

[SWTOR fan]

Is anyone building a file list for the retailclient besides myself? (sitting at 35% complete)

Yes, I am also currently working on a tool to do just that, but right now I am working with the final beta version. I hope to finish the tool soon so that I can compare the beta files with the retail files.