XeNTaX

Hi -

Just started using the EASYMPY to extract the files from TOR.

But my question is how did you get the filenames and correct directories to show?

All I have are some folders for csv, txt, xml, dds and all the filenames are some weird numbers / letters.

Wow, it's great to see how much response the topic got, thank you to everyone who wants to help out with the analyses!

badmp3 wrote:But my question is how did you get the filenames and correct directories to show?

The problem with the TOR archives is that the filenames themselves are not stored in the archives, but only their hashes. That's why you get these weird numbers as filenames, like DEADBEEF_9C022D76484096A0.txt.
However, some xml files contain file paths and we can guess some filenames manually, but unfortunately not all.
On the easyMYP Google Code page there is a TOR-found.rar file that has about 17% of all filenames. But even better is this file which has about 50% of all filenames.
Now to load these filenames into easyMYP, you have to first download and extract the archives. Then go to the folder where you have easyMYP saved and create a subfolder called "Hash". In this folder you can copy the file "hashes_download.txt" and then run easyMYP. It will automatically recognize the filenames if they are known, and extract the remaining ones as DEADBEEF_9C022D76484096A0.txt.
If you want to, you can of course try to find more filenames and publish them. I am confident we will soon have a list of all filenames, but with every new beta version new files are added or old ones removed so we always have to update the list.

DKK wrote:There seems to be a good amount of data stuffed into the bucket files.
I'm relatively new to this so any help extracting would be appreciated.

@DKK: I already started analysing the bucket files as well and will post what I have got so far in a separate post.

When I place that Hash file the 50% one into the Hash folder and start up the program it crashes!

says

EasyMYP has stopped working

im on Vista64 - not sure if thats the issue ... if i remove that file from the hash folder the program will start up again..


This is how the other generated ones show up as

Bucket file specification (.bck file/PBUK DBLB file)
The bucket files are an archive that contains small files with zlib encoding.
They can be found in assets_system_1.tor and are extracted into the folder /resources/systemgenerated/buckets/. The file extension is .bck, the header starts with PBUK.
The file names are sequentially numbered from 0.bck onwards, a list of all bucket files can be found in /resources/systemgenerated/buckets.info.
There are quite a few integers that I am not sure about what they mean, so any help is appreciated.

File header:
0-3: 4 byte identifier: PBUK (50 42 55 4B)
4-5: Unknown 16-bit little-endian integer (maybe number of DBLB sections?), always 02 00
6-7: Unknown 16-bit little-endian integer, always 04 00

DBLB block:
LOOP (for each DBLB block) {
--- 32-bit little-endian integer specifying the size of this block starting with DBLB, eg. 0C 00 00 00 or 64 A0 03 00
--- 4 byte identifier: DBLB (44 42 4C 42)
--- Unknown 32-bit little-endian integer, always 01 00 00 00
--- 32-bit little-endian integer specifying number of files in this block, eg. B6 00 00 00
--- LOOP (for each file in this block) {
--- --- Unknown 16-bit little-endian integer, always 0F 00
--- --- Unknown 16-bit little-endian integer, eg. 52 00 or 66 00 or 71 00
--- --- 6 unknown bytes (maybe CRC checksum?), eg. 01 FC 70 C1 12 01 or 20 32 63 27 01 or 80 0D 21 92 E2 0A
--- --- 4 bytes, always 00 E0 2E 00
--- --- Unknown 16-bit little-endian integer, always lower by one than the previous integer, eg. 51 00 or 65 00 or 70 00
--- --- Unknown 32-bit little-endian integer, eg. 17 00 00 00
--- --- 4 unknown bytes (maybe CRC checksum?), eg. 50 14 1E E2 or 0E CD 1A 01
--- --- 6 unknown bytes, always 00 00 00 40 00 00
--- --- Unknown 16-bit little-endian integer, eg. 58 00 or 68 00 or 78 00
--- --- Unknown 32-bit little-endian integer, eg. 61 00 00 00 or 63 02 00 00 or 74 01 00 00
--- --- Unknown 16-bit little-endian integer equal to the one two lines above, eg. 58 00 or 68 00 or 78 00
--- --- Unknown 16-bit little-endian integer, always 04 00
--- --- Two unknown bytes, always 01 03
--- --- String with a variable length until 00 00, specifying FQN of this file, eg. ipp.class.spy.a05.c02.s02.legs_v01
--- --- Separator/end of string, always 00 00
--- --- Beginning of a file, it is unknown where its length is stored. In most cases, the file starts with xœ (78 9C) and can be decoded with zlib.
--- --- 3 unknown bytes, identifying end of section, either 00 00 00 or 01 00 00
--- } END LOOP
} END LOOP

badmp3 wrote:When I place that Hash file the 50% one into the Hash folder and start up the program it crashes!

The error may occur because those filename lists are based on earlier versions where the TOR files had the prefix assets_ and not green_. That's also why the filenames are not recognized in the 17% file.
I do not yet have the current version so unfortunately I cannot help you with it. You could try looking in the readme and playing with the "Test filename list" settings.

So your saying if I was to download EasyMYP and download that 50% hash file, insert it into the hash folder and start it up ... to which it crashes .. is caused due to different file names that I havent scanned yet ..

Isnt that a tad weird?

Btw - Since you have the older verison did you happen to get any valueable data to which you may wanna share?

Alot of the big brand gaming db sites have tons of SWTOR information already and I can only assume its either from these client files or SWTOR gave them a database dump of sorts...

Did you find data about quests, skill tress, npcs, companions, icon files? Any data you can share?

Only things so far that I noted was as Ive been looking over the many .xml files I saw they had the full companion dialogs ..

Sorry, my mistake. The 50% file is actually based on a newer of version of easyMYP, no wonder why it crashes.
You can download the new, special TOR build of easyMYP here.
However, I cannot tell you if this tool will recognize the filenames from the newest beta version.

By the way, the 17% file can only be opened in the older version of easyMYP, but all of the files in there are also contained in the 50% file.

Edit:
As far as I know, the big gaming sites get these information from playing the beta versions. After all, the beta tests have been going one for a few months, and only a few days ago the NDA has been lifted so every one can talk about them. I doubt that someone has made any further progress in analysing the files than we currently have.
And yes, I have some exclusive story information but I want to follow Bioware's nonbinding request to not spoil any storylines until the game's release. And I don't want to convert this topic into "SWTOR spoiler information" since this would be off-topic. There are lots of gaming forums that have more than enough exclusive information. I recommend Wookieepedia or the SWTOR wiki as they are always on the frontline when it comes to spoilers, although I do not know if they will honor Bioware's request.
So sorry, I cannot help you with this, you have to look somewhere else. There is very much story in the game; I have only seen a small part of the files and even that took a long time to read.

SWTOR fan wrote:Sorry, my error. The 50% file is actually based on a newer of version of easyMYP, no wonder why it crashes.
You can download a new, special TOR build of easyMYP here.
However, I cannot tell you if this tool will recognize the filenames from the newest beta version.

By the way, the 17% file can only be opened in the older version of easyMYP, but all of the files in there are also contained in the 50% file.

That verison worked, the HASH file was taken in no problems

I have a really old SWTOR Beta client that has 1 big file called smallassests.p2v (7GB)

Would ya know a way to take a peek at whats inside this?

I tried rename the file to TOR and use EASYMYP but it will crash out ... it starts estimating files gets to like 460k then the program dies..

badmp3 wrote:I have a really old SWTOR Beta client that has 1 big file called smallassests.p2v (7GB)

Very interesting, I have not yet heard of that file. In which folder is this file stored? Is it the Assets/ folder?
If you are ever unsure how to open a file, you can use a hex editor to see its contents. I recommend HxD as it is a freeware standalone/portable hex editor.
You can open the file in there (fortunately, HxD is very fast and can easily open files this large). Just copy the first few lines (maybe 3 MB), save them as a new file and upload the small file.
This should be enough for us to analyse the basic format of the file but you may need to upload other parts later as well.

That file is from a veryyy early beta... its in the main folder.

heres the first lines

SWTOR fan wrote:Bucket file specification (.bck file/PBUK DBLB file)
The bucket files are an archive that contains small files with zlib encoding.
They can be found in assets_system_1.tor and are extracted into the folder /resources/systemgenerated/buckets/. The file extension is .bck, the header starts with PBUK.
The file names are sequentially numbered from 0.bck onwards, a list of all bucket files can be found in /resources/systemgenerated/buckets.info.
There are quite a few integers that I am not sure about what they mean, so any help is appreciated.

File header:
0-3: 4 byte identifier: PBUK (50 42 55 4B)
4-5: Unknown 16-bit little-endian integer (maybe number of DBLB sections?), always 02 00
6-7: Unknown 16-bit little-endian integer, always 04 00

DBLB block:
LOOP (for each DBLB block) {
--- 32-bit little-endian integer specifying the size of this block starting with DBLB, eg. 0C 00 00 00 or 64 A0 03 00
--- 4 byte identifier: DBLB (44 42 4C 42)
--- Unknown 32-bit little-endian integer, always 01 00 00 00
--- 32-bit little-endian integer specifying number of files in this block, eg. B6 00 00 00
--- LOOP (for each file in this block) {
--- --- Unknown 16-bit little-endian integer, always 0F 00
--- --- Unknown 16-bit little-endian integer, eg. 52 00 or 66 00 or 71 00
--- --- 6 unknown bytes (maybe CRC checksum?), eg. 01 FC 70 C1 12 01 or 20 32 63 27 01 or 80 0D 21 92 E2 0A
--- --- 4 bytes, always 00 E0 2E 00
--- --- Unknown 16-bit little-endian integer, always lower by one than the previous integer, eg. 51 00 or 65 00 or 70 00
--- --- Unknown 32-bit little-endian integer, eg. 17 00 00 00
--- --- 4 unknown bytes (maybe CRC checksum?), eg. 50 14 1E E2 or 0E CD 1A 01
--- --- 6 unknown bytes, always 00 00 00 40 00 00
--- --- Unknown 16-bit little-endian integer, eg. 58 00 or 68 00 or 78 00
--- --- Unknown 32-bit little-endian integer, eg. 61 00 00 00 or 63 02 00 00 or 74 01 00 00
--- --- Unknown 16-bit little-endian integer equal to the one two lines above, eg. 58 00 or 68 00 or 78 00
--- --- Unknown 16-bit little-endian integer, always 04 00
--- --- Two unknown bytes, always 01 03
--- --- String with a variable length until 00 00, specifying FQN of this file, eg. ipp.class.spy.a05.c02.s02.legs_v01
--- --- Separator/end of string, always 00 00
--- --- Beginning of a file, it is unknown where its length is stored. In most cases, the file starts with xœ (78 9C) and can be decoded with zlib.
--- --- 3 unknown bytes, identifying end of section, either 00 00 00 or 01 00 00
--- } END LOOP
} END LOOP

I'm wondering if Bytes 4-5 and 6-7 are some kind of version tag as my bkt files (as linked in my previous post) are all 02 00 05 00 as opposed to your 02 00 04 00
The same applies to the 32-bit integer following the first DBLB where yours read 01 00 00 00 mine read 02 00 00 00

For whatever reason these feel like compressed XML files to me

I found additional changes between these versions of the BKT files. Here's what I've come up with:

Bucket file specification (.bkt file/PBUK DBLB file)
The bucket files are an archive that contains small files with zlib encoding.
They can be found in assets_system_1.tor and are extracted into the folder /resources/systemgenerated/buckets/. The file extension is .bkt, the header starts with PBUK.
The file names are sequentially numbered from 0.bkt onwards, a list of all bucket files can be found in /resources/systemgenerated/buckets/

File header:
0-3: 4 Byte ID: PBUK (50 42 55 4B)
4-5: Unknown 16-bit little-endian integer (# of DBLB sections?), always 02 00
6-7: Unknown 16-bit little-endian integer (version?), always 05 00

DBLB block:
LOOP (for each DBLB block) {
--- 32-bit little-endian integer specifying the size of this block starting with DBLB, eg. 0C 00 00 00 or 64 A0 03 00
--- 4 byte identifier: DBLB (44 42 4C 42)
--- Unknown 32-bit little-endian integer, always 02 00 00 00
--- 32-bit little-endian integer specifying number of files in this block, eg. B6 00 00 00

--- LOOP (for each file in this block) {
--- --- 32-bit Null 00 00 00 00
--- --- 6 unknown bytes (maybe CRC checksum?), eg. 01 FC 70 C1 12 01 or 8D 20 32 63 27 01 or 80 0D 21 92 E2 0A
--- --- 4 bytes, always 00 E0 0F 00
--- --- Unknown 16-bit little-endian integer, always lower by one than the previous integer, eg. 51 00 or 65 00 or 70 00
--- --- Unknown 32-bit little-endian integer, eg. 17 00 00 00
--- --- 4 unknown bytes (maybe CRC checksum?), eg. 50 14 1E E2 or 0E CD 1A 01
--- --- 2 unknown bytes: typically 00 00 but values of 03 00, 04 00, and 05 00 have also been spotted
--- --- 8 unknown bytes, always 00 40 00 00 00 00 00 00
--- --- Unknown 16-bit little-endian integer, eg. 58 00 or 68 00 or 78 00
--- --- Unknown 32-bit little-endian integer, eg. 61 00 00 00 or 63 02 00 00 or 74 01 00 00
--- --- Unknown 16-bit little-endian integer equal to the one two lines above, eg. 58 00 or 68 00 or 78 00
--- --- Unknown 16-bit little-endian integer, always 05 00
--- --- Two unknown bytes, always 01 03
--- --- String with a variable length until 00 00 (I saw at least one file while scanning through one of the buckets, that seemed to end with 00 01 -- was very much a one-off thing and I couldn't find it replicated elsewhere), specifying FQN of this file, eg. ipp.class.spy.a05.c02.s02.legs_v01
--- --- Separator/end of string, always 00 00
--- --- Beginning of a file, it is unknown where its length is stored. In most cases, the file starts with xœ (78 9C) and can be decoded with zlib.
--- --- 3 unknown bytes, identifying end of section, either 00 00 00 or 01 00 00
--- } END LOOP
} END LOOP