Star Wars - The Old Republic - Beta

[CZW]

The contents of this post was deleted because of possible forum rules violation.

[chrrox]

where can you download the entire client from?
Also the compression is zlib.

Quickbms script.

Code: Select all

goto 0xC
get ftable long
goto 0x18
get files long
goto ftable
for i = 0 < files
get unk01 long
get unk02 long
get unk03 long
get offset long
get null01 long
get null02 long
get zsize long
get size long
get unk04 short
clog "" offset zsize size
next i

[CZW]

look on your PM

[CZW]

the script works.

but it seems we have different levels of data inside.

[chrrox]

Ill have to wait till the game comes out or is free to play i can not download the client.

[chrrox]

tor extractor
http://code.google.com/p/easymyp/downloads/list

[CZW]

good news ...

but your Quickbms script unpack into .DAT files
and your Hashcode list present all the tree of ressources with extension format


i hope test soon you TOR extractor but not present on your download list

[SWTOR fan]

Hello everyone! I have done some research on the file types and thought I might as well share it with you.
The wiki seems to be closed for creating pages or registering so I hope this is the right place for posting.

This file list could be used to improve the file recognition of easyMYP or to even build a complete SWTOR reader but is far from being finished.

To-do list (just to clarify: I do not plan to do all of this myself, so feel free to help out!):

• figuring out all file paths in the archives
• completing the list of all file extensions/file formats
• writing file specifications for the most important formats
• figuring out relationships between the files (eg. models/textures/animations belong together, so do dialogues/string tables/sound files) and possibly drawing a graph with the interdependencies
• writing a SWTOR tool that can handle all these files

EDIT: Note: This specification is outdated and only kept as a historical reference. The newest specification can be found on page 6 of this thread.

Archive files
These files are archives that contain other files.

File extension: .tor
File header: MYP
Description: Main archive for all files. In /Assets folder. Files start with either assets_, red_, or green_, depending on the type of beta version. Can be extracted with easyMYP.

File extension: .bkt
File header: PBUK........DBLB
Description: Bucket files, contain compiled XML files with zlib encoding. See below for a specification.

File extension: .bnk
File header: BKHD
Description: Soundbank that contains multiple .ogg sound files. Sections are named DIDX, DATA, HIRC, STID. Search for "bnkextr" tool in this topic.

File extension: .gom
File header: DBLB
Description: eg. client.gom. Contains files with zlib encoding.

File extension: .acb
File header: XX XX XX XX 00 00 00 00
Description: ACB soundbank that contains multiple .ogg sound files with the file extension .wem, similar to the BKHD archives. See page 4 of this thread for a specification.

Filelists
These files contain a list with references to other files.

File extension: .list
File header: SDEF or SIDS
Description: The first one references SCPT files, the other contains the files from assets_system_1.tor.

File extension: .info
File header: PINF or PBUK
Description: The first one references .node files, the other one .bkt files.

Compiled files
These files are binary files and can only be read in a HEX editor or designated reader.

File extension: .bik
File header: BIKi (42 49 4B 69)
Description: Regular Bink Video files, contained in the /Movies folder (ie. not part of the TOR archives). Can be opened with the RAD Video Tools.

File extension: .bin
Description: Very small file without a header. Seems to contain version information.

File extension: .dds
File header: DDS
Description: Normal DDS texture (Direct Draw Surface), can be opened in any DDS reader.

File extension: .elf
File header: .ELF (7F 45 4C 46)
Description: Normal ELF script (Executable and Linkable Format), contained in the SCPT files. Can be opened in any ELF reader, including IDA Pro.

File extension: .fxa
File header: FACE (46 41 43 45)
Description: Facial animations, matched to the localised voices in the sound files.

File extension: .gfx
File header: CFX or GFX
Description: See swtor_fonts.gfx.

File extension: .gr2
File header: )ÞlÀ (29 DE 6C C0)
Description: Granny Format by RAD Game Tools, contains the models. See this topic and that topic.

File extension: .jba
File header: 00 00 00 00
Description: Possibly contains animations. easyMYP automatically classifies them as zero.txt.

File extension: .mph
File header: 02 00 00 00
Description: Contains list of animations in a compiled format.

File extension: .node
File header: PROT.... (50 52 4F 54 02 00 04 00)
Description: Contains a compiled form of the XML dialogues and stringtables. See /resources/systemgenerated/prototypes/.

File extension: .ogg
File header: RIFF..6.WAVEfmt
Description: Ogg sound file with WWise encoding. Dialogue and sound effects are contained in .bnk/BKHD archives, music/soundtrack is stored in /resources/bnk/streamed/. See here for the decoding tool ww2ogg.

File extension: .swf
File header: CWS. (43 57 53 0A)
Description: Normal Adobe Flash file, apparently used for fonts.

File extension: n/a
File header: AMX (41 4D 58 20)
Description: Contains animations.

File extension: n/a
File header: ¥+..BNRY....LTLE
Description: One file is called mount_neu_model_e_scheme_04. The first two bytes can vary but BNRY and LTLE are always in there. This seems to be part of the GR2 model files and may just as well be an error of the MYP extractor.

File extension: (none)
File header: SCPT
Description: Contains header with some strings and a ELF script file (Hydra script?). See the folder /resources/systemgenerated/compilednative/. See below for a specification.

Human-readable files (txt/xml)
These files can be opened and edited in normal text or XML editors, although I would recommend using a hex editor for editing. The files should be pretty self-explanatory.
Note: Many XML files use a encoding like ÿþ<.S.u.r.v.e.y.I.n.s.t.a.n.c.e.>, ie. a Byte Order Mark for Unicode encodings. Also, some have an <!xml> tag before the first tag.

File extension: .dat
Description: Text file. Contains definitions of the rooms/maps. Comments are marked with an exclamation mark at the beginning of a line, eg. "! Room Specification". Some files contain textures(?), to do that copy the hex bytes into a hex editor and decode them using gzip.

File extension: .dyc
Description: Text file. Sometimes begins with "Version=2". See /resources/art/dynamic/spec/.

File extension: .fxspec
File header: <nodeWClasses>
Description: XML file.

File extension: .lod
File header: <LODSchemaGroup>
Description: XML file.

File extension: .mag
Description: Text file. Comments are marked with an exclamation mark at the beginning of a line, eg. "! Character Specification for ...".

File extension: .mat
File header: <Material> (EF BB BF )
Description: XML file.

File extension: .mpn
File header: <mapNoteTemplate>
Description: XML file. See also /engine/mapnoteinstance.mpn.

File extension: .not
File header: <v><k>mapNotes</k><e>
Description: XML file.

File extension: .rul
File header: <Rules>
Description: XML file.

File extension: .svy
File header: <SurveyInstance> (FF FE ÿþ)
Description: XML file. Survey/questionnaire for beta players about gameplay performance.

File extension: .tbl
File header: <DataTable>
Description: XML file. Contains a table with gameplay data, like amount of damage dealt.

File extension: .tex
File header: <TextureObject>
Description: XML file. Contains information on how to use the textures, eg. whether to stretch or clamp them.

File extension: .xml
File header: <aam> or <Assets> or <Palette>
Description: XML file.

There are a few more XML files, like <Ability>, <plcTemplate>, <spnTemplate>, <npcTemplate>, <Maps>, <Appearance>, <Item>, <HydraScriptMap>, <clientOnlySpawner>, <PlayerCustomizationSet>, <quest>, and <Sectors>, although I have not yet found out their file extensions.

Unknown engine files
I have seen these extensions mentioned in file paths but I have not yet found the respective files themselves.

File extension: .amk
Description: /engine/actionmark.amk.

File extension: .bil
Description: /engine/billboard.bil.

File extension: .bon
Description: /engine/bonetracker.bon.

File extension: .box
Description: /engine/genericbox.box.

File extension: .cam
Description: /engine/placeablecamera.cam.

File extension: .cdr
Description: /engine/cover/cover_dirty_region.cdr.

File extension: .cvr
Description: /engine/cover_point.cvr.

File extension: .ext
Description: /engine/projector.ext.

File extension: .fla
Description: /engine/flare.fla.

File extension: .fol
Description: /engine/follower.fol.

File extension: .fxp
Description: /engine/fxplaceable.fxp.

File extension: .hms
Description: /engine/heightmap.hms.

File extension: .lit
Description: /engine/light.lit.

File extension: .mir
Description: /engine/mirror.mir.

File extension: .osg
Description: In sectors XML: <CollisionFileName>data3D/sector_1011.osg</CollisionFileName>.

File extension: .p
Description: /engine/portal.p.

File extension: .phy
Description: /engine/physics.phy.

File extension: .phj
Description: /engine/physicsjoint.phj.

File extension: .rbd
Description: /engine/roombound.rbd.

File extension: .rgn
Description: /engine/region.rgn.

File extension: .sfq
Description: /engine/sf_gfx_quad.sfq.

File extension: .spn_activator
Description: /engine/spawnactivator.spn_activator.

File extension: .spn_c
Description: /engine/spawn_creature.spn_c.

File extension: .spn_crf
Description: /engine/spawn_crefac.spn_crf.

File extension: .spn_grp
Description: /engine/spawn_group.spn_grp.

File extension: .spn_lst
Description: /engine/spawn_list.spn_lst.

File extension: .spn_med
Description: /engine/spawn_medcenter.spn_med.

File extension: .spn_mov
Description: /engine/move_point.spn_mov.

File extension: .spn_p
Description: /engine/spawn_placeable.spn_p.

File extension: .spn_pt
Description: /engine/spawn_point.spn_pt.

File extension: .spn_rez
Description: /engine/spawn_rezpoint.spn_rez.

File extension: .spn_rly
Description: /engine/rally_point.spn_rly.

File extension: .spn_veh
Description: /engine/spawn_vehicle.spn_veh.

File extension: .stg
Description: /engine/stginstance.stg.

File extension: .trg
Description: /engine/trigger.trg.

File extension: .wtr
Description: /engine/basewater.wtr.

File extension: .wws
Description: /engine/wwiseaudio.wws.

File extension: .zzp
Description: /engine/zap.zzp.


Edit 1: Updated list with new information on some formats.
Edit 2: WEM archives added.
Edit 3: WEM archives updated (file extension: *.acb).

[TaylorMouse]

Where can we download the béta from ?

T.

[SWTOR fan]

Integer format specifications
In many compiled formats I have seen two types of integers being used, and I want to document them here so that I can refer to them in other file specifications.

I do not know much about compressed files, so these may be industry standards and have a special name. If you know more about them, please tell me so that I can correct the specification.

Fixed-sized integers
These integers are always four bytes long (or sometimes two or eight bytes).
They are stored in little-endian format and are often used to store the string length of the file name.

Example:
01 00 00 00 → Number 1
10 00 00 00 → Number 16
00 01 00 00 → Number 256

Variable-sized integers (C8/C9/CA/... format)
These integers take up different byte sizes depending on the size of the integer and use the big-endian format.
For sizes smaller than C0, the integer is written as one byte. Otherwise, a byte specifying the length is prepended before the integer:

Hex bytes and their meaning:
XX → Integers 00 to C0
C8 XX → Integer 00 to FF (1 byte/8 bit)
C9 XX XX → Integers up to 256^2 (2 bytes/16 bit)
CA XX XX XX → Integers up to 256^3 (3 bytes/24 bit)
CB XX XX XX XX → Integers up to 256^4 (4 bytes/32 bit)
CC XX XX XX XX XX → Integers up to 256^5 (5 bytes/40 bit)
CD XX XX XX XX XX XX → Integers up to 256^6 (6 bytes/48 bit)
CE XX XX XX XX XX XX XX → Integers up to 256^7 (7 bytes/56 bit)
CF XX XX XX XX XX XX XX XX → Integers up to 256^8 (8 bytes/64 bit)

Example:
C8 FF → Number 255
C9 01 00 → Number 256
C9 01 01 → Number 257
C9 01 02 → Number 258

So far, CF integers were the highest ones I have seen.
Sometimes, there is a C7 byte before an integer, and (like the CF byte) eight bytes follow, making it a total of nine bytes:
C7 XX XX XX XX XX XX XX XX → Integers up to 256^8 (8 bytes/64 bit)
I do not know what the difference between C7 and CF is.

Edit 1: Added description for C7 integers.

[stalja]

Things to note:

1. A lot of files have been removed from the latest versions of the client so it will be increasingly difficult to discover proper filenames. Namely - most datatables and human readable xml files have been removed from the client and they held most information about proper filenames. For example - <Item> xml files contained the names of the item icons and these are not in the client anymore (as far as I know - please correct me if I am wrong - but I can't find them in red_ nor green_).

2. PROT files are a bit strange. I was able to extract (without fully understanding what all the bytes in it mean) some stringtable PROT files (I managed to extract lists of strings from them), but there are other prot files that have a different structure, so my method is not universal enough. I am not very experienced in game archive extraction - are PROT files zlibed (they do not look like they are to me - feels like their propriety binary archive/storage format)

[SWTOR fan]

Just to let you know, I have already analysed most parts of the PROT files.
Those PROT files that are not stringtables are just a compiled version of the <Dialogue> xml files. They contain information on which dialogue options are possible and which answers follow after a sentence. Also, they contain the animations and camera movements during the dialogue.
Rest assured, I will probably write a specification for them as well, although the PROT files are currently not my highest priority.

It is worrysome that the newest archives no longer contain the xml files. I have not yet looked at the newest version so I can't confirm it.
However, that would explain why the dialogue and string files were included both in xml and compiled format (I had wondered what the difference between them is and which format is used by the program). Have you discovered any test files or are they also gone in the newest clients?

[SWTOR fan]

SCPT file specification (version 4)

EDIT: The newest script files now use version 5. See page 7 for an updated specification.
Original text:

For starters, I want to post the specification of the SCPT format as it seems to be one of the easiest ones.
The SCPT files are can be found in assets_system_1.tor and are extracted into the folder /resources/systemgenerated/compilednative/. The files do not have an extension but are named after a number, like /resources/systemgenerated/compilednative/14988172135558299366. However, the file header begins with SCPT, or 53 43 50 54 in Hex.
Each SCPT file contains a script name, optionally a list of strings, and finally an uncompressed ELF file (Executable and Linkable Format) that can be opened in a normal ELF disassembler like IDA Pro.
Unfortunately, I fear that except for the strings, there is not much use with looking at assembler language, unless someone manages to extract the function names and format the assembler code nicely.
I do not yet know the quickBMS format so I can only give a general outline and no code, sry

File header (16 bytes):
The file header is always the same.
0-3: 4 Bytes: SCPT (53 43 50 54)
4-5: file version as 16-bit integer (little-endian): 04 00
6-7: file version as 16-bit integer (little-endian): 04 00
8-11: Unknown 32-bit integer (little-endian): 01 00 00 00
12-15: Unknown 32-bit integer (little-endian): 00 00 00 00

CRC checksum:
16-21: 6-byte checksum (possibly CRC-32), always different for each file.
Examples:
6D B5 71 08 DF E9 (mµq.ßé)
5B 7C 43 5D 64 9D ([|C]d.)
24 FD 2C CB 2A B2 ($ý,Ë*²)
C0 3A D3 4D A5 B8 (À:ÓM¥¸)

22-23: Two unknown bytes, always: .Ð (00 D0)

File name:
24-27: 32-bit integer specifying string length (little-endian), eg. 21 (15 00 00 00)
28-...: A string with the specified length. This string equals the name of the script file.
Examples: _AiDebuggerClassMethods, _BaseClientClassMethods, _CharacterSystemClassMethods, _ContextMenus, _DebugClassMethods, _ExternalFunctions, _FxSystemClassMethods, _GUIBaseWindowClassMethods, _GUIButtonClassMethods, ...

Unknown lengths:
8 zero bytes: 00 00 00 00 00 00 00 00
Unknown 32-bit integer (little-endian), eg. 94 43 00 00 or 79 0B 00 00 or 6E 05 00 00
→ This integer could stand for the file size, although I have not yet verified this.

List of strings:
Variable-length integer specifying number of strings, eg. 04
LOOP (for each string) {
→ Variable-length integer specifying length of string, eg. 0E
→ String with the specified length, eg. <font color='
}

ELF file:
Unknown variable-length integer, eg. C9 09 EC or C9 3B 2C
→ This integer could stand for the ELF file size, although I have not yet verified this.
From this point on to the rest of the file follows the ELF file. It has the header .ELF (7F 45 4C 46) and can be opened in any ELF disassembler.

[stalja]

In the xml files they have left in there are still some files with the .test. identification. However, I think the only files left are those describing maps, rooms, 3d models and such. I am guessing they are streaming the rest.

[DKK]

The contents of this post was deleted because of possible forum rules violation.