Files extractors scripting

[aluigi]

the xor operation is like an interrupt: on or off.
it's applied to the read operations so if you use something like:

Code: Select all

filexor 0xaa
get NUM1 long
filexor ""
get NUM2 long

NUM1 will be read from the file and XORed with 0xaa while NUM2 is read as is

now imagine if lazeska had only the filename table XORed, in this case you needed to specify "filexor 0xaa" before "get FILEID long" and then "filexor """ ("" disable the xoring because it's an empty string of numbers) after "getdstring FILENAME 52"

exists also another idea, using the virtual MEMORY_FILE to pre-load the filename table (in this case you need to know its full size first) and then using it as an external file, the result is the same and avoids the continuous on/off of filexor

I will post more examples later, just yesterday I have added some feature which allowed me to replicate the spike girls files decoder using custom math operations applied to each single byte of the file

[Mr.Mouse]

I like the XOR feature. It's simple yet effective. If you go on like this, I'll have to update my own multiex.dll !!

[aluigi]

the only "limitation" I have seen in this interrupt-like solution is with nametables xored with multiple xor bytes.
for example if the name table is xored with 0x12 0x34 0x56 there is no problem to handle it but when we disable the xoring to dump the file (filexor "") we loose the current position inside the xor string (uhmm practically if the last position was at byte 0x34 when we will re-activate the xor it will start from 0x12).

so the only way at the moment for avoiding this problem is using the memory_file feature like in the following example:

Code: Select all

filexor "0x12 0x34 0x56"
log MEMORY_FILE NAMETABLE_OFFSET NAMETABLE_SIZE
filexor ""

get OFFSET long MEMORY_FILE
get SIZE long MEMORY_FILE
get NAME string MEMORY_FILE
...

I'm looking for other possible arguments to avoid this problem because obviously the same happens if the files are xored with multiple bytes and the nametable is clear or xored with another sequence

[aluigi]

ok for solving the above issue I have specified the string of numbers as a sequence of fixes bytes while previously I declared it as a variable (it was useful in case of dynamic xor values but in the real world a similar obfuscation is not much used by the games) so the position is automatically handled inside the tool.

[chrrox]

The contents of this post was deleted because of possible forum rules violation.

[Mr.Mouse]

These are UNICODE strings.

[Mr.Mouse]

Code: Select all

# Mexscript for Husky Express .PAK files
# By Mr.Mouse 2009
IDString 0 "BPFS" ;
Get U1 Long 0 ;
Get LastOffset Long 0 ;
Get U2 Long 0 ;
Set Start Long 140 ;
GoTo Start 0 ;
Do ;
Get Dummy Long 0 ;
Get Dummy Long 0 ;
SavePos FOO 0 ;
Get Offset Long 0 ;
SavePos FSO 0 ;
Get Size Long 0 ;
Get FNSize Long 0 ;
Set Name String "" ;
For T = 1 to FNSize ;
GetDString A 1 0 ;
Get Dummy Byte 0 ;
String Name += A ;
Next T ;
Get U3 Long 0 ;
Get Date1 Long 0 ;
Get Date2 Long 0 ;
Get Date1 Long 0 ;
Get Date2 Long 0 ;
Get U4 Long 0 ;
Log Name Offset Size FOO FSO ;
Math Offset += Size ;
Set Start Long Offset ;
GoTo Start 0 ;
While Start <= LastOffset ;

This is my MexScript (BMS means actually "Binary MultiEx Script", which is the compilation of MexScript into a numeric format that legacy versions of multiex.dll could process. The current multiex.dll can also handle MexScript, but simply by parsing it into BMS.

Anyway, the above script works and logs the correct filenames.

Here's a few examples I could get out of them:

data\gfx\char\musher_mari\ani\mari_anim.h

Code: Select all

// This file was automatically generated. It contains definitions for all the
// animations stored in the associated KFM file. Include this file in your
// final application to easily refer to animation sequences.

#ifndef MARI_ANIM_H__
#define MARI_ANIM_H__

namespace mari_Anim
{
    enum
    {
        EAT_FOOD                = 330,
        EMOTION_ANGER001        = 540,
        EMOTION_FAIL001         = 510,
        EMOTION_GIGGLE001       = 530,
        EMOTION_GLAD001         = 500,
        EMOTION_HI001           = 520,
        IDLE_CONTAINER001       = 321,
        IDLE_FOR_WEBSITE        = 1020,
        IDLE_GESTR001           = 240,
        IDLE001                 = 230,
        IDLE002                 = 234,
        IDLE003                 = 235,
        OPEN_CONTAINER          = 320,
        PROLG_CROUCH_IDLE       = 1030,
        RETIRED                 = 270,
        RETIRED_IDLE            = 271,
        RUN                     = 231,
        RUN_LOOKBACK            = 233,
        SHIVERING               = 820,
        SLED_BRAKE              = 253,
        SLED_BRAKE_TO_IDLE      = 254,
        SLED_GETOFF             = 251,
        SLED_GETON              = 250,
        SLED_HIGHSPD            = 221,
        SLED_HIGHSPD_JUMP       = 131,
        SLED_HIGHSPD_LEFT       = 223,
        SLED_HIGHSPD_RIGHT      = 222,
        SLED_IDLE               = 11,
        SLED_IDLE_GESTR001      = 12,
        SLED_IDLE_GESTR002      = 13,
        SLED_IDLE_TO_NOMALSPD   = 1,
        SLED_IDLE_TO_TURNLEFT_END = 22,
        SLED_IDLE_TO_TURNLEFT_HOLD = 21,
        SLED_IDLE_TO_TURNLEFT_START = 20,
        SLED_IDLE_TO_TURNRIGHT_END = 25,
        SLED_IDLE_TO_TURNRIGHT_HOLD = 24,
        SLED_IDLE_TO_TURNRIGHT_START = 23,
        SLED_LOWSPD             = 201,
        SLED_LOWSPD_JUMP        = 111,
        SLED_LOWSPD_LEFT        = 203,
        SLED_LOWSPD_RIGHT       = 202,
        SLED_LOWSPD_TO_IDLE     = 2,
        SLED_NORMALSPD          = 211,
        SLED_NORMALSPD_JUMP     = 121,
        SLED_NORMALSPD_LEFT     = 213,
        SLED_NORMALSPD_RIGHT    = 212,
        SLED_NORMALSPD_TO_BRAKE = 252,
        SLED_PEDALING           = 200,
        SLED_RETIRED            = 260,
        SLED_RETIRED_IDLE       = 261,
        SLED_SHIVERING          = 810,
        SLED_USE_CAMERA         = 341,
        TALK_GESTR001           = 400,
        USE_CAMERA              = 340,
        USE_PICKAXE             = 300,
        USE_SHOVEL              = 310,
        WALK                    = 232
    };
}

#endif  // #ifndef MARI_ANIM_H__

Cursors from the data.016.pak:

cursors_extracted.JPG

[aluigi]

the 8 bytes are not at the end of the file block but at its beginning (so before Get Offset Long), indeed the last file is not catched.

[Mr.Mouse]

the 8 bytes are not at the end of the file block but at its beginning (so before Get Offset Long), indeed the last file is not catched.

Ah yes, I had forgotten that they were at the beginning of the file. I've adapted the above script accordingly. The last file WAS catched, by the way, but not correctly The

Code: Select all

While Start <= LastOffset ; 

made sure the while loop stopped after processing the last file.

[chrrox]

That looks perfect Mr.Mouse thanks for the help with this file format

[aluigi]

finally I have released QuickBMS to public so it's no longer in the beta stage.
I have also added the support for unicode strings (UTF-16 in both little and big endian form) in "Get" and for unicode->string conversion in "Set".

so the Husky Express script can be also like this example

[chrrox]

Thanks Bugtest this script worked great now I can try to compare the 2 scripts and hopefully learn how to convert them between each other.

[Mr.Mouse]

Well, apart from the UNICODE string handling, the scripts are comparible. My script loads up a few more variables from the archive (I do this so I remember what I thought the format was like at the time I worked on it). Bugtest's script is straight to the point of extracting.

[aluigi]

well, finally a new version :)
other than the usual corrections I have added support for the PKWare Data Compression Library (also known as explode), lzma and gzip.
indeed all these 3 compressions or file formats (like in the case of gzip which is just deflate) are used in some games so was necessary to implement them, an example is Assaurl Heroes.

if someone is aware of other libraries commonly used in game and non-game archives they are welcome.
it's important that they are well used libraries because algorithms like lzw are totally useless due to the existence of 1000 variants.

just for remaining in theme of libraries, is someone aware of games/archives which use UCL/NRV?
it's used to compress the executables through the UPX tool but I have never heard of it used in other fields.

[chrrox]

is it possible to support this SLZ compression?
Here someone documented the format specifications.
http://www.romhacking.net/forum/index.p ... 790.0.html
This is used in all the games from that publisher and it would be great to get at those archives.
I can post archives from these if you think it will not be to much trouble to implement / if you want to add support.
Just let me know and Ill post some archives.