Mafia II SDS files?

[s0beit]

updated my script. adding some comments before i upload

Good job.

I don't know which luac script my friend used to make this:
http://pastie.org/1088974


but he claims there is corrupted (or something) bytes in my dumped Luas.

If we could really dump the true luac files, it would help us alot

[WRS]

final version #2 !

note:
dumped files will have 30 bytes or so of random data at the start. i cant find the size of this to remove it, so work with it

removed the naming-by-resource-type. you can name them yourself from the xml.


this should be all.

[s0beit]

final version #2 !

note:
dumped files will have 30 bytes or so of random data at the start. i cant find the size of this to remove it, so work with it

removed the naming-by-resource-type. you can name them yourself from the xml.


this should be all.

Hi, and thank you very much!

I noticed the beginning of the file chunk for compiled Lua files has those extra bytes you mentioned, looks like a header.

i will explain:

Code: Select all

6C 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 E3 21 35 A0 (SIZE OF DIRECTORY PATH) 00 (DIRECTORY)
03 00 00 00 60 22 A9 B3 3A BA 5C C9 11 19 61 FE 3D 37 41 (SIZE OF FILENAME PATH) 00 (FILENAME) E0 59 00 00 (FILE START)

which is, like this in the hex editor for me:
http://img685.imageshack.us/img685/2259/dadat.png

The highlighted byte is the start of the actual compiled Lua file, hope this helps!

EDIT:

Then again some .dat files with no directories in it do this:

Code: Select all

08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 A5 C7 AF F9 (SIZE OF FILENAME PATH) 00 00 00 (FILENAME)
01 00 00 00 (SIZE OF FILE CHUNK) 00 00 00  (FILE START)

It is complex, i'll give you that much.

Edit 2: I also noted that the number of entries in the XML match exactly the order and number of the actual files generated.

Maybe it should be more like:

FrameResource.sds.pak
FrameNameTable.sds.pak
Actors.sds.pak
MemFile.sds.pak
Script.sds.pak

Although i might work on a version in C++, i'm unsure of QuickBMS's full capabilities but i think i can handle the final dumping of the files from the combined file.

[WRS]

yeah, the xml description holds information for the engine to allocate memory as well as the filenames.

the headers have 16 bytes+, but some dds headers have 30 bytes, some 29 - and neither have filenames.

edit: the files may be handled by resource type - i left the headers commented out, but they might be needed to read these headers

[s0beit]

yeah, the xml description holds information for the engine to allocate memory as well as the filenames.

the headers have 16 bytes+, but some dds headers have 30 bytes, some 29 - and neither have filenames.

edit: the files may be handled by resource type - i left the headers commented out, but they might be needed to read these headers

I concur, they seem to be different for some files but not others.

At the moment I'm just going to write one for the mission files

EDIT: So far i'm here

Code: Select all

	virtual bool Split( char *buff, size_t size )
	{
		guidtech::Buffer::DynamicReadable BufferRead( buff );

		unsigned long TotalFileSize = BufferRead.Read< unsigned long >();

		printf( "TotalFileSize [%i]\n", TotalFileSize );

		BufferRead.Read( 12 ); // Unknown chunk

		unsigned long UnknownDword = BufferRead.Read< unsigned long >();

		printf( "UnknownDword [%i]\n", TotalFileSize );

		unsigned char LengthOfDirectory = BufferRead.Read< unsigned char >();

		printf( "LengthOfDirectory [%i]\n", LengthOfDirectory );

		BufferRead.Read< unsigned char >(); // 00

		char *pszDirectoryName = new char[ LengthOfDirectory ];

		for( unsigned long i = 0; i < LengthOfDirectory; i++ )
		{
			pszDirectoryName[ i ] = BufferRead.Read< char >();
		}

		pszDirectoryName[ LengthOfDirectory ] = '\0';

		printf( "DirectoryName [%s]\n", pszDirectoryName );

//		delete[] pszDirectoryName; // Crashy??.....

		unsigned char NumberOfFilesInArchive = BufferRead.Read< unsigned char >();

		printf( "NumberOfFilesInArchive [%i]\n", NumberOfFilesInArchive );

		BufferRead.Read( 2 ); // End of header for Script

		/*for( unsigned char f = 1; f < NumberOfFilesInArchive; f++ )
		{
			//
		}*/

		BufferRead.Read( 17 ); // Unknown chunk

		unsigned char LengthOfFileName = BufferRead.Read< unsigned char >();

		BufferRead.Read( 1 ); // 00

		printf( "LengthOfFileName [%i]\n", LengthOfFileName );

		char *pszFileName = new char[ LengthOfFileName ];

		for( unsigned long i = 0; i < LengthOfFileName; i++ )
		{
			pszFileName[ i ] = BufferRead.Read< char >();
		}

		pszFileName[ LengthOfFileName ] = '\0';

		printf( "FileName [%s]\n", pszFileName );

		// 

		return false;
	}

Code: Select all

Opening filename [00000007.dat]
TotalFileSize [56594]
UnknownDword [56594]
LengthOfDirectory [34]
DirectoryName [/scripts/missions/M05WayToElGreco/]
NumberOfFilesInArchive [3]
LengthOfFileName [56]
FileName [/scripts/missions/M05WayToElGreco/WayToElGrecoPolice.lua]

Code: Select all

Opening filename [00000004.dat]
TotalFileSize [65644]
UnknownDword [65644]
LengthOfDirectory [26]
DirectoryName [/scripts/missions/M05Main/]
NumberOfFilesInArchive [3]
LengthOfFileName [44]
FileName [/scripts/missions/M05Main/M05ListOfParts.lua]

[Tosyk]

final version #2 !

note:
dumped files will have 30 bytes or so of random data at the start. i cant find the size of this to remove it, so work with it

removed the naming-by-resource-type. you can name them yourself from the xml.


this should be all.

Thank you. Script work great But how can i rename files using xml file?

[s0beit]

Hizzah!

I've dumped /missionscripts/

Here is the pastebin for the code i used (DynamicBuffer i won't be giving out, but it is possible to rewrite this easily with structs, full source will be available later)

http://pastebin.com/PPKLG3Ne

EDIT: Removed link, the dumps were buggy but I'm almost there!

[Gocha]

So guys is there a way to extract text and font files for editing and to repack back? I mean trasnlation

[bird]

You can actually unpack the fontfile and the strings with WRS's Final Version #2 BMS-script using QuickBMS.

Fire up your command prompt and type:

Code: Select all

quickbms.exe sds_bmsbywrs_v2.txt text_default.sds C:\path_for_stuff_youd_like_to_extract_things_blah\

You'll get 4 files (00000000.NEO 00000001.dat 00000002.dat 00000003.dat 00000004.NEO):
Datfiles contain strings encoded in UTF-8. 00000001.dat and 00000002.dat is what you need.

Oh, and you have to cut 00000000.NEO's first 30 bytes to get a proper DDS file (as WRS mentioned it before), so load up your favourite hexeditor and wipe out those bytes and start editing!
As for now, I don't know how to repack these files. In order to get an intact .sds file, guess we have to wait until someone does the magic.

[s0beit]

The script .dat format is as follows

Code: Select all

(FILE HEADER)
[filesize (int)][12 bytes][filesize2 (int)][directory length (byte)][00 (byte)][directory name (str)][number_of_files (byte)][3 bytes (end header?)]

(HEADER FOR EACH ENTRY)
[hash (16 bytes, md5)][file name length (byte)][00 (byte)][file name (str)][size of file (int)][file data raw (bytes, size of file)]

[WRS]

Code: Select all

(FILE HEADER)
[filesize (int)][12 bytes][filesize2 (int)]....

the first 12 bytes of all the extracted files is another ram header:

Code: Select all

uint32 RAM;
uint32 VRAM;
uint32 OtherRAM;
uint32 OtherVRAM;

so dont rely on this value - use filesize2 instead!

[s0beit]

Code: Select all

(FILE HEADER)
[filesize (int)][12 bytes][filesize2 (int)]....

the first 12 bytes of all the extracted files is another ram header:

Code: Select all

uint32 RAM;
uint32 VRAM;
uint32 OtherRAM;
uint32 OtherVRAM;

so dont rely on this value - use filesize2 instead!

Nice find, will do, probably will fix an error i had previously

[sng]

SDS file structure:

Code: Select all

------------------------------------------
SDS header (SDS - Source Data Streams (?)) (0x0A + 4) (0x1A)
------------------------------------------
int32 sdsSign ("SDS/0")
int32 version (0x13)
int32 platformSign ("PC/0/0")
int32 sdsHeaderHash 0x5FFB74F3
------------------------------
fileInfo (0x34 + 4)
------------------------------
void* typeHeader 
void* zlibHeader
void* xmlFile
int32 totalSlotRam
int32 totalSlotVram
int32 totalOtherRam
int32 totalOtherVram
int32 ? const 0x01
char[16] const "73979+"
int32 resrcCount
int32 fileInfoHash
-----------------------
typeHeader
-----------------------
int32 typeCount
-----------------------
typeDesc[typeCount]
-----------------------
int32 typeId
int32 typeStrSize
char[typeStrSize] typeName
int32 ? typeFlags
------------------------
zlibHeader
------------------------
int32 zlibSign ("UEzl")
int32 uncompressedStreamSize (default 16384 bytes or 16 KBytes)
byte version
-----------------------
streamHeader
-----------------------
int32 ? compressedStreamSizeWithHeader
bool ? 
int32 ? uncompressedStreamSize (default 16384 bytes or 16 KBytes)
int32 ?
int32 ?
int32 ?
int32 compressedStreamSize
int32 ?
int32 ?
int32 ?
-----------------------
streamData
-----------------------

Unpacked data header:

Code: Select all

------------------------
data block header
------------------------
int32 id
int32 nextBlockOffset
int16 ?
int32 fileSize
int32 ?
...
------------------------

Hash-function:

Code: Select all

int getHash(unsigned char* data, unsigned int size, unsigned int initHash = 0x811C9DC5)
{
    unsigned int r = initHash;
    for(unsigned int i = 0; i < size; i++) r = data[i] ^ (r*0x01000193);
}

[WRS]

my version already has all that and more. hash function is useful though.

[westernsys]

Impressive stuff!
I tried to change luadec's lua_number to 4 byte (float), because looking the .luac file with hex editor i saw it said 04 on 11th byte.
So I changed luaconf.h line 505 to "#define LUA_NUMBER float" and compiled it.
But luadec still complains "bad header in precompiled chunk" when i run it on luac files...