Mafia 3 - .SDS

[iambosh]

Posting my research here in case anyone can put it to good use. I'll probably make my own tools eventually, but I don't have an ETA on that

Thanks to aluigi for his Mafia .SDS (PS3) script, it helped point me in the right direction.

NOTE: I'm probably wrong on a lot of this, so please feel free to make corrections. I'm most likely off on the chunkCount being used to determine how many SDSChunkData's there are.

Code: Select all

struct SDSFile {
    int magic; // 0x534453 ('SDS\0')
    
    int version; // 0x14 (v20 / v1.4?)
    
    enum SDSType : int {
        PC = 0x4350,
        
        /* These are ASSUMED, I only have the PC version */
        PS4 = 0x345350,
        XB1 = 0x314258,
    } type;
    
    int unk_0C; // 0x5DE53FDE
    
    int dataInfoOffset;
    int dataOffset;
    
    int unk_18; // always zero?
    int unk_1C; // offset / size of something?
    int unk_20; // always zero?
    int unk_24; // offset / size of something?
    int unk_28; // always zero?
    int unk_2C; // 1
    int unk_30; // always zero?
    int unk_34; // always zero?
    int unk_38; // always zero?
    int unk_3C; // always zero?
    
    int chunkCount; // how many data chunks? (educated guess)
    int reserved; // checksum, hash, or key of some sort?
    
    /* @ dataInfoOffset */
    struct SDSDataInfo {
        int count; // how many data types there are
        
        struct SDSDataType {
            int typeIndex; // local index to reference this type
            
            int strLen;
            char typeName[strLen]; // no null-terminator
            
            int reserved; // possibly a guard? (always zero?)
        } dataTypes[count];
    
    } dataInfo;
    
    /* @ dataOffset */
    struct SDSData {
        int magic; // 0x6C7A4555 ('UEzl')
        
        int unk_04; // 0x10000? (never seems to change)
        char unk_05; // 4?
        
        struct SDSChunkData {
            int dataSize; // size of SDSChunkData
            char dataType; // reference to SDSDataInfo.dataType
            
            struct SDSChunkData {
                int memorySize; // size in memory? (e.g. uncompressed size)
                
                int bufferOffset; // offset to start of data
                
                int memorySize_2; // SDSData.unk_04 again?
                
                short unk_0A; // number of something? (this format is confusing)
                short unk_0C; // 0x80F?
                
                short bufferSize; // this might be an int, but unlikely
                /*
                    <<< Data padding >>>
                */
                /* @ bufferOffset */
                char buffer[bufferSize];
            } data;
        } chunks[chunkCount];
    };
};

hey man do you have a compiled version? it wont compile for me for some reason in visual studio

[BitEmE]

Also the content seems to be encrypted, I've tried to open an encrypted DDS file -> exception.

They not encrypted, chunks are compressed with Zlib. About XML toc > not exist anymore. Content filenames hashed (fnv64a).

Code: Select all

uint64 m3_gethash(unsigned char *m_String, int dwLength)
{
   uint64 dwHash = 0xCBF29CE484222325;
   for(int i = 0; i < dwLength; i++)
   {
      dwHash *= 0x00000100000001B3;
      dwHash ^= m_String[i];
   }
  return dwHash;
}

Some names

Code: Select all

/gui/pop/garage.dds

For DDS you need skip additional header with 0x2F bytes after decompress. You can do it here in TextureViewer:

Code: Select all

		private void UpdatePreview()
		{
			...
		}

Try set to 0x2F for code (memory.Position = 0L;) and this will be enought i guess

Ok the thing with zlib, I get it.
But i have the following problem at the moment:

My .SDS File has 9 chunks, the alignment is 0x10000 (65335)
The first file has a size of 699239 bytes (682kb)


Of course I can't read the file, because there are too less chunks (at least I'm getting exceptions thrown in my face)
9 Chunks a. 65335 bytes alignment = 588015 bytes

So I will always fail to read that file.

Am I doing something with zlib wrong?
I think the TypeID and Version are correct? Maybe the file sizes too?

I am confused right now. Would be great if someone could tell me what I'm doing wrong or at least giving me a clue
Thanks!

[chipsman]

...

Zlib decompression is exactly the same as was in the M2

About files.. It looks like now the file header has bigger size than was before (36 bytes instead of 30, I have such assumption because there was extra 11 bytes between file header and DDS file magic in M2, if we follow the same logic - the header size has increased by 6 bytes)

here is my 010 Editor Templates for sds files and decompressed memory blocks https://github.com/Chipsman/mafia3-010ETemplates
may be it will help you somehow

p.s.: anybody knows where radio music is located? here Mafia III\sds_retail\audio or here Mafia III\edit\audio\packages ?

[GRiNDERKILLER]

\edit\audio\packages\radio_global.pck

[BitEmE]

Alright, thanks to your help guys, really apprechiate it!

I'm now able to extract/show DDS files. Unfortunately DDs file in Mafia 3 can have the compression types ATI and DXT10, I'm not quite sure how to fix those decompressions with the texture viewer, but that's just a minor problem. (Also extracting is kind of a problem as there are +4 bytes infront of the magic DDs identifier of the file, so external DDs viewer will fail to load the image)


Also I've modified the XML reading and get now 'readable' XML content. Again, unfortunately, the format has changed in Mafia 3. The Problem right now is that I'm not able to say that the next node in memory is an attribute of a node, is a value, a child, or another node.
For example, with this file it's working because the values are only numeric (I'm using a hack at the moment to detect values inside nodes -> TryDouble)


With this file, it's not working because there are node attributes (see xsi:type is a node instead of attribute)


The structure is weird because it looks like this:

node type = 4
6 bytes = 0
String (length is not written in memory, you have to check for null terminator)
String Terminator
node type = 4
6 bytes = 0
String (length is not written in memory, you have to check for null terminator)
... and so on and so on..

So basically there's always a 4 between all nodes (maybe seperator? idk)
I don't know how I should figure out what's a child node, what's an attribute etc..

Again, here's the source (Just based of Rick's Mafia 2 tool) https://github.com/MartinJK/Mafia3SDSExplorer (in case someone of you wants to try it out etc.)

Thanks for you help guys, really apprechiate it. I'm used to reverse engineer and mod games, this is the first time I'm trying to 'reverse' the files format and create a tool for it

[Ekey]

Some new names

Code: Select all

/gui/pop/missions.dds
/gui/pop/womaniser_voodoo_dancer.dds
/gui/pop/alternative_shop.dds
/gui/pop/classy_shop.dds
/gui/pop/cloth_shop.dds
/gui/pop/fashion_shop.dds
/gui/pop/gun_shop.dds
/gui/pop/item_shop.dds
/gui/pop/shades_shop.dds
/gui/bc_map_pcp_lab_a.gfx
/gui/config/maps/bc_map_pcp_lab_a
/gui/hideout_district_assignment.swf
/gui/images/icons/button_a.dds
/gui/images/icons/button_b.dds
/gui/images/icons/button_x.dds
/gui/images/icons/other_icons0031.dds
/gui/images/icons/other_icons0037.dds
/gui/images/icons/other_icons0099.dds
/gui/images/icons/other_icons0101.dds
/gui/images/icons/other_icons0103.dds
/gui/images/icons/other_icons0105.dds
/gui/images/icons/other_icons0106.dds
/gui/interact_collectable.swf
/gui/interact_object.swf
/gui/interact_weapon_pickup.swf
/gui/in_world_menu.swf
/gui/pop/garage.dds
/gui/pop/home.dds
/gui/scripted_test.swf

/xbin/cities.xlaybin
/xbin/plotlines.xlaybin
/xbin/missions.xlaybin
/xbin/quests.xlaybin
/xbin/game_events.xlaybin
/xbin/persistent_characters.xlaybin
/xbin/businesses.xlaybin
/xbin/speeches.xlaybin
/xbin/shops.xlaybin
/xbin/cities_game_group.xbin
/xbin/plotlines_group.xbin
/xbin/missions_group.xbin
/xbin/quests_group.xbin
/xbin/game_events_group.xbin
/xbin/persistent_characters_group.xbin
/xbin/businesses_group.xbin
/xbin/streammap_gameinit_group.xbin
/xbin/speeches_game_group.xbin
/xbin/generic_speech_mp.xlaybin
/xbin/generic_speech_mp_group.xbin
/xbin/generic_speech_situations_group.xbin
/xbin/generic_speech_variations_group.xbin
/xbin/shops_group.xbin
/xbin/game_main.xlaybin
/xbin/game_main_group.xbin
/xbin/expressiondata.xlaybin
/xbin/game_expressions_group.xbin
/xbin/charactertable.xlaybin
/xbin/charactertable_cinematics_group.xbin

/sds/maps/mapa_bourbon2.sds
/sds/maps/mapa_testscene.sds

/root/Situations
/root/Variations
/root/CharacterTableMP
/root/StreamMapTableMP
/root/GameEventsTableMP
/root/QuestsTableMP
/root/MissionsTable
/root/SpeechesTableMP
/root/PlotlinesTableMP
/root/BusinessesTableMP
/root/PersistentCharactersTableMP

/fsm/persistent_characters/alvarez_active
/fsm/persistent_characters/alvarez_inactive
/fsm/persistent_characters/biaggio_active
/fsm/persistent_characters/biaggio_inactive
/fsm/persistent_characters/cesar_active
/fsm/persistent_characters/cesar_inactive
/fsm/persistent_characters/clayton_active
/fsm/persistent_characters/clayton_inactive
/fsm/persistent_characters/creed_active
/fsm/persistent_characters/creed_inactive
/fsm/persistent_characters/danny_quest_active
/fsm/persistent_characters/danny_quest_inactive
/fsm/persistent_characters/doctor_active
/fsm/persistent_characters/doctor_inactive
/fsm/persistent_characters/donovan_active
/fsm/persistent_characters/donovan_inactive
/fsm/persistent_characters/mack_quest_active
/fsm/persistent_characters/mack_quest_inactive
/fsm/persistent_characters/marcano_active
/fsm/persistent_characters/marcano_inactive
/fsm/persistent_characters/maria_active
/fsm/persistent_characters/maria_inactive
/fsm/persistent_characters/martinez_active
/fsm/persistent_characters/martinez_inactive
/fsm/persistent_characters/professor_active
/fsm/persistent_characters/professor_inactive
/fsm/persistent_characters/sixto_active
/fsm/persistent_characters/sixto_inactive

/scripts/gamelogic/bc_openworld
/scripts/gamelogic/fpm/fpm_plotline
/scripts/gamelogic/milestones/milestone27/drug_farm_main
/scripts/gamelogic/milestones/milestone27/gameplay_bricks/shooting_from_boat
/scripts/gamelogic/milestones/milestone27/gameplay_bricks/shooting_from_car
/scripts/gamelogic/milestones/milestone28/steal_rum
/scripts/gamelogic/plotline_act2
/scripts/gamelogic/testscenes/bourbon_test_mission/bourbon_test_mission
/scripts/gamelogic/testscenes/districts/west_marais_district_structure_quest
/scripts/gamelogic/testscenes/districts/wharf_district_structure_quest
/scripts/gamelogic/testscenes/game_structure_init
/scripts/gamelogic/testscenes/game_structure_quest
/scripts/gamelogic/testscenes/open_world/main
/scripts/gamelogic/testscenes/racing_focustest/main
/scripts/gamelogic/testscenes/test_bartek/simple
/scripts/gamelogic/testscenes/test_racing_pod/main

[Skrillex]

Alright, thanks to your help guys, really apprechiate it!

Thanks for you help guys, really apprechiate it. I'm used to reverse engineer and mod games, this is the first time I'm trying to 'reverse' the files format and create a tool for it

We owe you bro, thanks. Can you share compiled version of this script? And i want a question, this tool does it work to subtitles.sds files, can you extract and repack clearly?

[Ekey]

Info for PCKG archives

Code: Select all

struct PCKGHeader
{
   uint32_t   dwID; // Always PCKG (0x50434B47)
   uint32_t   dwVersion; // 1
   uint32_t   dwTotalFiles;
   uint32_t   dwFlag; // 0xFFFFFFFF
};

struct PCKGEntry
{
   uint64_t   dwHash; // Hash of filename?
   uint32_t   dwOffset;
   uint32_t   dwSize;
   uint32_t   dwZSize;
};

[Godxon1]

Alright, thanks to your help guys, really apprechiate it!

I'm now able to extract/show DDS files. Unfortunately DDs file in Mafia 3 can have the compression types ATI and DXT10, I'm not quite sure how to fix those decompressions with the texture viewer, but that's just a minor problem. (Also extracting is kind of a problem as there are +4 bytes infront of the magic DDs identifier of the file, so external DDs viewer will fail to load the image)


Also I've modified the XML reading and get now 'readable' XML content. Again, unfortunately, the format has changed in Mafia 3. The Problem right now is that I'm not able to say that the next node in memory is an attribute of a node, is a value, a child, or another node.
For example, with this file it's working because the values are only numeric (I'm using a hack at the moment to detect values inside nodes -> TryDouble)


With this file, it's not working because there are node attributes (see xsi:type is a node instead of attribute)


The structure is weird because it looks like this:

node type = 4
6 bytes = 0
String (length is not written in memory, you have to check for null terminator)
String Terminator
node type = 4
6 bytes = 0
String (length is not written in memory, you have to check for null terminator)
... and so on and so on..

So basically there's always a 4 between all nodes (maybe seperator? idk)
I don't know how I should figure out what's a child node, what's an attribute etc..

Again, here's the source (Just based of Rick's Mafia 2 tool) https://github.com/MartinJK/Mafia3SDSExplorer (in case someone of you wants to try it out etc.)

Thanks for you help guys, really apprechiate it. I'm used to reverse engineer and mod games, this is the first time I'm trying to 'reverse' the files format and create a tool for it

Can you please post here builded version of your tool? Thx

[chipsman]

Audio packages format info (AKPK)

Code: Select all

struct Header
{
    char    dwMagic[4];   // AKPK (0x4B504B41h)
    DWORD   dwHeaderSize; // +0x4
    DWORD   _f8;
    DWORD   _fC;
    DWORD   _f10;
    DWORD   _f14;
    DWORD   _f18;
    DWORD   _f1C;
    DWORD   _f20;
    DWORD   _f24;
    DWORD   _f28;
    DWORD   _f2C;
    DWORD   _f30;
    DWORD   dwFileCount;    //+0x34
};

struct FileEntry
{
    DWORD   dwHash;     // +0x0
    DWORD   _f4;        // always 16,  block size ? or aligment ?
    DWORD   dwSize;     // +0x8
    DWORD   dwOffsetInBlocks; //+0xC // block size == 16
    DWORD   _f10;       // always 0, // or dwOffsetInBlocks is QWORD
};

Audio is stored in .wav container but the format/codec inside is unknown

here is 010 template https://github.com/Chipsman/mafia3-010E ... /m3-pck.bt

[iambosh]

any progress on decrypting the dat files?

[Skrillex]

Alright, thanks to your help guys, really apprechiate it!

Thanks for you help guys, really apprechiate it. I'm used to reverse engineer and mod games, this is the first time I'm trying to 'reverse' the files format and create a tool for it

We owe you bro, thanks. Can you share compiled version of this script? And i want a question, this tool does it work to subtitles.sds files, can you extract and repack clearly?

[trexjones]

I just wanted to add my support for anyone thinking of putting tools together for this game! Love it! Great character models, and vehicles, too! Would love to be able to get in and take a proper look at some of them...!

[daemon1]

any progress on decrypting the dat files?

they are not encrypted

[iambosh]

any progress on decrypting the dat files?

they are not encrypted

yeah when decrypted with the tool the results from the sds are dat files with random names and they are encrypted. so no actual assets