Dark Souls 3 *.bdt and *.bhd Files

HunterAP · Post by **HunterAP** » Mon Apr 18, 2016 7:38 am

From what I've seen, so far all menu items are like previous games, actual model's textures are in DDS format with DX10 headers (meaning they are indeed PBR). I haven't gone through the hassle of converting anything to actually look, but the DX10 header is almost always an indication that PBR is being used. Right now it's just really hard to find anything since nothing is structured.

Zoetropes · Post by **Zoetropes** » Mon Apr 18, 2016 10:41 pm

Swennet wrote:So, has anyone been able to successfully extract and view any of the game's files yet? Personally, I'm most interested in the texture files, item images and sound files. Has anyone been able to extract these in a usable format yet?

It's easy to get stuff like this:

http://imgur.com/a/7Lac9

It's hard to actually do anything with it

tehopeologist · Post by **tehopeologist** » Mon Apr 18, 2016 11:12 pm

Zoetropes wrote:
Swennet wrote:So, has anyone been able to successfully extract and view any of the game's files yet? Personally, I'm most interested in the texture files, item images and sound files. Has anyone been able to extract these in a usable format yet?
It's easy to get stuff like this:

http://imgur.com/a/7Lac9

It's hard to actually do anything with it

how were you able to extract those? i have the extracted files using the bindertool, but i haven't been able to figure out which files are textures/etc. could you elaborate on how you got those images? thanks!

HunterAP · Post by **HunterAP** » Wed Apr 20, 2016 10:49 pm

Like I said before, you can open the files if they don't have an extension in a Hex editor program. If the extension is TPF or there no extension and the Hex Editor shows the first thre characters are TPF, then it's a DDS with a special header that needs to be removed.
If the extension is HKX or the Hex Editor shows the first few characters are WaaW, then it's animation/skeletal bone files.

So far we can't seem to find any other files.

nyxo · Post by **nyxo** » Mon Apr 25, 2016 12:31 am

FROM Software has been stepping up their game in hiding their data with each iteration of Dark Souls.

Here's some info about the files so far:
The root archives are composed of 2 files each: A .BHD file (header), and a .BDT file (data). Historically, these files are slightly different than the similarly named .BHD/.BDT files contained inside, however this hasn't been confirmed yet for DS3, since we haven't gotten full unpacking done yet.

Data0 seems to be unused by the game? At the very least, I haven't found any info about it.
For Data1, Data2, Data3, Data4, and Data5:
The .BHD is encrypted using 2048bit RSA. Each of these files have their own keys, and the keys are encrypted and hardcoded into the .exe file.
Data1:

Code: Select all

Data2:

Code: Select all

Data3:

Code: Select all

Data4:

Code: Select all

Data5:

Code: Select all

Once decrypted, the .BHD format is as follows:

Code: Select all

//--------------------------------------
//--- 010 Editor v3.1.2 Binary Template
//
// File: Dark Souls 3 .BHD (Archive Header)
// Author: Jed "Nyxojaele" Lang
// Revision: 1
//--------------------------------------
SetBackColor(cLtBlue);

typedef struct
{
    ulong StringLength;
    char String[StringLength] <bgcolor=0xffff55>;
} StringStruct <read=ReadStringStruct>;
string ReadStringStruct(StringStruct &ss)
{
    return ss.String;
}

typedef struct
{
    ulong Hash <format=hex, bgcolor=0xffff55>;
    ulong BdtSize <format=hex>;
    quad BdtOffset <format=hex>;
    quad SaltedHashOffset <format=hex, bgcolor=0x22aaff>;
    quad AesKeyOffset <format=hex, bgcolor=0x77aaff>;       //Sometimes 0 (no encryption), 0x10 bytes each
    quad DataSize <format=hex>;  //Sometimes 0 (is compressed)

    local ulong oldPos = FTell();
    if (SaltedHashOffset != 0)
    {
        FSeek(SaltedHashOffset);
        byte SaltedHash[1] <bgcolor=0x22aaff>;    //Not actual size, just marking first byte
    }
    if (AesKeyOffset != 0)
    {
        FSeek(AesKeyOffset);
        byte AesKey[0x10] <bgcolor=0x77aaff>;
    }
    FSeek(oldPos);
} EntryStruct <read=ReadEntryStruct>;
string ReadEntryStruct(EntryStruct &es)
{
    if (es.AesKeyOffset != 0)
    {
        if (es.DataSize == 0)
            return "Encrypted, Compressed";
        else
            return "Encrypted";
    }
    else
    {
        if (es.DataSize == 0)
            return "                   Compressed";
    }
    return "";
}

typedef struct
{
    ulong EntryCount;
    ulong EntriesOffset <format=hex>;
} BucketStruct <read=ReadBucketStruct>;
string ReadBucketStruct(BucketStruct &bs)
{
    string ret;
    SPrintf(ret, "%d Entries", bs.EntryCount);
    return ret;
}

byte Signature[4];
ulong Version;
ulong EndianCheck;
if (EndianCheck != 0x01)
    BigEndian();

ulong ArchiveSize;  //Doesn't match up perfectly? There may be padding at the end
ulong BucketCount;
ulong BucketsOffset <format=hex>;

StringStruct ArchiveName;

FSeek(BucketsOffset);
struct
{
    BucketStruct Bucket[BucketCount] <optimize=false>;
} Buckets;
struct
{
    local int i;
    for (i = 0; i < BucketCount; i++)
    {
        if (Buckets.Bucket[i].EntryCount > 0)
        {
            FSeek(Buckets.Bucket[i].EntriesOffset);
            EntryStruct Entry[Buckets.Bucket[i].EntryCount] <optimize=false>;
        }
    }
} Entries;

If an entry has an AESKeyOffset defined, it's data in the .BDT file is encrypted using that key. The size of the bytes that need decrypting is the entry's BdtSize. After decrypting, the data can be truncated to the entry's DataSize. (This is because the form of AES encrypting used requires data to be padded to a certain size)

If the entry has a 0 DataSize (whether encrypted or not), it's DCX compressed. Decryption must always happen before decompression. The DCX compression hasn't changed much since Dark Souls 1 -- it's basically a custom (Big-Endian!) header around a gzipped blob:

Code: Select all

byte DCXMagic[4];
ulong Version;  //?
ulong DCXSize;
ulong DCXUnknown1[3];

byte DCSMagic[4];
ulong DCSUncompressedSize;
ulong DCSCompressedSize;

byte DCPMagic[4];
char DCPZlibCompSig[4];
ulong DCPSize;
ulong DCPLvl;
byte DCPUnknown1[12];
byte DCPFlags[4];

byte DCAMagic[4];
ulong DCASize;

byte[?] GZipBlob;

This GZipBlob is why some of you have had luck getting some files out of the archives, but it'll only work for compressed, non-encrypted, files.

As far as I know, this is pretty much everything anybody has figured out so far. Most of this I've put together myself, but I'd be lying if I said I did it all without some indirect help from Atvaark (author of BinderTool).

The current (rather large) issue that both my DS3Explore, and Atvaark's BinderTool exhibit is that while the encryption and compression data above is correct when used individually, any entry (this is most of them!) that is both encrypted AND compressed don't seem to work exactly as described above. Decryption seems to work fine (it generates a valid DCX header, all the way up until, and including, the 2 byte header for the GZipBlob). However, decompression fails with various internal errors. I believe the compressed data may be further encrypted, but I'm not 100% sure on this - I'm conducting further research into this at the moment.

Also worth noting is that there is no complete filename lookup dictionary available yet. This is going to be an ongoing process, as there's no simple way to just get all of the filenames.

Once these 2 issues are resolved, we'll have full unpacking of all of the main archives available, however internal archives (such as .BND, .TPF, .BHD/.BDT, etc..) may or may not work immediately. For all we know at this time, perhaps FROM Software has put further encryption on their internal file entries as well.

pewposterous · Post by **pewposterous** » Mon Apr 25, 2016 1:15 pm

Wow thanks for the hard work! Hopefully things go smoothly

atvaark · Post by **atvaark** » Mon Apr 25, 2016 7:49 pm

nyxo wrote:As far as I know, this is pretty much everything anybody has figured out so far. Most of this I've put together myself, but I'd be lying if I said I did it all without some indirect help from Atvaark (author of BinderTool).

The current (rather large) issue that both my DS3Explore, and Atvaark's BinderTool exhibit is that while the encryption and compression data above is correct when used individually, any entry (this is most of them!) that is both encrypted AND compressed don't seem to work exactly as described above. Decryption seems to work fine (it generates a valid DCX header, all the way up until, and including, the 2 byte header for the GZipBlob). However, decompression fails with various internal errors. I believe the compressed data may be further encrypted, but I'm not 100% sure on this - I'm conducting further research into this at the moment.

Nice summary of the current situation. I can also share some of my findings you didn't write about yet.

1.
Data0.bdt is just a single .bnd.dcx file which contains the following .param files. http://pastebin.com/raw/52Q2Mq2K
Data0.bhd isn't even used by the game and the decryption key is probably not even included in it.

2.
The DataX.bhd files use the BHD5 format while the other .bhd files use the BHD4 format (which at least contains unicode filenames).

3.
010 Editor typedefs of the AES key and the SHA Hash structs look like this:

Code: Select all

typedef struct {
   byte hash[32];
   int type; // 1 or 3 
   int size;
   int unknown;
   int size;
   int unknown;

   if (type == 3) {
     int size; // or -1
     int unknown;
     int size; // or -1
     int unknown;
     int size; // or -1
     int unknown;
     int size; // or -1
     int unknown;
   }
} Bhd5ShaHash;

typedef struct {
   byte key[16];
   int type; // 1 or 3
   int size;
   int unknown;
   int size;
   int unknown;

   if (type == 3) {
     int size; // or -1
     int unknown;
     int size; // or -1
     int unknown;
     int size; // or -1
     int unknown;
     int size; // or -1
     int unknown;
   }
} Bhd5AesKey;

4.
If the file size of an entry is 0 (DCX files), then the game loads the first 48 byte to determine how much of the data has to be read.

5.
DCX encryption is a bit overkill. The data that is read from the DataX.bdt files is:
-Decrypted with the AES key found in the DataX.bhd file
-All the data after offset 78 is processed two times by some unknown algorithm that is related to zlib
-All the data after offset 80 is then decrypted with the AES key a second time
-The data after offset 78 can then be decompressed with zlib

See: BinderTool Issue #3 Decrypting BDT5 files

6.
Besides exporting all the static file names I could find in the executable I haven't worked on recreating the file name dictionary.
The hash function looks like this, but I couldn't confirm it with some test cases yet.

Code: Select all

filePath.Replace('\\', '/')
        .ToLowerInvariant()
        .Aggregate(0, (i, c) => i * 137 + c)

See:
BinderTool Issue #4 Rebuilding the dictionary

Swennet · Post by **Swennet** » Tue Apr 26, 2016 12:16 am

Zoetropes wrote:
Swennet wrote:So, has anyone been able to successfully extract and view any of the game's files yet? Personally, I'm most interested in the texture files, item images and sound files. Has anyone been able to extract these in a usable format yet?
It's easy to get stuff like this:

http://imgur.com/a/7Lac9

It's hard to actually do anything with it

I would also like to know how you were able to extract and open these files. Could you provide a step-by-step guide for us? Thank you!

atvaark · Post by **atvaark** » Wed Apr 27, 2016 7:57 pm

I just released the BinderTool v0.4-pre1 binaries. I haven't checked if all files are supported now, but it looks good so far.
What's missing is support for Data0.bdt and DS30000.sl2 and a working file name dictionary.

nyxo · Post by **nyxo** » Wed Apr 27, 2016 8:50 pm

Atvaark and I solved this last piece together via chat -- while he put together his new release, I confirmed that everything directly inside the main archives extracts without errors. Presumably that means the files are all correct, but we won't know 100% for sure until we start looking more directly at them.

For those curious, here is an updated section of the 010 Template with the pertinent information:

Code: Select all

typedef struct
{
    quad OffsetStart;
    quad OffsetEnd;
} OffsetSetStruct <read=ReadOffsetSetStruct>;
string ReadOffsetSetStruct(OffsetSetStruct &oss)
{
    string ret;
    SPrintf(ret, "%X[%X]", oss.OffsetStart, oss.OffsetEnd-oss.OffsetStart);
    return ret;
}

typedef struct
{
    byte Hash[32] <format=hex, bgcolor=0x22aaff>;
    ulong OffsetSetCount <bgcolor=0xEEEEEE>;
    OffsetSetStruct HashOffsetSets[OffsetSetCount];
} SHAHashStruct;

typedef struct
{
    byte Key[16] <format=hex, bgcolor=0x77aaff>;
    ulong OffsetSetCount <bgcolor=0xEEEEEE>;
    OffsetSetStruct EncryptionOffsetSets[OffsetSetCount];
} AESKeyStruct;

typedef struct
{
    ulong Hash <format=hex, bgcolor=0xffff55>;
    ulong BdtSize <format=hex>;
    quad BdtOffset <format=hex>;
    quad SHAHashOffset <format=hex, bgcolor=0x22aaff>;   //Sometimes 0 (some, not all, unencrypted & uncompressed entries)
    quad AesKeyOffset <format=hex, bgcolor=0x77aaff>;       //Sometimes 0 (not encrypted)
    quad DataSize <format=hex>;                             //Sometimes 0 (DCX compressed)

    local ulong oldPos = FTell();
    if (SHAHashOffset != 0)
    {
        FSeek(SHAHashOffset);
        SHAHashStruct SHAHash;
    }
    if (AesKeyOffset != 0)
    {
        FSeek(AesKeyOffset);
        AESKeyStruct AESKey;
    }
    FSeek(oldPos);
} EntryStruct <read=ReadEntryStruct>;
string ReadEntryStruct(EntryStruct &es)
{
    if (es.AesKeyOffset != 0)
    {
        if (es.DataSize == 0)
            return "Encrypted, Compressed";
        else
            return "Encrypted";
    }
    else
    {
        if (es.DataSize == 0)
            return "                   Compressed";
    }
    return "";
}

I'm holding off on releasing my GUI-based archive tool until we've created a filename dictionary, but at that time, I'll release DS3Explore for those who are command line averse

HunterAP · Post by **HunterAP** » Thu Apr 28, 2016 3:00 am

Just updating in what I found now that the files are a little more structured:
TPF texture files are, like previous Dark Souls games, DDS files with a TPF header. This time around, however, they use DX10 compression which is used for PBR shaders (previous games had DXT1 and DXT5). This makes the conversion process a bit harder, but it's a matter of opening the files in a Hex Editor -> deleting anything before the phrase "DDS |" -> saving the file and changing the extension to DDS -> open the file with Noesis and export it.
I'm working on a program to get around this, just have to analyze more textures so it can be batch done.

FLVER model files are still in effect, except they are read incorrectly with tons of mesh errors with the current plugins. I have no idea how to get around this at the time, but I ca at least make out what I'm looking at. Some of the files just plainly don't open with the Noesis plugin, too.

This is the model for the Reanimated Corpse from the Cathedral of the Deep graveyard, you can see the model is really screwed. The issue is pretty much the same thing I came across when working with Dark Souls 2: Scholar of the First Sin models, which I'm assuming is because SotFS and Dark Souls 3 are both 64bit and thus have 64bit modeling format / compression.

Update: I also can't seem to find any texture maps aside from the Albedo (diffuse). It may be BinderTool's dictionary isn't complete, or the other maps are hidden away somewhere in the jumbled folder structure.

zaramot · Post by **zaramot** » Thu Apr 28, 2016 6:48 am

I can take a look at .flver and .tpf files if you'll send me a sample. It shouldn't be hard to edit current script for Dark Souls 2 or make Noesis plugin for textures

EDIT: As I though, format almost not changed from previous games, thanks a lot to HunterAP for sample. So if anyone willing to upload pack of models (characters/monsters) and their .hkx files we will get models with bones and weights pretty soon. I hope lol

HunterAP · Post by **HunterAP** » Thu Apr 28, 2016 11:18 pm

I'm packing up all the characters and their HKX files now lol. I'll PM you with it in a few minutes, can't wait to get the models working.

EDIT: Sent the files over.

pewposterous · Post by **pewposterous** » Fri Apr 29, 2016 1:21 am

Thanks for the hard work peoples! Cant wait!

dibe91 · Post by **dibe91** » Fri Apr 29, 2016 7:47 am

sorry, but you can also get the original skeletons with skin models?

XeNTaX

XeNTaX

Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files

Re: Dark Souls 3 .bdt and .bhd Files