Saints Row 2 XWB compressed wave

[FordGT90Concept]

There's an application (towav.exe) that attempts to convert XWB files to WAV. On one of the XWB files in SR2 (MUS MIX), it only manages to extract 8 files when there are 101. I finally figured out why: the other 93 appear to be compressed but I can't figure out what kind of compression it is (no header info) or how to go about uncompressing them.

XWB is a Microsoft XNA Wavebank format so the logical conclusion would be something in System.IO.Compression. I tried them both as well as ICSharpZipLib Zip.Compress.Inflate and all of them fail (most likely due to the lack of header.

I'm running out of ideas.


Every file appears to start with a int16 value of 5-7. I find this odd and it is probably important. I attached a zip with 5 of the raw files (WAVE header removed). If I play it with the WAVE header, it is just static.

Any incite would be greatly appriciated. So close...but yet so far.

[aluigi]

why don't you use unxwb?
it tells any information found about the file and automatically builds the needed headers to play/convert it in a second moment.

[FordGT90Concept]

I didn't know it existed because it got no hits on Google. I put it on the XBOX_XWB3#Compatible_Programs so more people will find it.

However, it didn't work. All the files I'm having trouble with (the compressed lot) your tool dumps what I attached (a short, followed by compressed data) with a WMA extension. Obviously, WMP can't play it because it doesn't have the appriorate header information.

[aluigi]

uhmmm so they get identified as wma files.

indeed they don't look like xma1 or xma2 files because xmaencode can't decode them.
they could be msadpcm but I doubt because this adpcm needs a particular value located in the xwb archive and after a scanning I have found no value good to hear a clean sound so it's not msadpcm too.

is this a x360 game?
could it be xwma? mah
everything I know about this matter is already implemented in unxwb so I have out of ideas

[GodOfWar]

is this a x360 game?

yeah, is 360 game.the big archive called real_music.vpp_xbox2 (1.05gb) contains all music from game (files have .mus extension). tool for PC version didn't work with x360 version =(

[FordGT90Concept]

It was extracted from music1.vpp_pc to music4.vpp_pc from the PC (Steam) version of Saints Row 2. The xwb format appears "normal" except for a few things:

Header:

Code: Select all

        public string IdentifyingChars = "WBND";
        public int Unknown44;
        public int Unknown42;  // This is odd -- no idea what this 42 is for.  Maybe it is the original version number and they incremented it to 44 for their changes.
        public int WaveBankInfoOffset;
        public int WaveBankInfoLength;
        public int FileRecordsOffset;
        public int FileRecordsLength;
        public int CompressionInfoOffset;  // This is somewhat odd
        public int CompressionInfoLength;  // This is very odd.
        public int Unknown1Offset; // Always 0
        public int Unknown1Length; // Always 0
        public int FileDataOffset;
        public int FileDataLength;

CompressionInfo is an array of ints. It counts up from zero except where it matches FileRecords for a compessed file (e.g. FileRecord[1] = compessed, CompressionInfo[1] = -1). How quickly it counts up seems to be determined by the size of the file. On MUS MIX.xwb, the Length of Compression Info is over 6000 but the counting up ends around 600. There are only 101 file records. No -1s appear after the 101 index. It appears that, so long as the value isn't -1, it doesn't mean anything.

Wave Bank Info:

Code: Select all

        public int Flags;
        public int FileCount;
        public string Name; // 64 bytes
        public int FileRecordLength;
        public int FileNameLength;
        public int FileDataOffset;
        public int Unknown1; // Always 0
        public int Unknown2;
        public int Unknown3;

Those Unknown2 don't seem to match up to what appears in the unxwb source (especially 2 and 3). I haven't been able to sort what they are. It might be that Unknown2 is a start time (or ticks from an epoch) and Unknown3 is a stop time.

I uploaded the smallest XWB with the CompressionInfo array here. There are no files in this particular XWB that are not compressed:
http://www.speedyshare.com/files/22145167/MUS_420.xwb

When I say compressed, I mean these files are way too small to be waves. MUS MIX.xwb has 8 uncompressed files and some are north of 20 MiB in size. The compressed files rarely exceed 1 MiB.

I am not certain the format flag (WAV/WMA) is correct. There's no way to be certain until the compression is dealt with.

I haven't tried DeflateStream or InflateStream without the first short but, on examining some other files, it appears that short isn't uniform across all XWB files.

[FordGT90Concept]

All WMA files start with 0500, 0600, or 0700. The ones that are obviously music actually have the first 17 bytes match. It would make sense if their uncompressed headers matched therefore causing a large chunk of the compressed data match. So I guess my question is this: What types of compression support headerless inflation? I know zlib has compress() and uncompress() but I doubt it can handle the ~2 MiB some of these files are (I don't see any indication there are parts).


Edit: I tried zlib and zlib1 uncompress(dest, destlen, src, srclen) and both return Z_DATA_ERROR.


Anyone have any ideas I could try?


Maybe this thread should be moved to the compressed forum.

[FordGT90Concept]

This new section of the XWB (apparently unique to Saints Row 2) I call "CompressionInfo" has been described:

Code: Select all

int[FileCount] compress_array_offset_from_end_of_this_array; // if -1, File is not compressed

// at each offset
int count;
int[count] uncompressed_begin_offset;

For example, compress_array_offset_from_end_of_this_array could contain:

Code: Select all

0
-1
12

Reading the offsets would turn that into an array:
File 0:

Code: Select all

count = 3
value = { 20, 40, 60 }

File 1: not compressed = null

File 2:

Code: Select all

count = 5
value = { 30, 60, 90, 120, 150 }

Basically, what I know:
-The file's total deflated size.
-How many inflated sections there are.
-The size of each inflated section.
-Zipping a compressed file results in virtually zero gain.
-Appears to be headerless. If it has a header, it is short.
-Every deflated section appears to be exactly 929 or 1487 bytes in length (depends on file).

What I don't know:
-The compression method used.


Name that compression!

[FordGT90Concept]

I think it uses a combination of compression and encryption. That is, it compresses x amount of data and then losslessly encrypts it to y block size (1487 and 929 for sure). The compression itself could be as simple as zlib inflate (the SR2 exe has references to it) but I have no idea on the encryption.

I extracted the list of most of the strings from the SR2 executable and they are attached. Maybe it has some clues as to what is happening in the XWB files.

[Liandril]

Hello and sorry - I know, the last posting in this thread was 3 months ago, but it's the best thread on the subject XWB that I found so far - so I hope, the original posters are still reading. And sorry for my English!
I'm writing a tool (a perl script actually) in order to insert sounds into an existing german game, that uses .xsb/.xwb files. And additionally, the tool is supposed to extract waves (nearly all of them are WMA) from the wavebanks. I wrote it, because unxwb can't (afaik) write valid/playable .wma files... and other tools (XWB_Extractor, EkszBox,...) weren't able to handle the .xwb files of the game at all. This is no wonder, because backward compatibility is not really needed in XACT(=the tool that creates .xwb/.xsb files) and therefor, Microsoft can change the format with each version of the .xwb/.xsb file format. So "old" extraction tools won't work with "new" xsb/xwbs.

First of all: Congratulations, FordGT90Concept - you did a very good job in "demystifying" the .xwb structure , but (sorry, I have to say that) you can get detailed xwb-info very easy, because the .xwb format (unfortunately NOT the xsb format) is documented in Microsofts DirectX SDK. Just download+install the SDK and search for the header file "xact3wb.h" (in older SDKs it's "xact2wb.h"). As a summary:

Code: Select all

typedef struct WAVEBANKHEADER {
    DWORD           dwSignature;                        // File signature "WBND"
    DWORD           dwVersion;                          // Version of the tool that created the file
    DWORD           dwHeaderVersion;                    // Version of the file format
    WAVEBANKREGION  Segments[WAVEBANK_SEGIDX_COUNT];    // Segment lookup table
}

so your "Unknown44" means, the .xwb file was created with tool(=XACT) version 44, and "Unknown42" is the xwb file format version.

Code: Select all

typedef struct WAVEBANKREGION {
    DWORD       dwOffset;               // Region offset, in bytes.
    DWORD       dwLength;               // Region length, in bytes.
}

Actually, there COULD be up to 5 Regions, but all .xwbs I saw so far had only 3 or 4, namely (+appearing in this order in the file): 1. WAVEBANKDATA, 2. ENTRYMETADATA, (3. SEEKTABLES),(4.ENTRYNAMES), 5. ENTRYWAVEDATA
Thus, a Header of a .xwb with 5 Region-entries is 3*4(for Signature,..) + 5*8(Regions)=52 B long

1. WAVEBANKDATA

Code: Select all

typedef struct WAVEBANKDATA {
    DWORD                   dwFlags;                                // Bank flags
    DWORD                   dwEntryCount;                           // Number of entries in the bank
.....rest: see xact3wb.h }

dwFlags (= Bank flags) = info about: is it a in-memory or streaming wavebank; are EntryNames and/or SeekTables inlcuded?....
dwEntryCount = number of wave files in the wavebank
2. ENTRYMETADATA
In the ENTRYMETADATA region, there is for each wave file a WAVEBANKENTRY which contains the following infos for the wave file:
Flags (has Loops; Enable stream read-ahead)
Duration(e.g. a 10 second long wave with sampling rate 44100 Hz has Duration=441000)
WAVEBANKMINIWAVEFORMAT (Typ (PCM,ADPCM,XMA,WMA); number of Channels; sampling rate (e.g.44100Hz); Align; 8or16bit PCM))
PlayRegion (Region within the wave data segment (=ENTRYWAVEDATA) that contains this entry)
LoopRegion
5. ENTRYWAVEDATA
the actual data as referenced by PlayRegion of the WAVEBANKENTRIES (/End-of-summary )

-Every deflated section appears to be exactly 929 or 1487 bytes in length (depends on file)

Name that compression!

WMA (quote from xact3wb.h: static const DWORD aWMABlockAlign[] = { 929, 1487,...}

So we seem to have the same problem, FordGT90Concept: we both try to extract .wma-files from .xwb wavebanks. But with different strategies: you try to decompress the wma file data in order to write them to .wav files, I try to simply write the compressed data and create appropriate WMA-Headers in order to have .wma files as a result.

This is what I did so far: I looked into the (very long and not very usefull) ASF/WMA-Specification.
A .wma file contains at least 4 Objects: FileProperties Object,StreamProperties Object, Header Extension Object and DATA OBJECT (consisting of Data Packets, each Data Packet has some header/info bytes). With the infos from WAVEBANKENTRY of the file and xact3wb.h, I'm (almost) able to write the first 3 Objects. The biggest problem is the WMA Data Object. I hoped that the PlayRegion, i.e. the data region of a file in the ENTRYWAVEDATA Region simply contains the WMA DATA OBJECT for a wma file, but this is not the case. As you already noticed:

All WMA files start with 0500, 0600, or 0700. The ones that are obviously music actually have the first 17 bytes match

Unfortunately, these bytes seem to have nothing to do with the header bytes of typical .wma-Data Packets

So, can anyone give hints on the Data Regions for wma data in the .xwb (especially the meaning of thos 0500, 0600, 0700 at the beginning) or/and the Data Packets in .wma files...? Please !

PS: Sorry for the long post!
PS2: If anyone needs help on .XSB (Soundbank) files - I've 'decrypted' most of the file format. But a complete description would result in a posting at least twice as long as this one
PS3:

The compression itself could be as simple as zlib inflate

Unfortunately not, I'm afraid. It's rather like the mp3 compression:
"The WMA format uses a lossy compression algorithm. It is a transform codec that uses a psychoacoustics model and employs the modified discrete cosine transform (MDCT) to process the WMA bit stream."

[Liandril]

@FordGT90Concept: In case you ever look into this thread again: could you re-upload the .xwb file? It's no longer on speedyshare. I think, I succeeded in extracting the wma streams . aluigi already mentioned the key to the solution: xWMA. First I tried to transform the wma streams from my .XWBs to valid .wma files...rather impossible, because there are so many undocumented entries/codec specific data in the WMA file format. xWMA is a lightweight wrapper for wma bitstreams and has (especially compared to .wma) a really easy file format. The only difficulties were to calculate the correct values for AvgBytesPerSec, BlockAlign and the DPDS chunk from the infos given in the .xwb file.
The resulting xWMA file can be converted to playable .wav files using Microsofts xwmaEncode. This worked for my xwb files

[GodOfWar]

could you re-upload the .xwb file?

i can upload files you need files from x360 or PC version?

[Liandril]

Thanks, GodOfWar.. so far, I only worked with PC versions... so the PC version would be better. So far, my tool supports only new versions of .xwb (Format Version 42-44), so it will probably not work with the Saints Row 2 .xwb files. But if I find the time, I'll try to adjust it.

[Liandril]

Hm, I'm still looking for a re-upload of the .xwb (PC version) mentioned above. So if anybody has it.. I'd like to test, if my tool is able to extract the xwma files to wav...

[IdolNinja]

Hm, I'm still looking for a re-upload of the .xwb (PC version) mentioned above. So if anybody has it.. I'd like to test, if my tool is able to extract the xwma files to wav...

I've hosted the mus 420.xwb from Saints Row 2 on my site for you:
http://finalack.com/Idol/mus_420.xwb.rar

I release and maintain the Gentlemen of the Row mod for Saints Row 2 PC, and was recently able to get custom missions scripted and working. I would love to be able to add my own sound banks for them, in addition to extracting the current ones if possible.

Hope to hear some good news with your tool, and would be happy to host any other audio files you might need. Just say the word.

More Saints Row 2 modding info here if anyone's interested:
http://idolninja.com
http://community.saintsrow.com/forums/topic/43953

EDIT:
I'm not sure if this helps, but the xml table files in the game reference the banks in the following format:

Code: Select all

    <Music_Set>
        <Name>1036fou_A</Name>
        <Tracks>
            <Music_Set>
                <Track>
                    <Filename>BarringtonLevy_HereICome.wav</Filename>
                    </Track>
                <Play_Time>3.44</Play_Time>
                </Music_Set>
            <Music_Set>
                <Track>
                    <Filename>BeanieMan_WhoAmI.wav</Filename>
                    </Track>
                <Play_Time>3.16</Play_Time>
                </Music_Set>
            <Music_Set>
                <Track>
                    <Filename>BornJamaicans_BoomShakATack.wav</Filename>
                    </Track>
                <Play_Time>4.04</Play_Time>
                </Music_Set>
            <Music_Set>
                <Track>
                    <Filename>BujuBanton_HeyBoy.wav</Filename>
                    </Track>
                <Play_Time>2.18</Play_Time>
                </Music_Set>
            </Tracks>
        <Play_Order>In Order</Play_Order>
        <Min_Dist>0.5</Min_Dist>
        <Max_Dist>30.0</Max_Dist>
        <Volume>0.55</Volume>
        <_Editor>
            <Category>Radio Stations:FOUR-20 103.6</Category>
            </_Editor>
        <AudioBanks>MUS 420</AudioBanks>
        </Music_Set>

The <AudioBanks> element entry of MUS 420 is a reference to audio_banks.xtbl's Name element as defined in the TableDescription at the end of the file. The Description reads: "Required field if the music comes from XACT... Leave unchecked if it comes from the volition music crunch tool."

The entry in audio_banks.xtbl:

Code: Select all

    <AudioBanks>
        <Name>MUS 420</Name>
        <Streaming>
            <SectorRead>128</SectorRead>
            </Streaming>
        <_Editor>
            <Category>Music:Radio Stations</Category>
            </_Editor>
        </AudioBanks>

The TableDescription for the elements in the audio_bank.xtbl is:

Code: Select all

<TableDescription>
    <Name>AudioBanks</Name>
    <Type>TableDescription</Type>
    <Display_Name>Audio Banks</Display_Name>
    <Element>
        <Name>Name</Name>
        <Type>String</Type>
        <Display_Name>Bank Names</Display_Name>
        </Element>
    <Element>
        <Name>Streaming</Name>
        <Type>Element</Type>
        <Description>Every cue from this bank is streamed when needed.</Description>
        <Required>false</Required>
        <Element>
            <Name>SectorRead</Name>
            <Type>Int</Type>
            <Display_Name>Sector Read</Display_Name>
            <Description>The size of a DVD sector is 2048 bytes. The optimal DVD size is a multiple of 16 (1 DVD block = 16 DVD sectors).</Description>
            <Default>16</Default>
            <MinValue>16</MinValue>
            <MaxValue>512</MaxValue>
            </Element>
        </Element>
    <Element>
        <Name>Managed</Name>
        <Type>Element</Type>
        <Description>The wavebank is streamed into memory when needed. Effectively causing it to be always in memory.</Description>
        <Required>false</Required>
        </Element>
    </TableDescription>

I have updated the hosted mus_420.xwb.rar with both table files for reference.