There's an application (towav.exe) that attempts to convert XWB files to WAV. On one of the XWB files in SR2 (MUS MIX), it only manages to extract 8 files when there are 101. I finally figured out why: the other 93 appear to be compressed but I can't figure out what kind of compression it is (no header info) or how to go about uncompressing them.
XWB is a Microsoft XNA Wavebank format so the logical conclusion would be something in System.IO.Compression. I tried them both as well as ICSharpZipLib Zip.Compress.Inflate and all of them fail (most likely due to the lack of header.
I'm running out of ideas.
Every file appears to start with a int16 value of 5-7. I find this odd and it is probably important. I attached a zip with 5 of the raw files (WAVE header removed). If I play it with the WAVE header, it is just static.
Any incite would be greatly appriciated. So close...but yet so far.
Important information: this site is currently scheduled to go offline indefinitely by end of the year.
Saints Row 2 XWB compressed wave
-
- advanced
- Posts: 47
- Joined: Thu Apr 15, 2010 7:16 am
- Has thanked: 3 times
- Been thanked: 13 times
Saints Row 2 XWB compressed wave
You do not have the required permissions to view the files attached to this post.
- aluigi
- VVIP member
- Posts: 1916
- Joined: Thu Dec 08, 2005 12:26 pm
- Location: www.ZENHAX.com
- Has thanked: 4 times
- Been thanked: 664 times
- Contact:
Re: Saints Row 2 XWB compressed wave
why don't you use unxwb?
it tells any information found about the file and automatically builds the needed headers to play/convert it in a second moment.
it tells any information found about the file and automatically builds the needed headers to play/convert it in a second moment.
-
- advanced
- Posts: 47
- Joined: Thu Apr 15, 2010 7:16 am
- Has thanked: 3 times
- Been thanked: 13 times
Re: Saints Row 2 XWB compressed wave
I didn't know it existed because it got no hits on Google. I put it on the XBOX_XWB3#Compatible_Programs so more people will find it.
However, it didn't work. All the files I'm having trouble with (the compressed lot) your tool dumps what I attached (a short, followed by compressed data) with a WMA extension. Obviously, WMP can't play it because it doesn't have the appriorate header information.
However, it didn't work. All the files I'm having trouble with (the compressed lot) your tool dumps what I attached (a short, followed by compressed data) with a WMA extension. Obviously, WMP can't play it because it doesn't have the appriorate header information.
- aluigi
- VVIP member
- Posts: 1916
- Joined: Thu Dec 08, 2005 12:26 pm
- Location: www.ZENHAX.com
- Has thanked: 4 times
- Been thanked: 664 times
- Contact:
Re: Saints Row 2 XWB compressed wave
uhmmm so they get identified as wma files.
indeed they don't look like xma1 or xma2 files because xmaencode can't decode them.
they could be msadpcm but I doubt because this adpcm needs a particular value located in the xwb archive and after a scanning I have found no value good to hear a clean sound so it's not msadpcm too.
is this a x360 game?
could it be xwma? mah
everything I know about this matter is already implemented in unxwb so I have out of ideas
indeed they don't look like xma1 or xma2 files because xmaencode can't decode them.
they could be msadpcm but I doubt because this adpcm needs a particular value located in the xwb archive and after a scanning I have found no value good to hear a clean sound so it's not msadpcm too.
is this a x360 game?
could it be xwma? mah
everything I know about this matter is already implemented in unxwb so I have out of ideas
Re: Saints Row 2 XWB compressed wave
yeah, is 360 game.the big archive called real_music.vpp_xbox2 (1.05gb) contains all music from game (files have .mus extension). tool for PC version didn't work with x360 version =(aluigi wrote:is this a x360 game?
-
- advanced
- Posts: 47
- Joined: Thu Apr 15, 2010 7:16 am
- Has thanked: 3 times
- Been thanked: 13 times
Re: Saints Row 2 XWB compressed wave
It was extracted from music1.vpp_pc to music4.vpp_pc from the PC (Steam) version of Saints Row 2. The xwb format appears "normal" except for a few things:
Header:
CompressionInfo is an array of ints. It counts up from zero except where it matches FileRecords for a compessed file (e.g. FileRecord[1] = compessed, CompressionInfo[1] = -1). How quickly it counts up seems to be determined by the size of the file. On MUS MIX.xwb, the Length of Compression Info is over 6000 but the counting up ends around 600. There are only 101 file records. No -1s appear after the 101 index. It appears that, so long as the value isn't -1, it doesn't mean anything.
Wave Bank Info:
Those Unknown2 don't seem to match up to what appears in the unxwb source (especially 2 and 3). I haven't been able to sort what they are. It might be that Unknown2 is a start time (or ticks from an epoch) and Unknown3 is a stop time.
I uploaded the smallest XWB with the CompressionInfo array here. There are no files in this particular XWB that are not compressed:
http://www.speedyshare.com/files/22145167/MUS_420.xwb
When I say compressed, I mean these files are way too small to be waves. MUS MIX.xwb has 8 uncompressed files and some are north of 20 MiB in size. The compressed files rarely exceed 1 MiB.
I am not certain the format flag (WAV/WMA) is correct. There's no way to be certain until the compression is dealt with.
I haven't tried DeflateStream or InflateStream without the first short but, on examining some other files, it appears that short isn't uniform across all XWB files.
Header:
Code: Select all
public string IdentifyingChars = "WBND";
public int Unknown44;
public int Unknown42; // This is odd -- no idea what this 42 is for. Maybe it is the original version number and they incremented it to 44 for their changes.
public int WaveBankInfoOffset;
public int WaveBankInfoLength;
public int FileRecordsOffset;
public int FileRecordsLength;
public int CompressionInfoOffset; // This is somewhat odd
public int CompressionInfoLength; // This is very odd.
public int Unknown1Offset; // Always 0
public int Unknown1Length; // Always 0
public int FileDataOffset;
public int FileDataLength;
Wave Bank Info:
Code: Select all
public int Flags;
public int FileCount;
public string Name; // 64 bytes
public int FileRecordLength;
public int FileNameLength;
public int FileDataOffset;
public int Unknown1; // Always 0
public int Unknown2;
public int Unknown3;
I uploaded the smallest XWB with the CompressionInfo array here. There are no files in this particular XWB that are not compressed:
http://www.speedyshare.com/files/22145167/MUS_420.xwb
When I say compressed, I mean these files are way too small to be waves. MUS MIX.xwb has 8 uncompressed files and some are north of 20 MiB in size. The compressed files rarely exceed 1 MiB.
I am not certain the format flag (WAV/WMA) is correct. There's no way to be certain until the compression is dealt with.
I haven't tried DeflateStream or InflateStream without the first short but, on examining some other files, it appears that short isn't uniform across all XWB files.
-
- advanced
- Posts: 47
- Joined: Thu Apr 15, 2010 7:16 am
- Has thanked: 3 times
- Been thanked: 13 times
Re: Saints Row 2 XWB compressed wave
All WMA files start with 0500, 0600, or 0700. The ones that are obviously music actually have the first 17 bytes match. It would make sense if their uncompressed headers matched therefore causing a large chunk of the compressed data match. So I guess my question is this: What types of compression support headerless inflation? I know zlib has compress() and uncompress() but I doubt it can handle the ~2 MiB some of these files are (I don't see any indication there are parts).
Edit: I tried zlib and zlib1 uncompress(dest, destlen, src, srclen) and both return Z_DATA_ERROR.
Anyone have any ideas I could try?
Maybe this thread should be moved to the compressed forum.
Edit: I tried zlib and zlib1 uncompress(dest, destlen, src, srclen) and both return Z_DATA_ERROR.
Anyone have any ideas I could try?
Maybe this thread should be moved to the compressed forum.
-
- advanced
- Posts: 47
- Joined: Thu Apr 15, 2010 7:16 am
- Has thanked: 3 times
- Been thanked: 13 times
Re: Saints Row 2 XWB compressed wave
This new section of the XWB (apparently unique to Saints Row 2) I call "CompressionInfo" has been described:
For example, compress_array_offset_from_end_of_this_array could contain:
Reading the offsets would turn that into an array:
File 0:
File 1: not compressed = null
File 2:
Basically, what I know:
-The file's total deflated size.
-How many inflated sections there are.
-The size of each inflated section.
-Zipping a compressed file results in virtually zero gain.
-Appears to be headerless. If it has a header, it is short.
-Every deflated section appears to be exactly 929 or 1487 bytes in length (depends on file).
What I don't know:
-The compression method used.
Name that compression!
Code: Select all
int[FileCount] compress_array_offset_from_end_of_this_array; // if -1, File is not compressed
// at each offset
int count;
int[count] uncompressed_begin_offset;
Code: Select all
0
-1
12
File 0:
Code: Select all
count = 3
value = { 20, 40, 60 }
File 2:
Code: Select all
count = 5
value = { 30, 60, 90, 120, 150 }
-The file's total deflated size.
-How many inflated sections there are.
-The size of each inflated section.
-Zipping a compressed file results in virtually zero gain.
-Appears to be headerless. If it has a header, it is short.
-Every deflated section appears to be exactly 929 or 1487 bytes in length (depends on file).
What I don't know:
-The compression method used.
Name that compression!
-
- advanced
- Posts: 47
- Joined: Thu Apr 15, 2010 7:16 am
- Has thanked: 3 times
- Been thanked: 13 times
Re: Saints Row 2 XWB compressed wave
I think it uses a combination of compression and encryption. That is, it compresses x amount of data and then losslessly encrypts it to y block size (1487 and 929 for sure). The compression itself could be as simple as zlib inflate (the SR2 exe has references to it) but I have no idea on the encryption.
I extracted the list of most of the strings from the SR2 executable and they are attached. Maybe it has some clues as to what is happening in the XWB files.
I extracted the list of most of the strings from the SR2 executable and they are attached. Maybe it has some clues as to what is happening in the XWB files.
You do not have the required permissions to view the files attached to this post.
Re: Saints Row 2 XWB compressed wave
Hello and sorry - I know, the last posting in this thread was 3 months ago, but it's the best thread on the subject XWB that I found so far - so I hope, the original posters are still reading. And sorry for my English!
I'm writing a tool (a perl script actually) in order to insert sounds into an existing german game, that uses .xsb/.xwb files. And additionally, the tool is supposed to extract waves (nearly all of them are WMA) from the wavebanks. I wrote it, because unxwb can't (afaik) write valid/playable .wma files... and other tools (XWB_Extractor, EkszBox,...) weren't able to handle the .xwb files of the game at all. This is no wonder, because backward compatibility is not really needed in XACT(=the tool that creates .xwb/.xsb files) and therefor, Microsoft can change the format with each version of the .xwb/.xsb file format. So "old" extraction tools won't work with "new" xsb/xwbs.
First of all: Congratulations, FordGT90Concept - you did a very good job in "demystifying" the .xwb structure , but (sorry, I have to say that) you can get detailed xwb-info very easy, because the .xwb format (unfortunately NOT the xsb format) is documented in Microsofts DirectX SDK. Just download+install the SDK and search for the header file "xact3wb.h" (in older SDKs it's "xact2wb.h"). As a summary: so your "Unknown44" means, the .xwb file was created with tool(=XACT) version 44, and "Unknown42" is the xwb file format version.
Actually, there COULD be up to 5 Regions, but all .xwbs I saw so far had only 3 or 4, namely (+appearing in this order in the file): 1. WAVEBANKDATA, 2. ENTRYMETADATA, (3. SEEKTABLES),(4.ENTRYNAMES), 5. ENTRYWAVEDATA
Thus, a Header of a .xwb with 5 Region-entries is 3*4(for Signature,..) + 5*8(Regions)=52 B long
1. WAVEBANKDATAdwFlags (= Bank flags) = info about: is it a in-memory or streaming wavebank; are EntryNames and/or SeekTables inlcuded?....
dwEntryCount = number of wave files in the wavebank
2. ENTRYMETADATA
In the ENTRYMETADATA region, there is for each wave file a WAVEBANKENTRY which contains the following infos for the wave file:
Flags (has Loops; Enable stream read-ahead)
Duration(e.g. a 10 second long wave with sampling rate 44100 Hz has Duration=441000)
WAVEBANKMINIWAVEFORMAT (Typ (PCM,ADPCM,XMA,WMA); number of Channels; sampling rate (e.g.44100Hz); Align; 8or16bit PCM))
PlayRegion (Region within the wave data segment (=ENTRYWAVEDATA) that contains this entry)
LoopRegion
5. ENTRYWAVEDATA
the actual data as referenced by PlayRegion of the WAVEBANKENTRIES (/End-of-summary )
So we seem to have the same problem, FordGT90Concept: we both try to extract .wma-files from .xwb wavebanks. But with different strategies: you try to decompress the wma file data in order to write them to .wav files, I try to simply write the compressed data and create appropriate WMA-Headers in order to have .wma files as a result.
This is what I did so far: I looked into the (very long and not very usefull) ASF/WMA-Specification.
A .wma file contains at least 4 Objects: FileProperties Object,StreamProperties Object, Header Extension Object and DATA OBJECT (consisting of Data Packets, each Data Packet has some header/info bytes). With the infos from WAVEBANKENTRY of the file and xact3wb.h, I'm (almost) able to write the first 3 Objects. The biggest problem is the WMA Data Object. I hoped that the PlayRegion, i.e. the data region of a file in the ENTRYWAVEDATA Region simply contains the WMA DATA OBJECT for a wma file, but this is not the case. As you already noticed:
So, can anyone give hints on the Data Regions for wma data in the .xwb (especially the meaning of thos 0500, 0600, 0700 at the beginning) or/and the Data Packets in .wma files...? Please !
PS: Sorry for the long post!
PS2: If anyone needs help on .XSB (Soundbank) files - I've 'decrypted' most of the file format. But a complete description would result in a posting at least twice as long as this one
PS3:
"The WMA format uses a lossy compression algorithm. It is a transform codec that uses a psychoacoustics model and employs the modified discrete cosine transform (MDCT) to process the WMA bit stream."
I'm writing a tool (a perl script actually) in order to insert sounds into an existing german game, that uses .xsb/.xwb files. And additionally, the tool is supposed to extract waves (nearly all of them are WMA) from the wavebanks. I wrote it, because unxwb can't (afaik) write valid/playable .wma files... and other tools (XWB_Extractor, EkszBox,...) weren't able to handle the .xwb files of the game at all. This is no wonder, because backward compatibility is not really needed in XACT(=the tool that creates .xwb/.xsb files) and therefor, Microsoft can change the format with each version of the .xwb/.xsb file format. So "old" extraction tools won't work with "new" xsb/xwbs.
First of all: Congratulations, FordGT90Concept - you did a very good job in "demystifying" the .xwb structure , but (sorry, I have to say that) you can get detailed xwb-info very easy, because the .xwb format (unfortunately NOT the xsb format) is documented in Microsofts DirectX SDK. Just download+install the SDK and search for the header file "xact3wb.h" (in older SDKs it's "xact2wb.h"). As a summary:
Code: Select all
typedef struct WAVEBANKHEADER {
DWORD dwSignature; // File signature "WBND"
DWORD dwVersion; // Version of the tool that created the file
DWORD dwHeaderVersion; // Version of the file format
WAVEBANKREGION Segments[WAVEBANK_SEGIDX_COUNT]; // Segment lookup table
}
Code: Select all
typedef struct WAVEBANKREGION {
DWORD dwOffset; // Region offset, in bytes.
DWORD dwLength; // Region length, in bytes.
}
Thus, a Header of a .xwb with 5 Region-entries is 3*4(for Signature,..) + 5*8(Regions)=52 B long
1. WAVEBANKDATA
Code: Select all
typedef struct WAVEBANKDATA {
DWORD dwFlags; // Bank flags
DWORD dwEntryCount; // Number of entries in the bank
.....rest: see xact3wb.h }
dwEntryCount = number of wave files in the wavebank
2. ENTRYMETADATA
In the ENTRYMETADATA region, there is for each wave file a WAVEBANKENTRY which contains the following infos for the wave file:
Flags (has Loops; Enable stream read-ahead)
Duration(e.g. a 10 second long wave with sampling rate 44100 Hz has Duration=441000)
WAVEBANKMINIWAVEFORMAT (Typ (PCM,ADPCM,XMA,WMA); number of Channels; sampling rate (e.g.44100Hz); Align; 8or16bit PCM))
PlayRegion (Region within the wave data segment (=ENTRYWAVEDATA) that contains this entry)
LoopRegion
5. ENTRYWAVEDATA
the actual data as referenced by PlayRegion of the WAVEBANKENTRIES (/End-of-summary )
WMA (quote from xact3wb.h: static const DWORD aWMABlockAlign[] = { 929, 1487,...}FordGT90Concept wrote: -Every deflated section appears to be exactly 929 or 1487 bytes in length (depends on file)
Name that compression!
So we seem to have the same problem, FordGT90Concept: we both try to extract .wma-files from .xwb wavebanks. But with different strategies: you try to decompress the wma file data in order to write them to .wav files, I try to simply write the compressed data and create appropriate WMA-Headers in order to have .wma files as a result.
This is what I did so far: I looked into the (very long and not very usefull) ASF/WMA-Specification.
A .wma file contains at least 4 Objects: FileProperties Object,StreamProperties Object, Header Extension Object and DATA OBJECT (consisting of Data Packets, each Data Packet has some header/info bytes). With the infos from WAVEBANKENTRY of the file and xact3wb.h, I'm (almost) able to write the first 3 Objects. The biggest problem is the WMA Data Object. I hoped that the PlayRegion, i.e. the data region of a file in the ENTRYWAVEDATA Region simply contains the WMA DATA OBJECT for a wma file, but this is not the case. As you already noticed:
Unfortunately, these bytes seem to have nothing to do with the header bytes of typical .wma-Data PacketsFordGT90Concept wrote: All WMA files start with 0500, 0600, or 0700. The ones that are obviously music actually have the first 17 bytes match
So, can anyone give hints on the Data Regions for wma data in the .xwb (especially the meaning of thos 0500, 0600, 0700 at the beginning) or/and the Data Packets in .wma files...? Please !
PS: Sorry for the long post!
PS2: If anyone needs help on .XSB (Soundbank) files - I've 'decrypted' most of the file format. But a complete description would result in a posting at least twice as long as this one
PS3:
Unfortunately not, I'm afraid. It's rather like the mp3 compression:FordGT90Concept wrote:The compression itself could be as simple as zlib inflate
"The WMA format uses a lossy compression algorithm. It is a transform codec that uses a psychoacoustics model and employs the modified discrete cosine transform (MDCT) to process the WMA bit stream."
Re: Saints Row 2 XWB compressed wave
@FordGT90Concept: In case you ever look into this thread again: could you re-upload the .xwb file? It's no longer on speedyshare. I think, I succeeded in extracting the wma streams . aluigi already mentioned the key to the solution: xWMA. First I tried to transform the wma streams from my .XWBs to valid .wma files...rather impossible, because there are so many undocumented entries/codec specific data in the WMA file format. xWMA is a lightweight wrapper for wma bitstreams and has (especially compared to .wma) a really easy file format. The only difficulties were to calculate the correct values for AvgBytesPerSec, BlockAlign and the DPDS chunk from the infos given in the .xwb file.
The resulting xWMA file can be converted to playable .wav files using Microsofts xwmaEncode. This worked for my xwb files
The resulting xWMA file can be converted to playable .wav files using Microsofts xwmaEncode. This worked for my xwb files
Re: Saints Row 2 XWB compressed wave
i can upload files you need files from x360 or PC version?Liandril wrote:could you re-upload the .xwb file?
Re: Saints Row 2 XWB compressed wave
Thanks, GodOfWar.. so far, I only worked with PC versions... so the PC version would be better. So far, my tool supports only new versions of .xwb (Format Version 42-44), so it will probably not work with the Saints Row 2 .xwb files. But if I find the time, I'll try to adjust it.
Re: Saints Row 2 XWB compressed wave
Hm, I'm still looking for a re-upload of the .xwb (PC version) mentioned above. So if anybody has it.. I'd like to test, if my tool is able to extract the xwma files to wav...
Re: Saints Row 2 XWB compressed wave
I've hosted the mus 420.xwb from Saints Row 2 on my site for you:Liandril wrote:Hm, I'm still looking for a re-upload of the .xwb (PC version) mentioned above. So if anybody has it.. I'd like to test, if my tool is able to extract the xwma files to wav...
http://finalack.com/Idol/mus_420.xwb.rar
I release and maintain the Gentlemen of the Row mod for Saints Row 2 PC, and was recently able to get custom missions scripted and working. I would love to be able to add my own sound banks for them, in addition to extracting the current ones if possible.
Hope to hear some good news with your tool, and would be happy to host any other audio files you might need. Just say the word.
More Saints Row 2 modding info here if anyone's interested:
http://idolninja.com
http://community.saintsrow.com/forums/topic/43953
EDIT:
I'm not sure if this helps, but the xml table files in the game reference the banks in the following format:
Code: Select all
<Music_Set>
<Name>1036fou_A</Name>
<Tracks>
<Music_Set>
<Track>
<Filename>BarringtonLevy_HereICome.wav</Filename>
</Track>
<Play_Time>3.44</Play_Time>
</Music_Set>
<Music_Set>
<Track>
<Filename>BeanieMan_WhoAmI.wav</Filename>
</Track>
<Play_Time>3.16</Play_Time>
</Music_Set>
<Music_Set>
<Track>
<Filename>BornJamaicans_BoomShakATack.wav</Filename>
</Track>
<Play_Time>4.04</Play_Time>
</Music_Set>
<Music_Set>
<Track>
<Filename>BujuBanton_HeyBoy.wav</Filename>
</Track>
<Play_Time>2.18</Play_Time>
</Music_Set>
</Tracks>
<Play_Order>In Order</Play_Order>
<Min_Dist>0.5</Min_Dist>
<Max_Dist>30.0</Max_Dist>
<Volume>0.55</Volume>
<_Editor>
<Category>Radio Stations:FOUR-20 103.6</Category>
</_Editor>
<AudioBanks>MUS 420</AudioBanks>
</Music_Set>
The entry in audio_banks.xtbl:
Code: Select all
<AudioBanks>
<Name>MUS 420</Name>
<Streaming>
<SectorRead>128</SectorRead>
</Streaming>
<_Editor>
<Category>Music:Radio Stations</Category>
</_Editor>
</AudioBanks>
Code: Select all
<TableDescription>
<Name>AudioBanks</Name>
<Type>TableDescription</Type>
<Display_Name>Audio Banks</Display_Name>
<Element>
<Name>Name</Name>
<Type>String</Type>
<Display_Name>Bank Names</Display_Name>
</Element>
<Element>
<Name>Streaming</Name>
<Type>Element</Type>
<Description>Every cue from this bank is streamed when needed.</Description>
<Required>false</Required>
<Element>
<Name>SectorRead</Name>
<Type>Int</Type>
<Display_Name>Sector Read</Display_Name>
<Description>The size of a DVD sector is 2048 bytes. The optimal DVD size is a multiple of 16 (1 DVD block = 16 DVD sectors).</Description>
<Default>16</Default>
<MinValue>16</MinValue>
<MaxValue>512</MaxValue>
</Element>
</Element>
<Element>
<Name>Managed</Name>
<Type>Element</Type>
<Description>The wavebank is streamed into memory when needed. Effectively causing it to be always in memory.</Description>
<Required>false</Required>
</Element>
</TableDescription>