Stargunner DLT compression

[Malvineous]

Hi all,

Following on from this post back in 2006 I had another look at the Stargunner compression format. It looks tantalisingly simple, but alas I'm struggling somewhat.

I've uploaded a couple of files (compressed and decompressed versions) so if anyone is willing to take a quick look that would be great.

From what I've made out so far, the files seem to be laid out like this:

- "PGBP" signature
- uint32 for decompressed size
- uint16 for some kind of "chunk size" (in smaller files is equal to the number of bytes until EOF)
- dictionary/lookup table, often until 00 00 which is frequently the last entry in the dictionary table (no idea how the dictionary size is set)
- another uint16 chunk size for the data (in smaller files is again equal to the number of bytes until EOF)
- actual data, with codes referring back to the dictionary. The FF code is often replaced with 00 00, which seems to coincide with 00 00 being the last bytes in the dictionary
- with larger files, at the end of the chunk the next chunk begins, seemingly with a new dictionary

If anyone has any insights it would be greatly appreciated!

[Malvineous]

Looking at this further, it seems there are actually two variants to the compression, but they all seem to be based around byte pair encoding. Focusing on one for the moment (the one in the files I posted above), it seems to be roughly in this structure (using data from title.mod as an example)

Code: Select all

50 47 42 50  // "PGBP" header
DC 20 05 00  // decompressed size
8C 02        // chunk size
FE 7F        // dictionary signature (different format follows if != 0x7FFE)
B0           // first code -1 (B0 means first code is B1)
F9 FD        // B1 expands to these two bytes (which in this case, each further expand to other bytes)
17           // this many byte pairs follow (although not always)
F9 F5        // B2 expands to these
ED DD        // B3 expands to these
B4 0A        // B4 expands to these (but maybe not as this is infinitely recursive)
...
FE 01        // length of data subchunk
B1           // Expanded as per code B1 above
53 74 61 72  // Passed through unchanged
...

This doesn't quite work for title.mod though (has some rubbish mixed in with the expanded data), but it does successfully decompress smaller files like menu.pal. Byte pair encoding only seems to use bytes that aren't in the source data as codewords, however I'm not sure how to tell from the dictionary in these files which byte values are codewords and which aren't.

[aluigi]

I gave a quick scan with my QuickBMS compression scanner but found nothing

[Malvineous]

Thanks for investigating.

I've had an e-mail from someone who disassembled the algorithm and coverted it to C. If anyone is interested, I will be documenting it over the coming weeks on the Game Modding Wiki and of course implementing it in Camoto/libgamearchive.