The Last Express HPF files

[pythonnoob]

Hello,

I am a complete noob at this, but here goes:

Is anyone able to take see if they can make any sense of the *.HPF files that contain the data for the 1997 game The Last Express?

As far as I kinow, no one has ever been able to extract data from the files, although that could be because the game flopped commercially. The file that I am attaching is a "cut" version of the 30 MB file HD.HPF (the file that contains the data stored on the hard disk). The rest of the game's data is stored on the game's 3 CD's and run from there. On each cd there is one HPF file about 500-600 MB in size, named CD1.HPF, CD2.HPF, etc.

As well, the game's demo (42 MB) can be downloaded from here: http://lastexpress.markmoran.net/download/expdemo.zip

You may not have much luck, as I believe that because of the radically different game approach (real-time as in 24), the game uses its own custom programming language.

I hope that someone will be able to figure out this format, as I would love to see what is inside after all these years.

[Deniz Oezmen]

I hope that someone will be able to figure out this format, as I would love to see what is inside after all these years.

Well, the archive structure seems to be quite straightforward. Try the attached program; it worked pretty well with the demo file.

You probably will not be able to do much with the extracted files, however. They have custom formats, so we will have to figure out those as well ...

Code: Select all

HPFExt.exe archive.hpf

[Deniz Oezmen]

Okay, here is a converter for BG background files as well ... now I need to refrain from disassembling for a while.

Code: Select all

BG2BMP.exe *.bg

[pythonnoob]

Wow. I am speechless.

I have been searching for a way to look at the files for years, and I post here and you're able to write programs that can extract and look at some of the data in a matter of hours. Isn't the internet a wonderful thing?

I am extremely gratified to you Deniz. Thank you so much!

Now, if I could just figure out how to listen to the SND files...

[Deniz Oezmen]

Isn't the internet a wonderful thing?

It can be, yes.

I am extremely gratified to you Deniz. Thank you so much!

Now, if I could just figure out how to listen to the SND files...

You're welcome.

I have already started looking at the SND files; I assume that they are compressed with a customized ADPCM variant -- applying the standard 4-bit IMA ADPCM decompression scheme gives somewhat audible results. I cannot promise anything, though, since I do not have enough time to work consistently at this project.

[pythonnoob]

I cannot promise anything, though, since I do not have enough time to work consistently at this project.

No problem. Whatever you can manage is much appreciated. Thanks again.

[Mr.Mouse]

W00t, Deniz strikes again! D

[Deniz Oezmen]

This might be a fine time for a third strike.

The files were indeed compressed by an IMA ADPCM variant as I suspected; I just was too blind to see the correct parameters at first. The attached program should convert SND files to standard wave files -- at least it worked correctly for the demo. (Though two seemingly corrupt files were packed into the demo archive for some reason.)

Code: Select all

SND2WAV *.snd

Note that there might be minor differences between the output of this program and the sound output the game generates. As the game executable uses a strangely customized decompression variant -- which I do not feel inclined to disassemble to its full extent -- I simply used the standard scheme instead. However, the differences should usually be so small as to go unnoticed by the human ear. (We're talking about something like two levels of deviance on a 65536 level scale here ...)

Anyway, let me know if this works or if there are still any problems.


Have fun,

Deniz

[Deniz Oezmen]

... and let's just throw in something unspectacular for a change. This here extracts the text from SBE "subtitle" files for easier reading. I assume the values I have called "start" and "stop" to specify the point in time and duration the text is displayed on screen while playing a sound file, but this might be a wrong guess.

Code: Select all

SBE2TXT *.sbe

[Savage]

[pythonnoob]

You are da man, Deniz! I never would have imagined that I'd be able to look into all of this stuff. Thank you so much again!

I assume the values I have called "start" and "stop" to specify the point in time and duration the text is displayed on screen while playing a sound file

Listening to some of the soundfiles (hurray!), it seems that a conversation between multiple characters is stored in one audio stream, so "start" and "stop" would probably tell the subtitles at what point in the audio stream to appear and disappear.

If you have some spare time, Deniz, could you do one more small check for me please?

I was wondering if you could look at the NIS and SEQ files to see if there's any recognizable data in them? I believe NIS stands for non-interactive-scene, which would mean that they store the data for cutscenes. The NIS files are quite large (compared to some of the other types), so I assume that there must be some multimedia data in there...and I guess that SEQ stands for "sequence" or something...

Anyway, you have made my day (yes!) and keep up the good work helping people!

[Deniz Oezmen]

so "start" and "stop" would probably tell the subtitles at what point in the audio stream to appear and disappear.

That's what I suspected. The only remaining question would be what unit of time these values are expressed in. But on the other hand, that is probably not important anyway ...

I was wondering if you could look at the NIS and SEQ files to see if there's any recognizable data in them?

Way ahead of you.

You are most probably right about the cutscenes. By the way: Could you check whether the music or sounds of cutscenes you know can be found in SND files? If not, it might be contained within the NIS data as well.

I suspect the SEQ files to contain the character animation frames, which are probably once again compressed. This time, however, I do not exactly know what to look for -- I have some structural ideas and have located some related parts of code within the demo .exe, but I cannot see the purpose of most of the data yet.


Maybe somebody else could support me here? I have somehow seen enough disassemblies for the upcoming three months ...

What I know so far:

- SEQ files start with an int32 "counter" value (probably for the number of frames). Afterwards, there are (at least) as many 68 byte structures as the "counter" value indicates. Within these structures, at least one int32 value (from the fifth byte on) is a file offset value for the respective frame. Afterwards, there are some int32 or int16 values that look suspiciously like width and height ...
- The frame data of SEQ files is probably uint16-formatted. A "conversion routine" in the .exe, located at 0x00410420, indicates this.
- The routine which probably handles SEQ files is located at 0x00410F40.
- NIS files also start with this int32 "counter", which is followed by as many 8 byte structures, the purpose of which are unknown to me.

Well ... that's about it. Not enough for me to see anything useful yet. Anybody?

Anyway, you have made my day (yes!) and keep up the good work helping people!

You're welcome. You got me hooked up on these formats ...

[pythonnoob]

By the way: Could you check whether the music or sounds of cutscenes you know can be found in SND files? If not, it might be contained within the NIS data as well.

I looked, and none of the cutscene sounds are in any SND files where they should be. FYI, the first three letters of a SND filename is the name of the character that the file involves, e.g. ANN is Anna, ALX is Alexei, and so forth. Because there were both DOS and Windows versions of the game, I assume that they had to stick to 8.3 filenames.

I suspect the SEQ files to contain the character animation frames, which are probably once again compressed.

I suspect that you are right. I'm not sure if you ran the demo I sent you, but the game uses a "rotoscope" effect on characters, which use a very limited colour palette (I think there are no more than 16 colours on any character). As well, the game uses very little FMV, instead using a sort of "slideshow" method for cutscenes. As such, most cutscenes might not use more than 10 frames for the whole thing. While there is FMV, it is very rare, and runs at about 15 frames per second, with no synchronized audio...as well, I suspect that the animations have alpha channels/transparency, as the same animations can occur in a variety of locations.

Is it possible that the NIS files are archives that contain more SEQ, SND and BG files?

That's just me rambling. Thanks, and keep up the good work!

[Deniz Oezmen]

I looked, and none of the cutscene sounds are in any SND files where they should be.

I see. Then we might find sound data within NIS files as well. Except maybe if they contain the audio-less FMV sequences you mentioned.

I suspect that you are right. I'm not sure if you ran the demo I sent you, but the game uses a "rotoscope" effect on characters, which use a very limited colour palette (I think there are no more than 16 colours on any character).

Yes, I ran the demo and saw what you mean. The largely uniform colouring makes compression easy -- though I am not entirely sure how the colours are represented within the sequence files. I do not even know if the data is always stored as "full frames" or if the format makes use of delta frames as well ...

as well, I suspect that the animations have alpha channels/transparency, as the same animations can occur in a variety of locations.

They will probably just have one special colour defined as transparent. Or are there any "real" transparency effects, such as half-lucent objects, in the game?

Is it possible that the NIS files are archives that contain more SEQ, SND and BG files?

I would not describe them as archives, but rather as a form of proprietary video format; at least that is what I suspect them to be.

Alas, there is much guesswork involved ...

[ltronic]

Hello Guys !

Bumping this thread, since I'm trying to rewrite the whole game engine in modern, cross-platform C/SDL.
Thanks to the great tools posted here, I've been able to extract and convert most of the data :
- SND2WAV ==> Ogg files
- BG2BMP ==> Png24 optimized files

I'm now digging into the following file formats mentioned in that thread : NIS, SEQ.
I've also found some LNK files while decompressing the HPF archives.

Has anyone progressed on the NIS/SEQ files ? Any clue about the LNK files ?
I've start to disassemble the whole game (ExpressW.exe) under IDA Pro.
Also tried to produce pseudo-C file with Hex-Rays, but does not seem to give useful results so far.

Anyone wanting to help ?

Cheers,