Guild Wars .DAT File

BlackDragon · Post by **BlackDragon** » Wed Oct 24, 2007 2:33 pm

Wow. Good work on this. For some reason, I'd never ran across this project before or I could've helped you guys out.

Oh well.

Do you by any chance have any of the ffna file specs figured out? Could you explain how the "hashes" work? I'll bet that is part of the code that was confusing me. Is that why they loop certain variables byte-by-byte against a table in the exec?

BD

saracen · Post by **saracen** » Wed Oct 24, 2007 5:14 pm

BlackDragon wrote:Wow. Good work on this. For some reason, I'd never ran across this project before or I could've helped you guys out.

Oh well.

Do you by any chance have any of the ffna file specs figured out? Could you explain how the "hashes" work? I'll bet that is part of the code that was confusing me. Is that why they loop certain variables byte-by-byte against a table in the exec?

BD

We have no information about the FFNA files at the moment =( As somebody has suggested in the past, the model files probably use Granny v2 format, I've compared the formats (the Granny website has some models you can download with their demo package) and the formats don't look identical, but then again two granny 3d files didnt look the same at all either.

The only thing that looked the same out of two different granny 3d files was the version number. Everything else is probably compressed, hence no structure looks the same.

As for hashes... theres not much to say. There's a hash for each file, I don't think its based on the files content or filename. I don't think we should even be calling them hashes, they're just unique identifiers, and as i said in my previous post, a unique identifier can have a alias identifer (points to the same file) which just makes GW want to update the file (because it doesn't have it).

The byte to byte lookup is probably comparing what the client has and hasn't got.

Hopefully we can do some trial by error, our emulator can load up any map we want (it just uses the hash - Bah, can we start calling it a unique id?!). Using your extractor I managed to get a few "dev maps" from the DAT file (no idea why they are there!). If i can now make a DAT writer and keep editing a map file we can:

a) see if it loads and spot differences or ...
b) see what lovely, detailed error dump GW gives us back.

We don't have a compressor, and I can't get my head around huffman... so maybe a flag that tells GW the file isn't compressed will do the job.

It might also be interesting to do some map swapping... Halloween Lions arch all year around? Why not!

saracen · Post by **saracen** » Wed Oct 24, 2007 5:21 pm

saracen wrote: We don't have a compressor, and I can't get my head around huffman... so maybe a flag that tells GW the file isn't compressed will do the job.

Oh, and just to add... if that works, a DAT decompressor and saver might be handy for some people. Some people assume that -image decompresses the files in their DAT file... I haven't seen that happen yet - it just gets all the files from the server.

If we can do the flag trick, and decompress everything (resulting in a large file!) it might get some faster load times. Who knows.

(Showoff: Not that my computer needs faster load times, I'm not running Guild Wars on a GameBoy)

Darkstar · Post by **Darkstar** » Wed Oct 24, 2007 8:11 pm

saracen wrote:Yeah, I request files from the server. The GWLP Project hasn't disappeared and I am a staff member, I rarely program for the Login/Game server though. The fileserver was never much of an interest to us, but I got bored a few day's ago and thought I'd write a client.

Glad to hear you guys are still around. Have any new (public) website/forum?

saracen wrote:The GWLP Project is closed source, mainly because it uses encryption (we don't want people exploiting official servers etc.)

But the fileserver isn't encrypted - and I see no reason to keep the protocol specs closed (especially how simple it is), so I will probably release the information soonish

As for other protocols... you're on your own =(

Yes, I figured as much. Sad to hear, but I still have the hope that someday, an open source project will emerge from the depths of the internet ^^ It would be a shame if what happened to Auto Duel would also happen to GW in a year or two...

saracen · Post by **saracen** » Wed Oct 24, 2007 9:37 pm

Darkstar wrote:Have any new (public) website/forum?

http://gwlp.mgcorp.org/

We're mainly on IRC: irc.rizon.net / #gwlp

Darkstar wrote:sad to hear, but I still have the hope that someday, an open source project will emerge from the depths of the internet ^^ It would be a shame if what happened to Auto Duel would also happen to GW in a year or two...

I have no idea what happened to Auto Duel, but if Arenanet had to close its servers for whatever reason, then we'd probably release the protocol specifications. We're not against sharing, we just don't like the idea of bot makers going wild and people attempting more exploits (hint: GW isn't very good at unexpected user input =()

db. · Post by **db.** » Thu Oct 25, 2007 4:11 pm

saracen wrote:
Darkstar wrote:Have any new (public) website/forum?
http://gwlp.mgcorp.org/

We're mainly on IRC: irc.rizon.net / #gwlp

Darkstar wrote:sad to hear, but I still have the hope that someday, an open source project will emerge from the depths of the internet ^^ It would be a shame if what happened to Auto Duel would also happen to GW in a year or two...
I have no idea what happened to Auto Duel, but if Arenanet had to close its servers for whatever reason, then we'd probably release the protocol specifications. We're not against sharing, we just don't like the idea of bot makers going wild and people attempting more exploits (hint: GW isn't very good at unexpected user input =()

it doesnt like derv's either ... or did you get that part fixed already lol.

lej · Post by **lej** » Thu Nov 15, 2007 9:06 am

Contrary to Saracen's earlier statement a file server has become necessary to the GWLP so we wrote one. I think it may be of use to the people working on the data archive format.

The GWLP file server implements the Guild Wars file transfer protocol which is used officially for updating the game. This software is interoperable with the official Guild Wars game client. It can be used for placing custom files in the gw.dat, downgrading/upgrading a Guild Wars installation to a particular version, and modifying existing files.

You can download the server installation package here: http://www.mediafire.com/?5s9byyeho7d EDIT: (Fixed a major bug)

For more information and support visit: http://gwlp.mgcorp.org/forums/viewtopic.php?t=23

Also, the FFNA files the client expects from the file server differ from the FFNA files stored in the data archive. Both in size and content. It seems to me that they have sections that are compressed. I've attached FFNA files compressed from Anet's servers, decompressed from Anet's servers, and from dat to this post if anyone is interested.

saracen · Post by **saracen** » Thu Nov 15, 2007 2:52 pm

lej wrote:Also, the FFNA files the client expects from the file server differ from the FFNA files stored in the data archive. Both in size and content. It seems to me that they have sections that are compressed. I've attached FFNA files compressed from Anet's servers, decompressed from Anet's servers, and from dat to this post if anyone is interested.

I hadn't had time to look, but i just did so now and it seems very easy to figure out. So there will probably be an update soonish.

On another note, i would just like to mention that i was eating an egg sandwich while writing this, and the egg just slipped out and went down my sleeve.

The moral of the story being: l3j, stop posting incomplete stuff and give me a second to have a break from work to look at simple file formats.

tobias · Post by **tobias** » Wed Jan 23, 2008 5:30 pm

First of all, thanks to everyone here for the progress they've made on this.

I've been using Irina's Dat Browser program to browse through the .dat file, specifically at the image and sound files. Unfortunately, the program's a little unwieldy, due to a lack of search functions and being able to only export one file at a time. Being able to dump everything into a Windows folder and browse it there would work a lot better for me.

I saw BoyC's Unpacker program on the XeNTaX wiki:

BoyC wrote:Hi guys, the "anonymus" coder here

http://digitaldynamite.demoscene.hu/pri ... packer.zip

Unfortunately, I'm not sure how to use it (running the .exe opens the program for a split second and then closes it), and as the Wiki entry states I wouldn't know what to do with it if I did get it running, since I don't know how to use command line programs. And given the wiki page is half a year old, I'm not expecting a user-friendly version or a how-to guide any time soon.

Is there anyone here who could explain how to use the Unpacker to dump all the files from the .dat file, or failing that, through some other means that's simpler than right-click -> saving every file individually?

Apologies for this being a noob question in an otherwise intelligent discussion, there's really not many other people so far that have been fooling around with this stuff.

Darkfox · Post by **Darkfox** » Wed Jan 23, 2008 7:06 pm

Guess you would have had to have grown up with DOS. Let me explain...

Usually a command line unpacking tool is ran by putting it in the same folder as the file you want to unpack. Usually it is done like this:

Toolname.exe [name of file you want unpacked]

replacing the thing in the brackets with the name of the file you want unpacked. You could also accomplish this by dragging the .dat file over the program and dropping it (unclicking it) and some programs run this way as well.

tobias · Post by **tobias** » Wed Jan 23, 2008 7:30 pm

Well, dragging the .dat file onto the unpacker works... but all it does is spit out some information about the file (number of entries, etc) and then closes after about a second. It doesn't seem to unpack any files when it runs.

Darkfox · Post by **Darkfox** » Wed Jan 23, 2008 7:31 pm

Then it seems it requires an output desination if it unpacks nothing.

tobias · Post by **tobias** » Wed Jan 23, 2008 7:37 pm

How would I assign a destination, then?

Darkfox · Post by **Darkfox** » Wed Jan 23, 2008 8:00 pm

Likely with a third option like "C:\GWOutput"

But I don't know this program. For all I know it could be as simple as a missing option.

Post by **Dinoguy1000** » Sat Jan 26, 2008 6:31 pm

Tobias, I'm assuming you're not trying to run the unpacker from within DOS? If not, you can open up a DOS session by clicking Start -> Run... and running cmd (if that doesn't work, try command). When you get the session running, you should navigate to the folder where you saved the unpacker exe (type in cd "full directory"). From there, type in the unpacker filename followed by /?, for instance, gw_unpacker /?. This should spit out some text. Copy the text and post it here and I'll be able to help you further.

If any of that didn't make sense, blame me and feel free to ask for a clarification.