Page 1 of 19
[Request] Guild Wars 2 DAT
Posted: Wed Mar 07, 2012 8:37 pm
by Nimba
I wanted to enlist some help on getting the DAT file for the GW2 client figured out. It shares similarities to GW1 from what I see, but its different, and I couldn't quite figure it out.
I can provide the 16mb patcher that you download, it downloads the rest of the game. I can also provide cuts of the archive file (what file cutter do you recommend?).
Just tell me which is the preferred route and how to get it to you guys.
Re: [Request] Guild Wars 2 DAT
Posted: Wed Mar 07, 2012 9:19 pm
by Ekey
aluigi wrote:and where are the samples?
PS: Or just link for download client.
Re: [Request] Guild Wars 2 DAT
Posted: Wed Mar 07, 2012 9:40 pm
by Nimba
* snip *
That is the link to the client patcher. 5 mb (rar file), 16mb extracted, it will download the rest of the client.
It downloads in place, so put it into a convenient folder.
The full size of the DAT after downloading the full client is 16gb
Re: [Request] Guild Wars 2 DAT
Posted: Wed Mar 07, 2012 10:00 pm
by Nimba
I did do a bit of work already from the limited knowledge I know.
I also think there is a master file numbered 13525, it seems to have just a massive list in it. Not sure how its coded. URLs to their CDN are in the EXE if you run it through OllyDBG or monitor traffic with Wireshark. All files are downloaded compressed, but if you remove the compressed tag from the URLs, you will get the Uncompressed versions added to the .DAT file. If you get the cookie information, you can download them directly.
Other than that, I thought the list of files would be somewhere in memory, but I could not figure out where yet. So not sure how much this helps.
Re: [Request] Guild Wars 2 DAT
Posted: Thu Mar 08, 2012 1:25 pm
by Ekey
Cutted
http://www.mediafire.com/?hsowrca6g562gzc
Code: Select all
Signsrch 0.1.7
by Luigi Auriemma
e-mail: [email protected]
web: aluigi.org
optimized search function from Andrew http://www.team5150.com/~andrew/
disassembler engine from Oleh Yuschuk
- open file "d:\GW2\Gw2Dev.exe"
- 22800384 bytes allocated
- load signatures
- open file d:\!Project\Game Tools\Aluigi\signsrch.sig
- 1774060 bytes allocated for the signatures
- 2278 signatures in the database
- WARNING:
the file loaded in memory is very big so the scanning could take many time
- start signatures scanning:
offset num description [bits.endian.size]
--------------------------------------------
0109fda0 31 Adler CRC32 (0x191b3141) [32.le.1024]
010a0da0 32 Adler CRC32 (0x191b3141) [32.be.1024]
010a01a0 33 Adler CRC32 (0x01c26a37) [32.le.1024]
010a11a0 34 Adler CRC32 (0x01c26a37) [32.be.1024]
010a05a0 35 Adler CRC32 (0xb8bc6765) [32.le.1024]
010a15a0 36 Adler CRC32 (0xb8bc6765) [32.be.1024]
00f66cf0 115 ADPCM index table (step variation) [32.le.64]
00f66d30 117 ima_adpcm step table [16.le.178]
01044940 135 libavutil ff_log2_tab [..256]
0109e3e8 142 ACSS reverse sbox [..256]
01022e48 145 SHA256 Hash constant words K (0x428a2f98) [32.le.256]
008027e7 147 SHA256 Initial hash value H (0x6a09e667UL) [32.le.32&]
010adfa0 167 Rijndael Te0 (0xc66363a5U) [32.le.1024]
010ae3a0 169 Rijndael Te1 (0xa5c66363U) [32.le.1024]
010ae7a0 171 Rijndael Te2 (0x63a5c663U) [32.le.1024]
010aeba0 173 Rijndael Te3 (0x6363a5c6U) [32.le.1024]
010aefa0 175 Rijndael Te4 (0x63636363U) [32.le.1024]
010af3a0 176 Rijndael Td0 (0x51f4a750U) [32.le.1024]
010af7a0 178 Rijndael Td1 (0x5051f4a7U) [32.le.1024]
010afba0 180 Rijndael Td2 (0xa75051f4U) [32.le.1024]
010affa0 182 Rijndael Td3 (0xf4a75051U) [32.le.1024]
010b03a0 184 Rijndael Td4 (0x52525252U) [32.le.1024]
010b07a0 185 Rijndael rcon [32.le.40]
00801ee2 307 SHA1 / SHA0 / RIPEMD-160 initialization [32.le.20&]
010a21c0 357 Zlib dist_code [..512]
00f48ac0 358 Zlib length_code [..256]
010a24c0 360 Zlib base_length [32.le.116]
010a2538 362 Zlib base_dist [32.le.120]
0118c4cc 373 possible mask reset used in arithmetic compression [..9]
0109c650 384 Jpeg dct 14 bit aanscales [16.le.128]
0109c610 388 Jpeg dct AA&N scale factor [double.le.64]
0114e29a 550 Windows CryptAcquireContext [..21]
0049d86b 568 classical random incrementer 0x343FD 0x269EC3 [32.le.8&]
00e3d940 815 libavcodec ff_zigzag_direct [..64]
00f632b0 851 libavcodec ASF asf_header [..16]
00f67d48 855 Generic bitmask table [32.le.128]
00f63cb0 973 asf_content_desc_guid [..16]
011974b0 1014 mp3lib huffman tab1 [16.le.14]
01196a6c 1016 mp3lib huffman tab2 [16.le.34]
011974c0 1018 mp3lib huffman tab3 [16.le.34]
011974e4 1020 mp3lib huffman tab5 [16.le.62]
01197524 1022 mp3lib huffman tab6 [16.le.62]
01197568 1024 mp3lib huffman tab7 [16.le.142]
011975f8 1026 mp3lib huffman tab8 [16.le.142]
01197688 1028 mp3lib huffman tab9 [16.le.142]
01197718 1030 mp3lib huffman tab10 [16.le.254]
01197818 1032 mp3lib huffman tab11 [16.le.254]
01197918 1034 mp3lib huffman tab12 [16.le.254]
01197a18 1036 mp3lib huffman tab13 [16.le.1022]
01197e18 1038 mp3lib huffman tab15 [16.le.1022]
01198218 1040 mp3lib huffman tab16 [16.le.1022]
01198618 1042 mp3lib huffman tab24 [16.le.1022]
01198a18 1044 mp3lib huffman tab_c0 [16.le.62]
01198a58 1046 mp3lib huffman tab_c1 [16.le.62]
01196a90 1048 mp3lib bandInfo [..2592]
01194970 1049 mp3lib intwinbase [32.le.1028]
00f67f68 1176 Vorbis FLOOR1_fromdB_LOOKUP [float.le.1024]
0109e3e8 1198 FFT and FHT routines rv_tbl [..128]
00f4b788 1228 rfc3548 Base 64 Encoding with URL and Filename Safe Alphabet [..
62]
00f61648 1237 B64EncodeTable [..64]
010a1b78 1525 zinflate_lengthExtraBits [32.le.116]
010a1c00 1529 zinflate_distanceExtraBits [32.le.120]
0114cc02 1767 anti-debug: IsDebuggerPresent [..17]
00f67d4c 1810 bitmask [32.le.128]
015be7b4 2253 PADDINGXXPADDING [..16]
00f48bc0 2254 unlzx table_one [..32]
00f67d48 2259 unlzx table_three [32.le.64]
00de0bd0 2269 DMC compression [32.le.16&]
0104d163 2271 compression algorithm seen in the game DreamKiller [32.le.12&]
01046217 2272 compression algorithm seen in the game DreamKiller [32.be.12&]
- 70 signatures found in the file
Re: [Request] Guild Wars 2 DAT
Posted: Fri Mar 09, 2012 5:03 am
by Nimba
So this is what I have found so far that could help:
/latest/102 : has the latest build # (ex. returns: 13525 2312313 12313123) the first number is the build number.
/manifest/program/102/{build#} : manifest file for build number (ex. 13525, 13603)
/program/102/0/0/{fileNumber}/ : the actual data file uncompressed (ex. 844474 - MP3 File of a Roar)
/program/102/0/0/{fileNumber}/compressed : the actual data file compressed (have not investigated the compression used)
The Manifest File:
It is a list of UInt32 in Little Endian order. once converted to decimals, they are the fileNumber used in the above links. They do not have filenames, the number is the filename.
The Cookie information, and the CDN address is in the GW2Dev.exe file, should not be hard to find for those that are working on it.
Also, randomly (well seems random to me) there are numbers in the Manifest files that are not Files. They are the size in bytes of the previous file downloaded.
Example:
1814180 - 345208 bytes
2519389 - 345036 bytes
345036
2054826 - 72964 bytes
2948595 - 104384 bytes
104384
There are a lot of files unknown to me. AMATGMRT, MAPCPARM, PIMGPG, AMSPAMSP, MODLMODL, CINPC, and hvkChavK. A lot of files seem to have messed up or modified headers. like a good portion of the DXT files. They have a ATEXDXT# in the header, instead of some of the ones that work that start with DDS|. I have no clue how to repair the header, if its even possible. But I did some reading, it sounds similar to the GW1 ATEXDXT files. Messed up header, seems the size of the image is there somewhere after the header. Missing Texels? Thats all I read.
Re: [Request] Guild Wars 2 DAT
Posted: Wed Mar 14, 2012 12:11 am
by Nimba
Anyone have any success or progress on this? Just curious.
Re: [Request] Guild Wars 2 DAT
Posted: Wed Mar 14, 2012 1:14 am
by chrrox
i got a script to look at the uncompressed files but it downloaded 50 gigs of files not sure how that's possible.
Re: [Request] Guild Wars 2 DAT
Posted: Wed Mar 14, 2012 1:24 am
by Nimba
chrrox wrote:i got a script to look at the uncompressed files but it downloaded 50 gigs of files not sure how that's possible.
Well, I dunno what compression they use, or if they have secondary manifests somewhere, but that is correct. Uncompressed size of the download is 50 gigs of files. But compressed it is 16 gigs inside the DAT file. It compresses each file individually and then inserts it into the dat.
If you modify the GW2Dev.exe file and remove the /compressed from the URLs that are coded into the EXE, it will download 50 gigs directly into the DAT file. At least it did the last time I checked.
You can try it yourself and see. Backup your 16 gig file, and download the 50 gig version. you can then use it to compare for finding how files are stored in the DAT file? and then you can compared the uncompressed to the compressed to see how ti was compressed.
Not sure what to tell you.
Re: [Request] Guild Wars 2 DAT
Posted: Wed Mar 14, 2012 1:45 am
by chrrox
now that would be very usefull especially if it downloads them in order.
ill give that a try.
Re: [Request] Guild Wars 2 DAT
Posted: Wed Mar 14, 2012 1:57 am
by Nimba
chrrox wrote:now that would be very usefull especially if it downloads them in order.
ill give that a try.
You can also try with small samples. Let it download 20-30 files uncompressed, then 20-30 comrpessed into seperate dat files. So you dont have to wait.
You can then also check if they are in order.
Also you can download the individual files with my script, or a modified version of my script, and find the data in the archives. I dont have the tools to do it, but maybe some how search for the contents of Uncompressed File 1 in Uncompressed DAT, and then search Compressed File 1 in Uncompressed DAT file. and it could help you figure out how the DAT is split up and so on.
I think from there you know more than I do.
I also believe it downloads them in the order of the manifest.
Re: [Request] Guild Wars 2 DAT
Posted: Sun Mar 18, 2012 8:29 pm
by Nimba
Any luck?
Re: [Request] Guild Wars 2 DAT
Posted: Sun Mar 18, 2012 9:55 pm
by chrrox
i never got the uncompressed data as one big file
Re: [Request] Guild Wars 2 DAT
Posted: Mon Mar 19, 2012 1:48 am
by Nimba
chrrox wrote:i never got the uncompressed data as one big file
Even after modifying the patcher?
At address 00F5F454 of the latest exe (Build 13703), there is a line that says /compressed (2F 63 6F 6D 70 72 65 73 73 65 64), if you 0 that out, it should download everything uncompressed into the data file. I just did it yesterday to test. I have a 34 gig DAT file atm. Still downloading.
[Edit] Just did it. First ~1000 files. Compressed file is 135MB, uncompressed file is 239MB. I am uploading them both atm to dropbox and will link them to you in a bit.
Re: [Request] Guild Wars 2 DAT
Posted: Mon Mar 19, 2012 2:58 am
by Nimba
Compressed DAT Part (132MB) :Gw2DevStream-Compressed.dat ( just the first ~1000 files downloaded)
Uncompressed DAT Part (233MB): Gw2DevStream-Uncompressed.dat ( hopefully the same ~1000 files )
The way i cut it off, i started the patcher with no Data file (backed up my old one) and launched it, when it reached 157000 remaining files, I shut off the patcher. and I believe it started at 158000ish. Then I 00'd out the /compressed in the EXE, and did the same with a new data file.
[Edit] I took down the links, so they aren't linked to my dropbox account, and I believe anyone that would need them, has them.