Beyond: Two Souls - Reversing | Porting (THAT'S ALL FOLKS?)

[HugoPeters]

____Big Update List

____01/12/2014 We all knew it had to happen some day!

• 20/10/2014 I talk about scripts and Lua and show some more shit. And eat chips.
• 10/10/2014 2 scenes ran by NATIVE SCRIPTS ONLY.
• 18/09/2014 TEXTURES ARE FINALLY COMING!!!
• 02/09/2014 Exploit found in the actual Lua kernel, let the fun begin!
• 22/08/2014 Very early demo on CHOICE PATHS!
• 16/08/2014 KERNEL PART ~2~: Executing multiple byte-code scripts, loading resources and shit actually doing stuff (see vid for more info so tired)
• 07/08/2014 KERNEL PART ~1~: First real-world test of new engine rewrite, with native kernel.
• 28/07/2014 MILESTONE: Full sequence playing with scenery, camera's, dialog, music, etc!
• 27/07/2014 Early progress on area's / scenery!
• 20/07/2014 Camera keyframe animation figured out + implemented (video link in post)
• 18/07/2014 Passed final exams, will move out of house!
• 06/07/2014 Models, now in Mogre, and beginnings of skeleton / bones!
• 02/07/2014 Models (catalogs) seem to have a bright future!
• 01/07/2014 Timeline structure figured out, parsing in such a shape that we can now playback sequences (audio only)!
• 07/06/2014 Announcement new tool "InfraViewer"

Model-to-do-list Disclaimer: list subject to change
Todo

• Make a to-do list

Doing

• Research SCRIPT_ZONE's (ZONE_SHAPE_BOX, ZONE_SHAPE_SPHERE, ZONE_SHAPE_HEXAHEDRON)
• Research areas.

Done
I forgot to update this list #lol.
Except for the skeleton & materials everything is done: meshes are parsed, textures decoded.

Sequence-to-do-list Disclaimer: list subject to change
Todo

• Implement MOVIE_VIEWPORT_FADE_FX_ACTION
• Implement MOVIE_PAD_VIBRATION_FX_ACTION

Doing

• Implement PAR_ELEMENT_STANDARD
• Implement PAR_ELEMENT_HOLD
• Implement PAR_ELEMENT_LOOP
• Implement PAR_ELEMENT_MASH
• Implement MPAR_ELEMENT_AXIS_TRANSLATION
• Implement MPAR_ELEMENT_AXIS_ROTATION
• Implement MPAR_ELEMENT_ABC (?)
• Implement MPAR_ELEMENT_STANDARD
• Implement MPAR_ELEMENT_HOLD
• Implement MPAR_ELEMENT_LOOP
• Implement MPAR_ELEMENT_MASH
• Implement MOVIE_USER_ACTION_CONDITION

Done

• Create a "flexible" interface for playing / controlling sequences.
• Implement SEQUENCE_CONTROLLER (SEQ_CHK_), which only defines basic (but important) stuff like the root movie.
• Implement MOVIE_SCRIPT_CONDITION
• Implement SEQUENCE_CONTROLLER_PLAY[
• Rewrite audio streamer for the third time (this time asynchronous)
• Make it better.
• Perform first tests!!
• Write a custom high-resolution Timer class (using a thread and a stopwatch!)
• Implement MOVIE_PLAY_ANIM_ACTION
• Implement MOVIE_GO_TO_MOVIE_CONDITION
• Implement PLAY_SCRIPT_TOP
• Implement MOVIE_PLAY_SHOT_ACTION
• Implement MOVIE_WATCH_FX_ACTION
• Implement MOVIE_CAMERA_MODIFIER_FX_ACTION
• Implement MOVIE_CAMERA_QUAKE_FX_ACTION
• Implement MOVIE_PLAY_SOUND_ACTION
• Implement MOVIE_DIALOG_GROUP_CONDITION
• Implement MOVIE_PLAY_DIALOG_ACTION
• Implement MPA_CHK_ and the other entity Communicators (camera shots, sounds, dialog, etc.)
• Research camera "sub-shots".
• Implement MOV_ELEM (also relies on other stuff)
• Implement MOV_MISC
• Implement MOV_GUID
• Implement MOV_ENT_ (relies on other stuff though)
• Find out what PAR/MPAR stands for (Physical Action Reaction / Motion Physical Action Reaction)

Kernel-to-do-list Disclaimer: list subject to change
Todo

• Implement version 2 of CAC_____ (as seen in 0802 The Party Clash -> main scene com).
• Check out different DATA_CONTAINER => DC_DATA version formats and implement
• Implement SCRIPT_ZONEs
• Start my own blog(???)
• Check out non-empty SC_FRM_W (in 1802 Hunted Train)
• Audio features (stop, pause, resume, seek, insert silence)

Doing

• Perform tests to make sure everything works!
• Set up MOGRE (now Paradox) environment
• Spam XeNTaX with pointless update videos

Done

• Research PAR/MPAR events and choice events.
• Implement Lua "actions" system (partly completed)
• Re-implement dynamic loading for SCRIPT_FUNCTION_CALL.
• Proper meta-data reading (needs: SINT_SEC reader rewrite, modification to DC_DATA_ reader and preloader FIX)
• FIX: preloading of resources
• Partly rewrite, partly port the Sequence parser/player.
• Port usable code over from older tools (mainly InfraViewer)
  

• Completely revise attribute reading and data loading.

• Finish audio streaming engine (needs minor performance adjustments)
• Change FMOD audio engine to BASS audio engine (much better support for custom data streaming and easier DSP mixing)
• Research loading of resources.
• MILESTONE: Execute -FULL- scene, including sub-scripts, dynamic loading (100% with Broken (scene after Prologue) and Prologue)
• Implement SendEvent(...)
• Make some sort of Lua execution manager to make sure there are no cross-thread executions. (seems to be fixed using <lock>)
• [HIGH PRIORITY] Dynamic loading for LOADING_ZONEs & LOADING_ZONE_TIMERs. (seems to work fine, needs more testing though!)
• Implement SCRIPT_FUNCTION and SCRIPT_FUNCTION_CALL (major)
• Reorder classes into separate .cs files...
• MILESTONE: perform test using scene name, -> load scene loadzone, load scene datacontainer, execute scene (and load resources) -> call <scene>.OnInit().
• Research COM_INSTANCE_LIST's
• Implement EndVarFreedom()
• Converter for PS3 (PPC, big endian) LUAC chunks to PC (little endian) chunks.
• Temporary video play management, with threaded pre-loading and threaded playing. Note: needs actual playing! XD
• Audio system (FMOD) with global functions, events, and threading. (3D support will be written later)
• Implement SCRIPT_DELAY
• Implement MESSAGE_LAUNCHER & MESSAGE_CONTROLLER (TESTED)
• Proper resource management and properly working clean up functions (stress tested!)
• Make multithreaded loading manager with managed stacks and event callbacks
• "BinaryCore" for BigFiles (IDM file reading, handle assignments)
• Implement runtime.sdat reading
• Implement PARAM.SFO reader
• Implement Dynamic Communicator system (register, attributes, events, paramaters, actions (partly), unregister, linking)
• Implement Lua "variables" system.
• Implement Lua "events" system.
• Implement Lua "attributes" system.
• Neatly working kernel enviroment with easy-to-plug-in features and support.
• Create basic "instances" environment.

Support

As you may know, this category of XeNTaX doesn't get as much attention as we'd like it to get!
That's why I made this badge:

Code: Select all

[url=http://forum.xentax.com/viewtopic.php?f=32&t=11411][img]http://i.imgur.com/EUcneAM.png[/img][/url]

So now, you can show your support of this project by putting the badge in your signature!

Donations

Some people have asked me if they can donate somewhere, so I decided to put a PayPal donation thing up here:



To anyone who donates: THANK YOU! You're the best.
After you donate, make sure to send the receipt to hugo ~a t~ djek ~d o t~ nl with your nickname, so you can be in the credits once any tools / ports are released.

Where will the money go to? I haven't really thought of any situations where this project would cost money, but it will be primarily spent on this project. This means we could for instance set up a website, or pay people to reverse some assembly for us It could also be spent on hardware / software (such as the PS4 version of Beyond, when it comes out).

WARNING!!! The info under here is inaccurate and very outdated!



Here's my second tool since joining here!

This tool can extract all data blocks from Beyond: Two Souls, including audio (SFX, Dialog, Music & Ambience (multi channel)), segs entries (and decompress them using QuickBMS), scripts and dialog text (very limited support right now).

It also partly works with Heavy Rain. Let me know if you want me to investigate those formats too.

Here's a screenshot:



What's coming?
Okay, first a little explanation about how Beyond works.
Beyond (and Heavy Rain) store its data in blocks called Communicators.

(The following text is correct for around 60% )
A Communicator is a table containing data for a certain function in the game. These are formatted as precompiled Lua "function chunks". The values of these "settings" are stored either compressed (as a segs entry, which is a split up ZLIB archive), or uncompressed (as a QZIP entry, which is funny, because "ZIP" normally indicates a compressed file entry). Aside from storing just values as Lua functions, modified versions of Communicators are also used for storing the "actual" scene scripts, model data, animation data, (very small amounts of) audio, video (like the sport scenes and cartoons on the in-game TVs), textures, dialog sequence text, and more.

Right now, I'm working on a universal parser for these blocks. That's basically what the extractor does, but the difference is that this new parser can get useful data out of the blocks, rather than just extracting them.

>> Download Beyond: Two Souls -Extractor- v1.0
ANOTHER NOTE: Even though this is v1.0 it's still very much a beta. Please let me know if you get errors during export!!

So, there you go! I wonder if I will be getting any comments on this one!

[cra0]

I bow down to you sir good fucken work, I gave up when all those seg blocks were chopped up textures and such. If we can get a decent unpacker for each game asset would be easier to reverse.

-EDIT-

Ok so I tested it on the main BIGFILE


I get this not sure why.
Also BigFile_PS3.d08 throws this


Have you looked at parsing the model/animation/texture blocks yet?

[HugoPeters]

I bow down to you sir good fucken work, I gave up when all those seg blocks were chopped up textures and such. If we can get a decent unpacker for each game asset would be easier to reverse.

-EDIT-

Ok so I tested it on the main BIGFILE


I get this not sure why.
Also BigFile_PS3.d08 throws this


Have you looked at parsing the model/animation/texture blocks yet?

Thanks!

Damn, that were some things I forgot to fix... sorry for that! I'll post an update later. Slammed with school stuff right now...

From your list, I have only been looking at the ANIMDATA blocks. I always first try to reverse engineer the file entries / tab header, etc. but I can't figure it out completely... will post some info below.

About the segs archives: I haven't really looked at them too much, but from what I can see is that we're still missing something. After each(?) segs archive there are more chunks that start with 0x10... these define their own sizes as well... not sure what's up with that, maybe they belong to the segs archives, because every individual file is aligned to 0x800 (2048 bytes), and these only come after the segs stuff, unaligned.

I've been looking through the decrypted version of EBOOT.BIN (EBOOT.ELF), and found some strings that might be able to give us some clues about these archives..
PSST, don't tell anyone: I got the source code for EdgeZlib (it's a special kind of Zlib that can be offloaded to the ?SPU? that comes with the PS3 Edge SDK), will be checking that out soon...

0x00EF86F0: EDGE ZLIB ERROR: outputUncompSkipBeginSize(%d) + outputUncompSkipEndSize(%d) > expectedUncompSize(%d)
0x00EF8760: EDGE ZLIB ERROR: edgeZlibFetchAndInflateLargeRawData failed (%d)
0x00EF87B0: EDGE ZLIB ERROR: edgeZlibInflateRawData failed (%d)
0x00EF87F0: EDGE ZLIB ERROR: DecompressInflateQueueElement returned (%d)
0x00EF8830: EDGE ASSERTION FAILURE: %s(%d)
0x00EF8850: 1.2.3.0-PS3-SPU-EDGE
0x00EF8870: EDGE ZLIB ERROR: inflate error (%d)
0x00EF88A0: incorrect header check
0x00EF88C0: unknown compression method
0x00EF88E0: invalid window size
0x00EF8900: unknown header flags set
0x00EF8920: header crc mismatch
0x00EF8940: invalid block type
0x00EF8960: invalid stored block lengths
0x00EF8980: too many length or distance symbols
0x00EF89B0: invalid code lengths set
0x00EF89D0: invalid bit length repeat
0x00EF89F0: invalid literal/lengths set
0x00EF8A10: invalid distances set
0x00EF8A30: invalid literal/length code
0x00EF8A50: invalid distance code
0x00EF8A70: invalid distance too far back
0x00EF8A90: incorrect data check
0x00EF8AB0: incorrect length check
0x00EF9380: EDGE ZLIB ERROR: inflateEnd error (%d)
0x00EF93B0: EDGE ZLIB ERROR: Stream decompressed to size different from expected (%d != %d)
0x00EF9520: invalid distance code
0x00EF9540: invalid literal/length code
0x00EF9560: invalid distance too far back
0x00EF972C: SPUNAME
0x00EF9734: ..\..\Output\PS3\SPU Release\Ker

0x00D80384: Out of memory while calling edgeZlibAddInflateQueueElement. The game will crash soon !

0x00D84448: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:70 (0 == ret)
0x00D84488: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:96 CELL_OK == ret
0x00D844CC: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:188 (maxNumQueueEntries > 0)
0x00D8451C: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:148 (bufferSize == edgeZlibGetInflateQueueSize( maxNumQueueEntries ))
0x00D84594: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:149 (bufferSize == (128 + (maxNumQueueEntries * 32)))
0x00D845FC: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:150 ((((uint32_t)pBuffer) & 0x7F) == 0)
0x00D8465C: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:164 (0 == ret)
0x00D8469C: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:167 (0 == ret)
0x00D846DC: EDGE ZLIB ERROR: Input compressed data is on stack (which is illegal)

0x00D84724: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:401 (!sys_process_is_stack((void*)pInputCompressedData))
0x00D84794: EDGE ZLIB ERROR: Output uncompressed data is on stack (which is illegal)

0x00D847E4: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:404 (!sys_process_is_stack(pOutputUncompPartialBuff))
0x00D8484C: EDGE ZLIB ERROR: outputUncompSkipBeginSize(%d) + outputUncompSkipEndSize(%d) > expectedUncompressedSize(%d)

0x00D848BC: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:408 (outputUncompSkipBeginSize + outputUncompSkipEndSize <= expectedUncompressedSize)
0x00D84944: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:410 (pWorkToDoCounter)
0x00D8498C: Warning: Compressed data at address 0x%08X is *bigger* than master data (%d > %d). Pointless?

0x00D849EC: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:423 ((((uint32_t)pWorkToDoCounter) & 0x3) == 0)
0x00D84A54: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:442 (0 == ret)
0x00D84A9C: Buff))
0x00D84AA4: dSize)
0x00D84AB4: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:583 (0 == ret)


Furthermore, here are the different types of Dynamic Communicator blocks I could find, most of them with a description of some sort.
I put them in a nice switch statement (yes, I did this manually...):

NOTE: an asterisk (*) means the description is a guess by me, and where there's just the description it means I haven't found the short name for that one yet.

Code: Select all

switch (BlockType)
{
    case "COM_CONT":  // COMMUNICATOR_CONTAINER
    case "LOADCONT": // *LOADING_CONTAINER
    case "LOADZONE":  // LOADING_ZONE
    case "LOADZTIM": // LOADZING_ZONE_TIMER
    case "SINT_SEC": 
    case "SFE_LOAD": 
    case "ANIMDATA": // *ANIMATION_DATA
    case "LOCALIZ_":  // *LOCALIZATION
    case "PARTITIO":  // COMMUNICATOR_PARTITION
    
    case "SFESAV  ": // SCRIPT_FRAMEWORK_ENTITY_CONTROLLER_ABC
    case "SCN_CHK ": 
    case "SINT_SCN": 
    case "VPCCTRL ": // VIDEO_PLAYER_CONTROLLER
    case "MCR_CHK ":
    
    case "SCRIPT  ":
    case "SCPT_DAT":
        
    /* Unknown */
    case "SC_FRM_W": // SCRIPT_FRAMEWORK
    case "SENT_CHK":
    case "MK_INFO_":
    case "LOADZCAM": // LOADING_ZONE_CAMERA
    case "SINT_WLD":
    case "SCSAV   ":
    case "SFESAV  ":
    case "VLACRFC ": // VIEWPORT_LAYOUT_ANIMATION_CONTROLLER_RUN_FROM_CURRENT
    case "VLACR   ": // VIEWPORT_LAYOUT_ANIMATION_CONTROLLER_RUN
    case "VCFOCTRL": // VIEWPORT_CONTROLLER_FADE_OPACITY
    case "VCFCTRL ": // VIEWPORT_CONTROLLER_FADE
    case "CP_ATTR ": // SCRIPT_FRAMEWORK_ENTITY_CONTROLLER_ABC
    case "SCN_CHK ":
    case "SCENE   ":
    case "ADDR_CHK":
    case "SINT_ADR":
    case "VPCCTRL ": // VIDEO_PLAYER_CONTROLLER
    case "MMCSAV  ":
    case "MENUSCPM": // MENUS_MASTER_COM_PARAMETERS
    case "LMCRCTRL": // LOAD_MENU_CONTROLLER_RUN
    case "FADECTRL": // FADE_CONTROLLER
    case "MCRCTRL ": // MENU_CONTROLLER_RUN
    case "SEQ_CHK ":
    case "GEN_SEQ ":
    case "SECSAVE ": // SCRIPT_EVENT_CONTROLLER
    case "SEQUENCE":
    case "SCDLPARA": // SCRIPT_DIALOG_PARAMETERS
    case "SDPARAMS":
    case "SD_PARMS":
    case "GAP_CHK ": // GENERIC_ANIM_PARAMETERS
    case "UA_CHK  ": // USER_ACTION
    case "PAR_CONT": 
    case "UACABC  ": // UA_CONTROLLER_ABC
    case "UAECABC ": // UA_EVENT_CONTROLLER_ABC
    case "EVENTCOM": 
    case "EVNTMPAR":
    case "UAECPAR ": // UA_EVENT_CONTROLLER_PAR
    case "MPARELAD": // MPAR_ELEMENT_AXIS_DIAGONAL
    case "PARELEM ": // MPAR_ELEMENT_NO_EVENT
    case "PARELSHK": // PAR_ELEMENT_BLOCK/HIT
    case "SINT_TPG":
    case "MPCTRL  ": // MAGNETIC_SEGMENT_CONTROLLER
    case "TPGCTRL ": // THIRD_PERSON_GAMEPLAY_CONTROLLER
    case "SINT_GC ":
    case "GCTRLER ": // GAME_CONTROLLER
    case "SINT_GM ":
    case "SINTGM  ":
    case "SINT_GRI": // GAME_RUNTIME_INFO
    case "MAGCON  ": // MAGNETIC_CONNECTION
    case "TPGNP_CK": 
    case "MPASAV  ": // MAGNETIC_PATH
    case "MPCSAV  ": // MAGNETIC_PATH_CONTAINER
    case "SDCTRL  ": // SCRIPT_DELAY
    case "STIM_CHK": 
    case "GMK_COMG": // GMK_MOVEMENT_GRAPH
    case "GMK_ANIM": // GMK_ANIMATION_DATABASE
    case "RBSACTIO":
    case "RBSACTOR":
    case "RBSBRAIN":
    case "MPAREABC": // MPAR_ELEMENT_ABC
    case "PARELABC": // PAR_ELEMENT_ABC
    case "MPARELAR": // MPAR_ELEMENT_AXIS_ROTATION
    case "MPARELAT": // MPAR_ELEMENT_AXIS_TRANSLATION
    case "PARELLOP": // MPAR_ELEMENT_LOOP
    case "PARELMSH": // MPAR_ELEMENT_MASH/PAR_ELEMENT_MASH
    case "PARELHLD": // PAR_ELEMENT_HOLD
    case "PARELLP ": // PAR_ELEMENT_LOOP
    case "PARELSTD": // PAR_ELEMENT_STANDARD
    case "CBSPGP  ":
    case "CBFLWR  ": // ?CAMERA_BEHAVIOR_FOLLOW_VEHICLE
    case "CBFOLWVH": // ?CAMERA_BEHAVIOR_SEQUENCE
    case "ADCI_CHK": 
    case "ADBSAV  ":
    case "GSCCSAV ": // GAME_SAVE_CONTROLLER
    case "GSD_CHK ": // ?GAME_SAVE_DATA
    case "GSDSAV  ":
    case "SRG_CHK ":
    case "AG_SAV  ":
    case "AORSAV  ":
    case "AROSAV  ":
    case "AGNC_CHK": // ANAEL_GAMEPLAY_NAVIGATION_CONFIG
    case "ABCSAV  ": 
    case "AGNC_CHK": // ANAEL_GAMEPLAY_SELECTOR_CONFIG
    case "ANCSAV  ": 
    case "AGNC_CHK": 
    case "ASCSAV  ":
    case "AEMCSAV ": // ANAEL_EASY_MODE_CONTROLLER
    case "AFCSAV  ": // ANAEL_FX_CONTROLLER
    case "PIPO    ": // ?not sure =
    case "AFCCSAV ": // ANAEL_FX_CONTROLLER_CONFIG
    case "AGCSAV  ": // ANAEL_GO_TO_CONTROLLER
    case "AICSAV  ": // ANAEL_INTERACTION_CONTROLLER
                     // ANAEL_ENERGY_MANAGER
                     // ANAEL_ENERGY_SOURCE_CONFIG
                     // COM_ATTRIBUTE_CONTAINER_ABC
    case "AESSAV  ":
    case "AEM_CHK ":
    case "AEMSAV  ":
    case "AFC     ":
    case "ATCSAV  ":
    case "ATFSAV  ":
    case "CARGCSAV": // CAR_GAMEPLAY_CONTROLLER
    case "CARGPIN ": // CTRL_TRAIL_SPEED
    case "IEPSAV  ": // IW_ENTITY_TAIL_SHADER_CONTROLLER
    case "IECSAV  ": // INFRAWORLD_ENTITY_CONFIG
    case "IEP_CHK ": // INFRAWORLD_ENTITY_PACK
    case "IEPSAV  ": // INFRAWORLD_ENTITY_PACK_CONTROLLER
    case "OCSAV   ": // ?PURSUIT_BEHAVIOR
    case "PSSCTCFG": // POSSESSION_CENTRE_CONFIG
    case "PCCSAV  ": // POSSESSION_CENTRE_CONTROLLER
    case "VTSSAV  ": // VIEW_TARGET_SELECTION_CONTROLLER
    
    case "GFCCFG  ": // GROUP_FOLLOW_CONTROLLER_CONFIG
    case "ARRVLPOS": // ARRIVAL_POSE
    case "APSAV   ":
    case "SGCSAV  ": // ?STABILIZE_GAMEPLAY_CONTROLLER
    case "STCTLSV ": // STEALTH_CONTROLLER
    
    case "FSSFCTRL": // FOOTSTEP_SURFACE_CONTROLLER
    
    case "ICCSAV  ": // IK_CONSTRAINT_CONTROLLER
                     // IK_CONSTRAINT_CONTROLLER_EFFECTOR
    case "IQCSAV  ": // IK_QUADRIPED_CONTROLLER
    case "WICSAV  ": // WATCH_ENTITY_CONTROLLER
    case "SUBMGPIN": // SUBMARINE_GAMEPLAY_INPUT
    case "BIKE_CHK": // BIKE_GAMEPLAY
    case "BGCSAV  ": // BIKE_GAMEPLAY_CONTROLLER
    case "PRCANMDT": // PROC_ANIM_COM_DATA
                     // PROJECTILE_MANAGER
    case "VICSAV  ": // VIBRATE_CONTROLLER
    case "LCSAV   ": // LOADING_CONTROLLER
    case "LCFSAV  ": // LOADING_CONTROLLER_FUNCTION
    case "MPST_CHK": // PLAY_SCRIPT_TOP
    case "TSFX_CHK": // MOVIE_TIME_SCALE_FX_ACTION
    case "ACT_CHK ":
    case "COND_CHK":
    case "FX_CHK  ":
    case "CCTSAV  ": // COLLISION_CONTROLLER
    case "TCTSAV  ": // THROW_CONTROLLER
    case "PSESAV  ": // PHYSICS_SPECIAL_EFFECTS
    case "REPSAVE ": // REPULSOR
                     // PHYSICS_LOD_CONFIGURATOR
    case "PHYLC   ":
    case "PLCSAV  ": // CPN_FLMG_CORRECTION
                     // CPN_OBSTACLE_CORRECTION
                     
    case "CDEC_SAV": // CHARACTER_DIALOG_EVENT_CONTROLLER
    case "CEGASAV ": // CONTROLLER_ENTITY_GOTO_ABC
                     // CONTROLLER_ENTITY_GOTO_ADDRESS
    case "CEGAASAV": // CONTROLLER_ENTITY_GOTO_ADDRESS_ABC
    case "CEOTTSV ": // CONTROLLER_ENTITY_ORIENT_TO_TARGET
    case "SECPRANM": // SCRIPT_ENTITY_CONTROLLER_PLAY_PROCEDURAL_ANIM
    case "SSMSAV  ": // SCRIPT_ENTITY_CONTROLLER_SKELETON_MAPPING
    case "SKEL_MAP":
    case "SMCSAV  ": // SKELETON_MAPPING_CONFIG
    case "STZOCTRL": // STAIRS_ZONE_CONTROLLER
    case "SEPSAV  ": // SYNCHRONIZE_ENTITY_POSITIONS_CONTROLLER
    case "PHAD_CHK":
    case "SLSAV   ": // SCRIPT_LOCATOR
    case "FBCSAV  ": // FX_BANK_CONTROLLER
    case "TS_FX   ": // TIME_SCALE_FX
    case "LCALTCFG": // LOCATION_ALERT_CONFIG
    case "LACSAV  ":
                     // LOCATION_ALERT_CONTROLLER
                     // RAISE_EVENT_COUNT_CONTROLLER
                     // RAISE_EVENT_PER_ITEM_CONTROLLER
    case "SWSAV   ": // SCRIPT_WIND
                     // SEQUENCE_INSTANCE
    case "MGCDE   ": // MIX_GROUP_CONTROLLER_DISABLE_EFFECT
    case "MGCEE   ": // MIX_GROUP_CONTROLLER_ENABLE_EFFECT
    case "MGCSFRE ": // MIX_GROUP_CONTROLLER_SET_FDN_REVERB_EFFECT
                     // MIX_GROUP_CONTROLLER_MIX_GROUP
    case "MGCSMCE ": // MIX_GROUP_CONTROLLER_SET_MULTICHANNEL_COMPRESSOR_EFFECT
    case "MGCSPSE ": // MIX_GROUP_CONTROLLER_SET_PITCH_SHIFTER_EFFECT
    case "SSDFFCTL": // SCRIPT_SOUND_CONTROLLER_DISTANCE_FILTER
                     // FOOTSTEP_SOUND_MANAGER
    case "FSMSAV  ": // COLLISION_SOUND_MANAGER
    case "COSOMG  ": 
    case "COSOMGSV": // SCRIPT_SOUND_BANK
    case "SBCSAV  ": // SCRIPT_SOUND_BANK_CONTROLLER
                     // SCRIPT_SOUND_BANK_INSTANCE
                     // SCRIPT_SOUND_MULTITRACK_CONTAINER
    case "S_SND_MC": // SCRIPT_SOUND_RANDOM_CONTAINER
                     // SCRIPT_SOUND_RANDOM_CONTAINER_MODE
    case "S_SND_RC":
    case "S_SND_SC":
    case "AI_AGENT":
    case "AIMSAV  ": // AI_PATH_INTERACTOR_WITH_ZONE
    case "AIPIWZON":
                     // CONTROLLER_AGENT_DELAY_PATH
                     // CONTROLLER_AGENT_FOLLOW
                     // CONTROLLER_AGENT_GOTO_ABC
                     // CONTROLLER_AGENT_GOTO_ADDRESS
                     // CONTROLLER_AGENT_GOTO_ADDRESS_ANIM
                     // CONTROLLER_AGENT_GOTO_AGENT
                     // CONTROLLER_AGENT_GOTO_ZONE
    case "SDCTRL  ": // CONTROLLER_SYNC
    case "GCMCTRL ": // GMK_COMMAND_MESSAGE_CONTROLLER
    case "MKSP_CHK": // ?GMK_SIMULATE_PARAMS
                     // MOTION_GRAPH_TESTER_CONTROLLER
                     // MOTION_GRAPH_PLANNING_CONTROLLER
                     // MOTION_GRAPH_CONTROLLER
                     // QUATERNION_INTERPOLATION_TESTER_CONTROLLER
    case "HDAMCT  ": // HEAD_ANIMO_CONTAINER
    case "WANIMO  ": // WEIGHTED_ANIMO
    case "AREA_CHK":
    case "SINT_ARA":
    case "SINT_CD ":
    case "CSB_CHK ":
    case "SINT_CSB":
    case "CBABC   ": // ?CAMERA_BEHAVIOR_ANAEL
    case "CBSPGP  ": // ?CAMERA_BEHAVIOR_CHASE
    case "CBDCIRC ": // CAMERA_BEHAVIOR_CONTAINER
    case "CBCONTAI": // CAMERA_BEHAVIOR_CROWD_CHASE
                     // CAMERA_BEHAVIOR_FIRST_PERSON
    case "PANIMOCT": // PLAYABLE_ANIMO_CONTROLLER
    case "TGUI_CK ":
    
    case "CATALOG ": // AREA_CATALOG
    case "QFOBJECT": // QPF_COM_CLASS
    
    case "SCPTBANK": // SCRIPT_BANK
    case "GCFG_CHK":
    case "GAME_CFG":
    case "CFG_ELEM": // GAME_CFG_ELEMENT_ABC
    case "RUNT_CFG": // GAME_CFG_RUNTIME_ELEMENT_ABC
    case "STAT_CFG": // GAME_CFG_STATIC_ELEMENT_ABC
    
    case "DLG_CFG ": // DIALOG_CFG
    case "GMG_CFG ": // GAME_MANAGER_CFG
    case "ST_MPAR_":
    case "MPAR_CFG":
    case "RT_ID_  ": 
    case "ID_CFG  ": // INPUT_DEVICE_CFG
    case "RT_LOC_ ":
    case "LOC_CFG ": // LOCALIZATION_CFG
    case "RT_TPG_ ":
    case "TPG_CFG ": // THIRD_PERSON_GAMEPLAY_CFG
    
    case "GENPAT  ": // GENERIC_PATH
    case "GENCON  ": // GENERIC_PATH_CONNECTION
    case "GENSEG  ": // GENERIC_PATH_SEGMENT
    
    case "STORYBRD": // *Storyboard
    case "SINTSTRY":
    case "STBRDELM": // STORYBOARD_ELEMENT
    case "STRBDSCN": // STORYBOARD_SCENE
    case "STBSCELM": // STORYBOARD_SCENE_ELEMENT
    
    case "SF_SAV  ": // SCRIPT_FUNCTION
    case "SFCSAV  ": // SCRIPT_FUNCTION_CALL
    
    case "CMODFXCK": // MOVIE_CAMERA_MODIFIER_FX_ACTION
    case "CQFX_CHK": // MOVIE_CAMERA_QUAKE_FX_ACTION
    case "PADVIB  ": // MOVIE_PAD_VIBRATION_FX_ACTION
    case "PLAVIDFX": // MOVIE_PLAY_VIDEO_FX_ACTION
    case "RVLA_CHK": // MOVIE_RUN_VIEWPORT_LAYOUT_ANIM_ACTION
    case "VPFX_CHK": // MOVIE_VIEWPORT_FADE_FX_ACTION
    case "MDGC_CHK": // MOVIE_DIALOG_GROUP_CONDITION
    case "SCPTCOND": // MOVIE_SCRIPT_CONDITION
    case "MSIC    ": // MOVIE_SOFT_INTERRUPT_CONDITION
    case "MUAC_CHK": // MOVIE_USER_ACTION_CONDITION
    
    case "MOV_ENT_":
    case "MOV_EVT_":
    case "MOV_TOP_":
    case "MOV_FX__":
    case "MOV_GUID":
    case "MOV_MISC":
    case "MOV_COND":
    case "MOV_ELEM":
    
    case "PARDISPL": // UA_ELEMENT_DISPLAY
    case "PARINPUT": // UA_ELEMENT_INPUT
    
    case "MPA_CHK_":
    case "MPABC___": // MOVIE_PLAYABLE_ABC
    case "MCAA_CHK": // MOVIE_COM_ATTRIB_ACTION
    case "MPAA_CHK": // MOVIE_PLAY_ANIM_ACTION
    case "MPDA_CHK": // MOVIE_PLAY_DIALOG_ACTION
    case "NSHA_CHK": // MOVIE_PLAY_SHOT_ACTION
    case "MSA_CHK_": // MOVIE_PLAY_SOUND_ACTION
    
    /* 3D objects */
    case "OBJ_____":
    case "MOV_ENT_":
    case "MESHDATA":
    case "PRIMEDGE":
    case "CLUPSKME":
    case "ENTITY  ":
    case "MESH____":
    case "NODE    ":
    case "STPHDATA": // STATIC_PHYSIC_DATA
    case "DYPHDATA": // DYNAMIC_PHYSIC_DATA
    case "SHADCUST":
    case "FX_EVNT_": // FX_EVENTS
    
    /* Audio */
    case "PARTOFFS": // OFFSET_PARTITION
    case "STREAMAB": // STREAMABLE
    case "HEADER__":
    case "S_SND_E_":
    case "TBOFFSET": 
}

About the ANIMDATA, I haven't researched it very deeply, but I did find out some things:

0x8: Block ID (ANIMDATA)
0x4: Unknown identifier (see below)
0x4: Unknown identifier (see below)
0x4: Unknown identifier (see below)
0x4: Unknown identifier (see below)
0x4: Big "tab" entries amount (after the header tab)
0x4: Unknown identifier... (see below)
0x4: Unknown identifier... (see below)
0x4: Header "tab" entries amount

Then the header tab starts, every entry consists of 6 ushorts(?):
0x2: Unknown ushort
0x2: Unknown ushort
0x2: Unknown ushort
0x2: Unknown ushort
0x2: Unknown ushort
0x2: Unknown ushort

Then there's 2 (padding?) bytes, and then the "big" tab thing starts, every entry consisting of 4 ushorts(?):
0x2: Unknown ushort
0x2: Unknown ushort
0x2: Unknown ushort
0x2: Unknown ushort

Then there's some more data... might be the same size as the header entries amount (??????)

So yeah, haven't looked at it very long

I may have a lead on this one as well though, I found this in the extracted strings from EBOOT.ELF:

0x00D7B0B0: ANIMDATA
0x00D7B0C0: ANIMPARM
0x00D7B0D0: TRACKDAT
0x00D7B0DC: ANIM_DATA
0x00D7B0E8: KEYFRAME_ANIMATION
0x00D7B108: Evt_Foot_Left
0x00D7B118: Evt_Foot_Right
0x00D7B128: Evt_SYNCHstep_Left
0x00D7B140: Evt_SYNCHstep_Right
0x00D7B15C: MBI_SKA
0x00D7B164: STREAMABLE_KEYFRAME_ANIMATION
0x00D7B184: STREAM_ANIMATION_OFFSET
0x00D7B19C: AnimData
0x00D7B1AC: Ref_AnimData
0x00D7B1BC: Ref Frame
0x00D7B1CC: Start Frame
0x00D7B1DC: End Frame
0x00D7B1EC: Blend Time
0x00D7B1FC: Animation Type
0x00D7B20C: Streamable AnimData
0x00D7B224: Ref Streamable AnimData
0x00D7B23C: SetAnimData
0x00D7B24C: GetAnimData
0x00D7B25C: SetStartFrame
0x00D7B26C: GetStartFrame
0x00D7B27C: SetEndFrame
0x00D7B28C: GetEndFrame
0x00D7B29C: SetRefAnimData
0x00D7B2AC: GetRefAnimData
0x00D7B2BC: SetReferenceFrame
0x00D7B2D4: GetRefFrame
0x00D7B2E4: SetStreamableAnimData
0x00D7B2FC: GetStreamableAnimData
0x00D7B314: SetRefStreamableAnimData
0x00D7B334: GetRefStreamableAnimData
0x00D7B354: Main
0x00D7B35C: Idle Frame
0x00D7B36C: Idle Duration Min
0x00D7B384: Idle Duration Offset
0x00D7B39C: Animation Types
0x00D7B3AC: One Shot
0x00D7B3BC: Loop
0x00D7B3C4: Attachment Offset Translation
0x00D7B3E4: Attachment Offset Rotation X
0x00D7B404: Attachment Offset Rotation Y
0x00D7B424: Attachment Offset Rotation Z
0x00D7B444: Attachment Animation
0x00D7B45C: Pause Attachment Anim
0x00D7B474: ANIMMOD
0x00D7B47C: ANIMATION_MODIFIER
0x00D7B490: LeftUpLeg
0x00D7B4A0: LeftLeg
0x00D7B4A8: LeftFoot
0x00D7B4B8: RightUpLeg
0x00D7B4C8: RightLeg
0x00D7B4D8: RightFoot
0x00D7B4E4: LeftToeBase
0x00D7B4F4: RightToeBase
0x00D7B504: LeftArm
0x00D7B50C: LeftForeArm
0x00D7B51C: LeftHand
0x00D7B52C: RightArm
0x00D7B53C: RightForeArm
0x00D7B54C: RightHand
0x00D7B55C: ANIMATION_OPTIONS
0x00D7B570: DisplayMotionKitCommands
0x00D7B590: DisplayInvisiblePlanes
0x00D7B5A8: DisplayMagneticPathConstraints
0x00D7B5C8: DisplayAnimationPoses
0x00D7B5E0: DisplayBoneLock
0x00D7B5F0: DisplayPoseValidityCheck
0x00D7B610: DisplayCanDebug
0x00D7B620: DisplayHierarchyFadeIn
0x00D7B638: EnableRemoteDebugging

And this:

0x00DAC61C: ThisCommunicator
0x00DAC630: None
0x00DAC638: InBounds
0x00DAC648: OnEvent
0x00DAC650: AliasCRC
0x00DAC660: nFrameLowerBound
0x00DAC678: nFrameUpperBound
0x00DAC690: sAnimDataName
0x00DAC6A0: EventNameCRC
0x00DAC6B0: TriggerOnce
0x00DAC6C0: nAnimDataID
0x00DAC6CC: ThisCommunicator
0x00DAC6E0: Invoke
0x00DAC6E8: Success
0x00DAC6F0: AliasCRC
0x00DAC700: EventNameCRC
0x00DAC714: Invoke
0x00DAC71C: Success
0x00DAC724: EventNameCRC
0x00DAC734: EntityCatalogID
0x00DAC744: EntityCatalogName
0x00DAC758: ThisCommunicator
0x00DAC774: Offset_Translation
0x00DAC78C: Offset_Rotation

I don't know if that helps at all, but it could give some hints to what the data means...

By the way, I haven't found any texture blocks yet.... do they start without a Communicator ID? Because I did see a few of those that looked like they could be textures...

[HugoPeters]

Major Update / Find / Thing

Alright, I decided to take a look at (the decrypted version of) BigFile_PS3.sdat first, which starts with the id QUANTICDREAMTABIDMEM.
I suspected this file would give indices into the other BigFile's, and I was right!

The file starts off with a header, which I haven't figured out completely yet, but at least it gives the amount of entries, and the block offset for each block (+ 0x18 for the header).

Then the very first block of entries in the sdat file are indices into BigFile_PS3.dat and are for the LOCALIZ_ blocks (PS3 error codes / Sony messages), the second block is also for BigFile_PS3.dat and this one only has one entry, for the (huge) uncompiled Lua initalization script.

Then after that there are huge blocks of indices. First up is a tab index I haven't figured out yet
After that come the indices into the files, each 0x14 in size. They give block offsets, block sizes and the file ID's.

Here's a sample output of 1 of the blocks of BigFile_PS3.sdat (only the first 100 entries of the total of 10,069...):

Code: Select all

META
{
    File: G:\Projects\Tools\BeyondTwoSouls\Indexing\BigFile_PS3.(s)dat
    Header ID: QUANTICDREAMTABIDMEM
    Version: 3
}

HEADER TAB
{
    HeaderTabEntries: 7
    UnknownPadding: 0x0
     ----------
    Data Type Entry #1: Unknown: 0x3F8, TypeEntries: 0x2, BlockOffset: 0x60
    Data Type Entry #2: Unknown: 0x1078, TypeEntries: 0x1, BlockOffset: 0x488
    Data Type Entry #3: Unknown: 0x869, TypeEntries: 0x1, BlockOffset: 0x89C
    Data Type Entry #4: Unknown: 0x1D, TypeEntries: 0x2755, BlockOffset: 0xCB0
    Data Type Entry #5: Unknown: 0x897, TypeEntries: 0x15, BlockOffset: 0x32354
    Data Type Entry #6: Unknown: 0x3F2, TypeEntries: 0x3398, BlockOffset: 0x328F8
    Data Type Entry #7: Unknown: 0xFFB, TypeEntries: 0x1FC8, BlockOffset: 0x734D8
}

/* TEMP TEMP TEMP: STARTING ANALYSE @ 0x10C4 */

TEMP FILE OUTPUT
{
    Entry #1: BlockID: 256, BlockOffset: 0xADD4000, BlockSize: 0x77431F, u1: 0x6ACBC7, FileID: 0 -> BigFile_PS3.dat
    Entry #2: BlockID: 1024, BlockOffset: 0x46A46000, BlockSize: 0x68, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #3: BlockID: 1792, BlockOffset: 0x12DAE800, BlockSize: 0x12B7D8, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #4: BlockID: 2048, BlockOffset: 0x4C783800, BlockSize: 0x29, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #5: BlockID: 2560, BlockOffset: 0x240EA800, BlockSize: 0x782A, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #6: BlockID: 2816, BlockOffset: 0xBDB3000, BlockSize: 0x4F5B, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #7: BlockID: 3584, BlockOffset: 0x5B259800, BlockSize: 0x3B118, u1: 0x3A6CB, FileID: 0 -> BigFile_PS3.dat
    Entry #8: BlockID: 4608, BlockOffset: 0x61F3000, BlockSize: 0x6953B, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #9: BlockID: 6400, BlockOffset: 0x7DE5A000, BlockSize: 0x2ADB6A, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #10: BlockID: 8192, BlockOffset: 0x4DD74000, BlockSize: 0x120C7, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #11: BlockID: 8960, BlockOffset: 0x5985D000, BlockSize: 0x9D14, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #12: BlockID: 9984, BlockOffset: 0x2A398000, BlockSize: 0xE59E7, u1: 0xE45C4, FileID: 2 -> BigFile_PS3.d02
    Entry #13: BlockID: 10496, BlockOffset: 0x39932800, BlockSize: 0xBCC1, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
    Entry #14: BlockID: 12032, BlockOffset: 0x551D7000, BlockSize: 0x123AFE, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
    Entry #15: BlockID: 12800, BlockOffset: 0x7DA49800, BlockSize: 0x3CFFD56, u1: 0x3C9F87B, FileID: 3 -> BigFile_PS3.d03
    Entry #16: BlockID: 13568, BlockOffset: 0x12BDB000, BlockSize: 0x1F90A2, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #17: BlockID: 13824, BlockOffset: 0xBADE000, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #18: BlockID: 14592, BlockOffset: 0x4FBEF800, BlockSize: 0x3B3, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #19: BlockID: 14848, BlockOffset: 0x6F185800, BlockSize: 0x1EA4D, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #20: BlockID: 16128, BlockOffset: 0x64C32800, BlockSize: 0x2B42, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #21: BlockID: 16640, BlockOffset: 0x39570800, BlockSize: 0xB8C8A, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #22: BlockID: 18176, BlockOffset: 0x1AEE6800, BlockSize: 0xB9D5B, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #23: BlockID: 19200, BlockOffset: 0x3094D800, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #24: BlockID: 19968, BlockOffset: 0x30D5C800, BlockSize: 0x3AE, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #25: BlockID: 20992, BlockOffset: 0x5A712000, BlockSize: 0x29, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
    Entry #26: BlockID: 22016, BlockOffset: 0x379F5800, BlockSize: 0x29, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
    Entry #27: BlockID: 22272, BlockOffset: 0x5D2F8800, BlockSize: 0x9C9B, u1: 0x3207, FileID: 1 -> BigFile_PS3.d01
    Entry #28: BlockID: 24064, BlockOffset: 0x2A7D800, BlockSize: 0x247E9B, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #29: BlockID: 24576, BlockOffset: 0x396D0000, BlockSize: 0x1A19F4D, u1: 0x19F050E, FileID: 4 -> BigFile_PS3.d04
    Entry #30: BlockID: 27904, BlockOffset: 0x68C5E000, BlockSize: 0x7A6BF, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #31: BlockID: 28672, BlockOffset: 0x5D495000, BlockSize: 0xB98BE, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #32: BlockID: 29696, BlockOffset: 0x6AF8A800, BlockSize: 0x4F5D2, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #33: BlockID: 30208, BlockOffset: 0x52ED3800, BlockSize: 0x22B1D, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #34: BlockID: 30976, BlockOffset: 0x31093000, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #35: BlockID: 31232, BlockOffset: 0x3A18000, BlockSize: 0x2073C, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #36: BlockID: 32512, BlockOffset: 0x4E9B800, BlockSize: 0x79CE38, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #37: BlockID: 32768, BlockOffset: 0x63FB8000, BlockSize: 0x81448, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #38: BlockID: 33024, BlockOffset: 0x4825000, BlockSize: 0x28BF5, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #39: BlockID: 35072, BlockOffset: 0x255CD800, BlockSize: 0x29, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #40: BlockID: 35840, BlockOffset: 0x3D411800, BlockSize: 0x11C954, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
    Entry #41: BlockID: 36608, BlockOffset: 0x33F23800, BlockSize: 0x5D884C, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #42: BlockID: 38912, BlockOffset: 0x372E2800, BlockSize: 0x2ABA9D, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #43: BlockID: 39680, BlockOffset: 0x1F782000, BlockSize: 0xE417, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #44: BlockID: 40448, BlockOffset: 0x73AAD000, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #45: BlockID: 40704, BlockOffset: 0xBE20800, BlockSize: 0x7D26, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #46: BlockID: 41216, BlockOffset: 0x564DE800, BlockSize: 0xED8A, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #47: BlockID: 43520, BlockOffset: 0x2BB60800, BlockSize: 0x2DFC, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #48: BlockID: 43776, BlockOffset: 0x6B89000, BlockSize: 0xADF, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #49: BlockID: 513, BlockOffset: 0x396000, BlockSize: 0x464, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #50: BlockID: 1025, BlockOffset: 0x46A44800, BlockSize: 0x68, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #51: BlockID: 1281, BlockOffset: 0x832A000, BlockSize: 0x6957A1, u1: 0x686825, FileID: 2 -> BigFile_PS3.d02
    Entry #52: BlockID: 1537, BlockOffset: 0x235C6800, BlockSize: 0x3F0C9, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #53: BlockID: 1793, BlockOffset: 0x12EDA000, BlockSize: 0xC79A, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #54: BlockID: 2817, BlockOffset: 0xBDB8000, BlockSize: 0x18201B, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #55: BlockID: 3585, BlockOffset: 0x5B295000, BlockSize: 0xFF4C7, u1: 0xA3EFB, FileID: 0 -> BigFile_PS3.dat
    Entry #56: BlockID: 3841, BlockOffset: 0xE388800, BlockSize: 0x70F77, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #57: BlockID: 4097, BlockOffset: 0x4DB8B800, BlockSize: 0x41AE2, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #58: BlockID: 5121, BlockOffset: 0x4FAA1800, BlockSize: 0x6B137E, u1: 0x6A0839, FileID: 1 -> BigFile_PS3.d01
    Entry #59: BlockID: 7681, BlockOffset: 0x63847800, BlockSize: 0x3067, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #60: BlockID: 8193, BlockOffset: 0x4DD86800, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #61: BlockID: 8961, BlockOffset: 0x59867000, BlockSize: 0x6D704, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #62: BlockID: 10497, BlockOffset: 0x3993E800, BlockSize: 0x58BE, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
    Entry #63: BlockID: 11009, BlockOffset: 0x1141E000, BlockSize: 0xBE6, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #64: BlockID: 12801, BlockOffset: 0x734AB800, BlockSize: 0x5A3062, u1: 0x58888A, FileID: 0 -> BigFile_PS3.dat
    Entry #65: BlockID: 13825, BlockOffset: 0xBADE800, BlockSize: 0x42B2A, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #66: BlockID: 14337, BlockOffset: 0x764FE800, BlockSize: 0x37FAA8, u1: 0x313500, FileID: 0 -> BigFile_PS3.dat
    Entry #67: BlockID: 16385, BlockOffset: 0x65162800, BlockSize: 0x12447A, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #68: BlockID: 19201, BlockOffset: 0x3094E000, BlockSize: 0x79A66, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #69: BlockID: 19969, BlockOffset: 0x30D5D000, BlockSize: 0x46CB0, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #70: BlockID: 20481, BlockOffset: 0x3CD5D000, BlockSize: 0x76FD0, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #71: BlockID: 20993, BlockOffset: 0x7251E000, BlockSize: 0x1BD8BD, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #72: BlockID: 22017, BlockOffset: 0x379F6000, BlockSize: 0x29, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
    Entry #73: BlockID: 24065, BlockOffset: 0x2CC5800, BlockSize: 0x2602F, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #74: BlockID: 30209, BlockOffset: 0x53F21800, BlockSize: 0xF4D, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #75: BlockID: 30977, BlockOffset: 0x31093800, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #76: BlockID: 31745, BlockOffset: 0x5EE4F800, BlockSize: 0x60066, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #77: BlockID: 32257, BlockOffset: 0x46CA800, BlockSize: 0x3B1, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #78: BlockID: 32769, BlockOffset: 0x64039800, BlockSize: 0x2F2B9, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #79: BlockID: 35073, BlockOffset: 0x255CE000, BlockSize: 0x2D872, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #80: BlockID: 35841, BlockOffset: 0x3D52E800, BlockSize: 0xED651, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
    Entry #81: BlockID: 37633, BlockOffset: 0xD529800, BlockSize: 0x1964, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
    Entry #82: BlockID: 37889, BlockOffset: 0x116C0000, BlockSize: 0x71E3, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #83: BlockID: 39681, BlockOffset: 0x1F790800, BlockSize: 0x41D28, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #84: BlockID: 40193, BlockOffset: 0x22F54000, BlockSize: 0x1A616, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #85: BlockID: 40449, BlockOffset: 0x73AAD800, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #86: BlockID: 41217, BlockOffset: 0x564ED800, BlockSize: 0x1DC63, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
    Entry #87: BlockID: 41729, BlockOffset: 0x61035800, BlockSize: 0x14044, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #88: BlockID: 43777, BlockOffset: 0x69D0800, BlockSize: 0x5BE12, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #89: BlockID: 258, BlockOffset: 0x8FA2000, BlockSize: 0x7CBC5D, u1: 0x72BFB2, FileID: 0 -> BigFile_PS3.dat
    Entry #90: BlockID: 770, BlockOffset: 0x5305C000, BlockSize: 0x241422B, u1: 0x237FBD7, FileID: 1 -> BigFile_PS3.d01
    Entry #91: BlockID: 1026, BlockOffset: 0x46A45000, BlockSize: 0x464, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
    Entry #92: BlockID: 1538, BlockOffset: 0x23606000, BlockSize: 0x29, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #93: BlockID: 2818, BlockOffset: 0xBF3A800, BlockSize: 0x7EA1, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #94: BlockID: 3586, BlockOffset: 0x5B394800, BlockSize: 0xAE8F9, u1: 0x79511, FileID: 0 -> BigFile_PS3.dat
    Entry #95: BlockID: 8450, BlockOffset: 0x39708800, BlockSize: 0x3CEB, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
    Entry #96: BlockID: 8962, BlockOffset: 0x598D4800, BlockSize: 0x20EBB, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #97: BlockID: 10242, BlockOffset: 0x265C2800, BlockSize: 0xF3203, u1: 0xBE680, FileID: 0 -> BigFile_PS3.dat
    Entry #98: BlockID: 11010, BlockOffset: 0x1141F000, BlockSize: 0x138212, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
    Entry #99: BlockID: 12034, BlockOffset: 0x552FB000, BlockSize: 0xA5FB2, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
    Entry #100: BlockID: 12802, BlockOffset: 0x7F43C000, BlockSize: 0xE2CC2, u1: 0xB8880, FileID: 0 -> BigFile_PS3.dat
}

There are a few things to note here:
1. The thing that I called "BlockID" (which I'm not sure of if that's what it actually is) gets bigger every entry but then suddenly gets smaller again. I'm suspecting these are "groups" of assets that belong to each other.
2. The u1 stands for Unknown1. No idea what this indicates yet, but it's only non-0x0 for some entries...

So, that's it for now!

[HugoPeters]

The "groups" inside BigFile_PS3.(s)dat seem to be neatly ordered by content! There's a group that indexes just .bik files (video) of the different BigFile's, then there's a group that seems to be indexing only the audio files, and there are groups that index all over the place (BigFile_PS3.dat - .d04). Then, within the big content groups, there seem to be 255 subgroups (always that number for some reason), that's the tab thing I didn't understand in my previous post. These subgroups are defined before the actual file indices start of that content group. I'm not sure if these groups actually mean something or not right now. Today I made a pretty good parser for the index sdat with basic GUI / treeview of the individual entries of every BigFile, together with sorting per content group / subgroup. Will port some of the better code of the extractor over tomorrow so I can check if these groups are actual groups.

[JayK]

I have some files taken from the ps3s cache whilst the game is running, I can send them to you if you think they may help.

[cra0]

PSST, don't tell anyone: I got the source code for EdgeZlib

hehehe XD nice! hope that comes in handy

Furthermore, here are the different types of Dynamic Communicator blocks I could find, most of them with a description of some sort.

Ah nice yeah those look about right

case "MESHDATA":
case "OBJ_____":

seem interesting most likely OBJ would be the static objects in the scenes.

By the way, I haven't found any texture blocks yet.... do they start without a Communicator ID? Because I did see a few of those that looked like they could be textures...

The textures are in DXT1 DXT5 (DXTATI2N - NORMALS), if I recall the segs script EKEY wrote dumped some texture files.

I have some files taken from the ps3s cache whilst the game is running, I can send them to you if you think they may help.

Someone's mentioned this to me before would you mind if I took a look at the ram dump as well to at least see how there model formats are stored and hopefully
figure them out for when Hugo's finished his parser tool?

Oh lastly great job HugoPeters

[HugoPeters]

I have some files taken from the ps3s cache whilst the game is running, I can send them to you if you think they may help.

Yeah, that would be very useful! That could help with identifying the different groups of files the game defines, or what to expect from the decompressed file entries. Btw, cache dump = RAM dump?

Furthermore, here are the different types of Dynamic Communicator blocks I could find, most of them with a description of some sort.

Ah nice yeah those look about right

case "MESHDATA":
case "OBJ_____":

seem interesting most likely OBJ would be the static objects in the scenes.

From what I can tell the objects are very complex. Here's some strings related to the model structure:

0x00DCA3FC: hkVersionPatchManager
0x00DCA414: hk_2011.1.0-r1
0x00DCA424: hk.DataObjectType
0x00DCA438: The object of class
0x00DCA451: is about to be removed and all references to it will be invalid.
However, the object is referenced (
0x00DCA4B8: ) from outside the world and it will lead to unexpected behavior or crash.
0x00DCA508: d:/Perforce/Main/Havok2011/Source/Common/Serialize/Data/Dict/hkDataObjectDict.cpp
0x00DCA55C: typeName
0x00DCA610: hk.PostFinish
0x00DCA654: __classnames__
0x00DCA664: variant
0x00DCA66C: memSizeAndFlags
0x00DCA67C: value
0x00DCA684: name
0x00DCA68C: time
0x00DCA694: namedVariants
0x00DCA6A4: className
0x00DCA6B4: hkRootLevelContainer
0x00DCA6CC: hkRootLevelContainerNamedVariant
0x00DCA6F4: hkReferencedObject
0x00DCA70C: hkBaseObject
0x00DCA71C: type
0x00DCA724: data
0x00DCA72C: hkxMaterial
0x00DCA73C: floatData
0x00DCA74C: floatStride
0x00DCA75C: uint8Stride
0x00DCA76C: hkxIndexBuffer
0x00DCA77C: userChannels
0x00DCA78C: mapping
0x00DCA794: hkxMaterialTextureStage
0x00DCA7AC: uint8Data
0x00DCA7BC: decls
0x00DCA7C4: hkxMeshSection
0x00DCA7D4: attributes
0x00DCA7E4: hkxVertexDescriptionElementDecl
0x00DCA804: numVerts
0x00DCA814: subMaterials
0x00DCA824: length
0x00DCA82C: indexBuffers
0x00DCA83C: annotations
0x00DCA84C: indexType
0x00DCA85C: uint16Data
0x00DCA86C: hkxMesh
0x00DCA874: uint32Stride
0x00DCA884: vectorData
0x00DCA894: attributeGroups
0x00DCA8A4: ambientColor
0x00DCA8B4: byteOffset
0x00DCA8C4: hkxVertexBufferVertexData
0x00DCA8E4: indices16
0x00DCA8F4: indices32
0x00DCA904: hkpSphereMotion
0x00DCA914: hkxVertexDescription
0x00DCA92C: hkxAttributeGroup
0x00DCA944: texture
0x00DCA94C: diffuseColor
0x00DCA95C: material
0x00DCA96C: stages
0x00DCA974: usage
0x00DCA97C: userChannelInfos
0x00DCA994: specularColor
0x00DCA9A4: mesh
0x00DCA9AC: vectorStride
0x00DCA9BC: byteStride
0x00DCA9CC: sections
0x00DCA9DC: hkxAttribute
0x00DCA9EC: uint32Data
0x00DCA9FC: desc
0x00DCAA04: uint16Stride
0x00DCAA14: tcoordChannel
0x00DCAA24: hkxAttributeHolder
0x00DCAA3C: vertexBuffer
0x00DCAA4C: emissiveColor
0x00DCAA5C: hkxVertexBuffer
0x00DCAA6C: vertexBaseOffset
0x00DCAA84: hkxAnimatedVector
0x00DCAA9C: vectors
0x00DCAAA4: hkxMeshUserChannelInfo
0x00DCAABC: usageHint
0x00DCAACC: isHierarchicalCompound
0x00DCAAE4: extraData
0x00DCAAF4: hkpShapeInfo
0x00DCAB04: childShapeNames
0x00DCAB14: localFrame
0x00DCAB24: childTransforms
0x00DCAB34: hkdShapesCollected
0x00DCAB4C: hkLocalFrame
0x00DCAB64: children
0x00DCAB74: floats
0x00DCAB7C: parent
0x00DCAB84: hkMemoryResourceHandleExternalLink
0x00DCABAC: hkMemoryResourceHandle
0x00DCABC4: hkResourceContainer
0x00DCABDC: references
0x00DCABEC: hkResourceHandle
0x00DCAC04: memberName
0x00DCAC14: hkMemoryResourceContainer
0x00DCAC34: hkResourceBase
0x00DCAC44: externalId
0x00DCAC54: resourceHandles
0x00DCAC64: near
0x00DCAC6C: materials
0x00DCAC7C: leftHanded
0x00DCAC8C: hkxTextureInplace
0x00DCACA4: angle
0x00DCACAC: selectedNodes
0x00DCACBC: position
0x00DCACCC: hkxScene
0x00DCACDC: cameras
0x00DCACE4: rootNode
0x00DCACF4: filename
0x00DCAD04: modeller
0x00DCAD14: bindPose
0x00DCAD24: selected
0x00DCAD34: hkxNodeAnnotationData
0x00DCAD4C: fileType
0x00DCAD5C: asset
0x00DCAD64: far
0x00DCAD6C: object
0x00DCAD74: meshes
0x00DCAD7C: userProperties
0x00DCAD8C: hkxNode
0x00DCAD94: externalTextures

There's a lot more, but I don't know if that has to do with the actual model file...
What's interesting is that (some of) the physics data is also stored in the model file.
It looks like the mesh data is stored in chunks... not sure if one chunk equals one primitive, but it's worth considering...

Btw, here's a screenshot from the output of the data as how it's stored in the BigFile indices file (BigFile_PS3.sdat):

It's the beginning of my all-in-one parser tool thingy :3

By the way, I haven't found any texture blocks yet.... do they start without a Communicator ID? Because I did see a few of those that looked like they could be textures...

The textures are in DXT1 DXT5 (DXTATI2N - NORMALS), if I recall the segs script EKEY wrote dumped some texture files.

Do the texture files start with a Communicator ID? like COM_CONT, ANIMDATA, PARTITIO_, etc?
Can you attach a sample texture file? So I'm not "groping in the dark"?

[JayK]

Nah the cache dump is different to the ram dump, but I dumped the ram as well soI can give you that also. The cache dump has the scene and and index and DAT file etc, like TheEmbassy.idx TheEmbassy.DAT there's lots of Segs and other stuff insid. I became less interested as time went on but I still have the files. I'll send them to you both soon

[HugoPeters]

Update!
Alright, got quite a lot of stuff going on!
cra0 and I are working on the ultimate Beyond tool!

I'm doing most of the tool itself, the GUI, parser, structure, base features, decompressor, extractor, etc. while cra0 is reversing the model format! And another big thing for me: I've made the switch from GameMaker to C# in Visual Studio..... and damn. It's like heaven. I really want to thank cra0 for helping we with all my noob questions, I'm already getting the hang of it!

Here are some screenshots from the tool in its current form (which is: very unfinished / very early):


Left is where you select the main entries group. The entries are defined from the decrypted BigFile_PS3.sdat file. Then once you loaded the entries group, you get a nice display of all the entries in that group!


Before you can actually begin looking in the entries to see what data they contain you must first Inspect them. You can Inspect every individual entry manually by selecting it, then going to the Inspector tab and then clicking the Inspect button, but you can also Inspect all entries at once by clicking the Tools menu.
Once entries have been Inspected you can do things like searching the entries for types of content (Communicators). The filter can be set by selecting the "Filter" tab on the left... you can also choose between two filter modes: the entry must contain all Communicators in the filter, or the entry must contain one or more of the Communicators in the filter! When you click "Add Communicator" you get a nice popup with a combo box consisting of all known Communicators.


You can also filter the currently loaded entries on offset!


This is the Inspector tab. On the Inspector's Base tab, you can see an overview of all Communicators found within the entries. As you can see in this example, it contains both Communicators set in the filter!


This is a feature in a very early stage: an output of how the entry is constructed.


Here's a neat one: when you double click a Communicator in the Inspector's Base tab, you will be taken to the Hex Viewer, and the selected offset will be automatically set to the offset of that Communicator!

Also, don't forget the decompressor! I found out more about how these files are built, and most of them consist of more than one segs archive!

So yeah, let me know what you guys think! If you have any suggestions, please let me know!

[HugoPeters]

another huge update with a title in a huge font

Thanks to JayK's files I reverse engineered the DC_INFO sections in Beyond's file structure.

Too tired for an explanation now and since no one ever replies here I'm not going to bother posting the structure until someone asks me too

Here's a sample:

[JayK]

Nice, glad them files came in handy.

[HugoPeters]

Alrighty, I now partly understand the way the game accesses its content.
Again, not going to explain too deeply because I suck at explaining and this is pretty complicated xD

Here's a quick look though:



This is for dialog. For example, the first red square gives an index into the FLOW_DIALOG content group, which is included in these:



The enum value for FLOW_DIALOG(_OFFSET) is 0xffc / 0xffb.
Then, the File ID is the next int, in this example it's 0x131a.

Using my tool I can quickly seek to the file bound to that ID, and then hear the audio, and it matches the sentence!

This ID, 0x3f2 stands for AUDIO_DATA. This is from one of JayK's files, The Party.dat or something like that. When I go to the file with that ID, surprise surprise, it's the "Beautiful Body" song!

[HugoPeters]

I've made a video demonstrating the tool and how I find matching audio with text dialog (FLOW_DIALOG):

https://www.youtube.com/watch?v=vunRjVMVXY0

[HugoPeters]

Texture format is reversed!