Important information: this site is currently scheduled to go offline indefinitely by end of the year.

Beyond: Two Souls - Reversing | Porting (THAT'S ALL FOLKS?)

Programs that are related to this or that game.
Hamers
beginner
Posts: 21
Joined: Fri Jul 22, 2011 4:39 pm
Has thanked: 21 times
Been thanked: 13 times

Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars

Post by Hamers »

This is so awesome and completely made my day. Pretty much exactly what I needed. I never thought I would see the day of this happening haha
I've got some experience in C++, Java and VB actually and would love to help, but I absolutely suck at everything related to file structures. I've studied the uncompiled lua file as well and extracted some concept art using a file inspector, but not much beyond that. If you need support with anything I might be able to do feel free to ask (testing for example).

Take all the time you need. Just don't disappear from the face of the earth before releasing the individual file extractor :D
In case I get the lua files decompiled I will let you know immediately ^^

Like some never included gameplay features (Aiden being called Amael and being able to drain energy from plankton and people's emotions according to some lua variables?) Beyond also has some deleted chapters I'm pretty sure you can find some leftovers in the files. Curious learning about them as well. Only know the codenames from concept art that may or may not be accurate. Chapters that apparently got the axe: First Steps, Opening Credits, The Senator, New Agent, Shaman. Also an unreleased chapter that may be set in France and/or Kuala Lumpur.

Will definitely use your tool to try to find something out about them :D
Image
User avatar
HugoPeters
veteran
Posts: 133
Joined: Thu Apr 03, 2014 9:49 am
Location: The Netherlands
Has thanked: 64 times
Been thanked: 168 times
Contact:

Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars

Post by HugoPeters »

Hamers wrote:This is so awesome and completely made my day. Pretty much exactly what I needed. I never thought I would see the day of this happening haha
I've got some experience in C++, Java and VB actually and would love to help, but I absolutely suck at everything related to file structures. I've studied the uncompiled lua file as well and extracted some concept art using a file inspector, but not much beyond that. If you need support with anything I might be able to do feel free to ask (testing for example).

Take all the time you need. Just don't disappear from the face of the earth before releasing the individual file extractor :D
In case I get the lua files decompiled I will let you know immediately ^^

Like some never included gameplay features (Aiden being called Amael and being able to drain energy from plankton and people's emotions according to some lua variables?) Beyond also has some deleted chapters I'm pretty sure you can find some leftovers in the files. Curious learning about them as well. Only know the codenames from concept art that may or may not be accurate. Chapters that apparently got the axe: First Steps, Opening Credits, The Senator, New Agent, Shaman. Also an unreleased chapter that may be set in France and/or Kuala Lumpur.

Will definitely use your tool to try to find something out about them :D
Haha, thanks again!

Yeah, trying to load First Steps will crash the game. I tried, lol.
There might be some leftovers of The Senator, as I did see that being mentioned deeper in the game's structure. As for other unused things.... I already found a few, somewhat unsettling, pieces of unused audio dialog that were intended for a scene at The Bar (Like Other Girls), and I recently found this....

Image

And, well... yeah.

Image

And I've also found some epic unused music intended for a (unused??) scene in Hunted, will try to get that up soon.
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
Hamers
beginner
Posts: 21
Joined: Fri Jul 22, 2011 4:39 pm
Has thanked: 21 times
Been thanked: 13 times

Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars

Post by Hamers »

Yeah I have seen your entry on The Cutting Room Floor (that was my first WTF where is this suddenly coming from moment). Pretty messed up, especially the subtitles you just showed. Seems like you were originally supposed to take control over Aiden in that scene and make the choice of interrupting or just watching, which is pretty crazy. Would have made the revenge scene a lot more intense, but hearing the dialogue alone makes me uncomfortable already. They likely wanted to avoid a huge controversy, especially since not even movies go that far most of the time. Not to mention the rating boards, so they probably made the right call in the end.

The changed dialogue in general is pretty interesting as well. I watched your videos on YT and the prologue dialogue you played is almost completely different from the one that's actually used in the game. Much longer too.

Too bad you can't load First Steps. Apparently you play Jodie as a baby in that scene. Sounds pretty interesting, but considering that's a lot different from the main game scripting wise I'm not surprised it crashes. Interesting that you actually found something on Senator.

The Hunted music has a high chance of being the leftover music from the Beta. Hunted was the first scene they build and showed at E3 2012, so they included some early music that wasn't composed by the actual composer. The scene is probably still in there considering you found the "Broken" chapter as well from E3 2012 featuring the uncut scene of the cop talking to Jodie.

FunnyML (I think he is registered here as well) managed to extract all of the music before (but only in small unnamed chunks) and went through the effort of putting everything together again. He released the entire album over at FFShrine and I uploaded some of the Beta music to my channel. So maybe you are referring to one of these tracks?

https://www.youtube.com/watch?v=nSizbVG1S0o
https://www.youtube.com/watch?v=CKEh2yOAwlk

If not, then it would be pretty interesting if you found even more music :D
Image
User avatar
HugoPeters
veteran
Posts: 133
Joined: Thu Apr 03, 2014 9:49 am
Location: The Netherlands
Has thanked: 64 times
Been thanked: 168 times
Contact:

Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars

Post by HugoPeters »

Hamers wrote:Yeah I have seen your entry on The Cutting Room Floor (that was my first WTF where is this suddenly coming from moment). Pretty messed up, especially the subtitles you just showed. Seems like you were originally supposed to take control over Aiden in that scene and make the choice of interrupting or just watching, which is pretty crazy. Would have made the revenge scene a lot more intense, but hearing the dialogue alone makes me uncomfortable already. They likely wanted to avoid a huge controversy, especially since not even movies go that far most of the time. Not to mention the rating boards, so they probably made the right call in the end.

The changed dialogue in general is pretty interesting as well. I watched your videos on YT and the prologue dialogue you played is almost completely different from the one that's actually used in the game. Much longer too.

Too bad you can't load First Steps. Apparently you play Jodie as a baby in that scene. Sounds pretty interesting, but considering that's a lot different from the main game scripting wise I'm not surprised it crashes. Interesting that you actually found something on Senator.

The Hunted music has a high chance of being the leftover music from the Beta. Hunted was the first scene they build and showed at E3 2012, so they included some early music that wasn't composed by the actual composer. The scene is probably still in there considering you found the "Broken" chapter as well from E3 2012 featuring the uncut scene of the cop talking to Jodie.

FunnyML (I think he is registered here as well) managed to extract all of the music before (but only in small unnamed chunks) and went through the effort of putting everything together again. He released the entire album over at FFShrine and I uploaded some of the Beta music to my channel. So maybe you are referring to one of these tracks?

https://www.youtube.com/watch?v=nSizbVG1S0o
https://www.youtube.com/watch?v=CKEh2yOAwlk

If not, then it would be pretty interesting if you found even more music :D
Small unnamed chunks... that's correct, the music is formatted like this:

uint: channel amount (always 3?)

uint: channel 1 chunk size
uint: channel 2 chunk size
uint: channel 3 chunk size

< channel 1 chunk >
< channel 2 chunk >
< channel 3 chunk >

Then relative alignment to 0x800 and then another block of these chunks.
This goes on until the end of the PARTITIO attribute.

You've probably already seen it, but I did already release a (poorly made lol) file extractor that also exports the audio to .mp3.

Anyways here is one fucking epic (unused) track, which I can't remember of where it came from:
http://www.djek.nl/shit/beyond/EB89.mp3

Then there's the chapter 1803 Hunted Bike... In the game the bike part is called 1802B Hunted Barricade. Then 1804 is Hunted Town... so I'm thinking Hunted Bike is
completely unused! EDIT: actually I don't think the -whole- chapter is unused :A that audio is though!

Here's some audio from it:

http://www.djek.nl/shit/beyond/1BBE.mp3 (love this track)
http://www.djek.nl/shit/beyond/1953.mp3
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
User avatar
HugoPeters
veteran
Posts: 133
Joined: Thu Apr 03, 2014 9:49 am
Location: The Netherlands
Has thanked: 64 times
Been thanked: 168 times
Contact:

Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars

Post by HugoPeters »

Here's a better demo video! :D

http://youtu.be/D59SAbFSIoU
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
Hamers
beginner
Posts: 21
Joined: Fri Jul 22, 2011 4:39 pm
Has thanked: 21 times
Been thanked: 13 times

Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars

Post by Hamers »

HugoPeters wrote:Small unnamed chunks... that's correct, the music is formatted like this:

uint: channel amount (always 3?)

uint: channel 1 chunk size
uint: channel 2 chunk size
uint: channel 3 chunk size

< channel 1 chunk >
< channel 2 chunk >
< channel 3 chunk >

Then relative alignment to 0x800 and then another block of these chunks.
This goes on until the end of the PARTITIO attribute.

You've probably already seen it, but I did already release a (poorly made lol) file extractor that also exports the audio to .mp3.

Anyways here is one fucking epic (unused) track, which I can't remember of where it came from:
http://www.djek.nl/shit/beyond/EB89.mp3

Then there's the chapter 1803 Hunted Bike... In the game the bike part is called 1802B Hunted Barricade. Then 1804 is Hunted Town... so I'm thinking Hunted Bike is
completely unused! EDIT: actually I don't think the -whole- chapter is unused :A that audio is though!

Here's some audio from it:

http://www.djek.nl/shit/beyond/1BBE.mp3 (love this track)
http://www.djek.nl/shit/beyond/1953.mp3
I think the EB89 one is actually used in Dragon's Hideout near the end if you wait too long or something. I certainly remember it. But the other two are completely new to me and weren't included in FunnyML's release (http://forums.ffshrine.org/f72/2013-|-l ... nd-164892/). They might also be from the Beta version of the Bike scene though. I'm not sure as I don't have CFW installed, but I think 1802B Hunted Barricade is the part where you have to steal the bike while 1803 Hunted Bike is the actual bike part. They definitely load another scene after you escaped the forest and then again after you take off with the bike, so that would make sense.

HugoPeters wrote:Here's a better demo video! :D

http://youtu.be/D59SAbFSIoU
Really great work with the storyboard parsing. I assume this means you could at some point extract all the assets and scripts used by a chapter, which would be really helpful considering the game has so many assets apparently all over the place. To find something specific it certainly is a lot easier to just search in the related chapter instead of the entire game.
Image
User avatar
HugoPeters
veteran
Posts: 133
Joined: Thu Apr 03, 2014 9:49 am
Location: The Netherlands
Has thanked: 64 times
Been thanked: 168 times
Contact:

Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin

Post by HugoPeters »

Alright, EPIC UPDATE

JayK helped me with decompiling the Luac scripts, and guess what, perfect match!

Image

So the music we get is: http://www.djek.nl/shit/beyond/MusicMenu_1653D.mp3!!!


hehe.

I've been trying to get Mogre to work, and finally I'm getting some results. I remade the loading / splash screen of Beyond!

Code for making the splashscreen appear:
Image

And to make it fade-in / out:
Image
(if there's a better way of doing that, please let me know!)

Result:
Image

Thought I'd brag about that a little bit :roll:
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
JayK
mega-veteran
mega-veteran
Posts: 172
Joined: Fri Jun 01, 2012 10:08 am
Has thanked: 35 times
Been thanked: 121 times

Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin

Post by JayK »

Great work, is Mogre any good?
User avatar
HugoPeters
veteran
Posts: 133
Joined: Thu Apr 03, 2014 9:49 am
Location: The Netherlands
Has thanked: 64 times
Been thanked: 168 times
Contact:

Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin

Post by HugoPeters »

JayK wrote:Great work, is Mogre any good?
I'm liking it thusfar (though getting it to work was a pain)! Getting framerates of around 6000 while just having the splash screen fading in/out on a GTX 770, so pretty good!
I've only done a splash screen so far, the code behind it first didn't make a lot of sense but while making it I began to understand how Mogre / Ogre works, and I like it! But for real 3D stuff I can't really share a solid opinion yet as I haven't done anything of that kind yet.
Last edited by HugoPeters on Tue May 27, 2014 11:31 pm, edited 1 time in total.
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
User avatar
HugoPeters
veteran
Posts: 133
Joined: Thu Apr 03, 2014 9:49 am
Location: The Netherlands
Has thanked: 64 times
Been thanked: 168 times
Contact:

Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin

Post by HugoPeters »

Check it out, unused loading screen audio!
http://www.djek.nl/shit/beyond/LoadingScreenAudio.mp3
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
Hamers
beginner
Posts: 21
Joined: Fri Jul 22, 2011 4:39 pm
Has thanked: 21 times
Been thanked: 13 times

Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin

Post by Hamers »

Great fucking work! Glad someone else is interested in the LUA scripts :D
Image
User avatar
HugoPeters
veteran
Posts: 133
Joined: Thu Apr 03, 2014 9:49 am
Location: The Netherlands
Has thanked: 64 times
Been thanked: 168 times
Contact:

Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin

Post by HugoPeters »

Hamers wrote:Great fucking work! Glad someone else is interested in the LUA scripts :D
Well, to be honest, my goal is to get them to actually execute on my PC.

The files JayK sent me before contained a lot of function "outputs" and were to source to finding the Communicators and their ID's.
Now, in the decrypted main executable, I can see the templates of these functions and that they're just strings where the Communicator names and ID's are filled in to. Because I already generated a complete enum of the Communicators I can easily remake the generation of these functions.

An example:

function QDT.GAME_MANAGER.GetType() return 4222 end (not sure if this is valid Lua, not on my PC right now :A)

That's the simplest function, there's also {Communicator}.NewPlaced, {Communicator}.NewPlacedCtx, {Communicator}.New, etc.

I use DynamicLua to use Lua in C#, and it's pretty dope.

To initialize this "dynamic database", I loop through the whole Communicator enum, getting the name and the ID of that Communicator.
Then I generate the functions as strings in a string array, just like the main executable seems to do.
After that I loop through the string array, executing them on the Lua instance.

It's really neat, to define a function I just have to do something like IW.Lua("function QDT.GAME_MANAGER.GetType() return 4222 end"); (IW being my main "glue" of all settings, sub-classes, etc.), and IW.Lua being the DynamicLua instance.

After that I just do this: IW.Lua.QDT.GAME_MANAGER.GetType()! Because it's generated on runtime I can use it like a native method :D
To get a correct output is a bit different though (not 100% sure, but it works).
To make it display on a message box I do: MessageBox.Show(IW.Lua("return QDT.GAME_MANAGER.GetType()").ToString());.
Still, that's pretty fucking neat :D

So once I have all of Beyond's kernel functions figured out I can call the Game Manager (which is a huge uncompiled Lua script which takes care of all game-related variables, a ton of game-related functions and also the functions which start the storyboard).

Beyond isn't actually that complicated, the whole system is pretty minimalistic, but there are two things that still twist my mind:
  • Why are the names of the Communicator attributes-"attributes" (the ones you see in the data containers, like COM_CONT, MESHDATA, PARTITIO, LOADCONT, etc.) still present? Isn't it much more efficient to just have a uint of the type of the attribute?? What's the advantage of having these strings still in there? Using uints would save a huge space considering all of these attribute names are 0x8 in size, every data container has a lot of 'em, and there are more than 30000 data containers!
  • The way the DC_DATA section is formatted makes no sense, there are no offsets, just sizes before each Communicator attribute (so not the attribute-attributes like COM_CONT, but the blocks that contain them). I'm suspecting these tables are Lua tables of some kind. That would make sense, as Lua tables can contain their own functions, and a lot of these data containers have compiled scripts in them. Then still, the only way of parsing these DC_DATA tables is to go through each and every attribute and either skip the size of them to go to the next one or parse the whole thing, which would take up too much memory for a PS3 I think, however, the bigger data containers that contain textures have a "head" and "tail", the head being one segs archive containing the DC_INFO and DC_DATA section, and the the tail being the (compressed) texture.
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
Hamers
beginner
Posts: 21
Joined: Fri Jul 22, 2011 4:39 pm
Has thanked: 21 times
Been thanked: 13 times

Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin

Post by Hamers »

Whoa. So you could potentially directly (or at least partially) send commands to the game engine while running and call functions? That would be incredible :eek:
Image
User avatar
HugoPeters
veteran
Posts: 133
Joined: Thu Apr 03, 2014 9:49 am
Location: The Netherlands
Has thanked: 64 times
Been thanked: 168 times
Contact:

Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin

Post by HugoPeters »

Hamers wrote:Whoa. So you could potentially directly (or at least partially) send commands to the game engine while running and call functions? That would be incredible :eek:
What I'm doing now is basically tying to remake Beyond's engine, but sending commands to the PS3's engine would be possible, yeah.
How that would work:

First, figure out a script that we want to put our own things in.

Then to patch it:
1. If compressed, decompress the data container
2. Decompile the function
3. Add our own stuff
4. Recompile it
5. Write it to the data container, modifiying sizes and such (not sure if there's any CRC's, but eventually recalculate those)
6. If it was compressed, recompress it
7. Write it to the game's files
8. Update the defined size in the SDAT file (decrypt first and eventually modify the ExSize if that entry has stuff like textures the re-encrypt)

This is pretty limited though I'm afraid, as the functionality depends on how much Lua can do, but sure we could use the native functions of the engine.

Though I'm more interested in remaking the engine :twisted:
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
Hamers
beginner
Posts: 21
Joined: Fri Jul 22, 2011 4:39 pm
Has thanked: 21 times
Been thanked: 13 times

Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin

Post by Hamers »

Well that would be the best thing ever probably. But how? I mean the renderer is obviously pretty advanced and custom tailored to the SPUs right? How would you make that running on PC? Or do you plan to write an emulator specifically for Beyond :D
Image
Post Reply