This is so awesome and completely made my day. Pretty much exactly what I needed. I never thought I would see the day of this happening haha
I've got some experience in C++, Java and VB actually and would love to help, but I absolutely suck at everything related to file structures. I've studied the uncompiled lua file as well and extracted some concept art using a file inspector, but not much beyond that. If you need support with anything I might be able to do feel free to ask (testing for example).
Take all the time you need. Just don't disappear from the face of the earth before releasing the individual file extractor
In case I get the lua files decompiled I will let you know immediately ^^
Like some never included gameplay features (Aiden being called Amael and being able to drain energy from plankton and people's emotions according to some lua variables?) Beyond also has some deleted chapters I'm pretty sure you can find some leftovers in the files. Curious learning about them as well. Only know the codenames from concept art that may or may not be accurate. Chapters that apparently got the axe: First Steps, Opening Credits, The Senator, New Agent, Shaman. Also an unreleased chapter that may be set in France and/or Kuala Lumpur.
Will definitely use your tool to try to find something out about them
Important information: this site is currently scheduled to go offline indefinitely by end of the year.
Beyond: Two Souls - Reversing | Porting (THAT'S ALL FOLKS?)
- HugoPeters
- veteran
- Posts: 133
- Joined: Thu Apr 03, 2014 9:49 am
- Location: The Netherlands
- Has thanked: 64 times
- Been thanked: 168 times
- Contact:
Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
Haha, thanks again!Hamers wrote:This is so awesome and completely made my day. Pretty much exactly what I needed. I never thought I would see the day of this happening haha
I've got some experience in C++, Java and VB actually and would love to help, but I absolutely suck at everything related to file structures. I've studied the uncompiled lua file as well and extracted some concept art using a file inspector, but not much beyond that. If you need support with anything I might be able to do feel free to ask (testing for example).
Take all the time you need. Just don't disappear from the face of the earth before releasing the individual file extractor
In case I get the lua files decompiled I will let you know immediately ^^
Like some never included gameplay features (Aiden being called Amael and being able to drain energy from plankton and people's emotions according to some lua variables?) Beyond also has some deleted chapters I'm pretty sure you can find some leftovers in the files. Curious learning about them as well. Only know the codenames from concept art that may or may not be accurate. Chapters that apparently got the axe: First Steps, Opening Credits, The Senator, New Agent, Shaman. Also an unreleased chapter that may be set in France and/or Kuala Lumpur.
Will definitely use your tool to try to find something out about them
Yeah, trying to load First Steps will crash the game. I tried, lol.
There might be some leftovers of The Senator, as I did see that being mentioned deeper in the game's structure. As for other unused things.... I already found a few, somewhat unsettling, pieces of unused audio dialog that were intended for a scene at The Bar (Like Other Girls), and I recently found this....
And, well... yeah.
And I've also found some epic unused music intended for a (unused??) scene in Hunted, will try to get that up soon.
"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
-
- beginner
- Posts: 21
- Joined: Fri Jul 22, 2011 4:39 pm
- Has thanked: 21 times
- Been thanked: 13 times
Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
Yeah I have seen your entry on The Cutting Room Floor (that was my first WTF where is this suddenly coming from moment). Pretty messed up, especially the subtitles you just showed. Seems like you were originally supposed to take control over Aiden in that scene and make the choice of interrupting or just watching, which is pretty crazy. Would have made the revenge scene a lot more intense, but hearing the dialogue alone makes me uncomfortable already. They likely wanted to avoid a huge controversy, especially since not even movies go that far most of the time. Not to mention the rating boards, so they probably made the right call in the end.
The changed dialogue in general is pretty interesting as well. I watched your videos on YT and the prologue dialogue you played is almost completely different from the one that's actually used in the game. Much longer too.
Too bad you can't load First Steps. Apparently you play Jodie as a baby in that scene. Sounds pretty interesting, but considering that's a lot different from the main game scripting wise I'm not surprised it crashes. Interesting that you actually found something on Senator.
The Hunted music has a high chance of being the leftover music from the Beta. Hunted was the first scene they build and showed at E3 2012, so they included some early music that wasn't composed by the actual composer. The scene is probably still in there considering you found the "Broken" chapter as well from E3 2012 featuring the uncut scene of the cop talking to Jodie.
FunnyML (I think he is registered here as well) managed to extract all of the music before (but only in small unnamed chunks) and went through the effort of putting everything together again. He released the entire album over at FFShrine and I uploaded some of the Beta music to my channel. So maybe you are referring to one of these tracks?
https://www.youtube.com/watch?v=nSizbVG1S0o
https://www.youtube.com/watch?v=CKEh2yOAwlk
If not, then it would be pretty interesting if you found even more music
The changed dialogue in general is pretty interesting as well. I watched your videos on YT and the prologue dialogue you played is almost completely different from the one that's actually used in the game. Much longer too.
Too bad you can't load First Steps. Apparently you play Jodie as a baby in that scene. Sounds pretty interesting, but considering that's a lot different from the main game scripting wise I'm not surprised it crashes. Interesting that you actually found something on Senator.
The Hunted music has a high chance of being the leftover music from the Beta. Hunted was the first scene they build and showed at E3 2012, so they included some early music that wasn't composed by the actual composer. The scene is probably still in there considering you found the "Broken" chapter as well from E3 2012 featuring the uncut scene of the cop talking to Jodie.
FunnyML (I think he is registered here as well) managed to extract all of the music before (but only in small unnamed chunks) and went through the effort of putting everything together again. He released the entire album over at FFShrine and I uploaded some of the Beta music to my channel. So maybe you are referring to one of these tracks?
https://www.youtube.com/watch?v=nSizbVG1S0o
https://www.youtube.com/watch?v=CKEh2yOAwlk
If not, then it would be pretty interesting if you found even more music
- HugoPeters
- veteran
- Posts: 133
- Joined: Thu Apr 03, 2014 9:49 am
- Location: The Netherlands
- Has thanked: 64 times
- Been thanked: 168 times
- Contact:
Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
Small unnamed chunks... that's correct, the music is formatted like this:Hamers wrote:Yeah I have seen your entry on The Cutting Room Floor (that was my first WTF where is this suddenly coming from moment). Pretty messed up, especially the subtitles you just showed. Seems like you were originally supposed to take control over Aiden in that scene and make the choice of interrupting or just watching, which is pretty crazy. Would have made the revenge scene a lot more intense, but hearing the dialogue alone makes me uncomfortable already. They likely wanted to avoid a huge controversy, especially since not even movies go that far most of the time. Not to mention the rating boards, so they probably made the right call in the end.
The changed dialogue in general is pretty interesting as well. I watched your videos on YT and the prologue dialogue you played is almost completely different from the one that's actually used in the game. Much longer too.
Too bad you can't load First Steps. Apparently you play Jodie as a baby in that scene. Sounds pretty interesting, but considering that's a lot different from the main game scripting wise I'm not surprised it crashes. Interesting that you actually found something on Senator.
The Hunted music has a high chance of being the leftover music from the Beta. Hunted was the first scene they build and showed at E3 2012, so they included some early music that wasn't composed by the actual composer. The scene is probably still in there considering you found the "Broken" chapter as well from E3 2012 featuring the uncut scene of the cop talking to Jodie.
FunnyML (I think he is registered here as well) managed to extract all of the music before (but only in small unnamed chunks) and went through the effort of putting everything together again. He released the entire album over at FFShrine and I uploaded some of the Beta music to my channel. So maybe you are referring to one of these tracks?
https://www.youtube.com/watch?v=nSizbVG1S0o
https://www.youtube.com/watch?v=CKEh2yOAwlk
If not, then it would be pretty interesting if you found even more music
uint: channel amount (always 3?)
uint: channel 1 chunk size
uint: channel 2 chunk size
uint: channel 3 chunk size
< channel 1 chunk >
< channel 2 chunk >
< channel 3 chunk >
Then relative alignment to 0x800 and then another block of these chunks.
This goes on until the end of the PARTITIO attribute.
You've probably already seen it, but I did already release a (poorly made lol) file extractor that also exports the audio to .mp3.
Anyways here is one fucking epic (unused) track, which I can't remember of where it came from:
http://www.djek.nl/shit/beyond/EB89.mp3
Then there's the chapter 1803 Hunted Bike... In the game the bike part is called 1802B Hunted Barricade. Then 1804 is Hunted Town... so I'm thinking Hunted Bike is
completely unused! EDIT: actually I don't think the -whole- chapter is unused :A that audio is though!
Here's some audio from it:
http://www.djek.nl/shit/beyond/1BBE.mp3 (love this track)
http://www.djek.nl/shit/beyond/1953.mp3
"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
- HugoPeters
- veteran
- Posts: 133
- Joined: Thu Apr 03, 2014 9:49 am
- Location: The Netherlands
- Has thanked: 64 times
- Been thanked: 168 times
- Contact:
Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
-
- beginner
- Posts: 21
- Joined: Fri Jul 22, 2011 4:39 pm
- Has thanked: 21 times
- Been thanked: 13 times
Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
I think the EB89 one is actually used in Dragon's Hideout near the end if you wait too long or something. I certainly remember it. But the other two are completely new to me and weren't included in FunnyML's release (http://forums.ffshrine.org/f72/2013-|-l ... nd-164892/). They might also be from the Beta version of the Bike scene though. I'm not sure as I don't have CFW installed, but I think 1802B Hunted Barricade is the part where you have to steal the bike while 1803 Hunted Bike is the actual bike part. They definitely load another scene after you escaped the forest and then again after you take off with the bike, so that would make sense.HugoPeters wrote:Small unnamed chunks... that's correct, the music is formatted like this:
uint: channel amount (always 3?)
uint: channel 1 chunk size
uint: channel 2 chunk size
uint: channel 3 chunk size
< channel 1 chunk >
< channel 2 chunk >
< channel 3 chunk >
Then relative alignment to 0x800 and then another block of these chunks.
This goes on until the end of the PARTITIO attribute.
You've probably already seen it, but I did already release a (poorly made lol) file extractor that also exports the audio to .mp3.
Anyways here is one fucking epic (unused) track, which I can't remember of where it came from:
http://www.djek.nl/shit/beyond/EB89.mp3
Then there's the chapter 1803 Hunted Bike... In the game the bike part is called 1802B Hunted Barricade. Then 1804 is Hunted Town... so I'm thinking Hunted Bike is
completely unused! EDIT: actually I don't think the -whole- chapter is unused :A that audio is though!
Here's some audio from it:
http://www.djek.nl/shit/beyond/1BBE.mp3 (love this track)
http://www.djek.nl/shit/beyond/1953.mp3
Really great work with the storyboard parsing. I assume this means you could at some point extract all the assets and scripts used by a chapter, which would be really helpful considering the game has so many assets apparently all over the place. To find something specific it certainly is a lot easier to just search in the related chapter instead of the entire game.
- HugoPeters
- veteran
- Posts: 133
- Joined: Thu Apr 03, 2014 9:49 am
- Location: The Netherlands
- Has thanked: 64 times
- Been thanked: 168 times
- Contact:
Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin
Alright, EPIC UPDATE
JayK helped me with decompiling the Luac scripts, and guess what, perfect match!
So the music we get is: http://www.djek.nl/shit/beyond/MusicMenu_1653D.mp3!!!
hehe.
I've been trying to get Mogre to work, and finally I'm getting some results. I remade the loading / splash screen of Beyond!
Code for making the splashscreen appear:
And to make it fade-in / out:
(if there's a better way of doing that, please let me know!)
Result:
Thought I'd brag about that a little bit
JayK helped me with decompiling the Luac scripts, and guess what, perfect match!
So the music we get is: http://www.djek.nl/shit/beyond/MusicMenu_1653D.mp3!!!
hehe.
I've been trying to get Mogre to work, and finally I'm getting some results. I remade the loading / splash screen of Beyond!
Code for making the splashscreen appear:
And to make it fade-in / out:
(if there's a better way of doing that, please let me know!)
Result:
Thought I'd brag about that a little bit
"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
- HugoPeters
- veteran
- Posts: 133
- Joined: Thu Apr 03, 2014 9:49 am
- Location: The Netherlands
- Has thanked: 64 times
- Been thanked: 168 times
- Contact:
Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin
I'm liking it thusfar (though getting it to work was a pain)! Getting framerates of around 6000 while just having the splash screen fading in/out on a GTX 770, so pretty good!JayK wrote:Great work, is Mogre any good?
I've only done a splash screen so far, the code behind it first didn't make a lot of sense but while making it I began to understand how Mogre / Ogre works, and I like it! But for real 3D stuff I can't really share a solid opinion yet as I haven't done anything of that kind yet.
Last edited by HugoPeters on Tue May 27, 2014 11:31 pm, edited 1 time in total.
"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
- HugoPeters
- veteran
- Posts: 133
- Joined: Thu Apr 03, 2014 9:49 am
- Location: The Netherlands
- Has thanked: 64 times
- Been thanked: 168 times
- Contact:
Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin
Check it out, unused loading screen audio!
http://www.djek.nl/shit/beyond/LoadingScreenAudio.mp3
http://www.djek.nl/shit/beyond/LoadingScreenAudio.mp3
"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
- HugoPeters
- veteran
- Posts: 133
- Joined: Thu Apr 03, 2014 9:49 am
- Location: The Netherlands
- Has thanked: 64 times
- Been thanked: 168 times
- Contact:
Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin
Well, to be honest, my goal is to get them to actually execute on my PC.Hamers wrote:Great fucking work! Glad someone else is interested in the LUA scripts
The files JayK sent me before contained a lot of function "outputs" and were to source to finding the Communicators and their ID's.
Now, in the decrypted main executable, I can see the templates of these functions and that they're just strings where the Communicator names and ID's are filled in to. Because I already generated a complete enum of the Communicators I can easily remake the generation of these functions.
An example:
function QDT.GAME_MANAGER.GetType() return 4222 end (not sure if this is valid Lua, not on my PC right now :A)
That's the simplest function, there's also {Communicator}.NewPlaced, {Communicator}.NewPlacedCtx, {Communicator}.New, etc.
I use DynamicLua to use Lua in C#, and it's pretty dope.
To initialize this "dynamic database", I loop through the whole Communicator enum, getting the name and the ID of that Communicator.
Then I generate the functions as strings in a string array, just like the main executable seems to do.
After that I loop through the string array, executing them on the Lua instance.
It's really neat, to define a function I just have to do something like IW.Lua("function QDT.GAME_MANAGER.GetType() return 4222 end"); (IW being my main "glue" of all settings, sub-classes, etc.), and IW.Lua being the DynamicLua instance.
After that I just do this: IW.Lua.QDT.GAME_MANAGER.GetType()! Because it's generated on runtime I can use it like a native method
To get a correct output is a bit different though (not 100% sure, but it works).
To make it display on a message box I do: MessageBox.Show(IW.Lua("return QDT.GAME_MANAGER.GetType()").ToString());.
Still, that's pretty fucking neat
So once I have all of Beyond's kernel functions figured out I can call the Game Manager (which is a huge uncompiled Lua script which takes care of all game-related variables, a ton of game-related functions and also the functions which start the storyboard).
Beyond isn't actually that complicated, the whole system is pretty minimalistic, but there are two things that still twist my mind:
- Why are the names of the Communicator attributes-"attributes" (the ones you see in the data containers, like COM_CONT, MESHDATA, PARTITIO, LOADCONT, etc.) still present? Isn't it much more efficient to just have a uint of the type of the attribute?? What's the advantage of having these strings still in there? Using uints would save a huge space considering all of these attribute names are 0x8 in size, every data container has a lot of 'em, and there are more than 30000 data containers!
- The way the DC_DATA section is formatted makes no sense, there are no offsets, just sizes before each Communicator attribute (so not the attribute-attributes like COM_CONT, but the blocks that contain them). I'm suspecting these tables are Lua tables of some kind. That would make sense, as Lua tables can contain their own functions, and a lot of these data containers have compiled scripts in them. Then still, the only way of parsing these DC_DATA tables is to go through each and every attribute and either skip the size of them to go to the next one or parse the whole thing, which would take up too much memory for a PS3 I think, however, the bigger data containers that contain textures have a "head" and "tail", the head being one segs archive containing the DC_INFO and DC_DATA section, and the the tail being the (compressed) texture.
"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
-
- beginner
- Posts: 21
- Joined: Fri Jul 22, 2011 4:39 pm
- Has thanked: 21 times
- Been thanked: 13 times
Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin
Whoa. So you could potentially directly (or at least partially) send commands to the game engine while running and call functions? That would be incredible
- HugoPeters
- veteran
- Posts: 133
- Joined: Thu Apr 03, 2014 9:49 am
- Location: The Netherlands
- Has thanked: 64 times
- Been thanked: 168 times
- Contact:
Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin
What I'm doing now is basically tying to remake Beyond's engine, but sending commands to the PS3's engine would be possible, yeah.Hamers wrote:Whoa. So you could potentially directly (or at least partially) send commands to the game engine while running and call functions? That would be incredible
How that would work:
First, figure out a script that we want to put our own things in.
Then to patch it:
1. If compressed, decompress the data container
2. Decompile the function
3. Add our own stuff
4. Recompile it
5. Write it to the data container, modifiying sizes and such (not sure if there's any CRC's, but eventually recalculate those)
6. If it was compressed, recompress it
7. Write it to the game's files
8. Update the defined size in the SDAT file (decrypt first and eventually modify the ExSize if that entry has stuff like textures the re-encrypt)
This is pretty limited though I'm afraid, as the functionality depends on how much Lua can do, but sure we could use the native functions of the engine.
Though I'm more interested in remaking the engine
"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
-
- beginner
- Posts: 21
- Joined: Fri Jul 22, 2011 4:39 pm
- Has thanked: 21 times
- Been thanked: 13 times
Re: [PS3] Beyond: Two Souls - Extractor | Reversing | Portin
Well that would be the best thing ever probably. But how? I mean the renderer is obviously pretty advanced and custom tailored to the SPUs right? How would you make that running on PC? Or do you plan to write an emulator specifically for Beyond