Code Submissions

[Mr.Mouse]

Bloody hell, I see what you mean. If the plugin would support an extension that is not yet supported, it would not register it. SILLY SILLY ME!!!!! OMG I'm so STUPID sometimes.

I clearly used the if statement for the purpose of CREATING a new extension if it was not supported yet, I must have simply forgotten after I tested some code. "Ah it works, let's move on to a new, more important function" Damn.

[EDIT]Okay, I will fix that, the way you proposed would be nice, but it won't work like that. I will do it immediately after it turns out that an extension is not yet in the current base of the Commander.

[Rahly]

I don't readily see the benefit of "*.arch?" over "arch?", so I think the "arch?" approach will do nicely for the plugins. Perhaps I need to fix that stuff also for the script-processed formats, but that will require a bit more work, as the ScriptBinder (MexBinder) needs to be fixed for that as well.

The benefit is for something in the future, like ntfs supports.

"*.gfx.arch?" "*.script.arch?" may be the same type of archive, but you might wanna add support for preview. or maybe "*.gfx.arch?" isn't compressed but the "*.script.arch?" is compressed, but there are no "flags" in the file to indicate it, so you have to treat them as seperate archives.

Sorry, prolly not the BEST of examples, but an example, nonetheless.

[Mr.Mouse]

I don't readily see the benefit of "*.arch?" over "arch?", so I think the "arch?" approach will do nicely for the plugins. Perhaps I need to fix that stuff also for the script-processed formats, but that will require a bit more work, as the ScriptBinder (MexBinder) needs to be fixed for that as well.

The benefit is for something in the future, like ntfs supports.

"*.gfx.arch?" "*.script.arch?" may be the same type of archive, but you might wanna add support for preview. or maybe "*.gfx.arch?" isn't compressed but the "*.script.arch?" is compressed, but there are no "flags" in the file to indicate it, so you have to treat them as seperate archives.

Sorry, prolly not the BEST of examples, but an example, nonetheless.

Ok point taken.

[Rahly]

Okay, I will fix that, the way you proposed would be nice, but it won't work like that. I will do it immediately after it turns out that an extension is not yet in the current base of the Commander.

So you do have to redim it I thought so, i wasnt sure if they changed that since my VB4 days.

[EDIT]There are too many languages that DO work that way now, perl and php are just 2 examples. I still do a lot of work in perl.

[Rahly]

Ok, removed mpGetCreateOptions() function and replaced it with

mpGetOptions(FormatIndex: Integer; OptionType: Integer; Options: PChar): Boolean;

This allows plugins to provide Export/Import options as well, like, if they wanna convert one of the files in it, from a proprietary format, to a more standard format, or one that works with an application.

[Mr.Mouse]

Okay, but the plugin will also need to convert them back to the one the format uses when they import, else the game will mostlikely fail to use the (new) resource.

On another note, also directed to WATTO, I'm working at the background (in free, free time) on a little app that will focus on identifying parts of formats in newly encountered archives. The most easy example would be, say, WAV files. Suppose an archive contains Microsoft Wave files, then you would find a string like "RIFF", exposing the beginning of the file in the archive. This offset might be found somewhere else in the file (as part of a header or tail). Now, you could also specify that the FileID entry has also a size variable (whcih the RIFFs have for instance) at at what distance from the start of the file it can be found, what type it is etc. ). That way you could also possibly find a Size variable somewhere in the archive in a header or tail. You see where this leads, and I think it is just an expansion what your basic Ripper would do, however, it would also try to MAP a header or tail. The possibilities are there I think to create a custom extractor, that users could try and run on an archive they encountered, and if the analysis of the tool would be succesful, they could at least extract stuff from it. The tool could create a text file that users might want to send to us, so we can have already some idea of the format before we delve into it with a hex editor. But most of all, it would help out users that want to get something out of non-supported formats.
What do you think?

[friendsofwatto]

On another note, also directed to WATTO, I'm working at the background (in free, free time) on a little app that will focus on identifying parts of formats in newly encountered archives. The most easy example would be, say, WAV files. Suppose an archive contains Microsoft Wave files, then you would find a string like "RIFF", exposing the beginning of the file in the archive. This offset might be found somewhere else in the file (as part of a header or tail). Now, you could also specify that the FileID entry has also a size variable (whcih the RIFFs have for instance) at at what distance from the start of the file it can be found, what type it is etc. ). That way you could also possibly find a Size variable somewhere in the archive in a header or tail. You see where this leads, and I think it is just an expansion what your basic Ripper would do, however, it would also try to MAP a header or tail. The possibilities are there I think to create a custom extractor, that users could try and run on an archive they encountered, and if the analysis of the tool would be succesful, they could at least extract stuff from it. The tool could create a text file that users might want to send to us, so we can have already some idea of the format before we delve into it with a hex editor. But most of all, it would help out users that want to get something out of non-supported formats.
What do you think?

Mr Mouse,

Yes, this sounds like a good idea, and I myself have contemplated creating a program like this before, however i think it would be quite painful to accomplish.

Firstly, we would need to fully understand that the program is developed for non-experienced users - not us programmers - because face it, we already know how to read archives . With this in mind, we would have to cram as much as possible into the program, but still keep it rather simple to use.

Secondly, we would need to program it to recognise as many different archive formats as possible

Thirdly, we would need to have the program recognise as many different header types as possible, so we may need to research this alittle.

I think this is a great idea, but alot of work. I would be happy to help out with the program, assuming it is in Java, VB, or to a lesser degree, .Net/C#/J# etc.

Sorry for sounding alittle down about it, I do think it would be a great tool to have!

WATTO
[email protected]
http://www.watto.org

[Rahly]

Okay, but the plugin will also need to convert them back to the one the format uses when they import, else the game will mostlikely fail to use the (new) resource.

Thats up to the plugin designer.

On another note, also directed to WATTO, I'm working at the background (in free, free time) on a little app that will focus on identifying parts of formats in newly encountered archives. The most easy example would be, say, WAV files. Suppose an archive contains Microsoft Wave files, then you would find a string like "RIFF", exposing the beginning of the file in the archive. This offset might be found somewhere else in the file (as part of a header or tail). Now, you could also specify that the FileID entry has also a size variable (whcih the RIFFs have for instance) at at what distance from the start of the file it can be found, what type it is etc. ). That way you could also possibly find a Size variable somewhere in the archive in a header or tail. You see where this leads, and I think it is just an expansion what your basic Ripper would do, however, it would also try to MAP a header or tail. The possibilities are there I think to create a custom extractor, that users could try and run on an archive they encountered, and if the analysis of the tool would be succesful, they could at least extract stuff from it. The tool could create a text file that users might want to send to us, so we can have already some idea of the format before we delve into it with a hex editor. But most of all, it would help out users that want to get something out of non-supported formats.
What do you think?

Sorry, I know you were talking to Watto, but.....

It sounds like a great idea, as long as file is still in raw format (not encrypted/compressed). It sounds almost like your trying to recreate the unix command "file". The only difference, is file, only works on the original file position, you want it to go through the entire file. ie "file bob.avi" -> "RIFF Video File" "file bob.wav" -> "Riff Audio Wave PCM file"

[Mr.Mouse]

Firstly, we would need to fully understand that the program is developed for non-experienced users - not us programmers - because face it, we already know how to read archives . With this in mind, we would have to cram as much as possible into the program, but still keep it rather simple to use.

Yes. It should have a Wizardish kind of feel.

Secondly, we would need to program it to recognise as many different archive formats as possible

Thirdly, we would need to have the program recognise as many different header types as possible, so we may need to research this alittle.

Yes, we need to do research on this. Not in the least, we should define a way the program would need to be coded, and how we should be able to easily "Feed" it new formats/header types/fileids etc.

As an example, and a very early prototype, here's my Mexscan I talked about:
http://www.xentax.com/downloads/oldcode ... exscan.zip
This little dos app takes script commands to do some of the work our new program should also be able to do. Please read it's docs for more.

I think this is a great idea, but alot of work. I would be happy to help out with the program, assuming it is in Java, VB, or to a lesser degree, .Net/C#/J# etc.

That would be great! We could even start a whole new open source project for it. I already have a new name for it: Eureka, although that may sound a little corny...

Sorry for sounding alittle down about it, I do think it would be a great tool to have!

I don't think you are sounding down about it!
Before we decide who codes what in what we'd best define first how the program should operate in my opinion. Gives us a proper way to start.

Sorry, I know you were talking to Watto, but.....

It sounds like a great idea, as long as file is still in raw format (not encrypted/compressed). It sounds almost like your trying to recreate the unix command "file". The only difference, is file, only works on the original file position, you want it to go through the entire file. ie "file bob.avi" -> "RIFF Video File" "file bob.wav" -> "Riff Audio Wave PCM file"

I was talking also to you!

Anyway, yes, compressed/encrypted resources in an archive might be problematic, but if you have identified a header or tail to belong to a specific type of format (one that you know uses a certain compression) you would not need to identify the compression from the compressed file. Even the latter may be accomplished for some compression formats.

I want it to go through the file, looking for a match of header patterns, meanwhile looking for file signatures etc, as part of Phase 1. In Phase 2, when it has not identified the archive yet, it should go through the whole archive again and try to find somewhere the offsets and sizes (as variables) of the found signatures, i.e. Phase 2 would be the actual mapping of regions where directory structures reside.

[Rahly]

Well, since we are working on the plugin system to MultiEx. I had another thought, this would make plugins even more generic, and possible to use in your new program.

Instead of passing filenames for the plugin to use, i was thinking of passing IStream's to it, this way, IStreams can point to a file, memory, or even a TCP/IP socket. This will allow the plugins to be used in multiple a variety of projects.

Code: Select all

function mpOpenArchiveBindStream(var ArchiveHandle: Integer; MyStream: IStream): Boolean;

function mpIsArchive(MyStream: IStream): boolean;

mpIsArchive tests for archive validity. RIFF files could be a plugin also, since its an archive of "streams". Also, with the new plugin system, you can define weither or not it can be tested, so you could say "riff" files can't extract, can't import, but can be tested for (has mpIsArchive). Then only use the plugins that have the option of "SUPPORTFLAG_CANTESTFOR".

[EDIT] http://msdn.microsoft.com/library/defau ... stream.asp [/EDIT]

[Mr.Mouse]

Well, since we are working on the plugin system to MultiEx. I had another thought, this would make plugins even more generic, and possible to use in your new program.

Instead of passing filenames for the plugin to use, i was thinking of passing IStream's to it, this way, IStreams can point to a file, memory, or even a TCP/IP socket. This will allow the plugins to be used in multiple a variety of projects.

Code: Select all

function mpOpenArchiveBindStream(var ArchiveHandle: Integer; MyStream: IStream): Boolean;

function mpIsArchive(MyStream: IStream): boolean;

mpIsArchive tests for archive validity. RIFF files could be a plugin also, since its an archive of "streams". Also, with the new plugin system, you can define weither or not it can be tested, so you could say "riff" files can't extract, can't import, but can be tested for (has mpIsArchive). Then only use the plugins that have the option of "SUPPORTFLAG_CANTESTFOR".

[EDIT] http://msdn.microsoft.com/library/defau ... stream.asp [/EDIT]

I can see you clearly have a good way of thinking! You must do this stuff for some time now! Professionally maybe? What do you do for a living?

These Istreams would have to be declared in MultiEx Commander as well then. How would this work out? Suppose I wish to open an archive (say .MyArch) and retrieve a resource from it (say MyFile.res), how would the commander procede?

[Rahly]

I can see you clearly have a good way of thinking! You must do this stuff for some time now! Professionally maybe? What do you do for a living?

Coding and Electrical Engineering, I like to think i have good logical skills as well as good problem solving skills. Unfortunately, having developed these skills, leaves me lacking in creativity skills(art, drawing, design). My last job was solely creating development tools for the developers .

These Istreams would have to be declared in MultiEx Commander as well then. How would this work out? Suppose I wish to open an archive (say .MyArch) and retrieve a resource from it (say MyFile.res), how would the commander procede?

It does HAVE to have it, only if it plans to use these functions. The way I had it planed out is, that I'll have One activex Plugin tool that you call, and it will handle the calls to all the other plugins for you. I have since found out VB cannot call functions out of normal dlls , at least dynamically. Statically, YES, but dynamically, no, you have to use a com/activex object. So I'll write an activex plugin manager, so that you can import that into. And Since, multiex commander won't call those functions (it will use the original methods (send filename)), it doesn't need to define IStream, but is available if we do plan to change multiex commander later. But your "scan" program would have to define it, look at the MS web site i posted, maybe you can gleam info on it from there. I'm DEFINATELY not familiar with importing Interfaces into VB.

This IStream GUID is {0000000C-0000-0000-C000-000000000046} if you need it. Fortunately Delphi already has it defined in the ActiveX module. I like to think of Delphi as "VB with power" as the syntax is as easy and nice as VB, but you get the power of C++. Just a personal preference, but unfortunately not a lot of companies use it, because its not marketed like MS markets tools. One of my projects is a new compiler plugin for Delphi, to add some features I would like it to have. MACROs also some new function types "constinconstout" and "compiletime". constinconstout tells the compiler that if constants are passed in, constants are passed out, and if the compiler sees that function with all constants, it executes it at compile time, and returns it as a constant, which if it can, can be further parsed down. Also, allowing for PIC (Position Independant Code) so that when you import a unit and you only use one function, you only import that function(and anything it uses). Also a couple of nuisances that piss me off when coding .

I really want macros to do something like this

Code: Select all

{$IFDEF DEBUGMEMORY}
{$MACRO AllocateMem(s) Track_GetMem(s, __FILE__, __LINE__)};
{$ELSE}
{$MACRO AllocateMem(s) GetMem(s)}
{$ENDIF}

this way a programmer can type AllocateMem and whereever it does that the compiler will insert the filename and the line number. That way when you, say debug your memory, and there was a place you didn't free the memory, you can find out the File and line number that you created the memory, but didn't free it.

You know, simple things like that, that people need. Right now, you'd have to pass the filename and line number everytime you did it, which is kind of a pain for a programmer.

[EDIT]what i'd really like to do is. is make the activex plugin manager, export functions as well as an activex control, this way, you can use either.[/EDIT]

[Mr.Mouse]

Rahly, you got me all excited. Love this stuff. Yeah, you can't use function from non-ActiveX dlls in VB, a TRUE and UTTER pain. you have to declare them statically, which limits you no end.

Another stupid thing (among may in VB) is the lack of unsigned variables, the lack of simple things like bitwise shift-left etc. and the lack of using some inline assembly whenever wanted. of course it has advantages, one being simplicity, but I have had to do a lot of additional coding in C++ for other VB projects, writing a support-program that could actually do stuff that was impossible in VB.

Rewriting all of the Commander in Delphi/C++ would be too much of a pain, so I appreciate you creating an ActiveX plugin-interface.

Even Pascal could handle inline assembly, come to think of it. And Commodore 64 BASIC let you load up assembly routines at certain memory locations, by letting you POKE DATA at these locations.

[Rahly]

Rahly, you got me all excited. Love this stuff. Yeah, you can't use function from non-ActiveX dlls in VB, a TRUE and UTTER pain. you have to declare them statically, which limits you no end.

LOL, i got you.. excited. ~wonders if he should be scared~

Another stupid thing (among may in VB) is the lack of unsigned variables, the lack of simple things like bitwise shift-left etc. and the lack of using some inline assembly whenever wanted. of course it has advantages, one being simplicity, but I have had to do a lot of additional coding in C++ for other VB projects, writing a support-program that could actually do stuff that was impossible in VB.

Rewriting all of the Commander in Delphi/C++ would be too much of a pain, so I appreciate you creating an ActiveX plugin-interface.

Actually it would be a "hybred" providing the original interface, which makes it easy for C++/Delphi to use. Come to think of it, I don't HAVE to make it an activex plugin interface, because you CAN link to static functions, which WILL be in the plugin interface, although, I might just make it an activex just for simplicity sakes. But its best we get all these ideas out NOW, that way we can be really fullied featured for the first version. Now, here is how the plugin system I think will work. Let me know if you think anything is out of order, or something.

Ok, multiex commander includes the MultiexPluginManager DLL, either through static methods or an activex control. After this connects, the DLL will go through the plugin directory (either a cfg file, or a registry entry) and attempt to load each plugin. First it attempts to loads the plugin dll as a C++/Delphi with static functions WINAPI standard calling convention. If valid (adds it to the list of valid plugins). If not valid, checks for the DLLGetClassObject, if its there it will attempt to register it if needed, and then try to load it at an activex object. (the purpose of this is, so you can write easy C++/Delphi(Static function) plugins and easy VB(activex) plugins. So we are not limiting anyone from making plugins, although, i think VB.NET can make proper dlls.

Also, a suggesting, I was thinking of renaming dlls to a different extension, i'm thinking of maybe "MXP" for MultieX Plugin.

I think I'll have a version of the static plugin up here in a few hours. So let me know what you think, i'll have to include the IStream idea in though.

Even Pascal could handle inline assembly, come to think of it. And Commodore 64 BASIC let you load up assembly routines at certain memory locations, by letting you POKE DATA at these locations.

Yeah I remember that, it took me a whole year after I loaded this one program to figure out what "00010 SYS 8510" meant. Course, I was 9 at the time. My first real program was a single player RPG typing game when i was 8 . Written in C64 Basic. And then it wasn't till i was 11 that i found the almight book C64 6510 Programming Reference. I never really liked the C128. I had an Amiga 2000 at one time, and i was so geeked about 4096 colors. ~drooled~ The only thing i wanted more was the Video Toaster. It took forever for the PC market to get more than 256. One of the things I did love, was that all of Babylon 5's graphics were done on an Amiga. When I read that, i was like .... hell yeah... amiga was soooo ahead of its time.... too bad it had such poor marketing. I think it would have been more popular than the PC, and would have soared.

[EDIT]I have some "fair" ability in debuging programs with no source ~cough~ [/EDIT]

[Rahly]

Another stupid thing (among may in VB) is the lack of unsigned variables, the lack of simple things like bitwise shift-left etc. and the lack of using some inline assembly whenever wanted.

Actually, if you declare the variable as a long or an integer type

doing

X := X * 2 (converts to a shl 1)
X := X / 2 (converts to a shr 1)

if you do

X := X * (2^A) (converts to a shl A)
X := X / (2^A) (converts to a shr A)

at least, delphi is smart enough to do that.

Although, i donno what VB does for this.