Spotlight: Señor Casaroja's Noesis

[Mr.Mouse]

http://www.arl.wustl.edu/~lockwood/clas ... m/toc.html

If you fancy being elite, go for it.

[MrAdults]

finale00, you may recall that a while ago I told you to pass in None for vertex components no longer in use by subsequent triangles/meshes when drawing in immediate mode. I now see that doing that would raise an exception, ha ha. My bad. I just fixed that in Noesis 3.861.

[MrAdults]

I just got to thinking about the nature of Noesis and its status as a half-open half-closed source project. Brain dump.

I licensed the foundation I eventually based Noesis on however many years ago (when it was something very different from what it is now), and I maintained rights to it but agreed not to make the source code public in the process. Contract bullshit. When I incorporated that foundation into my new (at the time) pipeline tool, though, I didn't see an issue with that, because while I expected it to get some use from people modding my game(s), I really otherwise didn't expect anyone else would use it for anything. Then I decided to start adding more exotic format support to that tool because it was the most convenient place for me to experiment, and people started using the tool for things completely unrelated to its content pipeline functions, like papercrafting and self-education (I know one of my own contract modelers improved 500% when I showed her some existing tricks used to model hands and general flow in some PSP games) and all kinds of random shit that I didn't foresee (some of it also extremely undesirable as far as I was/am concerned) cause I'd really never been into that whole scene. So after wrestling with the sometimes more terrible shit people were doing, I finally decided not to let some stupid assholes keep me from developing it into something more, and to give it a real frontend, and it became Noesis. Noesis also started out not having any particularly lofty goals, and I didn't even plan to facilitate other people in making crap for it. But once it actually became Noesis, I recognized a real lack of and need for the framework it was already kind of providing for my own use (a multi-platform set of utilities and functions targeted at interpreting and making reversal of all manner of game data for every game platform an easier, faster process, eliminating the need for a new pseudo-framework and interface in each of those many standalone tools people were making and still make... it still falls short in many categories, but it's what I'm working toward), so I exposed a native plugin API. That didn't get much use, but I saw there was a lot of potential when revelation started using it. And thanks to his patience and feedback, I continued improving the API and fixing shit up, and it grew into something almost respectable. Fast-forward a while later, and eventually I decided Python support would open it up to a lot more people, and it did.

Now I feel like the fact that its core is closed-source kind of conflicts with what it's become. But that's interesting too. I've noticed that software engineers, especially the kind that like to reverse data, are really fucking ego-driven. Just about as much so as graphics programmers. And I've noticed in the past that my public code snippets and modules will often get absorbed very quickly into other projects/tools without so much as a mention to me and my work. (when I strongly suspect this has happened, I'm sometimes sneaky and out of sheer curiosity disassemble the tool in question, then I see if the disassembled code matches my own, usually there's some quirk or read/write ordering that makes it unique enough to tell whether they just plopped my shit right in with a reasonable probability, even across different compilers, optimization settings, and configurations... this is impossible if they ported it or didn't compile to native code, but at least then they had to do a *little* bit of work) But because a lot of Noesis remains closed-source except for the stuff in the SDK and what I intentionally spread around, and people who might otherwise thanklessly steal raw source are far too lazy/stupid to disassemble what they need out of Noesis, it's forced people to actually use Noesis to do some of this stuff and get at some of this data.

I've noticed a tendency in these communities for people to gravitate very strongly toward making and using a lot of standalone tools, and people rip each other off really relentlessly, seemingly just for some silly sense of glory, or making a name/following for themselves amongst of a bunch of semi-retarded content thieves. I think that if I were trying to maintain an opensource codebase for Noesis, it would be getting torn apart without regard for its actual GPL terms on a very regular basis, and people would be developing their own standalone tools, going around to forums/users themselves and spreading those tools (cause I certainly make no effort on this front for Noesis, nor do I desire to, or want others to when it comes to a great many of those forums as their focus is simple theft), and far fewer people would actually end up using Noesis itself. That would mean even fewer bug/crash reports, even less feedback, and even fewer people writing plugins/scripts for it. I think this is the biggest reason we haven't seen an opensource project that does what Noesis does (specifically in the context of providing a framework for reversing game data, rather than just being a framework for mesh/texture/etc. processing) take off or be successfully, actively maintained over a period of years. I can't discount the larger draw I feel Noesis would have across a wider range of communities if it were open-source, but from what I've witnessed very specifically in this area of engineering interest (a typical GPL'd game engine for example does not have any of these problems, or has them to a barely-detectable degree), I do think the undesirable impact of ego and theft on such open-source initiatives has been a lot greater than people tend to realize.

So, I think in a strangely roundabout way, being closed-source has allowed Noesis to grow and make far greater improvements than it otherwise might, or at the very least has allowed it to improve in a more focused manner. It's also more motivating for me to be able to work in a closed framework to occasionally add new non-game-related experimental shit that I really don't want to make public, so that's been useful too. I don't think I could justify putting as much work into a public codebase, because that's code that I stand absolutely no remote chance of ever commercially profiting from, which means... outside of my base fascination with game data, I guess it would all have to be ego-driven for me too, which means I'd be far more likely to sputter out after seeing enough people thanklessly steal my code. As it is, I simply don't care if they do, and see no harm or loss of profit in making new format specs/code public - short of the fewer new Noesis users that will result from other tools supporting that format as well. It's interesting how it all ties together.

So, Noesis turned into something kind of interesting. I also hope that despite the best efforts of some of you selfish babykillers, Noesis doesn't one day have to die for your sins.

[greywaste]

and self-education

I don't understand all the floaty-endian stuff for the life of me but I can say that thanks to Noesis, scripts from Chrrox and others and an occasional tutorial, I can model pretty much anything I want to place in my mods by myself now.

and people rip each other off really relentlessly, seemingly just for some silly sense of glory, or making a name/following for themselves amongst of a bunch of semi-retarded content thieves.

Well personally I've abided by the readme 100% but yes, it's obvious others ignore it and extracted models and textures end up splattered all over facepunch/Xnalara/bethesda games mod site forums
and now that at least one of those venues has opened donation pools for "mod authors" content thieves can now profit from it too

But it would still happen regardless of Noesis I think - Daz/Renderosity community would get hit even harder especially I think.

So, Noesis turned into something kind of interesting. I also hope that despite the best efforts of some of you selfish babykillers, Noesis doesn't one day have to die for your sins.

It's been a great help - thanks again for keeping it available thus far!

[Mr.Mouse]

Somehow, Richie's rambling reminds me of my own. viewtopic.php?f=3&t=797&p=4338#p4338

[MrAdults]

I can definitely appreciate the "I'm turning 30" sentiment... I think software engineers are prone to having early midlife crises, I had my first one when I was 20.

[howfie]

Well personally I've abided by the readme 100% but yes, it's obvious others ignore it and extracted models and textures end up splattered all over facepunch/Xnalara/bethesda games mod site forums

Ha ha ha tell me about it. Recently, all of the models from TTT from Tekken Hybrid were posted somewhere using the python script I wrote lol. Most of the Neptunia ones were posted on another site as well. But I kind of have a different view of these people. Rather than content thieves I would like to think of the people who use my tools as budding game programmers and artists. While I wish that they didn't post them in batches like that, I would prefer that they use my tools to learn something from them. Look at my code... learn something. Look at the models generated by my tools... learn something. Learn to rig the models yourself rather than posting all of them and expecting somebody else to do it for you lol. That's why I do what I do. I want people to learn something. Unfortunately, as I learned from teaching at a university for several years, most students don't give a shit about learning. They are always thinking about an easier way out.

[finale00]

I don't mind if people decide to rip every model from every game that is supported by any tools or scripts that I post on the internet. I don't mind if they pass it off as their own work, or even go ahead and make money off it. In fact, if they get rich, I wouldn't mind either, cause if I had the business talent to do so I probably would have done it. But because I don't, I only do it for fun.

In any case, people are free to give what kind of image they want to give. As long as they take responsibility for their own actions and when black suits come knocking on their doors regarding intellectual property theft, they aren't going to start diverting all that spam to me.

Plus, C is unfriendly. I do not even want to look at noesis source and see what kind of horrors may be going on in there.

Anyone want to add obscure language support to noesis?
Let's try adding some ruby support to the plugin script list lol.

[MrAdults]

Might be of interest: http://www.richwhitehouse.com/index.php?postid=66

I wanted to update my web site cause it had been forever, but I really can't talk about a single damn thing I've been doing outside of Noesis crap. So I talked about Noesis crap.

[finale00]

Here's a problem I've been having that I don't know how to solve.

A lot of (PC) games use XML files to define which files should be loaded, what goes where, etc.
These are plain XML files so unless you get lucky there's really no way to determine which format you're looking at.

So I've decided not to bother with that route and go with a plain XML parser.

Some games use an XML file to store the model and a material library, and there is no way to determine the name of the material library from the model itself. I'd have to go through the XML file to load the model and the materials, so I would register some XML tags that say one tag is for the model and another tag is for the material. The XML parser will then load the model and the materials.

Ideally, I would like an XML parser that can be extended with plugins.

For example, if I'm working with a specific game, I would define it so that if I see a particular tag, I would do something specific with it. If I see a "model" tag, I would know that the value is the name of a model file. If I see an "animation" tag, I would know that the value is the name of an animation file.

Maybe each plugin can contain a XML handler that registers a bunch of tags when loading all the plugins.
When an XML file is selected, the XML parser will go through tag by tag. If one of the tags is registered, it will look at which plugin(s) they are registered to and try something with it.

That is, if two plugins register "mesh" for the model-loading tag, then noesis will try to load the specified file with the two plugins.

For the material library example, I would write a function to parse the material library, creating the appropriate material objects and stuff. The XML tag will be registered to call this function if noesis checks this plugin.

I don't know how this would work.

[MrAdults]

You could certainly make your own generic XML parser plugin with its own sub-plugins system, sure.

You could also just make a script for the model format. When the model is loaded, parse its path out to determine where on that path you should look for the central xml, then load and parse down to find whatever it is you need to determine the materials associated with the model. If that means loading all of a certain set of xml's and scanning them to find the one that references the model being loaded, then that's what you've gotta do.

My Duke Nukem Forever plugin actually does this, and indexes the full material list and such for the whole game every time a single model is loaded. Despite this, models still load pretty much instantly. Having to parse through tons of XML in Python might be a little less lightning-fast though, so you should consider that.

[finale00]

Can I write the XML parser in C++ and then extend it with python plugins?

Oh, and is there a way to load a default "background" in noesis? Like maybe if I put some textures or a model of some scenery somewhere and it'll be loaded automatically when I preview a model.

So previews look more interesting.

[Mr.Mouse]

I can definitely appreciate the "I'm turning 30" sentiment... I think software engineers are prone to having early midlife crises, I had my first one when I was 20.

I still wonder when the crisis ends.

[MrAdults]

Can I write the XML parser in C++ and then extend it with python plugins?

Oh, and is there a way to load a default "background" in noesis? Like maybe if I put some textures or a model of some scenery somewhere and it'll be loaded automatically when I preview a model.

So previews look more interesting.

Sure, it would be all on you to implement that though. And I thought you hated C++.

Nope, that's not an available feature at the moment. But you can do an ultrashot to get a nice antialiased snapshot with alpha channel, then overlay it on whatever you want in, say, PhotoShop. If pixxx are your concern. Then again, ultra-shot requires a FBO-capable GPU, which I've gathered you may not have.

I still wonder when the crisis ends.

My best guess is, death.

[finale00]

I don't like C++ but it's probably more suitable as a general parser rather than just a one-off thing.

Anyways for the QUAD options, I don't seem to be able to use them?
Noesis just tells me they're undefined.