Advanced compilers for Ogre's scripts

Threads related to Google Summer of Code
Post Reply
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Advanced compilers for Ogre's scripts

Post by Praetor »

I've been tinkering with creating a new compiler system for Ogre's scripts. As summer's coming up and I think SoC is a good opportunity to be able to work on this project. Discussion about it already started here: http://www.ogre3d.org/phpBB2/viewtopic.php?t=28685

I've also had some comments and such through the SoC system but I'm not sure those are public so I'll start this discussion going so the community can get involved. Basically I've worked out a basic list of requirements and features.

1) All current scripts shall compile under the new system, no changes needed.
2) Memory footprint and performance shall be reasonable.
3) All script elements (e.g. materials, techniques, particle affectors, etc.) can be inherited from other elements of the same type.
4) The lexer shall recognize flexible script formats in regards to whitespace (any tab, space, and newline combination is allowed so you can format the script any way you like).
5) Each compiler subtype shall support a listener, which will allow external systems to be notified of events and also to control some internal aspects of the compiler (e.g. how the material compile creates new materials).
6) External systems shall be able to override the internal compiler's processing of script nodes, so that existing elements and properties can specially compiled, or so that new script elements or properties can be added to existing compilers without the need to change Ogre's codebase.

That's the simple, easy version. I have a prototype of a material compile in the works, and the base system is quite stable (not much error handling yet though). If this project gets accepted into SoC I plan on getting all script compilers done, with all the goodies and advanced features.

One special script feature that has been discussed that is still being worked out is extending and pretty much replacing the texture_alias feature with a much more generalized and flexible system. Base elements would define special variables, which can then be overridden by inheriting elements. Imagine a base particle system which can parameterize lifetime and particle counts and then you simply need to inherit from it and override two parameters. You could right entire libraries of scripts in half the time with much less copy-paste action.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

This has already had quite a bit of discussion between the other thread and between myself and the mentors, but I thought I'd give a nice example so everyone can at least see what I'm talking about here, and why this is interesting.

Code: Select all

pass TwoTextureSeparateUV
{
    lighting off
    texture_unit{texture $tex1}
    texture_unit{
        texture $tex2
        tex_coord_set 1
    }
}

material Test1
{
    technique
    {
        pass : TwoTextureSeparateUV
        {
            $tex1 "BaseTexture.jpg"
            $tex2 "Texture With Spaces.png"
        }
    }
}
All sorts of goodies going on there. A global pass is declared outside of any material. This can now be referred to from within any material that can "see" it (basically, right now, anything in the same .material file). We have flexible input with some weird newlines going on. And you see the more flexible variable system totally taking over the texture_alias that is currently used. Right now, the plan would be to make $ a reserved character (as in, not valid in names for passes, particle systems, etc.) and when the compiler sees one it does post-process replacement. So, when it compiles Test1 it derives the first technique's pass from the global template, and based on the context the variables shown here will be replaced with the final concrete textures. Now imagine doing this with more than just texture names in more than just texture_units, with a whole library of materials (I'm sure most of you with sizable projects have a lot of materials to contend with). I still write most of my materials by hand, or at least tweak those that are exported by a tool, so this stuff really helps.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

I definitely like the idea of being able to pull in other aspects of the materials like techniques and passes rather than just entire materials. Being able to assemble materials like that from a library of existing passes and techniques with some substitutions would lend itself to high-level tools very nicely indeed. I can see the drag-drop-tweak sequence now ;)

The thing I was less keen on in your proposal was the addition of custom user sections in the materials, because I feel this can be abused quite badly if people start putting physical properties inside rendering materials, for example. I can see the capability might be useful for experienced users but it's one of those things that can be used to create very bad designs. So whilst I wouldn't say 'don't do that', I would say put that at the back of the feature list, because the stuff you just talked about is much more exciting and core to the material script's nature.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

I agree. It could lend itself to abuse. I'm not yet sure if the possibility of abuse is enough to warrant removing the ability. I'm thinking of extensions like spookyboo's fx plugin which could make use of a few custom parameters without having to modify the core. That sort of feature will probably take a back seat anyway to the other (done nearer the end) and maybe we can work out an alternative or a compromise.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Yeah - my thought exactly. In the right hands, and in the right circumstances, it will be a bonus, no doubt. So that's why I wasn't saying not to do it per se. But the first thing some people will do I bet is start adding all kinds of crap to material scripts ;) So I thought the other more universal things would be better prioritised.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

In total agreement.

The most important thing for external systems for me is not custom nodes, but listeners which can do things like control how materials are allocated.

As far as the variable system goes what I'm thinking is a stack-based context. Variables are declared in the base element, and then each element being compiled stores assigned variables, overriding as they go down. Example time.

Code: Select all

pass TwoTextureSeparateUV
{
    lighting off
    texture_unit{texture $tex1}
    texture_unit{
        texture $tex2
        tex_coord_set 1
    }
}

material Test1
{
    $tex1 "Base1.jpg"
    $tex2 "Base2.jpg"

    technique
    {
        pass : TwoTextureSeparateUV{}
        pass : TwoTextureSeparateUV
        {
            $tex1 "BaseTexture.jpg"
            $tex2 "Texture With Spaces.png"
        }
    }
} 
So the first pass would use "Base1.jpg" and "Base2.jpg" as its texture names, but obviously the second pass will use "BaseTexture.jpg" and "Texture With Spaces.png." The only concern is nested contexts can increase memory footprint and lower performance. We won't know until I prototype it. However, I can picture now the ease of having reusable per-pixel lit passes. How easy it would be to create a lot of high-level materials from a base library of components.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Yes, that's a good idea. If the stack was lightweight and only had an overhead when multiple levels of nesting were used, I don't see a problem with it.
User avatar
spookyboo
Silver Sponsor
Silver Sponsor
Posts: 1141
Joined: Tue Jul 06, 2004 5:57 am
x 151
Contact:

Post by spookyboo »

Preator, does your system takes keywords into account that are used in a different context? Consider the usage of the 'enabled' keyword (true/false) in both the context of an emitter and an affector. With the current compiler I have to bind 1 function to the 'enabled' keyword and I have to combine both the 'enabled' code for emitter and affector into 1 function (at least there is some shared code). If I am able to declare in front that I use function A if 'enabled' is used in the context of the emitter and use function B if 'enabled' is used in the context of the affector I have a cleaner separation. Hope I have made myself clear.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

The system doesn't use the same mechanism as the current compiler. Currently you must bind functions to handle incidents of a token. With this system, an abstract syntax tree is created and that is what is used during compilation.

Code: Select all

material Test
{
    lighting on
}
Generates a tree like this:

Code: Select all

"material" : MTID_MATERIAL
"Test" : TID_WORD
    "lighting" : MTID_LIGHTING
        "on" : TID_WORD
So, the mechanism is completely different. Since a tree is used we don't really care about registering specific handlers for specific token types. Instead we do a tried-and-true tree traversal. When you are inside an emitter and you encounter the "enabled" keyword, nothing is stopping you from sending the compilation down a separate path than if you encountered the keyword when inside an affector.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

I had a thought while going back over the prototype. Most likely each compiler subtype will be instanced within the appropriate manager. That is, a MaterialScriptCompiler is inside the MaterialManager, ParticleScriptCompiler is inside the particle system factory, etc. Well, the whole resource system is being fitted with threading capabilities. Should locks and such be within the compilers, or will those protection mechanisms exist around and outside them? Either way, I figured I wouldn't worry about threading until near the end, but I was wondering how it could be added in, and even if it needed to be.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Take a look at how the existing compilers are handled (like CompositorSerializer). Their parsing state is self-contained and the instance of the compiler is held in thread-specific storage, abstracted via OGRE_THREAD_POINTER, e.g. in CompositorManager:

Code: Select all

		/// Serializer - Hold instance per thread if necessary
		OGRE_THREAD_POINTER(CompositorSerializer, mSerializer);
The pointer is built and destroyed using the macros OGRE_THREAD_POINTER_SET and OGRE_THREAD_POINTER_DELETE.

Provided you keep state local to the instance this will work for you too. Locking will only occur during creation / loading of the individual resource so that parallelised parsing is allowed.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

After discussing a few things with spookyboo there would be a change to the schedule.

I would clean up and make robust the base compilers during the "get-to-know-the-project-and-community-phase." Once that ends, the first third of the summer would be focused on creating a working particle system compiler. This compiler would then be integrated with Ogre's base (only 1 week). Then, the next third would be taking lessons from the particle system compiler and creating similar compilers for other script types. Finally integration and polishing until the end of SoC.

I switched the particle system compiler for the material compiler because the work spookyboo is doing now would mesh well with a new compiler, and I'd like to involve as much of the community as possible, especially those with plugin experience and experience trying to extend and use the current compilers. There have also been a few particle system tools out there, so I'd like to elicit their input in the first stage as well.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Ok - just remember that the material compiler will be the most time consuming by far though, since it has easily the most directives, so make sure you keep enough time allocated for it.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

Yes, that will be the first one I do after the particle system. A prototype material compiler is already 1/4 finished though. It also will help that I have the most experience dealing with the materials and material compilation.
User avatar
spookyboo
Silver Sponsor
Silver Sponsor
Posts: 1141
Joined: Tue Jul 06, 2004 5:57 am
x 151
Contact:

Post by spookyboo »

:D
Tenttu
Halfling
Posts: 74
Joined: Mon Dec 13, 2004 1:56 pm

Post by Tenttu »

This would also fit nicely with the Particle System Improvements project proposed at: http://www.ogre3d.org/phpBB2/viewtopic.php?t=29922

It would be easier to keep the system backward compatible and to make sure that the compiler has the features that would be needed by the improvements. Also totally new features to particle systems could be added if these changes happen.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

One of the things I hope to focus on is allowing for the compilers to augmented in a much more flexible manner in the future. Plugin developers should be able to also plug in new scripting features along with whatever other features their plugin provides.
User avatar
jacmoe
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 20570
Joined: Thu Jan 22, 2004 10:13 am
Location: Denmark
x 179
Contact:

Post by jacmoe »

Really, really looking forward to this summer, Preator! :)

You need no introduction ... 8)
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

Thanks a lot jacmoe. I have some time this weekend to get my thoughts together and I'll probably make a post about what I want to make of this "get acquainted time." Suffice to say I feel very much acquainted and am ready to get to (some) work.
User avatar
Kencho
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 4011
Joined: Fri Sep 19, 2003 6:28 pm
Location: Burgos, Spain
x 2
Contact:

Post by Kencho »

I'm looking forward to this project as well. You'll probably feel me lurking on this progress... and eventually coming out to encourage :P
Image
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

So, here is a little update one what I'll be doing in the near future.

The first order of business is to set up a test framework. I'm going to take the plunge and do as much test-first development as possible. I think given the relative non-interactive nature of this project, automated testing can do well here. It'll be an incredibly simple test app system, shouldn't take very long at all.

After that I will be refactoring and cleaning up first the lexer, and then the parser. I've started to make the distinction between lexer, parser, and compiler. The lexer tokenizes the input, the parser creates the abstract syntax tree, and the compiler does the final business. So, my initial efforts will be making the lexer and parser nice reusable components, and a compiler base class. I'll take some time here to make sure there's solid error and handling and messages along the way.

I'll whip up a diagram to show this stuff better sometime soon.
User avatar
Kencho
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 4011
Joined: Fri Sep 19, 2003 6:28 pm
Location: Burgos, Spain
x 2
Contact:

Post by Kencho »

If you're interested, this weekend I'll post/wiki the tools and methods I'm using for development. Maybe you can get some ideas from that.
Just as a starter, I'm using CxxTest for the automated unit tests and premake to create the makefiles/scripts and the tests. Indeed, I've found premake pretty easy to use and great to automate a lot of things.
Then there's also ANTLR, though I think it shouldn't be appliable here, but worthy to take a look at.

I've got quite a good base on compilers this year at the uni, so count with my help for anything related to it!
Image
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

I've previously managed to make a lexer already for ogre's scripts. It has stood up to my informal testing thus far and we'll see how it does under more rigorous testing. However, from the lexer's point of view the scripting language is fairly simple, so a hand-made one should work fine. After all, it's just a state machine, what could possibly go wrong...?

I'll take a look at CxxTest, thanks. I assume we'll want to keep the tests and framework out of the repo. Should I just keep that part local to my machine? As far as premake goes... I do most development on windows. My machine is set up to dual-boot windows/ubuntu (though, I need to redo my installation), and I was planning on checking out KDevelop for some x-platform dev work. I'm really very rusty when it comes to make builds on *nix and especially library and app packaging and distribution. So, help with *nix issues will be appreciated.
User avatar
jacmoe
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 20570
Joined: Thu Jan 22, 2004 10:13 am
Location: Denmark
x 179
Contact:

Post by jacmoe »

Regarding CxxTest: it seems to be awfully out of date (but I could be wrong).
I like TUT better! -> http://tut-framework.sourceforge.net/ :)
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

Thanks jacmoe. I've used boost's test before, but tut looks nice. The main draw: no compiled library. My favorite parts of boost are the all-header parts. I have no problem with all-header libraries where everything is inlined into my code, and tut looks fairly small too.
Post Reply