UPDATED: Plugin to save/load binary script format

[CoreDumped]

Hi, I created a plugin that automatically saves the compiled script from memory (the Abstract Syntax Tree) to disk. Next time, when the script is about to be parsed and if its corresponding binary version is available, the compilation is skipped and the AST is loaded directly from the binary file and sent to the translator, which would speed up the loading process. This was pretty straight forward with Ogre's new script compiler.

The following scripts can be saved in binary format:
*.material
*.program
*.compositor
*.particle

There is no need for any code change. Just include this plugin to your plugins.cfg and it will automatically save the compiled script to disk when you run it the first time. Then on subsequent runs, the compiled version would be used from the cache, if available. I noticed that the the loading time of the sample browser was reduced from 7 seconds to 4.5 in my Core i7.

The text based scripts can also be replaced by the compiled version by removing their entries and adding the cache folder in resources.cfg. This could be useful when you dont want end-users to modify your scripts



Code released under MIT license
http://code.google.com/p/ogre-script-serializer/

The default location of the cache directory is ".scriptCache" in the working directory. This directory would be created if its unavailable.
The code was tested to an extent. The AST generated from the script compiler matches with one save and loaded from disk, for all the scripts in the sample media folder. All samples seem to work fine.

Pending Stuff:
1. [s]The compiled version should be discarded whenever the source file is modified. We can save the MD5 hash of the source file in the binary header and compare before reloading later. Or use the timestamp approach?[/s] - DONE
2. [s]Save/Load compiled shaders to disk[/s] - Already implemented in 1.8 by Assaf Raman. Using this lib in the plugin


EDIT 1: Check this post for details on the new update

[dark_sylinc]

Wow! this looks surprising! Congrats!

Pending Stuff:
1. The compiled version should be discarded whenever the source file is modified. We can save the MD5 hash of the source file in the binary header and compare before reloading later. Or use the timestamp approach?

MD5 could counter the benefit because you would need to read all script files, and HDD can be a bottleneck (I wonder if it actually is). So a time-stamp could probably be a lot faster, but this might be countered if the HDD must seek too much.
MD5 may benefit during development, since often a script may be reedited multiple times, only to be reverted to the way it was before.
Timestamps on the other hand, would benefit more the final release if the executable allows modding or something like that. And those who don't, don't need source file changes checks at all; just use the binary version.

So, my suggestion is to check timestamps, and if it's newer, compute MD5; with possibility of disabling one or both techniques.
Of course, what you really are willing to implement is up to you

I would go for CRC32 though, it's cheaper to implement, 4 times smaller, faster to compute, and it's OK as long as you don't use it to as a unique hash to compare against multiple files (chance of having a collision between two files is of 1 / 2^32, but if you start comparing against multiple files to see if they're "identical", the birthday paradox makes your chance alarmingly grow)

Keep up the good work!

[Jabberwocky]

An excellent and welcome feature!

[CoreDumped]

Thanks guys.

dark_sylinc, you are right. I'll go with the timestamp approach. However, I'm not sure how I can fetch the file stat of an ogre resource, since it might come from a filesystem, zip or even a remote web server. Any ideas? If it's not feasible, then we can go with CRC32.

Also, to check if the compiled version is valid, I'll also have to check if any of the base scripts (all the way to the top) were modified since the AST generated by the compiler is self-contained which contains the compiled code of the base scripts as well

EDIT: I'm not sure about the last point that the AST contain code of the base script

[so0os]

You're my saviour, many thanks!

[jacmoe]

This is really great!
Resource parsing/loading is really a time sink.

[syedhs]

I really think this should be part of Ogre core - if I am not mistaken there are some discussions toward doing this but nothing comes out from it..

[so0os]

I really think this should be part of Ogre core - if I am not mistaken there are some discussions toward doing this but nothing comes out from it..

I strongly approve! It takes approx 17 seconds on i5 to load all my materials and stuffs (and thus, boot the app) it's a narrow bottleneck, do it.

[spookyboo]

I would even go so far to move all Ogre scripting stuff to the plugin. Serializing/deserializing is not core functionality of a rendering system.

[CABAListic]

I would even go so far to move all Ogre scripting stuff to the plugin. Serializing/deserializing is not core functionality of a rendering system.

Interesting idea. It has some appeal to me

[Zonder]

Wow! this looks surprising! Congrats!

Pending Stuff:
1. The compiled version should be discarded whenever the source file is modified. We can save the MD5 hash of the source file in the binary header and compare before reloading later. Or use the timestamp approach?

MD5 could counter the benefit because you would need to read all script files, and HDD can be a bottleneck (I wonder if it actually is). So a time-stamp could probably be a lot faster, but this might be countered if the HDD must seek too much.
MD5 may benefit during development, since often a script may be reedited multiple times, only to be reverted to the way it was before.
Timestamps on the other hand, would benefit more the final release if the executable allows modding or something like that. And those who don't, don't need source file changes checks at all; just use the binary version.

So, my suggestion is to check timestamps, and if it's newer, compute MD5; with possibility of disabling one or both techniques.
Of course, what you really are willing to implement is up to you

I would go for CRC32 though, it's cheaper to implement, 4 times smaller, faster to compute, and it's OK as long as you don't use it to as a unique hash to compare against multiple files (chance of having a collision between two files is of 1 / 2^32, but if you start comparing against multiple files to see if they're "identical", the birthday paradox makes your chance alarmingly grow)

Keep up the good work!

The last modified date should be enough during development

[boyamer]

Absolutelly, this should go to Ogre trunk

[dark_sylinc]

+1 to put this in the trunk.

Thanks guys.

dark_sylinc, you are right. I'll go with the timestamp approach. However, I'm not sure how I can fetch the file stat of an ogre resource, since it might come from a filesystem, zip or even a remote web server. Any ideas? If it's not feasible, then we can go with CRC32.

Archive::getModifiedTime?
ResourceGroupManager::resourceModifiedTime?

[CoreDumped]

Archive::getModifiedTime?
ResourceGroupManager::resourceModifiedTime?

Ah great. Thanks, that would do

[so0os]

Hmm... i'd hate to complain, but I didn't get any boot speedup (or a minimal one) from using the plugin, Ogre.log says it skips scripts and processes sbins, but it still takes a year. Am I mising something, or is speedup not the intent here?

[CoreDumped]

Hmm... i'd hate to complain, but I didn't get any boot speedup (or a minimal one) from using the plugin, Ogre.log says it skips scripts and processes sbins, but it still takes a year. Am I mising something, or is speedup not the intent here?

I tested this in release mode and see that the compiler's script parsing speed is fast (my original stats were on debug mode). However the ratio is still the same and the difference would be noticeable if you have many scripts

With a large material file with 140k lines, 2000 materials with 2 passes each takes 6 seconds to parse normally, while the plugin takes 3 seconds. I think the translators take up that 2-3 seconds. I'll post proper profiling results later

However, we'll probably wont have that many materials in a game Script compilation probably wont take that much time to begin with anyway. I think the real bottleneck is the shader code compilation. I am working on passing the compiled shader code to the GpuProgram through a workaround (i dont see any directly interface to load from binary)

Also, I just updated the code with some optimizations. I was seeking past the header on every AST node, which meant thousands of slow seeks. Refactoring the data structures doesn't require this anymore.

[Rambus]

This is a very useful and important plugin. If you continue to provide it with some love over the next few days (like serializing compiled shaders) then I think it will get wide use.

Good job, and thank you!

[CoreDumped]

This is a very useful and important plugin. If you continue to provide it with some love over the next few days (like serializing compiled shaders) then I think it will get wide use.

Good job, and thank you!

Thanks


Added support for re-compiling the script if it was modified since last compile

Code updated in repository

EDIT: Also added version number in the script header to track data structure changes. So if you are trying to load an older binary version, it will throw an exception. Be sure to clear out your scriptCache folder after the update (rev 6)

[spookyboo]

I saw that it supports the basic Ogre script extensions. Can I also add other types? (i.e. *pu or *osm files)

[reptor]

Nice! This is clearly a good idea. Thank you for your work.


Did you notice Mr. Assaf Raman's work, a shader cache?

[CoreDumped]

I saw that it supports the basic Ogre script extensions. Can I also add other types? (i.e. *pu or *osm files)

I was thinking of trying the *.pu scripts out yesterday. I have a license of your library

As long as you use the ogre script compiler, it should be transparent and should work

However, for automatic detection of the binary scripts from resources.cfg file, i.e. if you deploy only the binary script, i hardcoded the search extensions to be either one of the four (program, material, particle, compositor). I'll move this to a config file.

[CoreDumped]

Nice! This is clearly a good idea. Thank you for your work.


Did you notice Mr. Assaf Raman's work, a shader cache?

Yep, I briefly worked with RTShaderLib while trying out the iPhone ES 2.0 renderer. Its a great lib. Not sure if I will find something related there since the shader source code is cached there and we want to cache the binary format. I'll check it out though

[dark_sylinc]

I think most developers here will agree that improving debug performance time is very important, regardless of how much Release mode actually improves.
It's awful when you're debugging something that took you to restart your application more than 3 or 4 times.

As for the shader binary, there's no abstract interface because all formats are too different.
In GLSL for example, you don't even get the binary or compiled representation at all, it's hidden from you inside OpenGL.
I believe HLSL starting from shader model 4.0 has the same issue.

You would need to cast the shader programs to the adequate derived class (HlslProgram, CgProgram, GLSLProgram, etc) and get direct access to their pointers.

[Wolfmanfx]

If you want to save shaders to the cache:
* Before the resource load call GpuProgramManager::getSingleton().setSaveMicrocodesToCache(true);
* When you want to save the cache to a file call - GpuProgramManager::getSingleton().saveMicrocodeCache(...);

When you want to use the cache:
* Before the resource load call - GpuProgramManager::getSingleton().loadMicrocodeCache(...);

http://www.ogre3d.org/forums/viewtopic.php?f=4&t=61004

I think Assaf did a shader cache which saves the bin representation of the shader.

[CoreDumped]

I think Assaf did a shader cache which saves the bin representation of the shader.

Oh great . I didn't know about this. So this is what reptor was talking about