UPDATED: Plugin to save/load binary script format
- CoreDumped
- Gremlin
- Posts: 177
- Joined: Sun Aug 05, 2007 3:55 pm
- x 14
UPDATED: Plugin to save/load binary script format
Hi, I created a plugin that automatically saves the compiled script from memory (the Abstract Syntax Tree) to disk. Next time, when the script is about to be parsed and if its corresponding binary version is available, the compilation is skipped and the AST is loaded directly from the binary file and sent to the translator, which would speed up the loading process. This was pretty straight forward with Ogre's new script compiler.
The following scripts can be saved in binary format:
*.material
*.program
*.compositor
*.particle
There is no need for any code change. Just include this plugin to your plugins.cfg and it will automatically save the compiled script to disk when you run it the first time. Then on subsequent runs, the compiled version would be used from the cache, if available. I noticed that the the loading time of the sample browser was reduced from 7 seconds to 4.5 in my Core i7.
The text based scripts can also be replaced by the compiled version by removing their entries and adding the cache folder in resources.cfg. This could be useful when you dont want end-users to modify your scripts
Code released under MIT license
http://code.google.com/p/ogre-script-serializer/
The default location of the cache directory is ".scriptCache" in the working directory. This directory would be created if its unavailable.
The code was tested to an extent. The AST generated from the script compiler matches with one save and loaded from disk, for all the scripts in the sample media folder. All samples seem to work fine.
Pending Stuff:
1. [s]The compiled version should be discarded whenever the source file is modified. We can save the MD5 hash of the source file in the binary header and compare before reloading later. Or use the timestamp approach?[/s] - DONE
2. [s]Save/Load compiled shaders to disk[/s] - Already implemented in 1.8 by Assaf Raman. Using this lib in the plugin
EDIT 1: Check this post for details on the new update
The following scripts can be saved in binary format:
*.material
*.program
*.compositor
*.particle
There is no need for any code change. Just include this plugin to your plugins.cfg and it will automatically save the compiled script to disk when you run it the first time. Then on subsequent runs, the compiled version would be used from the cache, if available. I noticed that the the loading time of the sample browser was reduced from 7 seconds to 4.5 in my Core i7.
The text based scripts can also be replaced by the compiled version by removing their entries and adding the cache folder in resources.cfg. This could be useful when you dont want end-users to modify your scripts
Code released under MIT license
http://code.google.com/p/ogre-script-serializer/
The default location of the cache directory is ".scriptCache" in the working directory. This directory would be created if its unavailable.
The code was tested to an extent. The AST generated from the script compiler matches with one save and loaded from disk, for all the scripts in the sample media folder. All samples seem to work fine.
Pending Stuff:
1. [s]The compiled version should be discarded whenever the source file is modified. We can save the MD5 hash of the source file in the binary header and compare before reloading later. Or use the timestamp approach?[/s] - DONE
2. [s]Save/Load compiled shaders to disk[/s] - Already implemented in 1.8 by Assaf Raman. Using this lib in the plugin
EDIT 1: Check this post for details on the new update
Last edited by CoreDumped on Fri Jan 14, 2011 5:26 pm, edited 4 times in total.
- dark_sylinc
- OGRE Team Member
- Posts: 5299
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1279
- Contact:
Re: Plugin to save/load binary script format
Wow! this looks surprising! Congrats!
MD5 may benefit during development, since often a script may be reedited multiple times, only to be reverted to the way it was before.
Timestamps on the other hand, would benefit more the final release if the executable allows modding or something like that. And those who don't, don't need source file changes checks at all; just use the binary version.
So, my suggestion is to check timestamps, and if it's newer, compute MD5; with possibility of disabling one or both techniques.
Of course, what you really are willing to implement is up to you
I would go for CRC32 though, it's cheaper to implement, 4 times smaller, faster to compute, and it's OK as long as you don't use it to as a unique hash to compare against multiple files (chance of having a collision between two files is of 1 / 2^32, but if you start comparing against multiple files to see if they're "identical", the birthday paradox makes your chance alarmingly grow)
Keep up the good work!
MD5 could counter the benefit because you would need to read all script files, and HDD can be a bottleneck (I wonder if it actually is). So a time-stamp could probably be a lot faster, but this might be countered if the HDD must seek too much.CoreDumped wrote: Pending Stuff:
1. The compiled version should be discarded whenever the source file is modified. We can save the MD5 hash of the source file in the binary header and compare before reloading later. Or use the timestamp approach?
MD5 may benefit during development, since often a script may be reedited multiple times, only to be reverted to the way it was before.
Timestamps on the other hand, would benefit more the final release if the executable allows modding or something like that. And those who don't, don't need source file changes checks at all; just use the binary version.
So, my suggestion is to check timestamps, and if it's newer, compute MD5; with possibility of disabling one or both techniques.
Of course, what you really are willing to implement is up to you
I would go for CRC32 though, it's cheaper to implement, 4 times smaller, faster to compute, and it's OK as long as you don't use it to as a unique hash to compare against multiple files (chance of having a collision between two files is of 1 / 2^32, but if you start comparing against multiple files to see if they're "identical", the birthday paradox makes your chance alarmingly grow)
Keep up the good work!
- Jabberwocky
- OGRE Moderator
- Posts: 2819
- Joined: Mon Mar 05, 2007 11:17 pm
- Location: Canada
- x 218
- Contact:
Re: Plugin to save/load binary script format
An excellent and welcome feature!
- CoreDumped
- Gremlin
- Posts: 177
- Joined: Sun Aug 05, 2007 3:55 pm
- x 14
Re: Plugin to save/load binary script format
Thanks guys.
dark_sylinc, you are right. I'll go with the timestamp approach. However, I'm not sure how I can fetch the file stat of an ogre resource, since it might come from a filesystem, zip or even a remote web server. Any ideas? If it's not feasible, then we can go with CRC32.
Also, to check if the compiled version is valid, I'll also have to check if any of the base scripts (all the way to the top) were modified since the AST generated by the compiler is self-contained which contains the compiled code of the base scripts as well
EDIT: I'm not sure about the last point that the AST contain code of the base script
dark_sylinc, you are right. I'll go with the timestamp approach. However, I'm not sure how I can fetch the file stat of an ogre resource, since it might come from a filesystem, zip or even a remote web server. Any ideas? If it's not feasible, then we can go with CRC32.
Also, to check if the compiled version is valid, I'll also have to check if any of the base scripts (all the way to the top) were modified since the AST generated by the compiler is self-contained which contains the compiled code of the base scripts as well
EDIT: I'm not sure about the last point that the AST contain code of the base script
- so0os
- Bugbear
- Posts: 833
- Joined: Thu Apr 15, 2010 7:42 am
- Location: Poznan, Poland
- x 33
Re: Plugin to save/load binary script format
You're my saviour, many thanks!
Sos Sosowski
http://www.sos.gd
http://www.sos.gd
- jacmoe
- OGRE Retired Moderator
- Posts: 20570
- Joined: Thu Jan 22, 2004 10:13 am
- Location: Denmark
- x 179
- Contact:
Re: Plugin to save/load binary script format
This is really great!
Resource parsing/loading is really a time sink.
Resource parsing/loading is really a time sink.
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
- syedhs
- Silver Sponsor
- Posts: 2703
- Joined: Mon Aug 29, 2005 3:24 pm
- Location: Kuala Lumpur, Malaysia
- x 51
Re: Plugin to save/load binary script format
I really think this should be part of Ogre core - if I am not mistaken there are some discussions toward doing this but nothing comes out from it..
A willow deeply scarred, somebody's broken heart
And a washed-out dream
They follow the pattern of the wind, ya' see
Cause they got no place to be
That's why I'm starting with me
And a washed-out dream
They follow the pattern of the wind, ya' see
Cause they got no place to be
That's why I'm starting with me
- so0os
- Bugbear
- Posts: 833
- Joined: Thu Apr 15, 2010 7:42 am
- Location: Poznan, Poland
- x 33
Re: Plugin to save/load binary script format
I strongly approve! It takes approx 17 seconds on i5 to load all my materials and stuffs (and thus, boot the app) it's a narrow bottleneck, do it.syedhs wrote:I really think this should be part of Ogre core - if I am not mistaken there are some discussions toward doing this but nothing comes out from it..
Sos Sosowski
http://www.sos.gd
http://www.sos.gd
- spookyboo
- Silver Sponsor
- Posts: 1141
- Joined: Tue Jul 06, 2004 5:57 am
- x 151
- Contact:
Re: Plugin to save/load binary script format
I would even go so far to move all Ogre scripting stuff to the plugin. Serializing/deserializing is not core functionality of a rendering system.
Gui generator tool https://github.com/spookyboo/Magus ==> Windows binaries https://github.com/spookyboo/Magus_bin
HLMS editor https://github.com/spookyboo/HLMSEditor ==> Windows setup https://github.com/spookyboo/HLMSEditor ... e?raw=true
HLMS editor https://github.com/spookyboo/HLMSEditor ==> Windows setup https://github.com/spookyboo/HLMSEditor ... e?raw=true
-
- OGRE Retired Team Member
- Posts: 2903
- Joined: Thu Jan 18, 2007 2:48 pm
- x 58
- Contact:
Re: Plugin to save/load binary script format
Interesting idea. It has some appeal to mespookyboo wrote:I would even go so far to move all Ogre scripting stuff to the plugin. Serializing/deserializing is not core functionality of a rendering system.
- Zonder
- Ogre Magi
- Posts: 1168
- Joined: Mon Aug 04, 2008 7:51 pm
- Location: Manchester - England
- x 73
Re: Plugin to save/load binary script format
The last modified date should be enough during developmentdark_sylinc wrote:Wow! this looks surprising! Congrats!
MD5 could counter the benefit because you would need to read all script files, and HDD can be a bottleneck (I wonder if it actually is). So a time-stamp could probably be a lot faster, but this might be countered if the HDD must seek too much.CoreDumped wrote: Pending Stuff:
1. The compiled version should be discarded whenever the source file is modified. We can save the MD5 hash of the source file in the binary header and compare before reloading later. Or use the timestamp approach?
MD5 may benefit during development, since often a script may be reedited multiple times, only to be reverted to the way it was before.
Timestamps on the other hand, would benefit more the final release if the executable allows modding or something like that. And those who don't, don't need source file changes checks at all; just use the binary version.
So, my suggestion is to check timestamps, and if it's newer, compute MD5; with possibility of disabling one or both techniques.
Of course, what you really are willing to implement is up to you
I would go for CRC32 though, it's cheaper to implement, 4 times smaller, faster to compute, and it's OK as long as you don't use it to as a unique hash to compare against multiple files (chance of having a collision between two files is of 1 / 2^32, but if you start comparing against multiple files to see if they're "identical", the birthday paradox makes your chance alarmingly grow)
Keep up the good work!
There are 10 types of people in the world: Those who understand binary, and those who don't...
- boyamer
- Orc
- Posts: 459
- Joined: Sat Jan 24, 2009 11:16 am
- Location: Italy
- x 6
Re: Plugin to save/load binary script format
Absolutelly, this should go to Ogre trunk
- dark_sylinc
- OGRE Team Member
- Posts: 5299
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1279
- Contact:
Re: Plugin to save/load binary script format
+1 to put this in the trunk.
ResourceGroupManager::resourceModifiedTime?
Archive::getModifiedTime?CoreDumped wrote:Thanks guys.
dark_sylinc, you are right. I'll go with the timestamp approach. However, I'm not sure how I can fetch the file stat of an ogre resource, since it might come from a filesystem, zip or even a remote web server. Any ideas? If it's not feasible, then we can go with CRC32.
ResourceGroupManager::resourceModifiedTime?
- CoreDumped
- Gremlin
- Posts: 177
- Joined: Sun Aug 05, 2007 3:55 pm
- x 14
Re: Plugin to save/load binary script format
Ah great. Thanks, that would dodark_sylinc wrote:Archive::getModifiedTime?
ResourceGroupManager::resourceModifiedTime?
- so0os
- Bugbear
- Posts: 833
- Joined: Thu Apr 15, 2010 7:42 am
- Location: Poznan, Poland
- x 33
Re: Plugin to save/load binary script format
Hmm... i'd hate to complain, but I didn't get any boot speedup (or a minimal one) from using the plugin, Ogre.log says it skips scripts and processes sbins, but it still takes a year. Am I mising something, or is speedup not the intent here?
Sos Sosowski
http://www.sos.gd
http://www.sos.gd
- CoreDumped
- Gremlin
- Posts: 177
- Joined: Sun Aug 05, 2007 3:55 pm
- x 14
Re: Plugin to save/load binary script format
I tested this in release mode and see that the compiler's script parsing speed is fast (my original stats were on debug mode). However the ratio is still the same and the difference would be noticeable if you have many scriptsso0os wrote:Hmm... i'd hate to complain, but I didn't get any boot speedup (or a minimal one) from using the plugin, Ogre.log says it skips scripts and processes sbins, but it still takes a year. Am I mising something, or is speedup not the intent here?
With a large material file with 140k lines, 2000 materials with 2 passes each takes 6 seconds to parse normally, while the plugin takes 3 seconds. I think the translators take up that 2-3 seconds. I'll post proper profiling results later
However, we'll probably wont have that many materials in a game Script compilation probably wont take that much time to begin with anyway. I think the real bottleneck is the shader code compilation. I am working on passing the compiled shader code to the GpuProgram through a workaround (i dont see any directly interface to load from binary)
Also, I just updated the code with some optimizations. I was seeking past the header on every AST node, which meant thousands of slow seeks. Refactoring the data structures doesn't require this anymore.
-
- Greenskin
- Posts: 100
- Joined: Tue Aug 01, 2006 6:50 am
- Location: Canada
- x 6
- Contact:
Re: Plugin to save/load binary script format
This is a very useful and important plugin. If you continue to provide it with some love over the next few days (like serializing compiled shaders) then I think it will get wide use.
Good job, and thank you!
Good job, and thank you!
- CoreDumped
- Gremlin
- Posts: 177
- Joined: Sun Aug 05, 2007 3:55 pm
- x 14
Re: Plugin to save/load binary script format
Rambus wrote:This is a very useful and important plugin. If you continue to provide it with some love over the next few days (like serializing compiled shaders) then I think it will get wide use.
Good job, and thank you!
Thanks
Added support for re-compiling the script if it was modified since last compile
Code updated in repository
EDIT: Also added version number in the script header to track data structure changes. So if you are trying to load an older binary version, it will throw an exception. Be sure to clear out your scriptCache folder after the update (rev 6)
- spookyboo
- Silver Sponsor
- Posts: 1141
- Joined: Tue Jul 06, 2004 5:57 am
- x 151
- Contact:
Re: Plugin to save/load binary script format
I saw that it supports the basic Ogre script extensions. Can I also add other types? (i.e. *pu or *osm files)
Gui generator tool https://github.com/spookyboo/Magus ==> Windows binaries https://github.com/spookyboo/Magus_bin
HLMS editor https://github.com/spookyboo/HLMSEditor ==> Windows setup https://github.com/spookyboo/HLMSEditor ... e?raw=true
HLMS editor https://github.com/spookyboo/HLMSEditor ==> Windows setup https://github.com/spookyboo/HLMSEditor ... e?raw=true
-
- Ogre Magi
- Posts: 1120
- Joined: Wed Nov 15, 2006 7:41 pm
- Location: Finland
- x 5
Re: Plugin to save/load binary script format
Nice! This is clearly a good idea. Thank you for your work.
Did you notice Mr. Assaf Raman's work, a shader cache?
Did you notice Mr. Assaf Raman's work, a shader cache?
- CoreDumped
- Gremlin
- Posts: 177
- Joined: Sun Aug 05, 2007 3:55 pm
- x 14
Re: Plugin to save/load binary script format
I was thinking of trying the *.pu scripts out yesterday. I have a license of your libraryspookyboo wrote:I saw that it supports the basic Ogre script extensions. Can I also add other types? (i.e. *pu or *osm files)
As long as you use the ogre script compiler, it should be transparent and should work
However, for automatic detection of the binary scripts from resources.cfg file, i.e. if you deploy only the binary script, i hardcoded the search extensions to be either one of the four (program, material, particle, compositor). I'll move this to a config file.
Last edited by CoreDumped on Thu Jan 13, 2011 10:59 pm, edited 1 time in total.
- CoreDumped
- Gremlin
- Posts: 177
- Joined: Sun Aug 05, 2007 3:55 pm
- x 14
Re: Plugin to save/load binary script format
Yep, I briefly worked with RTShaderLib while trying out the iPhone ES 2.0 renderer. Its a great lib. Not sure if I will find something related there since the shader source code is cached there and we want to cache the binary format. I'll check it out thoughreptor wrote:Nice! This is clearly a good idea. Thank you for your work.
Did you notice Mr. Assaf Raman's work, a shader cache?
- dark_sylinc
- OGRE Team Member
- Posts: 5299
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1279
- Contact:
Re: Plugin to save/load binary script format
I think most developers here will agree that improving debug performance time is very important, regardless of how much Release mode actually improves.
It's awful when you're debugging something that took you to restart your application more than 3 or 4 times.
As for the shader binary, there's no abstract interface because all formats are too different.
In GLSL for example, you don't even get the binary or compiled representation at all, it's hidden from you inside OpenGL.
I believe HLSL starting from shader model 4.0 has the same issue.
You would need to cast the shader programs to the adequate derived class (HlslProgram, CgProgram, GLSLProgram, etc) and get direct access to their pointers.
It's awful when you're debugging something that took you to restart your application more than 3 or 4 times.
As for the shader binary, there's no abstract interface because all formats are too different.
In GLSL for example, you don't even get the binary or compiled representation at all, it's hidden from you inside OpenGL.
I believe HLSL starting from shader model 4.0 has the same issue.
You would need to cast the shader programs to the adequate derived class (HlslProgram, CgProgram, GLSLProgram, etc) and get direct access to their pointers.
- Wolfmanfx
- OGRE Team Member
- Posts: 1525
- Joined: Fri Feb 03, 2006 10:37 pm
- Location: Austria - Leoben
- x 99
- Contact:
Re: Plugin to save/load binary script format
http://www.ogre3d.org/forums/viewtopic.php?f=4&t=61004If you want to save shaders to the cache:
* Before the resource load call GpuProgramManager::getSingleton().setSaveMicrocodesToCache(true);
* When you want to save the cache to a file call - GpuProgramManager::getSingleton().saveMicrocodeCache(...);
When you want to use the cache:
* Before the resource load call - GpuProgramManager::getSingleton().loadMicrocodeCache(...);
I think Assaf did a shader cache which saves the bin representation of the shader.
- CoreDumped
- Gremlin
- Posts: 177
- Joined: Sun Aug 05, 2007 3:55 pm
- x 14
Re: Plugin to save/load binary script format
Oh great . I didn't know about this. So this is what reptor was talking aboutWolfmanfx wrote:I think Assaf did a shader cache which saves the bin representation of the shader.