TaaTT4 wrote:
dark_sylinc wrote:
We offer an easy workaround though:saveMicrocodeCache & loadMicrocodeCache.
Use these functions to save compiled shaders to disk (e.g. write on exit, load on startup) and these long compile times will go away for the next time.
What is the workflow of the
loadMicrocodeCache?
I have to call it before starting to parse and load the HLMS material scripts (and, in some magical way, the
D3DCompile function will never be called anymore)?
When loading the
microcode cache, one of the best places to do it is right after you've registered the Hlms implementations. Perform:
Code: Select all
GpuProgramManager::getSingleton().setSaveMicrocodesToCache( true ); //Make sure it's enabled.
DataStreamPtr shaderCacheFile = root->openFileStream( "D:/MyCache.cache" );
GpuProgramManager::getSingleton().loadMicrocodeCache( shaderCacheFile );
When saving (at exit, before the RenderSystems are shut down):
Code: Select all
DataStreamPtr shaderCacheFile = root->createFileStream( "D:/MyCache.cache", ResourceGroupManager::DEFAULT_RESOURCE_GROUP_NAME, true );
GpuProgramManager::getSingleton().saveMicrocodeCache( shaderCacheFile );
Note these calls can throw if there are IO errors (a folder doesn't exist in the path, the file didn't exist, you don't have write or read access, etc) so make sure to wrap the calls in a try/catch block.
Can I save and then load different set of caches (to have, for example, one cache for metal materials, another one for concrete materials and so on)?
Technically you could. But the cache is only one and loading a new cache destroys the current one. It is easier to treat it like a big can where you throw all your shaders. The cache is not specific to just Hlms shaders. It's for all types of shaders (i.e. low level materials included).
The
microcode cache computes a 128-bit hash of the shader's source code, and compares that against the cache to retrieve the bytecode before compiling. If you change the shader's source code, the cache will continue to grow indefinitely and probably these entries will never be used, but won't do any harm other than growing its size (except if you have hundreds of thousands of shaders, and thus hash collisions could happen). You clear the cache by simply deleting the file
As for the reason why it takes to long, I only have guesses. As you can Google, lots of industry people have complained about this. And they all resort to the same (save the
microcode/bytecode cache). Btw I did not know that the latest D3D compilers (the one shipped with Windows 8 SDK) was faster. That's cool to know!
My guesses is our array sizes. HLSL compiler takes particularly long unrolling loops and with big arrays (probably some quadratic O(N^2) behavior triggering when parsing the syntax, because the issue is there whether you use the array or not, and whether optimizations are turned on and off).
Our shaders generated by the Hlms don't have loops (we unroll them ourselves with the Hlms script parser), but we do have huge arrays, like Material materialArray[273].
Unfortunately these arrays are key to our AZDO optimizations so they can't go away.
The
microcode cache solved our problems in production.