Question regarding HLMS microcode cache

Discussion area about developing with Ogre-Next (2.1, 2.2 and beyond)


Post Reply
aymar
Greenskin
Posts: 145
Joined: Fri Jun 12, 2015 6:53 pm
Location: Florianopolis, Brazil
x 17

Question regarding HLMS microcode cache

Post by aymar »

My app have 1 directional light, but since it has day/night cycle, this light keeps changing it's intensity, going from 0,0,0 to almost white in a full day cycle.

I noticed that everytime I get close to a city, if it's a different clock time than the previous time I was there, it recompiles the shaders (freezes for a few seconds).

What I thought is that maybe the HLMS hash is equal but the shader is different if it's night or day, then if I was in the city during the day, the microcode stored one set of shaders (with the light on), and then, when I go there at night, it detects that it needs to recompile the shaders, but then overwrite the day shaders with the night ones, since they share the same hash (assuming they do...) Then, if I ever go back to the city during the day it'll need to recompile again the day shader, since it dumped the previous one(got overwritten), even though it should be stored somewhere in the cache.

I DID notice that different shaders share the same filename, for instance, the shader 538607746PixelShader_ps.hlsl may have some content now, but if I backup this file, and come back to it after playing a little, the shader content will be different than before, I did this and compared the backep up file with the current one, and they were completely different shaders, one had normals, the other had different UVs, etc.

Does any of this make sense?
Hotshot5000
OGRE Contributor
OGRE Contributor
Posts: 226
Joined: Thu Oct 14, 2010 12:30 pm
x 56

Re: Question regarding HLMS microcode cache

Post by Hotshot5000 »

Are you absolutely sure that the only thing that you're changing is just the light's colour? Can you post the different shaders that have the same filename?
aymar
Greenskin
Posts: 145
Joined: Fri Jun 12, 2015 6:53 pm
Location: Florianopolis, Brazil
x 17

Re: Question regarding HLMS microcode cache

Post by aymar »

I'm changing a lot of things, but I couldn't think of anything that could cause this.
I'm changing the Sky shader, Fog shader, light color, light power scale, light direction (it emulates a sun movement).
I'm NOT: changing shadow setup, adding more lights, removing lights (on top of zeroing out the main light color), changing any mesh related stuff or textures.
Hotshot5000 wrote:Can you post the different shaders that have the same filename?
Sure thing, I've picked only the files that have the same name but different content from two runs:

Backup #1:
https://www.dropbox.com/s/wswk3lzj8mygz ... p.zip?dl=0

Backup #2:
https://www.dropbox.com/s/l23yiusohqtj1 ... 2.zip?dl=0

Thank you.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Question regarding HLMS microcode cache

Post by dark_sylinc »

Hi!

The names of the shader may vary per runs, that is to be expected. However the microcode cache hashes the contents of the generated shader and its macro preprocessor options; therefore even if the filenames are different, the cache will still work.

The reason has to be somewhere else, possibly:
  • The shader loading / saving routine is not set correctly (e.g. overwriting previous caches, corrupting it, etc). Things to look out for: Accidentally calling GpuProgramManager::clearMicrocodeCache, never calling loadMicrocodeCache, not calling saveMicrocodeCache at shutdown (thus cache is not updated), calls to removeMicrocodeFromCache are being made, two processes trying to access the cache with write access at the same time, thus the second process overwrites the cache or even corrupts it(*).
  • You're using very different settings for some reason between each run, thus you're encountering something new that wasn't in the cache.
(*) IIRC your game has a server and a client exe; make sure the server doesn't try to save the cache. If you have multiple clients running in the same machine at the same time, make sure only one has write access.

To debug what's going on place breakpoints at GpuProgramManager::getMicrocodeFromCache at see what's going on.
aymar
Greenskin
Posts: 145
Joined: Fri Jun 12, 2015 6:53 pm
Location: Florianopolis, Brazil
x 17

Re: Question regarding HLMS microcode cache

Post by aymar »

dark_sylinc wrote:The names of the shader may vary per runs, that is to be expected. However the microcode cache hashes the contents of the generated shader and its macro preprocessor options; therefore even if the filenames are different, the cache will still work.
This is what I needed to know, so the filenames aren't important, the cache is.

I'll try to sort this out then.

Let me ask you another question: Should I distribute the cache to my players? Will they use the same D3D11 cache even if they have a different machine (GPU, etc)?
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Question regarding HLMS microcode cache

Post by dark_sylinc »

For D3D11 the cache can be distributed to other players with different CPU/GPUs and it will work as intended.
Post Reply