I just fixed an obscure bug in the texturing code. It occurred where a material had more layers than could be multitextured (say 5),and the fallback to multipass happened just before an alpha-blended texture, AND another alpha-blended texture occurred afterwards within the range of the next multitexture pass. Got that? đŸ˜‰
Because Ogre uses multitexturing hardware as much as it can rather than multipassing all the time, it optimised the material into 3 passes on my GeForce2 (passes of 2,2, and 1 texture units). On the second pass though, both textures had alpha blend patterns. Ogre correctly realised that it had to turn the bottom layer into a scene blend for the second pass and turned the upper layer into a multitexture blend, but because it was only using the alpha value from the first layer for the scene blend, the second layer ends up getting stencilled out by the bottom layers alpha value which looks wrong.
This would not be an issue if either a) I had a card with 5 texture units (hmm, no…) or b) I always used multipass rendering because each alpha pattern is used independently. I did not intend to do the latter since I’m pleased with Ogre’s multitexture/multipass auto-handling which saves a lot of extra passes especially on high-end cards like the GF3.
Anyway, it was simply a case of detecting the multipass fallback scenario and setting the multitexture alpha operation to ADD, thus making the scene alpha blend to be based on a combination of the alpha from BOTH layers, whilst making the multitexture blend for the upper layer be based on the alpha in the upper layer only. Phew!
As an aside, I also fixed the unusual release build problem I had which was due to a read buffer overrun which the debug build didn’t catch because of the extra heap padding.