About the implementation of the batch page

Hello JohnJ,

It has been a while.
I just wanted someone to confirm me this. If we have two different entities having the same materials and vertex informations they will be put in the same subbatch, is that right?

If it is true, we can make different models of the same tree species without cost.

Yes, with BatchPage you can add multiple trees using the same set of materials and it will perform as well as a single tree type, but unfortunately the same is not true with ImpostorPage. Sharing materials won't help ImpostorPage, because it has to generate a unique texture for each tree type. Although with some modification it wouldn't be impossible to use a single impostor texture for a whole set of geometric variations of a tree - although it won't look perfect, it would perform very well.

Thank you, lately we use only the batch pages because of the texture memory the impostors take. We noticed that using only the the batches in some cases is faster ( and better looking) than impostors.

Yeah, on a little research project a while back, I found that it can often be more efficient to use batched geometry than impostors if you're rendering reasonably close to the camera, due to fill rate bottlenecks - impostors don't spare any screen-space since they're basically square, while the full geometric version only takes up as much pixels as are needed. Basically, if you use impostors, it won't help if they're too close.

It still about the batch pages but this time it is about the materials.

I cannot seem to be able to disable the lighting of the materials used as batch. The lighting of the vertical planes are off since the normal faces the opposite way.

I do not understand as well why you divide the vertexcolor by the material ambient color.


vertexProgSource +=
//Perform lighting calculations (no specular)
" float3 light = normalize(objSpaceLight.xyz - (iPosition.xyz * objSpaceLight.w)); \n"
" float diffuseFactor = max(dot(normal, light), 0); \n"
" oColor = lightAmbient + diffuseFactor * lightDiffuse; \n";


vertexProgSource +=
" oColor *= (iColor / matAmbient); \n"; // vertex color divided by the ambient material ?


I tried Lighting off and putting emissive.

Ok, I'll check and see why the lighting isn't being properly disabled.

I'm not sure why the vertex color was divided by the ambient color, but I think it was supposed to make it so the tree color doesn't lower the output color below the ambient color level. Either way, it doesn't belong there (division by zero for black ambient colors), so I'll fix this too. I don't know what I was thinking when I wrote that

I don't know what I was thinking when I wrote that

It proves I was not stupid to think something was odd with this division.

I updated to the latest CVS version and my trees were not loading anymore when using bounds for the pagedgeometry. I had to put it on InfiniteMode.

It is probably my bounding size that is not correct or something. If someone else has the problem, you could hint to look in this direction. For the moment it is fine with me to stay in infinite mode.

It looks like lighting wasn't being disabled because I neglected to add the ability for BatchPage to switch off the lighting calculations in the shader when you disable lighting. It should work now, so give the CVS a try.

Also I fixed the division thing. I think I did it because without the division, the impostors will almost always appear 2x as bright as the real geometry. I must have thought that dividing by the ambient would "properly" fix this, but obviously that wasn't a good idea. Anyway, it seems to work fairly well now by simply multiplying by the vertexcolor * 2.

I haven't tested the bounds yet, but I will when I get more time later. Bounds may be important if you run split-screen multiplayer because in infinite mode if two players separate far enough, performance will drop drastically because the infinite system only has a limited cache space and can't keep two separated areas in memory at once.

Ok thanks for the fix and the note about the infinite mode, i will add this to my huge todo list.

I am lucky to have you.

Can you enlight me on the lighting of a vertex shader without fragment shader please, I am quite confused. I am not used to use Vertex shaders without fragment shaders.

"void main( \n"
" float4 iPosition : POSITION, \n"
" float3 normal : NORMAL, \n"
" float2 iUV : TEXCOORD0, \n"
" float2 iUV2 : TEXCOORD1, \n"
" float4 iColor : COLOR, \n"

" out float4 oPosition : POSITION, \n"
" out float2 oUV : TEXCOORD0, \n"
" out float2 oUV2 : TEXCOORD1, \n"
" out float4 oColor : COLOR, \n"
" out float4 oFog : FOG, \n";

if (lightingEnabled) vertexProgSource +=
" uniform float4 objSpaceLight, \n"
" uniform float4 lightDiffuse, \n"
" uniform float4 lightAmbient, \n";

if (fadeEnabled) vertexProgSource +=
" uniform float3 camPos, \n";

vertexProgSource +=
" uniform float4x4 worldViewProj, \n"
" uniform float fadeGap, \n"
" uniform float invisibleDist )\n"
"{ \n";

if (lightingEnabled) vertexProgSource +=
//Perform lighting calculations (no specular)
" float3 light = normalize(objSpaceLight.xyz - (iPosition.xyz * objSpaceLight.w)); \n"
" float diffuseFactor = max(dot(normal, light), 0); \n"
" oColor = (lightAmbient + diffuseFactor * lightDiffuse) * (iColor * 2); \n";
else vertexProgSource +=
" oColor = iColor; \n";

if (fadeEnabled) vertexProgSource +=
//Fade out in the distance
" float dist = distance(camPos.xz, iPosition.xz); \n"
" oColor.a *= (invisibleDist - dist) / fadeGap; \n";

vertexProgSource +=
" oUV = iUV; \n"
" oUV2 = iUV2; \n"
" oPosition = mul(worldViewProj, iPosition); \n"
" oFog.x = oPosition.z; \n"
"}";


What confuses me is the use of the vertex color and how the material are dealt in the rest of the pipeline.


I just want to make sure I understand to not have to change all the materials of the trees once again.

Note that I added 2nd UVs to do DarkMaps.

Note that all our vertex colors are all 1 1 1.

The way the vertex shader works is relatively simple - it takes the material and light's diffuse / ambient / etc. colors and the light's relative position, calculates an output vertex color, which is then multiplied by the inpout vertex color assigned to the vertex.

Before I explain this, you'll also need to understand how PagedGeometry uses vertex colors: every entity added to PagedGeometry has a variety of per-tree values, like "position", "scale", and also "color". Every entity added to PagedGeometry can have it's own color value applied. This is necessary for trees to be darkened / lightened from crude lightmaps, or some other purpose. Anyway, the way this color value is applied to the tree is simple: for impostors, they're simply tinted with vertex colors on the billboard. For batches, each tree is given vertex colors which are used to tint the tree the desired color.

Now the reason the vertex color is multiplied as an after-step is because it's meant to tint the color of the whole mesh rather than modify the lighting parameters as vertex colors are usually used for. This is done to match up with the way the impostors are lit - otherwise things start looking wrong with lightmaps applied to trees.

Using a vertex shader without a pixel shader is pretty simple - without a pixel shader, the fixed-function pipeline will simply take the values your vertex shader puts out and draws it appropriately. Just as long as you give an output vertex position, color, texture coordinate, and fog value (a hassle, but necessary for compatibility with fixed function fog under OpenGL I've found), it should work just fine. The only drawback I've found in using a vertex shader on its own is that you can't use multitexturing with one set of UVs.

(Somewhat off-topic:) Personally I wish every video card that doesn't support vertex and pixel shaders could be erased from existence . Shaders are extremely powerful and useful tools, and aren't relatively slower than fixed function as they used to be. It's been so much hassle maintaining both fixed function and shader support for PagedGeometry (basically requiring two coexisting branches of code), I doubt any of my future projects will support fixed function (though PagedGeometry v1.x will continue to support it of course). For example there's no way the planet rendering technology I'm currently working on would ever be possible without shaders.

Fortunately shaders are pretty much a universal standard now, so making a shader-only game isn't nearly as bold as it would have been years ago.

Thanks for the explanation.

The only last part I do not understand is how the Material Ambient, Material Diffuse, Material Specular and Material Emissive are considered in the equation of the rest of the fixed pipeline.

If you look at the code that assigns the shader parameters, lightDiffuse and lightAmbient are set to ACT_DERIVED_LIGHT_DIFFUSE_COLOUR and ACT_DERIVED_AMBIENT_LIGHT_COLOUR, which is basically give the product of the light color and the material color. I probably should have named the shader parameters something else, because lightDiffuse and lightAmbient does sound confusing.

OH! My bad. I did not noticed the "DERIVED" at all, everything makes sense now.

Thanks.

I updated to the latest CVS version and my trees were not loading anymore when using bounds for the pagedgeometry. I had to put it on InfiniteMode.

It is probably my bounding size that is not correct or something. If someone else has the problem, you could hint to look in this direction. For the moment it is fine with me to stay in infinite mode.
So far the bounded mode seems to work fine for me, as long as I give setBounds() the correct boundary info. If you still have problems with bounded mode, just give me your PagedGeometry setup code and I'll try to reproduce.

Lighting off and ambient has not the same behavior than Ogre. When Lighting off is activated the trees still have the vertex colors multiplied by the ambient parameter. Is it wanted?

You may want to remove the vertex color calculations when lighting off is set to reduce memory use and calculations.

Lighting off and ambient has not the same behavior than Ogre. When Lighting off is activated the trees still have the vertex colors multiplied by the ambient parameter. Is it wanted?
No, I forgot that the vertex color generation is being multiplied by the ambient color. I'll fix this soon.

You may want to remove the vertex color calculations when lighting off is set to reduce memory use and calculations.
Vertex colors are used when lighting is on or off - vertex colors are used for tree lightmaps, and are independent from dynamic lighting. For example you might want to disable lighting on your trees, but still use a lightmap to darken trees in shadows of your mountain.

However, it would be good to disable vertex colors when you don't use a tree colormap, although I don't know if that's possible. I'll put it on my todo list though.

Ahhh that makes sense for the vertex colors, thanks.

I really like your batch system.

I've fixed both problems (vertex color ambient lighting mistake, and now BatchedGeometry doesn't use vertex colors when they're not needed to save memory). Unfortunately, I'm having trouble getting the new ogreaddons SVN working, so it may be a while before everything's updated.

Ok, the SVN seems to have fixed itself, so the latest version should be available now.

Sorry it took so long (I was gone most of the day).

I just updated to the latest SVN version and it seems the following code causes problems with trees without VertexColors:


" oColor = (lightAmbient + diffuseFactor * lightDiffuse) * iColor; \n";


"*iColor" should not be there in that situation.

Those trees were all black.

That's strange. If no vertex colors are used, I would think the input vertex color to the vertex shader would be (1,1,1,1), as it appears to be on all the non-lightmapped examples. Can you confirm that all the trees are black on PagedGeometry examples 1/2/3/5/6?

If they appear black, then I can remove the vertex shader's iColor parameter when vertex colors aren't in use I think. If the examples don't appear black, however, I would suspect that you're using a black lightmap, or something else that's providing a black color to addEntity() when adding entities to PagedGeometry.

I found what caused the problem but i do not know yet why.

I had modified the batch page to support a 2nd set of UV to create a darkmap for the trees and it seems it is what causing the bug.

I'll pm you a sample of the trees and here's the patch to apply to recreate the bug.

Apply this patch on OgreAddons/forest/


Index: source/BatchPage.cpp
===================================================================
--- source/BatchPage.cpp (revision 2444)
+++ source/BatchPage.cpp (working copy)
@@ -174,10 +174,12 @@
" float4 iPosition : POSITION, \n"
" float3 normal : NORMAL, \n"
" float2 iUV : TEXCOORD0, \n"
+ " float2 iUV2 : TEXCOORD1, \n"
" float4 iColor : COLOR, \n"

" out float4 oPosition : POSITION, \n"
" out float2 oUV : TEXCOORD0, \n"
+ " out float2 oUV2 : TEXCOORD1, \n"
" out float4 oColor : COLOR, \n"
" out float4 oFog : FOG, \n";

@@ -210,6 +212,7 @@

vertexProgSource +=
" oUV = iUV; \n"
+ " oUV2 = iUV2; \n"
" oPosition = mul(worldViewProj, iPosition); \n"
" oFog.x = oPosition.z; \n"
"}";



Ok, I'll try to recreate the problem and debug it.

I noticed the tree causing the problem does not have a 2nd UV but those with two UVs work. But earlier they were working!

Are there any other changes? I see the shader adds a second UV set for the darkmap you refer to, but I don't see any code or anything in the material scripts (or any media for that matter) related to your darkmap. If you add a second set of UVs to the vertex shader, and don't provide the UV data when generating the mesh, or don't apply a second texture unit, it should be no mystery that the mesh is showing up black.

We have trees with 2nd UV to do dark maps on trees but some of them do not have dark map.

In my view, if in the material script there is no mention of the 2nd UV and there is no texcoord1 in the vertices of the mesh, the tree should not be black even though there is a texcoord1 in the vertex shader. Removing the "* iColor" cancels the fact it gets black, I don't get it.

Ok, I think I misunderstood what the darkmap was supposed to be used for.

Anyway, this time I've reproduced the black trees problem, and I'm working on finding what's causing it. It seems that adding a second UV coordinate makes the vertex colors turn black. Obviously my code is providing the correct vertex colors since the only thing that changes is the shader, so it must be some shader issue.

I am currently logged on gtalk if you have time to talk about the different issues.

Ok, I've implemented the changes we've discussed (HLSL shaders on DirectX, and better support for multiple texture units in BatchPage) and they're available from SVN.

While adding the automatic shader texture coordset code generator, I encountered an interesting problem - tree leaves disappeared, and alpha fading did not work! I tried to debug in nvperfhud, and then the tree trunks turned black again.

I solved the problem by not only ensuring that the vertex shader accesses the proper texture coord sets, but that it only tries to access vertex colors when they exist! Otherwise, the behavior is undefined. This explains all the strange behavior with the iColor parameter, etc.

Anyway, now it's all done "properly", and seems to be working perfectly with whatever mesh configuration you have. Let me know if this fixes the black trees problem (I hope it does - it should).