[2-1 PSO] Terra System

Discussion area about developing with Ogre-Next (2.1, 2.2 and beyond)


Post Reply
User avatar
SolarPortal
OGRE Contributor
OGRE Contributor
Posts: 203
Joined: Sat Jul 16, 2011 8:29 pm
Location: UK
x 51
Contact:

[2-1 PSO] Terra System

Post by SolarPortal »

Hi,

We have moved over to the latest PSO branch in order to use the new Terra System in our engine and have come across a couple of issues with it.
Just grabbing notepad lol as we had no internet for a month which is why i havent been around haha :P and have been writing notes on problems found and things to pass back to the community.

(Major) Issue 1) Forward 3D
- This crashes the demo instantly and is a feature that is really required on the terrain for us.
- Also after a tweak to the Terra code, i got it too stop crashing, but the moment a point light is entered into the frustum for lighting, it CTD.

Here is the tweak.
Line 521 - approx:
add this after the mGlobalLightListBuffer = 0; variable set.

Code: Select all

if (!casterPass)
{
Line 550-559- approx:
find this block:

Code: Select all

		if (shadowNode)
			{
				//Six variables * 4 (padded vec3) * 4 (bytes) * numLights
				mapSize += (6 * 4 * 4) * numLights;
			}
			else
			{
				//Three variables * 4 (padded vec3) * 4 (bytes) * numDirectionalLights
				mapSize += (3 * 4 * 4) * numDirectionalLights;
			}
after it, Add:

Code: Select all

		}else
		{
			mapSize += (2 + 2) * 4;
		}

This will close the condition above

Line 590-599- approx:
after:

Code: Select all

mPreparedPass.shadowMaps.clear();
add:

Code: Select all

if (!casterPass)
		{
Line 815-822- approx:
after:

Code: Select all

Forward3D *forward3D = sceneManager->getForward3D();
			if (forward3D)
			{
				forward3D->fillConstBufferData(renderTarget, passBufferPtr);
				passBufferPtr += forward3D->getConstBufferSize() >> 2;
			}
add:

Code: Select all

}
		else
		{
			//vec2 depthRange;
			Real fNear, fFar;
			shadowNode->getMinMaxDepthRange(camera, fNear, fFar);
			const Real depthRange = fFar - fNear;
			*passBufferPtr++ = fNear;
			*passBufferPtr++ = 1.0f / depthRange;
			passBufferPtr += 2;
		}



Issue 2) Terra's fast Shadows that shadow over the landscape only work in DX11 correctly, in OpenGL, they are there but faint and broken up...

Issue 3) Normals maps applied to the ground kill the shader completely in both DX11 and OpenGL. I consider this quite important to be fixed as a terrain needs normal maps to be applied.

Issue 4) This may not be an issue, but the top layer applied in the shader seems to have no control in the blendweights map and is applied across the entire terrrain.


Thats all i have to report on the Terra system so far and i look forward to your responses and fixes as this will make the terra completely usable in end game :)

Thanks so much for all your hard work so far and every time i pop into Ogre3D, advancements are always seeming to be made on the newer renderers. Keep it up! :D
Lead developer of the Skyline Game Engine: https://aurasoft-skyline.co.uk
User avatar
lingfors
Hobgoblin
Posts: 525
Joined: Mon Apr 02, 2007 12:18 am
Location: Sweden
x 79

Re: [2-1 PSO] Terra System

Post by lingfors »

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1280
Contact:

Re: [2-1 PSO] Terra System

Post by dark_sylinc »

Thanks a lot for the feedback!

The Terra system is quite new and I didn't get the chance to test it thoroughly in a real environment as I was sucked into the Metal port. Thanks for doing that for me :)
SolarPortal wrote: (Major) Issue 1) Forward 3D
- This crashes the demo instantly and is a feature that is really required on the terrain for us.
- Also after a tweak to the Terra code, i got it too stop crashing, but the moment a point light is entered into the frustum for lighting, it CTD.
Ah! Gotcha, will check. (what does CTD mean?)
SolarPortal wrote: Issue 2) Terra's fast Shadows that shadow over the landscape only work in DX11 correctly, in OpenGL, they are there but faint and broken up...
Can you upload a picture?
Is the shadow broken in the main sample?
Is it possible your tweak to get it to run with Forward3D broke the shadows?
Can you provide the material (or how you setup the terrain)?

I'm just hoping it's not an NVIDIA vs AMD problem because it used to run fine on my machine (I use AMD). Hopefully it just broke by some other unrelated change and I just didn't notice.
SolarPortal wrote: Issue 3) Normals maps applied to the ground kill the shader completely in both DX11 and OpenGL. I consider this quite important to be fixed as a terrain needs normal maps to be applied.
Can you send me the material and you're using to set it up? (and textures too if you're kind :P)
SolarPortal wrote: Issue 4) This may not be an issue, but the top layer applied in the shader seems to have no control in the blendweights map and is applied across the entire terrrain.
You mean detail map 0? If so that's a bug.
SolarPortal wrote:Just grabbing notepad lol as we had no internet for a month which is why i havent been around haha :P
The horror!

Overall, Metal port has been keeping me busy. I'll try to take a look when I get a chance.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1280
Contact:

Re: [2-1 PSO] Terra System

Post by dark_sylinc »

Got Forward3D to work. It was embarrassingly broken in more than one way.

Image

Regarding Terrain shadows looking wrong on OpenGL: I can't reproduce that. I'm hoping your tweaks was that caused it. Otherwise please upload a picture.
Issue 3) Normals maps applied to the ground kill the shader completely in both DX11 and OpenGL. I consider this quite important to be fixed as a terrain needs normal maps to be applied.
I'll be waiting for your material / setup info so I can take a better look.
Issue 4) This may not be an issue, but the top layer applied in the shader seems to have no control in the blendweights map and is applied across the entire terrrain.
I should be able to check this on my own, assuming you're talking about detail map #0?
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1280
Contact:

Re: [2-1 PSO] Terra System

Post by dark_sylinc »

Issue 4) This may not be an issue, but the top layer applied in the shader seems to have no control in the blendweights map and is applied across the entire terrrain.
I just tested this and it's working as intended. The only thing I did notice is that from the blendweights map all four RGBA channels are used, thus to hide the 4th detail map, the alpha channel must be 0 which may be counterintuitive or easy to miss.
Perhaps you think it does something else? What do you expect it to do? Perhaps I'm misunderstanding the problem?
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1280
Contact:

Re: [2-1 PSO] Terra System

Post by dark_sylinc »

Fixed detail normal maps. Thanks for the bug report.

I'll be waiting for more info from your 2 other issues.
User avatar
SolarPortal
OGRE Contributor
OGRE Contributor
Posts: 203
Joined: Sat Jul 16, 2011 8:29 pm
Location: UK
x 51
Contact:

Re: [2-1 PSO] Terra System

Post by SolarPortal »

wow, thank you for the quick work. I will get the other information to you as quickly as possible :)

Thanks for your continued support of the community :)
Lead developer of the Skyline Game Engine: https://aurasoft-skyline.co.uk
User avatar
SolarPortal
OGRE Contributor
OGRE Contributor
Posts: 203
Joined: Sat Jul 16, 2011 8:29 pm
Location: UK
x 51
Contact:

Re: [2-1 PSO] Terra System

Post by SolarPortal »

Hey, just wanted to say sorry for the amount of time to get the information to you.

This weekend has suddenly become very busy and should be able to respond tomorrow with info. First job.
Lead developer of the Skyline Game Engine: https://aurasoft-skyline.co.uk
User avatar
SolarPortal
OGRE Contributor
OGRE Contributor
Posts: 203
Joined: Sat Jul 16, 2011 8:29 pm
Location: UK
x 51
Contact:

Re: [2-1 PSO] Terra System

Post by SolarPortal »

Hey, finally got the videos and information you wanted. I had to reset all my Terra code to ogre3d's version and update to the latest pso branch in order to compile with the OptimizationStrategy.
So Latest source and identical shaders and Terra code. it is a lot better than it was now, but some problems remain :)
As with ourselves and our users, things tend to only get fixed when someone actually uses the codebase system and finds it needs fixed lol :P So we understand how this feels :) Keep it up! :D

Here are the materials: 1 is glossy and the other is fully rough.
Terra Materials.zip
(1.17 KiB) Downloaded 94 times
I have pm'd you the textures to use :)

Lets look at the issues now :)

Issue 1) Forward 3D. This is mostly solved and works very well with lights that do not cast shadows. The moment you add a spot light with shadow casting in DX11 it will crash with this error in the Log:

Code: Select all

High-level program 3758096645PixelShader_ps encountered an error during loading and is thus not supported.
OGRE EXCEPTION(-2147467259:RenderingAPIException): Cannot compile D3D11 high-level shader 3758096645PixelShader_ps Errors:
E:\00_Skyline_SDK\Skyline - New Renderer\Win32_Release\3758096645PixelShader_ps.hlsl(128,14-26): error X4510: maximum ps_5_0 sampler register index (16) exceeded
 in D3D11HLSLProgram::compileMicrocode at E:\00_Skyline Integrations\Ogre3D SDK\Source\v2-1-pso\RenderSystems\Direct3D11\src\OgreD3D11HLSLProgram.cpp (line 545)
High-level program 3758162181PixelShader_ps encountered an error during loading and is thus not supported.
OGRE EXCEPTION(-2147467259:RenderingAPIException): Cannot compile D3D11 high-level shader 3758162181PixelShader_ps Errors:
E:\00_Skyline_SDK\Skyline - New Renderer\Win32_Release\3758162181PixelShader_ps.hlsl(128,14-26): error X4510: maximum ps_5_0 sampler register index (16) exceeded
 in D3D11HLSLProgram::compileMicrocode at E:\00_Skyline Integrations\Ogre3D SDK\Source\v2-1-pso\RenderSystems\Direct3D11\src\OgreD3D11HLSLProgram.cpp (line 545)
WARNING: Texture instance '[Hash 0x3a8c545d]' was defined as manually loaded, but no manual loader was provided. This Resource will be lost if it has to be reloaded.
And in OpenGL, this happens :) If you keep adding spotlights, you will eventually lose all textures.

[youtube]k4fo9W-S214[/youtube]

The more spot lights added removes the other layers. I suspect the shadow maps are overwriting the other textures.
Does this mean we can have lights on terrain, but we cant have any shadows?

Issue 2) (Terra's fast Shadows that shadow over the landscape)
I have recorded a video from our engine, but the same happens in the demo.
Firstly, the scene with a skybox is OpenGL and the Black Sky is DX11 as the postFX sky issue seems to be there still from early 2.1

[youtube]_hNJbfQv3vM[/youtube]

Issue 3) Normal maps are now working and have nothing to report on them [SOLVED]

Issue 4)I can happily report that the Alpha channel does work well and can use all 4 textures correctly. [SOLVED]
The only thing that mislead me was the use of a png which does not support alpha, but transparency only using a layer mask, which didnt work to well.
So i used a DXT5 DDS and it seemed to resolve the problem :)


Anyway, here is a screenshot:
Image


--------------------- Thats the end of the older issues --------------------------

Now for the newer issues i have located while testing, and have recorded videos if possible.

Issue 5) "reflection_map": Like the normal maps originally. You cannot add a envmap to the terrain. It simply does not generate the shaders. Again like shadows, maybe running out of slots in the shader.

Issue 6) A strange blackening when looking towards the lights direction with full roughness or black moving mess with Glossy.
This just makes the terrain look a bit strange and have tried to capture it the best i can for you in a video :)
Its hard to describe so best to watch the videos :)

Glossy Terrain material from folder:
[youtube]epCvPvmntoA[/youtube]

Rough Terrain material from folder:
[youtube]RF14k9RFEfg[/youtube]

--------------------- Just some extra info from a users perspective --------------------------

CTD means - Crash To Desktop

Q) How many terrains do you see with metallic textures? I understand its too provide a ful PBR shader for the terrain but most use cases this is not required and would be great to turn off.
Q) Will there be a base layer that covers the entire terrain with no blend map like in the older terrain system.

Another pointer is texture packing, if the shader is running out of texture space, then how about packing them like:
  • Diffuse: (RGB - Diffuse) (A - Roughness), since roughness is a monochrome texture.
  • Normal: (RGB - Normal) map (A - Displacement) - Would love to have POM on the terrain.
  • (Optional) Metallic: (RGB - Metallic) (A - Detail map) - The detail map is a monochrome texture that adds more detail to the surface of the terrain so rocks that are large at a distance still have detail when close. Like Cryengine or Torque3D did.
Even in this packed form, it has the same amount of textures for full PBR and more, but saves another 4 textures. This could be another 4 shadow maps or something else.

Other things we would love to see, is a global normal map to pop those mountains right out haha :P

Q) How would we go about getting more paint layers?

wow. so much typing and videos. lol :P I hope this helps :D
We understand that this is still being developed and have full faith that you will be able to pull it off :)

Good work so far!
Thanks for your time again :)
Lead developer of the Skyline Game Engine: https://aurasoft-skyline.co.uk
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1280
Contact:

Re: [2-1 PSO] Terra System

Post by dark_sylinc »

Whoa!
That's a lot of useful feedback! Thanks!!!
The moment you add a spot light with shadow casting in DX11 it will crash with this error in the Log:
(...)
And in OpenGL, this happens :) If you keep adding spotlights, you will eventually lose all textures.
Ugh. This API limitation is beginning to become a problem.

In theory array textures are supposed to mitigate this problem. But you're probably running with 1 texture per array which would cause 1 sampler register for each texture. And given the amount of textures the terrain needs, you're running out of slots. Try to keep everything at same resolution and packed into arrays.
Issue 2) (Terra's fast Shadows that shadow over the landscape)
I have recorded a video from our engine, but the same happens in the demo.
Firstly, the scene with a skybox is OpenGL and the Black Sky is DX11 as the postFX sky issue seems to be there still from early 2.1
Mmm... sounds like an NVIDIA problem, or if our problem... it doesn't repro on AMD. Unfortunately I don't have any NV card close to test. The ones I have died :(
Issue 5) "reflection_map": Like the normal maps originally. You cannot add a envmap to the terrain. It simply does not generate the shaders. Again like shadows, maybe running out of slots in the shader.
Could be running out of tex units, or could just be bad code. Gotta check.
Issue 6) A strange blackening when looking towards the lights direction with full roughness or black moving mess with Glossy.
This just makes the terrain look a bit strange and have tried to capture it the best i can for you in a video :)
Its hard to describe so best to watch the videos
Interesting indeed. I'm hoping it's just a blending problem (i.e. you ask metalness = 1 but for some reason the blending math says "finalMetalness = 0.5")
Q) How many terrains do you see with metallic textures? I understand its too provide a ful PBR shader for the terrain but most use cases this is not required and would be great to turn off.
The idea was that it was optional. Still is.
Diffuse: (RGB - Diffuse) (A - Roughness), since roughness is a monochrome texture.
That's not a bad idea. Just a question: Do you expect the diffuse and roughness to be more or less of the same resolution? AFAIK on PBS normally we get: Diffuse low res, Roughness high res, Metalness depends, normals high res.
Q) How would we go about getting more paint layers?
You mean more than 4? Once we figgure all the current stuff we may go onto that :lol:
User avatar
SolarPortal
OGRE Contributor
OGRE Contributor
Posts: 203
Joined: Sat Jul 16, 2011 8:29 pm
Location: UK
x 51
Contact:

Re: [2-1 PSO] Terra System

Post by SolarPortal »

No problem. Happy to help :)
I will give the texture arrays a try and see if this mitigates the problem, but i don't think it will, but happy to be surprised.

Unfortunately, i have no other cards to send or give you atm; i will try it on my laptop though as this has a dual GFX card. An intel HD 4000 and GT710m.
I seem to be the opposite and have no AMD cards to try.

Good point on the diffuse being smaller, but we are still going upto 1024 sizes even though it would be quicker with smaller textures. What if that was optional also for packing?
However, most games have low quality textures and get away with it haha :P
The idea was that it was optional. Still is.
When i took the metallic out the material, the same amount of slots were still used as the shadows still broke the terrain textures at the same numbers. That's why i mentioned it...
You mean more than 4? Once we figgure all the current stuff we may go onto that
lol. That would be awesome! :D But as you say, lets get this up and running first.

Thanks again and look forward to your responses :)
Lead developer of the Skyline Game Engine: https://aurasoft-skyline.co.uk
User avatar
SolarPortal
OGRE Contributor
OGRE Contributor
Posts: 203
Joined: Sat Jul 16, 2011 8:29 pm
Location: UK
x 51
Contact:

Re: [2-1 PSO] Terra System

Post by SolarPortal »

The fast shadows issue exists on my laptop aswell when using the integrated gfx and high powered gfx card.

I have yet to try the texture arrays :)
Lead developer of the Skyline Game Engine: https://aurasoft-skyline.co.uk
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1280
Contact:

Re: [2-1 PSO] Terra System

Post by dark_sylinc »

Good to know. I should try my Intel card
User avatar
SolarPortal
OGRE Contributor
OGRE Contributor
Posts: 203
Joined: Sat Jul 16, 2011 8:29 pm
Location: UK
x 51
Contact:

Re: [2-1 PSO] Terra System

Post by SolarPortal »

The texture pack at default settings does the exact same effect with lights with cast shadows on.
Hope this helps :D
Lead developer of the Skyline Game Engine: https://aurasoft-skyline.co.uk
Post Reply