Performance in Large Worlds

Hi,

A few of you may know that I'm using Hydrax in Project Utopia. Well, I'm starting to think about performance, and Hydrax is next on my list.

Okay, the way it currently works is:
- The world consists of one large body of land (I mean, really large). As a server is connected, the island increases in size.
- The idea is that around the outside of the 'island' is an infinite ocean. So the end of the world will basically be like looking out at sea into nothingness (as opposed to the boring 'surround it with mountains' approach). Okay, so the projected grid is perfect for that.
- Smaller water bodies inland are rendered on simple planes.

The major problem is that Hydrax is still rendered even when inland as a projected grid. So basically everywhere under the terrain the water is being rendered. Unfortunately Hydrax is a huge performance hit, especially with paged geometry; and with the underwater/depth/etc. effects it has to render everything in 3/4 passes (as far as I remember), so it's really expensive. Chances are, you're not going to be hanging around the coast most of the time.

I've thought about it, and here are some possible solutions:

1) Only load Hydrax in the 'outside' zones. So when a player enters an outside zone, Hydrax is loaded and the player can suddenly see the sea. The problem with this is that the game's performance in these outer zones would be noticeably less than in the inland zones, coupled with a considerable loading time each time a player enters and exists the zone. Plus, the zones are generally huge anyway, so a player may not even see the ocean until he/she is near the coastline.

2) Have Hydrax decide when to render according to the depth technique. The idea being that Hydrax always exists under the land, but where a certain depth is detected (i.e. the ocean surface), it is rendered as normal. Another way may be to have the water occluded and only rendered when it is in the camera view. Or even have Hydrax cut a massive hole out of its mesh. I have no idea if any of this would be possible, but that would be down to Xavyiy.

3) Use the simple grid and have multiple instances of Hydrax act like a zone. Imagine zones like tiles; so say if I had 4 inland zones, I would need 8 zones around the outside defined as water. I'm not sure how this would fare, or if Hydrax can handle it, and it may be more effort than its worth...

Out of the options, I would go with the first since each server has its own set of settings, and I can easily enable/disable Hydrax accordingly. But it's not very swimmingly (NPI).

If anyone has any suggestions/thoughts as to how Hydrax could perform at its best in a large world, that would be a great help.

(Oh and btw if anyone has any idea how to get the depth component working with PLSM, that would be greatly appreciated. It's just refusing to work, and causes PLSM to go crazy and crash out.)

Cheers!

Hi Daniel,

I think you can mix some of the options you've written:

For first, implement the option 1: disable Hydrax if the player is inland(just checking if the sea is in the camera frustum), I suggest you waiting to Hydrax 0.5 for the Hydrax::setVisible(const bool& Visible) funtion. As you say, you're going to experiment less performance in the zonez where Hydrax is actived, but there's no solution about this(Or just disabling Hydrax fps-kill features, like depth rendering, foam, projected grid complexity, etc...).
Maybe you could be interested in adding all underwater objects to another Ogre::RenderQueue and do some modifications to Hydrax/RttManager.cpp to render only this RenderQueue in refraction/depth/underwaterdepth rtt's, but it's a little more complex.

For inland and individual bodies of water:
you can activate hydrax when these zones are in the camera frustum, and just use the projected grid with a 2*2 complexity using GPU generated normals and disabling underwater rendering, etc...(There's a little example in the Editor, GPUWater.hdx), but... this options is only viable is just ONE body of water is in the camera frustum, for diferent bodies of water in the camera frustum this option isn't viable..
If you need multiple bodies of water at the same time and each one in different y-position(for depth calculations..) you'll need to create your own shader for simple waters and just add a reflection/refraction effect, without using Hydrax for that. [Best option I think]

My opinion is that you should use Hydrax for the ocean, where a high level of realism is needed(like an eye-candy , and use simple water planes for little bodies of water(to save performance).

The last and more complex option is to code your own Hydrax::Module(Modiffying the existent ProjectedGrid.cpp and updating only the vertex that are over the terrain, or maybe creating your own geometry approach...).

-----------------------------------------
I think you can be interested in read this article about how the "Anno 1404" programmers have solved the problem you've:
http://anno.de.ubi.com/devdiary.php?dia ... ary_1404_2 [In german, I've used the google traductor to read it and I've understood it ]
Basically, they've used somethink like Hydrax(complex water) for ocean and simple planes with reflection/refraction effect for inland water.

Probably for future Hydrax versions(0.8, 0.9...)I'll add a little abstraction noise module to get really waves simulation near the shore, or creating waves/ripples behing ships, etc, like they've done for Anno 1404, it'll be glad to have something like that:

Hydrax::Noise::InteractiveNoise* mInteractiveNoise = new Hydrax::Noise::InteractiveNoise(new Hydrax::Noise::FTT() /* Original hydrax noise, the interactive noise is just an abstraction module to get interactive water effects over the original hydrax noise */)
InteractiveNoise->createRipple(vector2 worldripplepos, float power, etc...)
InteractiveNoise->createWave(vector2 pos, vector2 dir, etc...)
etc...
P.D.: Sorry for my little English, and if you need specific support to doing modifications to Hydrax, just send me an e-mail [xavyiy [at] gmail] and I'll help you in all I can

Good luck with your awesome project!
Xavi

About your PLSM crash, are you using PSLM2?
I've used Hydrax with PLSM2 before and works fine(Addind correctly the hydrax depth technique). So, what's the code you are using to add the depth technique?

Xavi

Thanks for your reply Xavyiy, much appreciated.

I suggest you waiting to Hydrax 0.5 for the Hydrax::setVisible(const bool& Visible) funtion.
That sounds very useful. I'll look forward to that.

Maybe you could be interested in adding all underwater objects to another Ogre::RenderQueue
This may be useful in the future, but for now I don't plan on having any objects underwater. Currently, I have no choice since PagedGeometry is procedurally generated across the terrain. Should be ok once I can manually place geometry.

My opinion is that you should use Hydrax for the ocean, where a high level of realism is needed(like an eye-candy , and use simple water planes for little bodies of water(to save performance). ... I think you can be interested in read this article about how the "Anno 1404" programmers have solved the problem
Yeah, that's what I plan on doing -- definately the best approach. (OT: I'm also interested in doing something similar to HeroEngine's ribbon water; it's seriously awesome.)

About your PLSM crash, are you using PSLM2?
I've used Hydrax with PLSM2 before and works fine(Addind correctly the hydrax depth technique). So, what's the code you are using to add the depth technique?
Yes, PLSM2. I'm not confident that I'm doing it correctly at all; it's just the fact that it crashes even when I don't specify a depth technique yet enable the component -- does it normally crash in that case? When I've worked with TSM, when I didn't specify a depth technique with the component enabled, it never crashed.

The problem I have is that the terrain is split into pages, and I have no idea about a global material to apply the depth technique to. In the end I came up with this:

Ogre::Vector3 p = Ogre::Vector3(0,0,0);
void* myOptionPtr = &p;
mSceneMgr->getOption("getMaterialPageName", myOptionPtr);
std::string terr_mat = **static_cast<std::string**>(myOptionPtr);
mHydraxModule->getMaterialManager()->addDepthTechnique(
static_cast<Ogre::MaterialPtr>(Ogre::MaterialManager::getSingleton().
getByName(terr_mat))->createTechnique());
Obviously that doesn't work. How did you manage it?

Thanks again!

The fact is that PLSM2 clone the base material to one per terrain page, so, this is the correct way to add the depth technique:

int w, h;

mEscena->mSceneManager->getOption("Width", &w);
mEscena->mSceneManager->getOption("Height", &h);

Ogre::String Mapa;
Ogre::String Formato;

mEscena->mSceneManager->getOption("CurrentMap", &Mapa);
mEscena->mSceneManager->getOption("CurrentTextureFormat", &Formato);

for (int i = 0; i < w; i++)
{
for (int j = 0; j < h; j++)
{
Ogre::String NombreMaterial = Formato + "." + Ogre::StringConverter::toString(i) + "." + Ogre::StringConverter::toString(j) + "." + Mapa;

mat = Ogre::MaterialManager::getSingleton().getByName(NombreMaterial);

if (!mat.isNull())
{
try
{
// Add depth technique
mHydrax->getMaterialManager()->addDepthTechnique(mat->createTechnique());
}
catch (...)
{
}
}
}
}
Sorry for the Spanish-Code
If should work;)

Xavi

Speedy reply. Great! That does look right.

I tried it though, and it crashes the same way it has always crashed as soon as I include the depth component.

23:32:45: [Hydrax] Depth material created.
23:32:45: [Hydrax] Creating underwater material...
23:32:45: Texture: UnderwaterDistortion.jpg: Loading 1 faces(PF_R8G8B8,256x256x1) with hardware generated mipmaps from Image. Internal format is PF_X8R8G8B8,256x256x1.
23:32:45: [Hydrax] Underwater material created.
23:32:45: [Hydrax] Materials created.
23:32:45: [Hydrax] Creating god rays...
23:32:45: [Hydrax] Perlin destroyed.
{application exits}

Like I say, it shouldn't crash even without the depth technique defined.

EDIT: Actually, I tell a porky. The application just exits at that point without error.

Since you don't get that problem, it's a tricky one. It's not easy to debug; it could be anything. Any ideas?

What hydrax components are you setting?, maybe there's a bug with a especific conbination of them, It's really extrange :S.

Edit: Rencently I've fixed some bugs that leads to crashes if your call Hydrax->create() more than one time(Something related with listeners that weren't deleted in remove() function), so maybe this is what's happend.
If you post some of your Hydrax-realted code maybe I can know what's the problem.

Xavi

Okay, more specifically, it's definitely something wrong with PLSM2. I don't think Hydrax is at fault. Just to mention, I tried using the default config for Hydrax which has all components enabled, and same crash, and I'm not doing anything wrong in the initialisation.

What PLSM does is call destroyCamera() randomly; I thought that when it did this, it was calling a shutdown. But in fact, it all it does is call destroyCamera() at the point where I stated {application exits}. The same thing happened with PagedGeometry, when I called mTree->update(), PLSM would try to destroy the ImpostorCam created by PagedGeometry and hence the application would exit. I fixed this with a PLSM hack in destroyCamera() by skipping the function if it was trying to destroy "ImpostorCam". Does that make any sense in terms of Hydrax?

If the camera pointer that is passed to Hydrax constructor is deleted before Hydrax was, it should crash. It's maybe this, but if it only crashes when depth component is enabled things change.
Anyway, there isn't a breaktrace with VS debugger?, it is extrange.. only with depth component, maybe something that is initialised when the component is enabled is fighting with PLSM2, but seems odd.

Edit: Try disabling god rays component (It creates a internal camera, maybe...)

Edit: Try disabling god rays component (It creates a internal camera, maybe...)
I just tried that before you posted -- and it worked! The god rays component does indeed create an internal camera.

Okay, so... I could fix that by doing the same thing I did with PagedGeometry. But I probably won't be enabling the underwater components for now.

Also, depth looks nicer for water rendering inland, but it has problems with exponential fog for long distance sea; looks very ugly. I want to keep my precious fog. [image 3]

And the water's depth technique is completely disabled when my camera approaches the water - it looks fine from a distance, but as soon as I go up close to the water, it has all kinds of nasty artifacts. I'm thinking it may be to do with the fact that the camera is attached to the scene node, and maybe Hydrax is switching to underwater before it gets there? [images 1+2] Plus, for some reason the geometry that is underwater's material is canceled out (white). [image 1]

The performance is affected even more with depth; 5fps really isn't good, as opposed to 50 without Hydrax. But I'm sure that once I can control where geometry is placed, it will be much better.

Here's some screenshots.

Wow, very impressive!

I think the best option to debug Hydrax is creating some overlays and setting the hydrax rtt's textures in them.
#define _def_Hydrax_Reflection_Rtt_Name "HydraxReflectionMap"
#define _def_Hydrax_Refraction_Rtt_Name "HydraxRefractionMap"
#define _def_Hydrax_Depth_Rtt_Name "HydraxDepthMap"
#define _def_Hydrax_Depth_Reflection_Rtt_Name "HydraxDepthReflectionMap"
#define _def_Hydrax_API_Rtt_Name "HydraxAPIMap"
#define _def_Hydrax_GPU_Normal_Map_Rtt_Name "HydraxNormalMap"

Try to disable fog just to see if the problems are solved. Anyway, in Hydrax 0.5 fog is disable in water materials(depth and overwater/underwater materials). You can disable it in 0.4 version just Editing MaterialManager.cpp. (setting setFogEnable(false) in material passes), but before try just disabling caelum fog management.

About the camera attaching to a scene node, it should work as expected(You need the RttManager.cpp path, but I remember you had applied it!).

About performance: You scene is quite complex and Hydrax have to render it for 3 more times: Reflection, Reflaction and Depth, and another one if underwater reflections are enabled, only when the camera is underwater, of course.

I think you should just solve the visual artifacts, and waiting to next Hydrax release(v0.5) for perfomarmance modifications. Anyway I'm going to centrate my efforts for Hydrax 0.6 in improving hydrax performance and provide some tricks to keep up performance in complex scenes.

Xavi

Hi again; took a break for a while.

I upgraded to version 0.5 without problems, so great! Is there anything new I should know about in 0.5 that would help?

I noticed the camera node/distance problem is fixed! Also, performance is (slightly) better by a few FPS, but it's still clunky @ 14 FPS with only "Sun|Depth|Smooth|Caustics" and reduced PG Complexity set.

I'm still having battles with the fog though. You mentioned that you had fog disabled in 0.5, but that doesn't seem to be the case.

I assumed the Hydrax material needed "fog_override true none".

So I tried to add this to MaterialManager::_createWaterMaterial:

WM_Technique0_Pass0->setFog(true, Ogre::FOG_NONE);
[also tried]
WaterMaterial->setFog(true, Ogre::FOG_NONE);
But no cigar.

Wow, very impressive!
Thanks, lol, but they're supposed to be bad screens!

Finally I haven't disable the fog because after I've thinked that maybe people would have fog on the water, but there's something that I've to do for the 0.6 version:
Disable fog in all hydrax depth materials and in underwater material(And underwater god rays, etc.., because the fog is only over the water surface ), and only enables it in the over-water material.

You can disable the fog with PASS->setFog(true, Ogre::FOG_NONE); in all hydrax materials(Which means, all depth materials, and underwater/overwater materials).

Try it and post results:)

Xavi

Okay, here are the results.

I know I shouldn't use Hydrax for inland rendering, but it's just a demo.

Thanks again for your help.

Wow, looks very nice!
Does the Hydrax depth technique work finally with your PLSM?

Also you can be interested in: viewtopic.php?f=20&t=9235 If you've planned put underwater grass or semethink with alpha channel in the water!

Xavi

Okay, here are the results.

I know I shouldn't use Hydrax for inland rendering, but it's just a demo.

Thanks again for your help.

(..)

hello, are you using paged geometry?

if so how do you fix the artifact caused by the waving grass and hydrax?

thanks!

Woot. It happens that the depth technique actually didn't load for those screenshots that I posted! -- I thought they looked odd.

So, here are the new screenshots with the depth technique for real!

I don't care about the rules, I'm using Hydrax for inland water rendering! Need to tweak it to make it more inland friendly though - might use the GPU settings to make the water flatter, a bit like Perfect World's water, and obviously it's a bit too blue atm.






Unfortunately I can't apply the alpha fix to PagedGeometry right now, since it complains when shaders are used for its materials. I'll need to add fade techniques to each material. I'll post how to get it working when I figure it out.