Ogre Terrain Component FAQ        
Print


Table of contents

General:

 

help The demo loads a single 513x513 terrain. How do we load for example a 8192x8192 terrain? Does OgreTerrain support heightmap chunks like 16x16 chunks, each chunk being 513x513, or do we need to pass all of the 8192x8192 data?

A. Theoretically you can add as large an area as you like to one terrain instance and it will be decimated according to 16-bit index sizes and the batch sizes you configure. However in practice you're likely to want to split up your terrain into multiple pages of terrain, each of which is a Terrain instance. The code for doing this with the Paging component is already there to a degree but this is hugely untested so far. You can also use multiple Terrain instances outside the paging system if you want.

help In the case of big terrains how will the layers treated? One layer list for the entire terrain? One layer list per chunk? Let’s say i have an 8192x8192 terrain, will I be limited to only 6 normal mapped textures for the whole terrain, or 6 unique normal mapped textures per chunk of it?

A. A layer list for each Terrain instance. So each chunk or page of terrain can have a different number of layers with different textures applied, but within one Terrain instance you have a single layer list. This is because of the batching - the terrain chunk has to be capable of being rendered as a single batch at the lowest level of detail.

help Will page sources be handled by the current terrain implementation, or do we need to supply our own to handle the situations above?

A. The new terrain has its own storage format for quick loading, and this is what the paging system will use, but you can populate it yourself too since it's inherently editable. No 'page source' is required as with TSM because you can edit it directly.

help It looks like the Terrain's dimensions will always have the same width/depth? Did I misread, or are there factors that make it difficult to have terrain with differing width/depth dimensions?

A. Although Ogre::Terrain covers a square area, actually, OgreTerrain is just a page. You can add as many pages as you like, an extreme case below:
P01 P02 P03 P04
P05 P06 P06 P07
P08 P06 P06 P09
As you can see above, you can combine the pages to make any shape you like, even some pages like P06 in the example can cover much more space than other pages with few vertexes (imagine a desert area in the middle of usual hills)
Also each Page (Ogre::Terrain) can have its unique set of layers / textures / normal maps / colourmaps / lightmaps...
NOTE: Also by using terrain alignments, you can create a Page (Ogre::Terrain) with vertical alignment and a relatively low vertex count to simulate overhangs etc..

InfoNote: By the way, using a layout like above you can use P01-p09 as for example 513x513 and p06 as 65x65, but of course such setups would need very good editing support to make borders compatible.

help So if I have a height map of size 512 x 1024, and I want a terrain that matches it in world units (512 wide by 1024 deep), I'd need to create multiple pages and put them together? (pages of size 512) What about a narrow terrain level that was 1000 x 300?

A. The sides must be power of 2 anyway(*)(**) and 2 dimensions both are power of 2 always have a common divisor so your terrain can only be 1025x257 which means you can put four pages of 257x257 together
Please don’t think that having 4 pages would behave like having 4 ETMs in a scene, its completely different because it uses paging, think it as PLSM2, you have a page size of 257x257 and a map of 4x1 pages.... (but again, this is much more stable and each page can have its own set of layers/textures....)
(*) Why power of 2? You know its the golden rule of nearly all kinds of terrain rendering engines since lod is achieved by dividing by 2
(**) By saying power of 2 i mean the tile count per side, the vertex count is of course ((power of 2) + 1)
Power of two (+1) is a requirement, it's the only way that the terrain can divide by two at each LOD level. Each page has to be square because the hierarchical approach requires it - each level absorbs 4 children into one parent, a non-square arrangement would stop the hierarchy batching the same way.

If you want non-square worlds, use multiple square pages. Each terrain page instance is no more expensive than if the terrain page was non-square anyway, because the same limitations on batching / hierarchy would apply (that's why it's pointless complicating the code to make the hierarchy algorithms stop when they hit non-square conditions). If you configure them with the same min/max batch settings, they will be entirely equivalent of a non-square single page.
Other systems could support non-square (but still power of two +1) terrains because they decimated the terrain into fixed patches, which always remained separate from each other (therefore always a minimum number of batches). My system batches things up which means many things:

  • fewer batches at distance
  • much larger LOD range (in theory if your min batch size is 33, the entire terrain patch could reduce to 33x33 vertices at distance, something that is impossible in systems where each page is divided into fixed patches)
  • the ability to drop more detailed geometry at distance, saving VRAM. Because we have to be able to support rendering the entire terrain in one batch, the terrain data is held at multiple resolutions to deal with the fact that 16-bit indexes can't possibly address the entire terrain at the most detailed level, but have to be able to at low LOD levels. *at distance, the higher resolution terrain data can be purged from VRAM to free up space, leaving only the lower resolutions to serve the low LOD levels. This makes the system much more scalable.

 

help Can I load raw height maps without using an image?

A. Yes

Terrain::ImportData imp;
imp.terrainSize = 513;
imp.worldSize = 12000;
imp.inputScale = 1.0;
imp.inputFloat = pRawFloats;
terrain->prepare(imp);

 
Basically, the new terrain doesn't need to use Image to load the height data, that's just a convenience. It far prefers it if you just give it a list of unscaled floats!
If your floats are just in a flat file, it's easy:

MemoryDataStream rawFloats(ResourceGroupManager::getSingleton().openResource(filename, groupName));
imp.inputFloat = (float*)rawFloats.getPtr();

Be aware though of endian issues when you're using raw files like this. This file will only work if saved / loaded on machines of the same endianness (this applies to Myrddin too).
Basically we don't do this via a filename because it's far more flexible just to accept packaged data like this, the float data can come from anywhere. Also, you can load / construct / modify it in a background thread and even call prepare() in a background thread so the entire terrain can be bootstrapped without blocking. Again, I would expect you only to do it this way at import time, the prepare(stream)/save/load functionality of Terrain is what you'd use after that.

help I did not see an option to set the visibility flags of the terrain. Is there an API to do this? If not, I would like to request this functionality. In my game I have a built in MeshViewer, and when it appears, I want to make everything in the scene invisible by setting the camera's visibility filter, however the terrain is always visible.

A. Yeah ok, I'll put this in. There will be a default in TerrainGlobalOptions for convenience and a per-Terrain setting too.

Textures:

 

help Will there be a limit on how many different materials there can be splatted?

A. There is theoretically no limit, but it depends on what your material generator has been designed to handle. I'm concentrating on high-quality terrain, so whilst if I was doing basic splatting I could probably do 12 layers in one pass, I can only realistically do 5, maybe 6 with normal/parallax and specular mapping plus shadows. Now, you can of course do more than one pass if you want more, but with my generator I'm shooting for primarily single-pass terrain if I can. This is all changeable with pluggable material generators and you're free to define your own texture layouts if you want.

help Is a "material generator" some c++ code that outputs a material, perhaps generating some shader code based on some kind of config file parameters?

A. Yes. It's a 2-way thing - the generator is a C++ class which tells the Terrain object what information it needs (normals, light maps), and the terrain tells the generator how many layers there are, and what is in them. The generator also uses a 'profile' class which is sort of like a material technique or material scheme, but higher level because each one can request different types of data. A fixed-function profile might request a fully generated static lightmap for example, whilst a higher end profile would request a global normal map plus use shadow maps instead so it can use dynamic lighting.

help If I add, say, 20 texture layers to a terrain with this new terrain engine, is the engine always computing the 20 texture layers? Or is it divided into chunks and only the textures used in every chunk are rendered?

A. You can't add 20 layers to a single Terrain instance, the maximum is currently 5 with the default normal/parallax/specular/colourmap mapping enabled. Different material generators (they're pluggable) might have different limits. There is no auto-splitting if you try to exceed this, you're simply limited to it within that Terrain instance and must create separate Terrain instances manually if you want more variation (although 5 layers is plenty for most modern commercial games when combined with other detailing - people who think they need more should play some terrain-heavy games and look for themselves; nothing uses anywhere near 20 layers). It's up to the content creator to split appropriately, break things up using colour maps, detail objects etc. Any automatic approach would just result in inefficient terrain anyway.

help is it possible to have one of the layers be a multipurpose layer where different textures are used for blending at different parts of the terrain?

Secondly, I have this problem with decals, I thought I'd report it. Disregard the fact that the terrain is black, I just don't have textures atm, but the problem happens with textures loaded.

help I want to know if it’s possible to access terrain material directly from external, and if not, I would like to request this! Need this for integration with Hydrax and its depth technique!

 

Ogre::Technique *t = m_Handle->getMaterial()->getBestTechnique();
Ogre::Pass* p = t->createPass();

I personally do it this way. One limitation is that material is internally recreated when you modify texture list, so any changes are "cancelled" when you change splatting textures.

LOD:

 

help What sort of controls are provided to specify LOD transitions?

A. It's a max pixel error metric again - this seems to be the simplest, and of course it automatically adapts based on the viewport size (better than the old TSM did).
Batching is highly configurable, since you set a 'min batch size' and 'max batch size'. These 2 metrics determine how much the terrain is split up. A batch is any patch of triangles at a hierarchy level. The leaves of the quadtree that represents the hierarchy can range from the maximum batch size, to the minimum batch size at their lowest LOD, and then after that 4 adjacent patches at the min batch count get grouped into one parent patch at the min batch size to drop LOD. So, a large difference between the min and max batch size gives you more LOD levels at lowest level of the quad tree, so more flexibility in LOD picking (because each low-level patch can make its own LOD decision - when moving up the tree all 4 patches have to agree that it's time to drop LOD for it to happen), but at the expense of less efficient batching. You can tweak the absolute and relative values of these batch sizes to get the balance you need between vertex count and batch count.

help Sinbad, you mentioned in your initial post that with the hierarchical batching "there's also the opportunity to go to a lower overall resolution and to drop detail geometry in the far distance compared to non-hierarchical approaches." Would be possible to make this work with paging so distant terrain pages load lower resolution data instead of the full resolution maps?

A. Yes. Because of the hierarchical batching, the principle is that the entire terrain should be renderable as a single batch at the lowest possible level of detail (in practice you're more likely to get down to just 4 because of the pixel error unless you render right out to the horizon or the terrain isn't that irregular). This means you can't use the same vertex data at every LOD, because the lower LOD groupings need to access much more dispersed vertices within one batch than can be held in a single buffer structure (and addressed with 16-bit indexes) at the higher levels of detail. So, the terrain figures out how to create hierarchical copies of the data at lower resolutions for use in the distant LODs. This of course means that the higher level LOD vertex data becomes unreferenced beyond a certain distance and could be released. With the introduction of the composite map, we could also drop some textures potentially.

help Has any consideration been made on how to deal with loading higher detail terrain pages in a background thread while continuing to display the lower resolution terrain until the newer terrain data is ready, and once ready, swapping it out for the higher detail data? I'm not sure if this is clear but I'm referring to distant pages and then moving the camera towards them. I've worked with a few paging terrain systems, such as the Paging Landscape Manager and AdVantage Terrain System and the former always loaded the full resolution maps, which was a problem for memory consumption, while the latter dealt with it nicely by supporting hierarchical loading and rendering of only what was necessary. I'm obviously not suggesting that your terrain system need to implement it, only whether you think there'd be a way for others like myself to fit it into the architecture.

 
A. I've thought about this already and there are places where this is intended be implemented, but it's planning only right now. Firstly, the terrain when saved uses its own efficient binary format which is much faster to load than traditional methods of using images etc (you can obviously build the terrain from these components but when saved it's in a sort of 'compiled' form). This means that it's already faster to page data in than previous methods.
Secondly, the new core paging system already supports the concept of multiplicity at several levels, allowing for what I call 'external' and 'internal' LOD. 'External' LOD is a switch which requires a physical load to happen, and 'internal' LOD is done within already loaded structures. Currently the terrain's LOD is entirely internal, obviously, because it's done within a single instance (in the paging system's terminology, it's contained in PageContent). My approach to implementing 'external' LOD would be for the terrain's page-related classes (specifically, TerrainPageContentCollection) to simply be able to load 2 different versions of the terrain page based on 'first contact distance', either at full detail or at low detail, and just switch them out at transition time. This means that the Terrain class itself has no knowledge of this (making it simpler), except perhaps being able to clone itself into a lower detail version if asked (this is how the low detail version would be created for the purposes of saving).
PageContentCollection was always designed to handle 'external' LOD like this (and maybe other things - any reason for load-time multiplicity of a page really) so this is what I intend to use. Eventually. I put quite a lot of thought into these designs and hopefully it'll all pay off in the end.

help Does this mean you're saving out multiple levels of detail of the terrain when the terrain is saved to binary format? For example, are you saving out various hierarchy levels, or just a low and high level of detail?

A. I'm only saving the high detail data, the hierarchy (which is a quadtree) is built from that, and it's very fast to do so because of the regular grid and that all the other data is already determined or an under sampled version of the same data (plus, it's all done in the background thread - I build CPU versions of all the buffer structures and vertex declarations etc and just clone them across to the GPU in the render thread when they're ready).
My intention is to continue to do that, and that the low-detail version of the terrain will just appear to be a regular terrain with less base resolution from an internal standpoint. This keeps things simple, as well as decoupling any data dependence between the load levels, meaning they can be processed entirely separately (important when you're bringing in and processing new data in the background). Essentially then transition between loaded levels is just a case of switching out one Terrain instance for another. I favor this separable approach over trying to make the terrain instance itself aware of high & low detail loaded levels, because I think it overcomplicates things to have internal re-loading going on. That's why I made a distinction between 'internal' and 'external' LOD.

help Finally - and this one is a bit of a stretch - would it be possible to perform simplification on the terrain geometry? I can imagine this not being possible for a regular grid of vertices especially while editing, but I ask because the implementation sounds similar to Thatcher Ulrich's Chunked LOD implementation (and also AdVantage Terrain), which basically just stores terrain tiles in various hierarchy levels as mesh geometry without them being stored as a grid of vertices.

A. This is in theory possible, but I have no intention of supporting it myself. One of the reasons is that you can make lots of simplifying assumptions if you know you're always dealing with a regular grid, including about how much data you need to hold (positions only need one value - height- rather than being explicit), how you transition between LODs (one delta factor instead of 3 for full 3D morphing), how you deal with collision & ray queries (you can use a fast Bresnhams algorithm), and also even that your vertex shader calculation resolution is regular and predictable, meaning it's easier to avoid artifacts due to irregular tesselation & interpolation. I don't believe that the vertex overhead you save with an irregular terrain in the areas where you don't need the resolution is necessarily worth it. I'll be adding extra vertex compression later on which will exploit the regular grid too.

help I much prefer the way the lower detail is handled by sampling in a second thread right now over having separate low detail data saved. In our project we stream and page a lot of data from disk and so whenever we can efficiently get away with not touching the disk (such as in this case) it is very preferable compared to racing to load assets (even if it is in a p-queue it is likely to contend with other assets much of the time).

A. Having the lower detail data available can actually reduce your disk overhead, assuming that you see more distant pages than you actually visit. Of course, if you load the lower detail version and then subsequently visit it, you've overall incurred some additional loading (the low-detail version), but that's likely to be outweighed by the other pages that you've loaded only the low-detail version of and never got close enough to require that the higher detail version was paged in. This is assuming a large terrain with large draw distances though.
In any case, and low/high external LOD will be an option rather than a requirement (once it's added - the core paging of just the standard terrain tiles comes first) so you can pick what works best for you.

help So to clarify, does that mean that for distant terrain pages (when pages are eventually implemented by yourself or someone else), the highest level of detail data needs to be loaded, then the lower levels of detail are generated and the higher levels of detail can be discarded until they are later needed (if the camera moves towards them, for example)?

A. Yes, although the generation of the lower detail version would occur at save time when you've finished editing the terrain, not at runtime (although you could make it do it that way too), so that the runtime would only need to load the low detail level version in the first instance.
Probably because there's only a generalized structure to hold it right now (PageContentCollection) - it's deliberately abstract because I want to give people options for how they might use dynamic switch-out of PageContent instances; external LOD is the most obvious but I didn't want to bake that concept in there, I'm sure there are other reasons why someone might want to have multiple representations of a PageContent and switch them out based on some criterion.

Paging:

 

help How will this integrate with indoor rendering? Do you have some sort of indoor rendering support planned? Currently only the PCZSM handles indoor rendering properly - aka occludes the terrain and non-visible zones (rooms). But it's a scene manager in itself, and requires that a plugin be written especially for it to work. Will PCZSM be updated to support the new terrain/paging system, or will you develop a new system separately, or not at all?

A. Paging is about loading/unloading, it won't have anything to do with portals. However, pages can be owned by different SceneManagers so provided your paging strategy can cope with determining when these different pages come in and out of scope, you can use it to bring in indoor and outdoor pages. It's up to the scene managers to handle the visibility, as it always has been. So your page load could set up zones and portals within PCZ if you wanted (the page data structure is extensible to handle things like this). I'm not sure when I'd get to that though, I'd probably look for help from PCZ people like l3ft-h4nd. Eventually Ogre v2.0 will merge all these things into one glorious whole and PCZ won't be necessary any more. In some theoretical future The paging system as-is is supposed to be a bridge from here to there.
the TerrainGroup feature is now completed - this is a class that you can use to define and manage a grid of Terrain instances more easily. It will handle background loading for you, make sure neighbour relationships are set up, do ray queries and getHeightAt calls across multiple instances, that kind of thing. It doesn't do anything you couldn't already do manually with multiple Terrain instances, it just makes everything much easier.

Using this I defined a 9-page terrain much quicker than I could have done otherwise and found that it actually performed as well as my single page of terrain on my 9800. This is precisely what I wanted to see, since the new terrain system is primarily optimised to deal with many instances without the kind of batching issues you got in the old systems.
The worst-case scenario on this 9-page terrain was 61 batches (this also includes all the GUI). Compare this to a typical fixed-tile terrain (which is what TerrainSceneManager and most other terrain systems used) where you were guaranteed to have a fixed number of batches per terrain, in this configuration 8x8=64 per page, or a worst-case scenario of over 500 batches for a 9-page terrain. My system runs at under 50 batches for all 9 pages (for just the terrain) regardless of how much terrain you can see because distant areas are merged into a single batch. Adding more pages adds very few batches because they're distant enough to be aggressively merged.
Note that this isn't 'paging' per se yet. The TerrainGroup class provides all the features you need to add and remove terrain instances in a grid, each loaded in the background, but it doesn't actually decide to do that on its own, you have to tell it when to add/remove/load. The final piece of the paging puzzle will be to make the bridge between the Terrain and Paging components which literally just hooks the page events into the group slots. This actually isn't a big step, most of the work is already done.
The demo calls TerrainGroup::loadAllTerrains with the 'synchronous' option set to 'true' which forces the base geometry load to happen before return. I did this because I wanted the 9 pages to be immediately available, (but for the lightmapping etc to happen in the background if it's importing). I think that's fairly normal for a startup configuration - you'd then use threaded loading to add new terrain instances on the fly as you move.

 

Physics:

help How do I get existing physics (OgreNewt, for example) to shadow and match dynamically rendered terrain? Or, will this new terrain rendering system simplify physics integration? Or, will existing physics (OgreNewt, OgreOde, nxOgre, OgreBullet) need to be modified in order to work with this beautiful terrain renderer? Or, will we need a whole new kind of physics model altogether?

A. I've (I think) made it easy to extract the data you need from the terrain geometry in order to build physics objects from it. At the end of the day the terrain is one big patch of triangle strips, which I just split up to fit within 16-bit index limits (for efficiency and compatibility) and batch up based on an internal quadtree. You can extract all this information from the terrain page so physics integration is doable. You can also perform height queries and ray tests using the internal calculations.

help I want to use the Ogre Terrain to create a NavMesh, but I don't see any APIs giving access to the vertex/triangle data. Is there an easy way to derive this, or is it too much to request an API to give access to this data? I know that visually the vertices and triangles change due to LOD, but I want data at the highest level of detail.

A. Actually, the Terrain object doesn't hold this information. The terrain is what I call "hierarchically batched" which means there is no set of vertex data at the highest LOD which covers the entire terrain - instead there are a series of hierarchical nodes which each store a specific range of LODs, each of which has a different coverage of the terrain. The only batch which has the whole terrain stored in one are the lowest LOD levels - used when the terrain is very far away. This allows us to efficiently render the entire terrain in one batch when far away, but closer up smaller (physically) batches are used for higher LODs but overall the vertex data for each batch is of the same size (or within a small range). This also allows us to deal with terrains that would be impossible to address with 16-bit indexes - any patch with more than 256 vertices on each side is actually impossible to address as one batch anyway without 32-bit indexes, which I avoid for compatibility. My hierarchical batch system allows very large terrain patches while still respecting 16-bit indexes and generally giving better performance. Unfortunately, it can never be as simple as a single top-level set of vertex data.
So, if I gave you access to what we use internally, I think you'd just be very confused You really do just need to extract the raw heights or just walk across the terrain using getPoint() if you want something 'raw'. I suppose I could provide an API which dumps unindexed full-LOD triangles into a buffer (or maybe with 32-bit indexing), but this will be really inefficient if you then have to re-process the buffer yourself anyway. It's much better just to hook out the points and plug those into your system directly.

Performance:

 

Nvperfhud. Intels' GPA. I'm sure there is an OpenGL out there somewhere. VTune. etc... all are a great help.

http://msdn.microsoft.com/en-us/library/ee417062%28VS.85%29.aspx(external link)
http://developer.nvidia.com/object/nvperfhud_home.html(external link)
http://software.intel.com/en-us/articles/intel-gpa/(external link)

nvperfhud has a "scissor rect" to clip the rendering output, set that to 1x1 and see if the perf gets better. If it does get a lot faster then you're shader/render limited, if it only gets a tiny bit better then you're geometry or setup limited.

I also think that nvperfhud (and I know that intel GPA) will let you force all textures to 1x1 or 2x2 sizes so you can easily rule out texture loads.

There's a neat write-up over at Intel on optimizing a real game, even if you don't try GPA the ideas and most of the terms and data will help your work in perfhud.
http://software.intel.com/sites/billboard/archive/become-more-godlike.php(external link)


Contributors to this page: uzik958 points  , spacegaier5891 points  and jacmoe171980 points  .
Page last modified on Thursday 18 of August, 2011 14:27:34 UTC by uzik958 points .


The content on this page is licensed under the terms of the Creative Commons Attribution-ShareAlike License.
As an exception, any source code contributed within the content is released into the Public Domain.