Terrain and Paging System Improvements for Very Large Worlds
Posted: Sun Mar 20, 2011 7:57 am
Hi,
My name is David Whittaker. I am working on my Ph.D. at the University of Alabama at Birmingham. I haven't completely nailed down my Ph.D. topic yet, but I am aiming in the direction of terrain generation techniques based on geologically accurate simulations. This is obviously not something I can do in a summer, but I think combining all the work and ideas I've generated so far to get me to this point into one project is. To begin my proposal, let me list some of the work I've done so far that has led me to this point:
displacementmapterrain: A looong time ago, one of my first OGRE projects was implementing displacementmapterrain - still in ogreaddons today. Its underlying idea - using a standard patch over and over again, only providing height data to each patch, and letting the vertex shader combine the height data with offset and scale information and the underlying patch prototype to greatly reduce graphics memory requirements is the underpinning of all my massive terrain work since.
Magna Terra: Fast forward to last year - this project was an over-specified, under-designed, and never implemented terrain engine for OGRE (and specifically, RoR). Many of the ideas of this project are applicable, but a more generalized terrain engine, for which these ideas could be one set of plugins, would be much more widely useful. So, the goal is to create an engine which could eventually accommodate all the ideas in this project, but not be limited to this particular implementation. I was a bit too focused on speed and optimization, and not on something that could be generally useful to the widest possible audience for this project. I will probably keep the OGRE project under this name.
MonkeyHorizons: This project for jMonkeyEngine represents my latest ideas, and most complete implementation. It has been partially released as a POC, but there is still work to do. So far, I have a system that can represent 8192 km^2 down to 1m resolution, with a single low-res bitmap to define the entire world down to 1km resolution, and a set of classes to generate noise to add procedural detail. The low-res bitmap can be as low-res as 2x2 (meaning the entire world is generated procedurally), or as high-res as the world is (meaning there is no procedural detail). There is still work to be done to make the texture splatting more realistic and controllable, but the geometry is working as intended. The new Magna Terra will begin as a port of this terrain engine to OGRE's paging and terrain concepts.
GSoC Proposal:
Deliverables:
Paging System - Implement a PageStrategy which works on MonkeyHorizons' page replacement system - successively replace 1 page with 4 more detailed pages as they get closer to the camera, while merging 4 pages into 1 as the camera moves away. In essence, the current TerrainQuadTreeNode class maps to a single page in the new PageStrategy. Much of the Terrain code will be integrated directly into the paging strategy, removing the need for LOD determination at two different levels (one at the PageStrategy level to determine which pages to show, and again the same types of calculations within a page to determine which nodes to show). Another way of thinking of this system is as taking the quadtree approach within a single terrain page and extending it to the entire PagedWorldSection, rather than having two different concepts. The current MonkeyHorizons demo has a far clip plane set to 10,000km due to this approach, and is running at 100fps.
Terrain Generation system - this system is de-coupled from the paging or terrain system, to allow a variety of methods of generating terrain. A mix of reading raw heightmaps from disk and generating procedural content should be supported. The demo will include the North American landscape at 1024m resolution, with the Grand Canyon area down to 32m resolution, and perlin noise added in everywhere to add high-frequency detail down to 1m. The terrain generation system is also responsible for generating "climate maps" which are grayscale bitmaps at the same resolution as the height data which are used as input to the splatting system. These can be physical things like average temperature, rainfall, or water table depth, observed climates in the region, or procedural content like noise functions or calculations from surrounding height data.
Splatting System - this system is based on rules. Two GLSL or other shader language function will be required which maps from the terrain height, normal, latitude, longitude, and any climate maps which have been supplied to a set of textures and their percentages. Future work may include specifying these rules in a configuration file of some type and dynamically generating the shader functions. Either way, one of these functions will be called within the vertex shader to work with climate values which don't interpolate linearly (such as observed climate types at each location), and the other will be called in the fragment shader to work with interpolated values and choose the right textures and blending on a per-pixel basis. This system is still under development in MonkeyHorizons and may change somewhat by this Summer, but the basic idea is to provide a framework that will allow procedural engines to generate believable splattings, while leaving the possibility of fully defining the world and just reading the data in from disk, or even defining the world on a high-level basis - such as general climate information like rainfall amounts and average temperature - at very low resolution, adding some noise to make things interesting, and combining that data with the terrain information to make some believable worlds from low-res data.
Possible deliverables, probably future work:
Static object system - integrating the LOD system with the paging terrain system seems simple enough, since the current terrain system already does this, but the above deliverables already spell out a pretty full summer for a single developer, so unless this ends up being practically automatic if the other systems are designed well, it may be out of the scope of a single GSoC project. Either way, the idea would be to simply map the LOD of all the objects on a page to the LOD of the page itself, and probably combine many objects into one on pages far from the camera.
Flora System - extend the static object system to work with tons of objects, fading out grass at a certain distance and bushes at a farther distance, and probably even replacing an entire forest of trees with a single texture at a certain distance. It would be great to integrate ngPlant or a similar software to create a fully open source replacement for SpeedTree.
World Editor - Add heightmaps at different resolutions over different parts of the world, add layers of noise to add detail, paint noise amplitudes over the terrain, paint climate maps and edit noise function parameters to generate detailed climate maps. All the editing should be done in real-time on a live instance of the world, and data should be saved as needed before unloading edited data (along with undo information). Finishing this is obviously out of the scope of a summer project, but it may be worthwhile to get a basic editor in place to make the engine more immediately accessible.
So, what do you think of the idea, and where should I draw the line for the deliverables? I'm usually too ambitious for my own good, so I drew the line earlier than I would have liked, but I'd rather go above and beyond my commitments within the alloted timeframe than not make them because I aimed for Mars and hit the moon.
David
My name is David Whittaker. I am working on my Ph.D. at the University of Alabama at Birmingham. I haven't completely nailed down my Ph.D. topic yet, but I am aiming in the direction of terrain generation techniques based on geologically accurate simulations. This is obviously not something I can do in a summer, but I think combining all the work and ideas I've generated so far to get me to this point into one project is. To begin my proposal, let me list some of the work I've done so far that has led me to this point:
displacementmapterrain: A looong time ago, one of my first OGRE projects was implementing displacementmapterrain - still in ogreaddons today. Its underlying idea - using a standard patch over and over again, only providing height data to each patch, and letting the vertex shader combine the height data with offset and scale information and the underlying patch prototype to greatly reduce graphics memory requirements is the underpinning of all my massive terrain work since.
Magna Terra: Fast forward to last year - this project was an over-specified, under-designed, and never implemented terrain engine for OGRE (and specifically, RoR). Many of the ideas of this project are applicable, but a more generalized terrain engine, for which these ideas could be one set of plugins, would be much more widely useful. So, the goal is to create an engine which could eventually accommodate all the ideas in this project, but not be limited to this particular implementation. I was a bit too focused on speed and optimization, and not on something that could be generally useful to the widest possible audience for this project. I will probably keep the OGRE project under this name.
MonkeyHorizons: This project for jMonkeyEngine represents my latest ideas, and most complete implementation. It has been partially released as a POC, but there is still work to do. So far, I have a system that can represent 8192 km^2 down to 1m resolution, with a single low-res bitmap to define the entire world down to 1km resolution, and a set of classes to generate noise to add procedural detail. The low-res bitmap can be as low-res as 2x2 (meaning the entire world is generated procedurally), or as high-res as the world is (meaning there is no procedural detail). There is still work to be done to make the texture splatting more realistic and controllable, but the geometry is working as intended. The new Magna Terra will begin as a port of this terrain engine to OGRE's paging and terrain concepts.
GSoC Proposal:
Deliverables:
Paging System - Implement a PageStrategy which works on MonkeyHorizons' page replacement system - successively replace 1 page with 4 more detailed pages as they get closer to the camera, while merging 4 pages into 1 as the camera moves away. In essence, the current TerrainQuadTreeNode class maps to a single page in the new PageStrategy. Much of the Terrain code will be integrated directly into the paging strategy, removing the need for LOD determination at two different levels (one at the PageStrategy level to determine which pages to show, and again the same types of calculations within a page to determine which nodes to show). Another way of thinking of this system is as taking the quadtree approach within a single terrain page and extending it to the entire PagedWorldSection, rather than having two different concepts. The current MonkeyHorizons demo has a far clip plane set to 10,000km due to this approach, and is running at 100fps.
Terrain Generation system - this system is de-coupled from the paging or terrain system, to allow a variety of methods of generating terrain. A mix of reading raw heightmaps from disk and generating procedural content should be supported. The demo will include the North American landscape at 1024m resolution, with the Grand Canyon area down to 32m resolution, and perlin noise added in everywhere to add high-frequency detail down to 1m. The terrain generation system is also responsible for generating "climate maps" which are grayscale bitmaps at the same resolution as the height data which are used as input to the splatting system. These can be physical things like average temperature, rainfall, or water table depth, observed climates in the region, or procedural content like noise functions or calculations from surrounding height data.
Splatting System - this system is based on rules. Two GLSL or other shader language function will be required which maps from the terrain height, normal, latitude, longitude, and any climate maps which have been supplied to a set of textures and their percentages. Future work may include specifying these rules in a configuration file of some type and dynamically generating the shader functions. Either way, one of these functions will be called within the vertex shader to work with climate values which don't interpolate linearly (such as observed climate types at each location), and the other will be called in the fragment shader to work with interpolated values and choose the right textures and blending on a per-pixel basis. This system is still under development in MonkeyHorizons and may change somewhat by this Summer, but the basic idea is to provide a framework that will allow procedural engines to generate believable splattings, while leaving the possibility of fully defining the world and just reading the data in from disk, or even defining the world on a high-level basis - such as general climate information like rainfall amounts and average temperature - at very low resolution, adding some noise to make things interesting, and combining that data with the terrain information to make some believable worlds from low-res data.
Possible deliverables, probably future work:
Static object system - integrating the LOD system with the paging terrain system seems simple enough, since the current terrain system already does this, but the above deliverables already spell out a pretty full summer for a single developer, so unless this ends up being practically automatic if the other systems are designed well, it may be out of the scope of a single GSoC project. Either way, the idea would be to simply map the LOD of all the objects on a page to the LOD of the page itself, and probably combine many objects into one on pages far from the camera.
Flora System - extend the static object system to work with tons of objects, fading out grass at a certain distance and bushes at a farther distance, and probably even replacing an entire forest of trees with a single texture at a certain distance. It would be great to integrate ngPlant or a similar software to create a fully open source replacement for SpeedTree.
World Editor - Add heightmaps at different resolutions over different parts of the world, add layers of noise to add detail, paint noise amplitudes over the terrain, paint climate maps and edit noise function parameters to generate detailed climate maps. All the editing should be done in real-time on a live instance of the world, and data should be saved as needed before unloading edited data (along with undo information). Finishing this is obviously out of the scope of a summer project, but it may be worthwhile to get a basic editor in place to make the engine more immediately accessible.
So, what do you think of the idea, and where should I draw the line for the deliverables? I'm usually too ambitious for my own good, so I drew the line earlier than I would have liked, but I'd rather go above and beyond my commitments within the alloted timeframe than not make them because I aimed for Mars and hit the moon.
David