Ogres Volume Component

[PhilipLB]

Sorry for the late answere, here is some crunchtime in the office...
Sounds like a big approach, could work.

Whereas I thought about it in a simpler way which won't scale to infinity but would allow quite big worlds. Just one static big ChunkTree where only chunks near to the player are loaded with geometry. As the player moves along, now near chunks load their geometry and now far away chunks unload it. Drawback: The ChunkTree would have to be present completly which consumes memory. Advantage: Shouldn't be hard to implement.

[PhilipLB]

I just added the async-loading-support which included a bit of refactoring. And the loaded-callback is modified to include lod-level and amount
of items still to be processed and will always be called when a chunk is loaded, even if there are no triangles generated by the chunk. This way, some kind of progressbar can be created or the terrain be displayed, as soon as a certain LOD-level is loaded while the finer details are still loading in the background.

[holocronweaver]

As usual, many thanks for your continued work on this project Philip.

If I understand correctly, to enable asynchronous loading you just set ChunkParameters.async to true. When I do this I get a silent crash and occasionally see the the following error:

pure virtual method called
terminate called without an active exception

This only occurs when async is enabled.

I have a more pressing problem though. A month or so ago I made a few hacks in Chunk and MeshBuilder so that I can call load on the same Chunk multiple times (over different regions) without having the SceneManager complain about the Entities already being bound. Unfortunately when I did a recent wipe and clone of my fork, I forgot to commit these changes. Now I cannot for the life of me remember what I did exactly - my attempts to reproduce it result in silent crashes during my second load attempt.

So basically I want to be able to:

Code: Select all

 
mVolumeRoot->load(mVolumeRootNode, Ogre::Vector3::ZERO, Ogre::Vector3(15), 1, &mParameters);
mVolumeRoot->load(mVolumeRootNode, Ogre::Vector3(15), Ogre::Vector3(30), 1, &mParameters);

and so on.

Oddly, I am now getting a crash I wasn't getting before the recent merge. When I try to load two separate chunk/node pairs with the same parameters over different regions, like so:

Code: Select all

 
mVolumeRootA->load(mVolumeRootNodeA, Ogre::Vector3::ZERO, Ogre::Vector3(15), 1, &mParameters);
mVolumeRootB->load(mVolumeRootNodeB, Ogre::Vector3(15), Ogre::Vector3(30), 1, &mParameters);

I get a silent crash during the second load. Again, this did NOT occur before the merge.

Now that async loading is available I was hoping to finish up my paging system this weekend and demo it, but I am stuck being unable to load multiple regions or use the new async feature. I will keep trying to figure out a way around this, but any help would be greatly appreciated. I am excited to finally get paging out and in the wild and want to do a mad push to make it happen.

[PhilipLB]

You are right with just setting async to true.

This parallel updating of multiple regions is an untested scenario, I'll investigate. This should be fixed, soon as it looks like a common use case.

[holocronweaver]

You are right with just setting async to true.

This parallel updating of multiple regions is an untested scenario, I'll investigate. This should be fixed, soon as it looks like a common use case.

Sorry, I should have been more clear. The second and third crashes mentioned above occur when async is FALSE. So this is sequential loading of multiple regions for both the same Chunktree and separate Chunktrees.

For paging purposes, I am currently using the latter method, creating new Chunktrees as needed and destroying the old ones as they are replaced. This no longer works after the recent upstream merge, even if async is false. I will investigate the cause myself, but if you happen to know what change caused the problem please let me know.

[holocronweaver]

I solved the crash when loading two different volumes, A and B, with async=false:

Code: Select all

mVolumeRootA->load(mVolumeRootNodeA, Ogre::Vector3::ZERO, Ogre::Vector3(15), 1, &mParameters);
mVolumeRootB->load(mVolumeRootNodeB, Ogre::Vector3(15), Ogre::Vector3(30), 1, &mParameters);

Inside the Chunk::load class you need to add back the lines for removing the request handlers:

Code: Select all

if (!parameters->async)
{
          while(mShared->chunksBeingProcessed)
          {
             LogManager::getSingleton().stream() << "CHUNKS BEING PROCESSED";
             OGRE_THREAD_SLEEP(0);
             wq->processResponses();
          }

++       wq->removeRequestHandler(workQueueChannel, this);
++       wq->removeResponseHandler(workQueueChannel, this);
}

It is not clear to me why this would cause trouble among different Chunktrees, but it does.

Philip, would you mind making these changes in your repo? Also, might this be related to async being nonfunctional?

Ok, back to finishing up paging.

[PhilipLB]

Ah, good find. I'll make a pull request in like 1-2h (need to fork again and to compile and test as some warnings have been fixed in the main repo).
I moved the deregistering to the first frame-update for the async loading. This never gets called when you are not async and call load twice.

So after your patch, I'll think about how to handle the WorkQueue more flexible and less errorprone as there is a high chance that it will cause problems again when updating multiple regions simultanously. Could be better moved to another class or so which just registers once and never deregisters.

Your paging-approach seems to be suitable for giant worlds. Will it be simple to use? If it's a bit tough, we might also integrate this paging on the ChunkTree-level which only needs one parameter in the ChunkParameters like "MaxAcceptedGeometryicErrorUntilChunkGetsUnloaded" (better name to come ). This type of paging wouldn't be hard to implement as it needs just some loadGeometry/unloadGeometry-functions. So basically the question is, whether both solutions would have a reason to co-exist or your more scalable approach is better anyway.

// Edit: Pull request created.

[PhilipLB]

Update:

• Refactored the WorkQueue handling to a separate class. The handler is only registered once and never gets deregistered so the complicated handling of it (and causing the last crash) is gone now.
• Added an async parameter to the terrain config files.

[PhilipLB]

After some silence:

The first complete round on http://www.volume-gfx.com/ is finished, the LOD-algorithm is completly explained. Now that this is complete, I can write some surrounding articles everywhere (like CSG).

Coding-whise: First of all, I discovered that the 256^3 3D texture from Acropora hasn't the world dimensions 256^3. This means that a density value of 3 isn't 3 world units. This fact explains some of the weird constants I had to use now and then. This can be fixed now by adding a function to the Source class like "Real getToWorldDimensionFactor(void)" with which those constants are actually calculated.

After cleaning this up, finally the paging can be attacked. holocronweaver: As you are in GSoC-mode now, I expect you don't have much time to finish what you got, right? That's totally OK of course. Is it in a state where I can pick it up? I also could start with a Source-implementation reading from multiple density files. Or I could implement my approach of loading/unloading single chunks in a bigger ChunkTree. This wouldn't scale to infinity, but would make at least really big worlds possible.

Another thing to do after the constants cleanup is the detailed documentation of the various loading parameters in the wiki.

All in all, more than enough to do. I haven't had any time or motivation to program here in the last few weeks as I had a bad crunch time at my day to day job and after a 12-14h session full of backend stuff, I didn't feel like programming anything at the evening. But this project now comes to an end and I hope to have more spare time for this one here.

[PhilipLB]

Ok, the hard coded factors from above are gone now.

• Refactored the VolumeSource so it delivers a factor to convert volume coordinates to world coordinates which is used for generating the Octree and not anymore a hardcoded, manually set factor from the loading parameters.
• Shortened the sample label of the CSG demo so it no longer breaks the UI.

[PhilipLB]

Small fix:

• Checking in the rendering call, whether the current camera actually has a viewport.

Thanks to mmatt for spotting this!

[holocronweaver]

Sorry for taking so long to respond! I must have missed previous notifications.

holocronweaver: As you are in GSoC-mode now, I expect you don't have much time to finish what you got, right? That's totally OK of course. Is it in a state where I can pick it up? I also could start with a Source-implementation reading from multiple density files. Or I could implement my approach of loading/unloading single chunks in a bigger ChunkTree. This wouldn't scale to infinity, but would make at least really big worlds possible.

Yep, GSoC will be my focus until the end of the summer. My paging code is barely documented and is somewhat broken. I need figure out a clean way to call ManualObject.end() in the main thread during volume extraction. This is because OpenGL requires API calls to only occur in the thread that created the GL context. Ideas?

If I can solve this, I should be able to get the code in working order pretty quickly and will upload them to BitBucket where you can fix up remaining bugs.

In the mean time, I see no reason why you couldn't implement your octree paging scheme.

Another thing to do after the constants cleanup is the detailed documentation of the various loading parameters in the wiki.

This and a tutorial would make things much easier for beginners. Once GSoC is over I can help with both.

[mmatt]

Thanks to mmatt for spotting this!

No problem, happy to help.

[PhilipLB]

I need figure out a clean way to call ManualObject.end() in the main thread during volume extraction. This is because OpenGL requires API calls to only occur in the thread that created the GL context. Ideas?

For what do you use a ManualObject? The current Chunk implementation uses the WorkQueue from Ogre which offers a callback in the main thread when the Work part is done.

In the mean time, I see no reason why you couldn't implement your octree paging scheme.

Maybe I find time for it, maybe you are first. (Spare-)Time will show.

This and a tutorial would make things much easier for beginners. Once GSoC is over I can help with both.

Yep, this definitely comes first.

[PhilipLB]

mmatt talked with me about how to better hand over the triangles of the volume to a physics engine for example. The outcome is that the MeshBuilderCallback now also gets the RenderOperation so the physics engine can be directly pointed to the vertices/indices memory.

[PhilipLB]

mmatt needed a unique something in the MeshBuilderCallback to identify the Chunk where the callback was fired from to update only specific parts of the physics representation, that's why a pointer to the Chunks parent class SimpleRenderable is now handed in. Not the Chunk itself to not create a dependency from the MeshBuilder to the Chunk.
And there where some flaws in the editing which appeared after the correction of some scaling factors mentioned above.

All in all:

• Fixed editing of GridSource which took some scaling factors wrong into account. So the editing of the terrain demo works again.
• Refactored some GridSource (Texture and HalfFloat) clamping from "if"s to tenary operators as Google claims that this is the fastest solution
• Check for the Root in the ChunkHandler destructor as it might already be deleted when Ogre is already shutdown.
• Handing in the SimpleRenderable (Parent Class of Chunk) into the MeshBuilderCallback for convinient teamwork with physics libraries for example.

There are still situations, where the editing behaves strange, but they will be squashed soon.

[mmatt]

Great work PhilipLB, looking forward to seeing those in the main repository.

[PhilipLB]

Found the reason why the editing was strange after the fixes with the constants. Fixed and it behaves like before now.

[PhilipLB]

A big update of the documentation:

• The Startpage got some explanations of what is what
• All ChunkParameters are finally described!
• Added an example of volume terrain editing
• Corrected some outdated code snippets

[Klaim]

A big update of the documentation:

• The Startpage got some explanations of what is what

This page would benefit a lot from a short description of what features this component is supposed to provide.
I say that because I have no idea and I just got on this page and it start directly by a tutorial where we see code.


EDIT> After re-reading the first post, that I totally forgot I already read previously, I think a copy paste of it in the wiki would be good.

[PhilipLB]

That's true, I added some general information.

[PhilipLB]

Interesting, some people started to port this component to the JMonkeyEngine.
http://hub.jmonkeyengine.org/forum/topi ... -cubes-co/

Project: https://code.google.com/p/jmedualmarchingcubes/
Source: http://jmedualmarchingcubes.googlecode. ... alContour/

Quite a 1:1 port. Will be interesting, how this performs in Java.

[OgerLord]

Hi,

Interesting, some people started to port this component to the JMonkeyEngine

And I am one if them

One question about the interpolate method, in my code I changed

Code: Select all

 if (fabs(val0.w - ISO_LEVEL) <= FLT_EPSILON)
            {
                normal.x = val1.x;
                normal.y = val1.y;
                normal.z = val1.z;
                return v0;
            }

to

Code: Select all

if (fabs(val0.w - ISO_LEVEL) <= FLT_EPSILON)
            {
                normal.x = val0.x;
                normal.y = val0.y;
                normal.z = val0.z;
                return v0;
            }

and it fixes some normal bugs. Or is it my fault because it is a long way the normals get passed through serveral methods and I made a mistake?
(https://bitbucket.org/sinbad/ogre/src/ is always the newest Ogre code?)

Besides I cannot find this bug:

The rest of the mesh generation is working fine. It looks like an error in the dualgrid generation but I looked through the whole code twice searching for the mistake and I did not find anything.

[PhilipLB]

You are right with the fix. Also done here.
Yep, the repository always contains the latest code, in the 1.9 branch.

For your DualGrid-problem: It massivly helps there to have a debug visualization of the DualGrid and the Octree. You instantly see then, if something is wrong there.

[OgerLord]

Hm, the dual grid look okay: http://i.imagebanana.com/img/qvukpm3b/dual_grid.jpg
It seems like the problem occurs on the Border of very small and very large OctreeNodes. Maybe the Problem is the Octree Generation and that the Tree is not detailed enaugh at this positions?