How to build geometry in a thread?

Hi.

I'm currently experimenting a little with pointclouds and billboardsets. Some of the calculations are taking a while longer (building billboardsets is one of them),
so i tried to separate them in a different thread. Unfortunatly i'm stuck with some nasty errors and have no clue, if it is a limitation regarding (M)Ogre or i'm doing
something fundamentally wrong. If i remember correctly i'm using the Version 1.7.1 (Cthugha) from the .Net 4.0 SDK i think.
The basic render-loop looks like this and runs fine if i create the geometry in the main thread.


while (KeepRendering) {
Application.DoEvents();
if (!PauseRendering) {
Engine.MogreRoot.RenderOneFrame();
Engine.MogreRoot._updateAllRenderTargets();
WindowEventUtilities.MessagePump();
}
}


In my FrameEnded-Handler i'm calling something like this:

public static bool FrameEnded(FrameEvent evt) {
//Move camera around and do some other stuff ...
Engine.GeometryManager.HandleCameraMovement(CameraNode.Position);
}


Followed by the creation of a backgroundworker thread to construct some stuff.

public void HandleCameraMovement(Vector3 cameraPosition) {
if (!Worker.IsBusy) {
BillBoardCloudConfig conf = new M3DBillBoardCloudConfig();
....
//Start a background-workerthread to calculate and create billboards
Worker.RunWorkerAsync(conf);
}
}

//This is running in the background-thread. It is the function invoked by Worker.RunWorkerAsync(conf);
public static void CreateDynamicGeometry(object sender, DoWorkEventArgs e) {
BillboardSet bbs = BillBoardCloud.CreateBillBoardCloud((e.Argument as M3DBillBoardCloudConfig));
e.Result = bbs;
}

//If the thread finishes building an updated billboardset, a callback function is processed.
public static void Worker_Finished(object sender, RunWorkerCompletedEventArgs e) {
engine.PauseRendering(); //This fires an event, that sets the PauseRendering flag in my main loop to true

//Release old billboardset.
if (billBoardCloud.NumAttachedObjects() > 0) {
foreach (var ob in billBoardCloud.GetAttachedObjectIterator()) {
sceneMgr.DestroyBillboardSet((BillboardSet)ob);
}
}

if ((e.Result != null) && ((e.Result as BillboardSet).NumBillboards > 0)) {
bbs = (BillboardSet)e.Result;
//Attach new billboardset
billBoardCloud.AttachObject(bbs);
}
engine.ContinueRendering(); //This fires an event, that sets the PauseRendering flag in my main loop to false. Continues with renderoneframe() ...
}



If i comment out the line "billBoardCloud.AttachObject(bbs);", there is not much to see of course, but it doesn't crash. As soon as i try to attach the object, the application crashes with various
errors ranging from "AccessViolation" to strange "Debugger can not unwind this frame. Attempt to unwind the callstack failed" to "The runtime has encountered a fatal error. The address of the error was at 0x675ef852, on thread 0xe40. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack." and several different Ogre-Exceptions.

I suppose this is because of the main ogre thread and the worker thread accessing the same objects at the same time. Or trying to access objects i already freed withing the other thread. Strangely enough it seems ok to call stuff like "sceneMgr.DestroyBillboardSet((BillboardSet)ob);" without any errors, but not to add scenenodes. But how may i create scenenodes, billboards or other objects from a different thread if not like this? Is it possible at all, with the current Mogre-Version? Might it work to create objects in a separate thread with SceneManager.Create...() but add them to the scenegraph and dispose them within the main ogre thread? Or is it just that i have to get a Mogre version that is compiled with multithread-support? Does 1.7.1 already have this?

If anyone has experience with stuff like this, please give me a hint in the right direction. Thanks!

I'm not too familiar with how thread safe Mogre is but I'm assuming that as long as only 1 thread is accessing it at any given time, you should be ok. (I do believe you can compile Ogre with multi-threading support but I believe that's only for resource loading, etc. Any threading outside of that, you need to manage yourself)

I believe what's happening in your case is that you fire the event which does indeed set PauseRendering to true, however, the main rendering thread is still in the middle of rendering a frame. When you come back from changing PauseRendering and start adding the billboards, the rendering is still happening and all hell breaks loose.

You should be able to add a simple synchronization point somewhere in there as a test using something as simple as a lock to make sure that your rendering thread has exited the rendering and is now sitting and waiting.

Something similar to this:

void RenderingThread()
{
lock( myLock )
{
if (!PauseRendering)
{

Engine.MogreRoot.RenderOneFrame();
Engine.MogreRoot._updateAllRenderTargets();
WindowEventUtilities.MessagePump();
}
}
}



and then inside of the event that changes PauseRendering, you use the same lock


void SetPauseRendering( bool _shouldPause )
{
lock( myLock )
{
PauseRendering = _shouldPause;
}
}


It's not a shining example of threading but it should at least tell you whether or not it'll work from a different thread. This will block any thread that attempts to change the pause state until the current frame is done. After this, if you told it to pause, you are free to modify Ogre values (theoretically).

Hope that helps as a starting point.
Shawn

Hi and thanks for the answer!

I tried to use static lockobjects on the renderloop to give the threads exclusive access, but that also didn't work.
It seems that AttachObject causes most of the problems. Maybe one has to lock access to the Mogre.Root-Object or the
sceneManager or something like that. I'll have to experiment a bit more, when i have more time, to check if it is only the Scenegraph
that can not be modified by another thread. That would allow to create stuff somewhere else, and let it be added to the scene later
by the main thread. Hopefully something like that works.

A little update on what i found out in the meantime:

It seems that Mogre generally is not threadsafe, since Ogre itself isn't. I read somewhere that Ogre has some #defines in the source that allows
it to compile with a flag "OGRE_THREAD_SUPPORT" That can have values 0 (Default), 1 or 2 (and by today maybe a 3) for different kinds of locking.
If i understand that process correctly, this compile results in a different Ogre.dll that, contains a linked boost or other libraries (like tbb? or something)
used for threadsafe resource locking.

Depending on the value of the flag ogre supports background loading and some other stuff, but i'm still not sure that it allows any kind of threaded
access to things like hardwarebuffers. I guess manual object and billboardsets use them, but not sure of that.

I also read, that they are working on an message-queue like system, that allows threaded access to the rendersystem itself. But i have no idea if
that is available/finished in 1.7.1. I think this aims at having 1 renderthread, but multips threads accessing geometry/texturedata. Generally there
seems to be a problem using threaded access with OpenGL. For Direct3d9 there was an update called Direct3d9Ex comming with Windows Vista,
that supports creation of a threadsafe renderdevice and Ogre seems to have support of some kind for this.

Currently i'm creating ManualObjects and Billboardsets in a different thread. Add them to a synchronzied Queue and attach them to scenenodes
within the main thread. Most errors are comming from "Billboardset.BeginBillboards" which should lock the used resources, but doesn't seem to do so.
After running it for some minutes i always end up with something like:
23:20:47: OGRE EXCEPTION(3:RenderingAPIException): Cannot lock D3D9 vertex buffer: Access to this surface is being refused because no driver exists which
can supply a pointer to the surface. This is most likely to happen when attempting to lock the primary surface when no DCI provider is present.
Will also happen on attempts to lock an optimized surface.
in D3D9HardwareVertexBuffer::updateBuffer at ..\..\..\ogre\RenderSystems\Direct3D9\src\OgreD3D9HardwareVertexBuffer.cpp (line 323)


Does anyone know, what value that OGRE_THREAD_SUPPORT-flag has in the Ogre.dll delivered with the 1.7.1 Mogre binaries? And how much work it would
be to compile and provide it with different settings? In the meantime i'll try to find out something more abount that workqueue-system in the ogre-forum.

This is a topic seems to discuss general Ogre handling (independent of the Mogre wrapper).
So if you open a topic in the Ogre main forum, there is a good chance that more well-known users can reply. (Most Ogre users don't read the Mogre forum.)
If you do so, please give us the link to the second topic.

Good luck

I did yesterday morning, but didn't get helpfull answers until now.

Here is the link:
http://www.ogre3d.org/forums/viewtopic.php?f=2&t=63101

And if i understood correctly, it may be a problem of both, ogre and mogre. Ogre has a #define called OGRE_THREAD_SUPPORT,
that has to be set before compiling ogre and mogre on top of it. It seems Gantz has tried that a while ago here, but had some problems:
http://www.ogre3d.org/addonforums/viewtopic.php?f=8&t=7826&start=0

So threading is possible with 1.7.1, if that OGRE_THREAD_SUPPORT is set to 1 and i don't know what value it has, in the ogre.dll that is used
to compile the mogre binaries provided by mystoke and the other guys. Maybe i should post in the feature wish-list to compile a threading
capable mogre-version?

Mogre as always use OGRE_THREAD_SUPPORT = 0. I don't know if the version of mystoke use a different settings, but i suppose it's not the case since it add a boost dependencies to Mogre. That said, i know some user have been able to compile mogre with OGRE_THREAD_SUPPORT enabled, but i haven't been able to check it myself. You have to know that there is 3 different values for thread support :

0 disabled
1 full multi threading support (that also make direct x thread safe, since it use the D3DCREATE_MULTITHREADED flag when creating the device, note that using this setting is not without consequence, check the direct x documentation for more informations)
2 multi threading only for resources loading

Also, keep in mind that enabling thread support come with a performance loss, so that's also the reason why mogre is usually compiled with thread support disabled. If you want to have multi threading enabled in mogre, you have to recompile it, following the instruction here : https://bitbucket.org/mogre/mogre/src/e6cef16544a1/BUILD. Or ask for a multi threading enabled version, since compiling mogre is quite complex.

I don't know if the version of mystoke use a different settings, but i suppose it's not the case since it add a boost dependencies to Mogre.

I found this boost related note in the wiki (here)
Lucky users of Visual Studio 2003 - 2010 can use the Boostpro Boost Installers.
Select your compiler and "Multithreaded" and "Multithreaded Debug".
Be sure to install Boost Date Time and Boost Thread. The default selection (all) makes sure it's installed.

I found a multithreading note for Ogre 1.8 in the Byatis Notes:

New threading model
* No general thread safety on majority of classes except 'join points' - which is WorkQueue and handlers
* Task-based, data encapsulation (see new Terrain in 1.7)
* Default to processing all GPU interaction in one thread, but allow multi-threaded later where supported (e.g. Dx11)
* Greater support for localised data ownership for tasks

Hi and thanks for the answers.

Mogre as always use OGRE_THREAD_SUPPORT = 0. I don't know if the version of mystoke use a different settings, but i suppose it's not the case since it add a boost dependencies to Mogre. That said, i know some user have been able to compile mogre with OGRE_THREAD_SUPPORT enabled, but i haven't been able to check it myself. You have to know that there is 3 different values for thread support :
0 disabled
1 full multi threading support (that also make direct x thread safe, since it use the D3DCREATE_MULTITHREADED flag when creating the device, note that using this setting is not without consequence, check the direct x documentation for more informations)
2 multi threading only for resources loading


Yes. I think i need the full threading stuff. I'm pretty sure, creating billboardsets or manual-objects requires access to hardwarebuffers or something in the underlying classes. And those have to be synchronized or locked by direct3d or ogre, at least i wouldn't know how to do that myself. I also found out, that there is another flag to set before compiling. The file OgreConfig.H contains the two settings "OGRE_THREAD_SUPPORT" and "OGRE_THREAD_PROVIDER". The provider seems to decide which library to use. I think "boost" would be favorable, although i have no experience using it, i've read alot of good things about there threading support.

#define OGRE_THREAD_PROVIDER 0

Provider for threading functionality, there are 4 options.

OGRE_THREAD_PROVIDER = 0 No support for threading. OGRE_THREAD_PROVIDER = 1 Boost libraries provide threading functionality. OGRE_THREAD_PROVIDER = 2 Poco libraries provide threading functionality. OGRE_THREAD_PROVIDER = 3 TBB library provides threading functionality.

Definition at line 159 of file OgreConfig.h.

http://www.ogre3d.org/docs/api/html/OgreConfig_8h.html#a654b381200aff99c37d46d6ea7683da1

Also, keep in mind that enabling thread support come with a performance loss, so that's also the reason why mogre is usually compiled with thread support disabled.

Meanwhile i'm aware of that. But as far as i know aquiring a lock is quite fast and getting a worker-thread is pretty fast when using a thread-pool. The major slowdown seems to be system designs, when threads are using the same resources to often and have to wait for each other most of the time. At gamedev.net people reported a performance loss of 1% to as much as 50%, depending on the design and number of threads/cores. But at least in my simple test application i can get twice the framerate if i create billboardsets in another thread ... until it crashes. I suppose i simply have to test it to know for sure.

If you want to have multi threading enabled in mogre, you have to recompile it, following the instruction here : https://bitbucket.org/mogre/mogre/src/e6cef16544a1/BUILD. Or ask for a multi threading enabled version, since compiling mogre is quite complex.

I'll try to get someone else compile that. I tried 2 times to compile Mogre last year and sucked at it. After wasting 2-3 days each time i've simply given up on that. Probably the descriptions are a lot better now, but if someone has already set up his system to compile it, it shouldn't be too much work. I'll put a proposal in the feature request thread.

I found a multithreading note for Ogre 1.8 in the Byatis Notes:

Yes ... that's why i'm so confused. In ogre forums and in 1.7 changelog you can read, there already is a task-queue in 1.7 used for the new terrain. But on the other hand they say it will be included and/or changed in 1.8 or with D3D11.

I'll try to get someone else compile that. I tried 2 times to compile Mogre last year and sucked at it. After wasting 2-3 days each time i've simply given up on that. Probably the descriptions are a lot better now, but if someone has already set up his system to compile it, it shouldn't be too much work
Good news. Look to this fresh topic:
Mogre.Builder - a tool for easy, automatic build of Mogre

A build script works well. Now amirabiri wrote a similar build application in C# for better usage.
It's not ready, but look forward and be happy

It may not be exactly what you want, but I'm pretty sure that some of the buffers can be written to in a multi-threaded manner, even without any special compile options, because the underlying data structures are inherently thread safe.

For instance, I do (paraphrased a little):

TexturePtr texture = TextureManager.Singleton.CreateManual(textureName, ResourceGroupManager.DEFAULT_RESOURCE_GROUP_NAME, TextureType.TEX_TYPE_2D,
TEXTURE_DIMENSION, TEXTURE_DIMENSION, 0, PixelFormat.PF_FLOAT32_RGBA, (int)TextureUsage.TU_DYNAMIC_WRITE_ONLY);

using(HardwarePixelBufferSharedPtr pixBuf = texture.GetBuffer())
{
unsafe
{
try
{
pixBuf.Lock(HardwareBuffer.LockOptions.HBL_DISCARD);
}
catch
{
return null;
}

PixelBox pixBox = pixBuf.CurrentLock;
byte* buffer = (byte*) pixBox.data.ToPointer();

//Important bit!
Parallel.For(0, 10000, i=>
{
buffer = values; //Fine as the underlying structure is just an array AFAIK
});

pixBuf.Unlock();
}
}



My code is currently pretty stable (so long as you don't press any buttons!), and I can happily run this overnight doing this every frame without it crashing, and inserting ~32,000 float[4] per frame.

In your case, if creating ManualObjects, you might be able to access the Vertex etc buffers more directly and in a multi-threaded way, or look at the ManualObject source code to see what they're doing.

I got an idea.

Use a second scene manager instance or even a second Mogre instance, which uses an other thread.
There you build your (complex?) 3D objects and the result you copy to the main thread.
I don't know if you can copy a 3D object (e.g. mesh) from one Mogre instance to an other one.
If not, there would be 2 ways:
Grabb all basic information (content of buffers, which contains vertices, etc.) and recreate a new 3D object from it in the main Mogre instance. I suppose it's only possible for geometry content, and not easy/possible for material information.
An other way: The background thread saves the generated meshes to files and a the main thread loads them. It's not so fast, but maybe it's useful. Maybe you can write the mesh files to a MemoryStream? So it should be more quickly.

I don't know if my ideas are fine, but I wanted to tell them.

I'm not shure if it was in this topic ... maybe.
Someone said, there isn't much information available about vertex buffer programming (or similar topic).

Now if found a section about Hardware Buffer in the Ogre manual.
Look to section 5:
http://www.ogre3d.org/docs/manual/manual_toc.html