[Guide] Compile Mogre with thread support

I want to show you, how to enable thread support for (M)Ogre.
There are three avaiable options: None, background resource loading and full thread support.
I will focus on option number two: background resource loading.

The "thread provider" will be Poco (alternatives are Boost and TPM). I chooose Poco just because Boost caused some trouble with Mogre, nothing more.

But first some remarks:

1. This is an advanced tutorial, not a really step-by-step bullet proof tutorial[/*]
   
2. Although I managed to compile (M)Ogre with Poco successfuly and also set up an test project, this guide is some how "theoretical". The first reason is: Ogre is not desinged to be thread safe, because thread safe does not means "faster" but "more difficult to maintain" (and there is an additional overhead). Thread safety is accomplished by using mutexes. But mutexes are tricky...[/*]
   
3. Ogre 1.7.* (and higher) have a "bug", the second reason why this guide is "theoretical". There is a race condition and possible dead lock caused by the mutexes in both the resource manager and D3D. As a result, as soon as you start your project, ogre freezes because it locked itself (and not only the background thread): both main thread and background thread are wainting for each other[/*][/list:u]
   
   Let's start:
   
   1. Download poco from here, extract it somewhere (e.g. C:\) and compile it.[/*]
      
   2. We have to compile both mogre and ogre. For that we will use MogreBuilder. Compile it, too, and make sure that you can compile mogre without any modifications[/*]
      
   3. Start mogre builder and wait until it compiled the dependencies. Kill it now (completely).[/*]
      
   4. Open cmake-gui and navigate to the ogre source (e.g. C:\...\Main\OgreSrc\ogre\). The build directory will be "C:\...\Main\OgreSrc\build\" (you may need to create the directory).[/*]
      
   5. Press "configure" and fill in the fields for "Poco": POCO_INCLUDE_DIR = [The include dir of poco, e.g. "C:/poco-1.4.4/Foundation/include"] and POCO_LIBRARY_DBG = [the PocoFoundationd.lib, e.g. "C:/poco-1.4.4/lib/PocoFoundationd.lib"][/*]
      
   6. Press "Generate" (if there are any erros, try to solve them )[/*]
      
   7. Now download the attached "attributes.xml"Copy patch and apply it and replace it with the original in "C:\...\Codegen\AutoWrap\". This modified file includes some fixes which include the class "ResourceBackgroundQueue" into the auto wrapper process and "restores" the workqueue id.[/*]
      [/list:u]
      diff -r c2be4e87dc8b Codegen/AutoWrap/Attributes.xml
--- a/Codegen/AutoWrap/Attributes.xml Wed Apr 18 02:28:03 2012 +0200
+++ b/Codegen/AutoWrap/Attributes.xml Tue Sep 11 11:09:44 2012 +0200
@@ -215,7 +215,7 @@
<class name="ControllerFunctionRealPtr" Ignore=""/>
<class name="ManualResourceLoader" WrapType="Interface"/>
<class name="MaterialSerializer" WrapType="PlainWrapper"/>
- <class name="ResourceBackgroundQueue" Ignore=""/>
+ <!-- <class name="ResourceBackgroundQueue" Ignore=""/> -->

<!-- to avoid name clash with System.Exception -->
<class name="Exception" WrapType="PlainWrapper" ReadOnly="" Rename="OgreException">
@@ -302,7 +302,15 @@
<class name="Listener" WrapType="NativeDirector"/>
</class>
<class name="ResourceBackgroundQueue">
- <class name="Listener" WrapType="Interface"/>
+ <_CustomIncDeclaration>
+ public: typedef unsigned long BackgroundProcessTicket&amp;
+ </_CustomIncDeclaration>
+ <class name="Listener" WrapType="Interface">
+ </class>
+ <function name="canHandleRequest" Ignore=""/>
+ <function name="handleRequest" Ignore=""/>
+ <function name="canHandleResponse" Ignore=""/>
+ <function name="handleResponse" Ignore=""/>
</class>
<class name="SceneManager">
<class name="SkyBoxGenParameters" WrapType="ReadOnlyStruct"/>
@@ -898,8 +906,8 @@
<_CustomIncPreDeclaration>
public: typedef Ogre::WorkQueue::RequestID RequestID;
</_CustomIncPreDeclaration>
- <typedef name="RequestID" Ignore="" />
- <function name="addRequest" Ignore="" />
+ <typedef name="RequestID" ReplaceBy="unsigned long" />
+ <!-- <function name="addRequest" Ignore="" /> -->
<class name="Request" WrapType="PlainWrapper">
<function name="Request" Ignore="" />
</class>

      Sorry, the board attachment quota has been reached. Could the admin please fix this?
      
      
      1. Open "C:\...\Main\OgreSrc\ogre\OgreMain\include\OgreConfig.h" and change
         #define OGRE_THREAD_SUPPORT 0 to #define OGRE_THREAD_SUPPORT 1
         and #define OGRE_THREAD_PROVIDER 0 to #define OGRE_THREAD_PROVIDER 2Save the file.[/*]
         
      2. Fire up MogreBuilder again and let it finish (if the AutoWrapper step is skipped, do it manually by starting "C:\...\Codegen\AutoWrap\bin\Debug\AutoWrap.exe" and pressing "Produce".[/*][/list:u]
         
         And that's it (do not forget to copy "PocoFoundationd.dll" into your app directory). Ogre now supports background resource loading (which is not working properly ).
         
         Edit: Renamed topic to match content. This is not a question but a guide.

I have wondered if a CLR Threading provider could be added to ogre.

I have tended to make stuff single threaded anyway for the front end stuff but I always multi-thread .net related code if it's applicable.

Actually since you are talking resource loading could you use .NET to load resources instead I have never tried it but that way you can multi thread safely.

I have wondered if a CLR Threading provider could be added to ogre. [...]
Actually since you are talking resource loading could you use .NET to load resources instead I have never tried it but that way you can multi thread safely.
You are right, it was just a experiment whether it is possible to use (m)ogre with tread support.
There are some topics somewhere in this forum regarding thread support with Boost, which all have no "happy end".

I have actually thought about doing this in managed code instead letting ogre do it for me. Integrating a CLR thread provider seems not to be possible (ogre has nothing to do with .Net anyway).
The problem is: Getting the informations from managed world to ogre.
E.g. I load all my textures using .net but ogre still does not know them. I have to "inject" the ready-to-use data into ogre somehow. And material/mesh/skeleton parsing still has to be done with ogre.
This seems to be the "best" solution, because doing it this way there are no hidden "native traps" (native mutexes ). The question is just, how could we do it?

Yeah this is back to the resource manager not been replaceable due to the way things load like from materials, I forgot about this read a thread on main forum

Putting in a CLR thread provider should be possible though you just have to have a managed delegate that calls the native ogre delegate. But OGRE it self doesn't need to know about the CLR then again this comes down to MOGRE not been statically linked there was a reason for this and I can't remember for the life of me

Only scanned this briefly it goes about loading stuff dynamically

http://misterblue.com/programming/dynam ... -resources

Thanks, really interesting.

Great post, going to test it

I was wondering if this deadlock "bug" you describe definitely ALWAYS happens when using background resource loading, or just in some rare cases?
Basically your guide sounds pretty much like the results are not usable at all?

I also found lots of Ogre threads about deadlocks in background resource loading but most of them seem to be linked to special conditions when they appear...

What I'd like to accomplish isn't really a "background loading", it's more like a multi-threaded resource loading at startup.
I have to load quite a lot resources at startup. Right now it's single-threaded with loadingbar and takes like a minute.
I could get along with the loadingbar (no other actions performed while loading), but if I can speed it up by x2 or x4 depending on cpu cores/threads it would be awesome...

I was wondering if this deadlock "bug" you describe definitely ALWAYS happens when using background resource loading, or just in some rare cases? [...]
I also found lots of Ogre threads about deadlocks in background resource loading but most of them seem to be linked to special conditions when they appear...
It happens as soon as you try to access a resource (I am not sure if it is a specific type ...). Somebody in the main ogre forum tracked that down.
For me, the dead lock happend randomly with a minimal example scene.


What I'd like to accomplish isn't really a "background loading", it's more like a multi-threaded resource loading at startup.
I have to load quite a lot resources at startup. Right now it's single-threaded with loadingbar and takes like a minute.
I could get along with the loadingbar (no other actions performed while loading), but if I can speed it up by x2 or x4 depending on cpu cores/threads it would be awesome...
In the comments above, Zonder posted a very good link for "dynamic" loading. Maybe you could "adapt" it ...
By the way, the biggest bottle-neck are shaders and materials. You could cache them using buildin methods (ogre >=1.8, not wrapped by mogre) or the script plugin (reduced functionality)

I want to show you, how to enable thread support for (M)Ogre.
Thank you for publishing your useful guide.
I think it would be good, when we add it to the wiki, because there people find it much more quickly - especially in the future, when this topic get hidden in the depth of the forum.
On the wiki page we can add a link to this forum topic for questions and feedback.

Sorry, the board attachment quota has been reached.
Yes, this disturbs. I can't change the quota.
You can write a message to jacmoe. (But not here, send the message inside of the main forum!)
Either he has administration rights or at least he should know who's responsible.
(Yes, there is a list of forum administrators, but all of them weren't logged-in for years. Sinbad is the only one, but he's mostly retired and not responsible anymore.)

I think it would be good, when we add it to the wiki, because there people find it much more quickly - especially in the future, when this topic get hidden in the depth of the forum.
On the wiki page we can add a link to this forum topic for questions and feedback.
I will do it, as soon as I have some free time. Done, I added it as a child page on "Building Mogre from source".
And I sent an pm to jacmoe, stay tuned for the answer.