Multithreading - OgreBullet Ogre Interaction

Hi.

I was thinking about running the ogre rendering loop and the bullet physics step loop in 2 separated threads. Due to more and more multi core cpus this sounds reasonable to me.
I already realized something similar with vrs3d and ode and it worked quiet well.
So i was looking for the point where ogrebullet syncs the positions and orientations back to ogre, so calls sceneNode->setPosition() or something like that.

The only thing i found is in DynamicsWorld::stepSimulation(const Ogre::Real elapsedTime):
static_cast <btSimpleDynamicsWorld *> (mWorld)->stepSimulation(elapsedTime);
But except some debug draw stuff, nothing else happens in this method. So i do not find the point where i should start synchronizing.

Any hints? Any ideas?

Thanks in advance, rTi

I'm thinking of the same thing exactly, and actually I also did it for OGRE ODE before! Even though Bullet itself can run on multiple threads (everything but the broadphase, I think) this must entail some extra work, and it is not really worthwhile on dual-core... well this is my feeling anyway.

Synchronisation is done in btMotionState's setWorldTransform method which should be overridden in OgreBullet somewhere, I suppose (haven't actually tried to implement this yet).

How were you thinking of implementing this? I would like to avoid locking a mutex for synch of every object (in the ODE version I just locked once per timestep), and I was thinking of putting incoming data (to Bullet) through a lock-free queue. Also, I'd like to avoid outgoing data from using a queue because the physics time-step would be smaller than graphics timestep... so I was thinking of using some sort of shared memory area containing positions of the active objects. Then setWorldTransform() would change this transform, instead of the Ogre object directly. Whenever Ogre draws frame, it can lock the are and copy all positions. Of course, this relies on the assumption that Bullet calls setWorldTransform() for all objects in sequence rather that whenever they are ready, which I haven't checked yet... tonight, maybe.

Actually, ideally I'd like to have the user select whether to use queues or shared memory for synching, so I could benchmark which one is faster in different scenarios... maybe through template parameters, for efficiency.

Hope any of this makes sense... well, the btMotionState part at least. About the rest, I am not concentrating on this for the moment, but maybe later I could help?

The overridden setWorldTransform, and getWorldTransform are in OgreBulletCollisionsObjectState.h/cpp

It's a reall slow day at work....
setWorldTransform on each body is called at line 149 of btDiscreteDynamicsWorld.cpp or line 195 of btSimpleDynamicsWorld.cpp and, yes it is done after all simulation has been carried out... so it should be equally easy to add a threaded interface to Bullet as to ODE.

Thanks for your input.
I will have a deeper look into it soon.

Ok. I was thinking of a quiet naive implementation:
* let ogre render, let bullet step the world
* lock ogre while Root::renderOneFrame()
* as soon as bullet has finished his world step it has to wait for ogre
* as soon as ogre finished it's rendering, let bullet sync back to ogre

This has the downside, that bullet always has to wait for ogre, but multiple physics steps between to frames does not make sense (in most cases) anyway.

May basic goal is to have the physics world step and the graphics rendering in parallel for one time tick / frame, then sync both, and then start from the beginning. That should give the most performance boost. Or am i wrong here?

I did not know bullet is multithreaded in itself. So i somehow wonder whether it makes really sense to run bullet updates in an own thread. Do you have any documentation about the bullet threading? I did not find anything on the first view. My thread viewer did not show any extra threads in my demo application. So maybe bullet threading has to get activated somehow?

greetings, rTi

Finally i found something. But it is the only thing i found:
http://www.continuousphysics.com/Bullet ... 8e12df21d9

It's said, that they plan to bring multithreading back from PS3 version to open source version. So it does not really sound like there is multithreading available for us.

Somehow strange...

greetings, rTi

I read a bit into btDiscreteDynamicsWorld::stepSimulation().

As i see it, bullet can do multiple simulation sub steps in one stepSimulation() call. This is a nice feature, but the problem is, if several sub steps are preformed, the transformations are always synced back. So in the ogrebullet case, the transformation of an object may be updated several times in one stepSimulation() call. This is very bad for my approach, because this would lead to ogrebullet waiting for ogre as soon as the first sub step it done.

Ogrebullet only hands the timeStep value to bullet, so setSimulation is called with maxSubSteps == 1. So the multiple updating should not happen. But at the buttom of btDiscreteDynamicsWorld::stepSimulation() they run through all the substeps (so 1 in our case) and sync back the transformation. After this is done, the very last call before return syncs back transformations again... This is somehow strange. I do not see the point here. On one hand, it is unnecessary updating, and on the other hand the multi update problem mentioned above would happen.

So slowly i get the point of your queue approach. If i got it right, you want to replace the setWorldTransform() implementation of ogrebullet and save the update first in a queue. After stepSimulation() returns, you want to iterate through the queue and update all the objects. Am i right here?
This is the bullet -> ogre direction. Where do you need the bullet -> ogre synchronization direction you talked about?

Am i right with the unnecessary updating? Is this a bullet bug? A simple else there would solve this. (...some time passed by...) Hmm, the more i read the code the more i get the feeling the last call to synchronizeMotionStates() is completely useless. Please correct me.

greetings, rTi

Here some prove of concept implementation of a "transformation cache".

The normal world step is still done by:
DynamicsWorld::stepSimulation(Ogre::Real)
But to apply the changes to the ogre scene an additional call is needed.
DynamicsWorld::synchronizeToOgre()
This enables parallel updating of physics and graphics, and locking while synchronizing. This is independent from bullets sub stepping.
Index: dep/OgreBullet/Dynamics/include/OgreBulletDynamicsWorld.h
===================================================================
--- dep/OgreBullet/Dynamics/include/OgreBulletDynamicsWorld.h (revision 16)
+++ dep/OgreBullet/Dynamics/include/OgreBulletDynamicsWorld.h (revision 17)
@@ -48,6 +48,8 @@

void stepSimulation(const Ogre::Real elapsedTime);

+ void synchronizeToOgre();
+
void addRigidBody (RigidBody *rb);

void setDebugDrawer(OgreBulletCollisions::DebugDrawer *debugdrawer)
Index: dep/OgreBullet/Dynamics/src/OgreBulletDynamicsWorld.cpp
===================================================================
--- dep/OgreBullet/Dynamics/src/OgreBulletDynamicsWorld.cpp (revision 16)
+++ dep/OgreBullet/Dynamics/src/OgreBulletDynamicsWorld.cpp (revision 17)
@@ -27,6 +27,7 @@
#include "OgreBulletDynamics.h"

#include "OgreBulletCollisionsShape.h"
+#include "OgreBulletCollisionsObjectState.h"

#include "OgreBulletDynamicsWorld.h"
#include "OgreBulletDynamicsObjectState.h"
@@ -76,6 +77,7 @@
if (mDebugDrawer)
mDebugDrawer->clear ();

+ // step the world
static_cast <btSimpleDynamicsWorld *> (mWorld)->stepSimulation(elapsedTime);

if (mDebugDrawer)
@@ -99,6 +101,19 @@
}
}
// -------------------------------------------------------------------------
+ void DynamicsWorld::synchronizeToOgre()
+ {
+ // sync all object back to ogre
+ std::map<Object*, btTransform>::const_iterator it =
+ OgreBulletCollisions::ObjectState::transformationCache().begin();
+
+ while(it != OgreBulletCollisions::ObjectState::transformationCache().end())
+ {
+ (*it).first->setTransform((*it).second);
+ it++;
+ }
+ }
+ // -------------------------------------------------------------------------
void DynamicsWorld::removeConstraint(TypedConstraint *constraint)
{
getBulletDynamicsWorld()->removeConstraint(constraint->getBulletTypedConstraint());
Index: dep/OgreBullet/Collisions/include/OgreBulletCollisionsObjectState.h
===================================================================
--- dep/OgreBullet/Collisions/include/OgreBulletCollisionsObjectState.h (revision 16)
+++ dep/OgreBullet/Collisions/include/OgreBulletCollisionsObjectState.h (revision 17)
@@ -33,7 +33,7 @@
namespace OgreBulletCollisions
{
class ObjectState : public btMotionState
- {
+ {
public:
ObjectState(Object *parent);
~ObjectState();
@@ -41,9 +41,15 @@
virtual void getWorldTransform(btTransform& worldTrans ) const;
virtual void setWorldTransform(const btTransform& worldTrans);

+ static const std::map<Object*, btTransform>& transformationCache()
+ {
+ return m_transformationCache;
+ }
+
private:
Object *mObject;
btTransform mWorldTrans;
+ static std::map<Object*, btTransform> m_transformationCache;
};
}
#endif //_OGREBULLETCOLLISIONS_ObjectState_H
Index: dep/OgreBullet/Collisions/src/OgreBulletCollisionsObjectState.cpp
===================================================================
--- dep/OgreBullet/Collisions/src/OgreBulletCollisionsObjectState.cpp (revision 16)
+++ dep/OgreBullet/Collisions/src/OgreBulletCollisionsObjectState.cpp (revision 17)
@@ -34,14 +34,19 @@

namespace OgreBulletCollisions
{
+ std::map<Object*, btTransform> ObjectState::m_transformationCache;
+
// -------------------------------------------------------------------------
ObjectState::ObjectState(Object *parent):
mObject(parent)
{
+ // create an empty entry in the objects transformation cache
+ m_transformationCache[mObject] = btTransform();
}
// -------------------------------------------------------------------------
ObjectState::~ObjectState()
{
+ m_transformationCache.erase(mObject);
}
// -------------------------------------------------------------------------
void ObjectState::getWorldTransform(btTransform& worldTrans) const
@@ -57,7 +62,10 @@
{
assert (mObject);

- mObject->setTransform (worldTrans);
+ // use transformation cache instead of syncing directly
+ // mObject->setTransform(worldTrans);
+ m_transformationCache[mObject] = worldTrans;
+
mWorldTrans = worldTrans;
}
}

Btw. Sinbad thinks having threaded physics does not make really sense...
http://www.ogre3d.org/wiki/index.php/Threading


greetings, rTi

Your solution is very similar to what I had in mind, but I had not thought of using a std::map for syncing. It makes the task really simple and elegant! Isn't it a bit slow, though? Since you want to lock the transformationCache for as little time as possible to avoid the Ogre thread from waiting, I mean. Of course it is only in the getWorldTransform that you use operator[], in synchronizeToOgre you just iterate over the whole map... so maybe as a total the locking time is negligible?

There is still the issue of other type of communication between Ogre/main thread and Bullet. This is where I think a message queue should be used. If you want to add an object to Bullet, you first create it and then send a message with the pointers (or maybe you send info on the object and let Bullet create it, whatever works best...). Also, the input system needs to be able to send messages to Bullet, to allow user interaction. There's a library for threading called LDK which I am planning to use and which seems efficient and kindof easy to use. Among other things it has an implementation of a lockless queue. (http://ldk.sourceforge.net/)

About multithreading in bullet itself - there is both a demo and a section in the Bullet manual. So it is very much reality... but as I said, it does not apply to the broadphase collision detection, and there should be some overhead (potentially colliding pairs are divided up in chunks and distributed among the threads). Also, it's windows only.

and, btw, threaded physics does make sense. It is being used today by high-profile studios. For example the Ageia SDK is multithreaded, running either on a Physics Processing Unit (the one they are trying to sell) or on a second processor.

My thread viewer did not show any extra threads in my demo application. So maybe bullet threading has to get activated somehow?

Yes, it does. Apparently it is activated by using a special dispatcher (SpuGatheringCollisionDispatcher). It's covered int the manual.

This has the downside, that bullet always has to wait for ogre, but multiple physics steps between to frames does not make sense (in most cases) anyway.
But it does make sense! First, in order to have a stable simulation you need a constant time step (which is not desirable for graphics, usually). Second, the time step should be something like 1/60 s, which is too much for graphics, where the desirable frame rate is something like 30fps, right?

Anyhow, the right way to do it is like you started: to keep both threads running in parallel, with the only interaction being to poll a shared resource every step.

I made some further tests and the locking really seems not to slow down anything. I created a separated lock just for the transformation cache. Then i was able to run physics and graphics updates in a ratio of 1:10 and also 10:1. So it runs smoothly imho.

I used the std::map because of bullets multiple update steps. So if the transformation of a certain object gets updated several times (and it will as soon as bullet runs substeps), the transformation will be overwritten in the map and synced back to ogre only once after all the substeps.

I think the map access via operator[] is ok since it is in O(log(n)) or at least O(n*log(n)). Should be ok i think.

The physics command queue is indeed still needed for the ogre -> bullet way of communication. But I see a little problem here. What if you want to read values from bullet, like getVelocity()? Such a queue does not support any kind of value return.
So i just came up with the idea of an "interaction thread". The input handler (which lives in the ogre thread and reads input after each frame) just passes commands into a command queue of the interaction thread. This one loops forever, and as soon as there are commands in the queue available (like MOUSE_MOVE(x,y) or KEY_RELEASE(code)) it could lock graphics or physics and do things. since it is a separated thread it should be possible to implement it without frightening dead locks, and since you can completely lock for example the physics, you can call functions as usual and also retrieve return values.

I am not sure wether i am going to use ldk, i somehow get too much dependencies for my taste. i think a queue based on std::list with a mutex for locking while adding and removing elements should do the thing too.

Multithreading in bullet: i am sitting on MacOS X, so it is indeed not available for me atm.

Which bullet manual are you refering to? I was looking in Bullet_User_Manual.pdf from the source distribution and i still do not find anything about threading...

About multiple physics substeps: i think this should be handled by bullet itself since it does already support it. I think OgreBullet just needs some more tweaking in stepSimulation().

greetings, rTi

Even better and simpler.

Since i have a separated lock for the transformationCache now, my physics update loop now looks like this: lockPhysics();
PhysicsManager::get()->update(updateTime);
unlockPhysics();

lockGraphics();
PhysicsManager::get()->synchronize();
unlockGraphics();

This means physics and graphics are never locked at the same time. So if i detect a key press in the graphics thread, i can just lock the physics thread, apply forces and stuff, unlock physics again and everything is fine.

This makes the graphics always wait for physics completion before applying forces etc. This may be a downside, but atm this locking seems not to cause any slowdown.
So the physics command queue is not needed at all atm.

greetings, rTi