Augmented Reality: Integration of Ogre with ARToolkit (Plus)

[futnuh]

The ARToolkit http://www.hitl.washington.edu/artoolkit/ is a "library for building Augmented Reality (AR) applications. These are applications that involve the overlay of virtual imagery on the real world." Using camera-tracked marker cards as shown in the image below, a virtual character can be superimposed with correct apparent position and orientation into the original video image.



Searching the Ogre forums, several people have tried to use Ogre as the rendering engine for the ARToolkit - some successfully, some not. Here are a couple of relevent message threads:

- http://www.ogre3d.org/phpBB2/viewtopic.php?p=163290
- http://www.ogre3d.org/phpBB2/viewtopic.php?t=17725

I too am trying to get this working. I thought I'd write up my experiences as I went along, perhaps moving it over to a Wiki page if and when the remaining bugs are worked out. However, rather than use the ARToolkit, I'm instead using ARToolkitPlus (http://studierstube.icg.tu-graz.ac.at/handheld_ar/artoolkitplus.php ), an optimized version of the former that, in addition to providing a C++ interface, purportedly has a better pose estimator. The downside is that ARToolkitPlus has absolutely no support and doesn't include camera capture functionality. You need to have a separate means of obtaining the camera image data.

What I've done so far:

- installed ARToolkit (original) and ran the included demo applications.
- used the ARToolkit's calibration programs to obtain a camera calibration file specific to my Logitech QuickCam Orbit (ARToolkitPlus is able to read these calibration files).
- installed and built ARToolkitPlus
- ARToolkitPlus comes with a simple test application and sample image. I confirmed that the test image is "trackable".
- printed out my own marker (from the ARToolkitPlus), glued it to a piece of cardboard, captured a frame from my webcam and hacked the demo app to check that I could "track" this single image.
- To bring the webcam images into Ogre. I used Timme's plug-in, http://www.ogre3d.org/phpBB2/viewtopic.php?t=16572. It offers notification of a new frame's arrival via a listener class. It's in this listener class constructor that I instantiate the ARToolkitPlus, and the "new frame" method that I search for the marker card.
- Combined, I now have a simple Ogre application with the webcam showing up as a background (see http://www.ogre3d.org/wiki/index.php/Displaying_2D_Backgrounds) and a small test mesh being moved around based on calculations coming from ARToolkitPlus.

My problem is the lack of correspondance between these two images I.e., the camera/world according to Ogre and that being used by the ARToolkitPlus don't coincide. As a rough first go, I simply set the camera as:

Code: Select all

    virtual void createCamera(void)
    {
        mCamera = mSceneMgr->createCamera("SceneCamera");
        mCamera->setPosition(Vector3(0,0,0));
        mCamera->lookAt(Vector3(0,0,-5));   // back along -ve z axis
        mCamera->setNearClipDistance(0.01);

        // vertical opening angle of webcam measured roughly as 85cm at a distance of 160cm distant
        // let Ogre calculate horizontal FOV from viewport aspect ratio
        mCamera->setFOVy(Radian(0.5163)); 
    }


To check the positioning (ignore rotations for the moment), I created a small .mesh test cube with sides of 0.01m (1/10 the size of the external black border of my printed marker tile). To position this entity,

Code: Select all

        float* mv = (float*) (mTracker->getModelViewMatrix() );
        Ogre::Vector3 position = Ogre::Vector3(-*(mv+12), *(mv+13), -*(mv+14));
        std::cout << "Position: " << position << std::endl;
        mNode->setPosition(position);


As we move the marker tile around now, the cube should track its centre. What I find though is that the y coordinate tracks fine but that there is a discrepency in the horizontal position. Video (DivX, 2.5Mb) of the result can be found http://www.edmstudio.com/ogre/06G19-artoolkitplus-translation.avi.

Part of the mismatch is coming from ARToolkitPlus. The following image shows the marker tile held at the position that gives (x,y) coordinates of (0,0).



The calculated modelview matrix is

Code: Select all

Matched marker: 0 with confidence: 100
  -0.8475  0.0245  -0.5302  0.0000
  0.0070  0.9994  0.0349  0.0000
  0.5307  0.0259  -0.8472  0.0000
  0.0051  0.0057  1.6646  1.0000
Position: Vector3(-0.00508792, 0.00572599, -1.66464)


The red dot is where Ogre renders the test cube (0,0,-1.66). Note the horizontal difference between the middle of the marker tile and the image centre (white lines). I'm guessing that this is a camera calibration issue - though why (0,0) is offset from the image centre is strange.

With this tracking at least partly correct, let's consider rotations. Begin by replacing the tiny marker with a 0.1^3 cube sized to match the marker tile. For orientation, we follow haffax's advice (http://www.ogre3d.org/phpBB2/viewtopic.php?p=163290) and convert the ARToolkitPlus modelview matrix into an Ogre quaternion:

Code: Select all

        float* mv = (float*) (mTracker->getModelViewMatrix() );
        Ogre::Vector3 x = Ogre::Vector3(*(mv+0), *(mv+1), *(mv+2));
        Ogre::Vector3 y = Ogre::Vector3(*(mv+4), *(mv+5), *(mv+6));
        Ogre::Vector3 z = Ogre::Vector3(*(mv+8), *(mv+9), *(mv+10));

        Ogre::Quaternion orientation = Ogre::Quaternion(x.normalisedCopy(), y.normalisedCopy(), z.normalisedCopy());
        mNode->setOrientation(orientation);


The results are shown in this video (DivX, 2.5Mb) http://www.edmstudio.com/ogre/06G19-artoolkitplus-orientation.avi with an image still shown below.



Rotations are obviously being calculated but two of the directions are reversed. This is where I'm currently up to ...

[Falagard]

Not much advice from me other than just trying to play with the vectors to reverse them and see if it works ...

x.normalizedCopy() * -1.0f, etc.

Looks awesome though! I watched the vid a couple times in a row because it's so cool

[futnuh]

An addendum, rather than the naive camera approach taken above, I've been trying to use the projection matrix calculated by ARToolkitPlus

Code: Select all

float* mp = (float*) (mTracker->getProjectionMatrix());

    3.2125  0.0000  0.0000  0.0000
    0.0000  -3.7410  0.0000  0.0000
    0.0469  0.0125  1.0202  1.0000
    0.0000  0.0000  -0.2020  0.0000


Passing this to Ogre as follows

Code: Select all

    Ogre::Matrix4 projection = Ogre::Matrix4( *(mp+0), *(mp+4), *(mp+8), *(mp+12),
                                              *(mp+1), *(mp+5), *(mp+9), *(mp+13),
                                              *(mp+2), *(mp+6), *(mp+10), *(mp+14),
                                              *(mp+3), *(mp+7), *(mp+11), *(mp+15) );
    camera->setCustomProjectionMatrix(true, projection);


the mini-cube now tracks the marker tile in the horizontal direction, albeit with a constant offset from the marker centre. Video (DivX, 2Mb) can be seen here, http://www.edmstudio.com/ogre/06G19-artoolkitplus-projmatrix.avi

Of course, part of the reason for writing this up is the hope that others can chime in with suggestions and fixes for the problems I'm having ...

[futnuh]

Rotations are working now as you can see in this video (DivX, 2Mb), http://www.edmstudio.com/ogre/06G20-artoolkitplus-rotations.avi. I realized that while ARToolkit and Ogre both use right-handed coordinate systems, there was a 180 degree Y rotation needed to bring things into alignment.

Code: Select all

        Ogre::Vector3 xr = Ogre::Vector3(*(mv+0), *(mv+1), *(mv+2));
        Ogre::Vector3 yr = Ogre::Vector3(*(mv+4), *(mv+5), *(mv+6));
        Ogre::Vector3 zr = Ogre::Vector3(*(mv+8), *(mv+9), *(mv+10));
        Ogre::Vector3 x = xr.normalisedCopy();
        Ogre::Vector3 y = yr.normalisedCopy();
        Ogre::Vector3 z = zr.normalisedCopy();
        Ogre::Quaternion orientation = Ogre::Quaternion(x, y, z);
        Ogre::Quaternion y180 = Ogre::Quaternion(Ogre::Radian(Ogre::Degree(180.0f)), Ogre::Vector3::UNIT_Y);
        mNode->setOrientation(y180*orientation);


Question: is this possible via selected sign changes in the matrix elements? You can't just arbitrarily negate x (or y or z) as this changes the handedness of your orthonormal basis - and you get strange scaling effects as the object turns.

While the above video is promising, there is still a very noticeable difference between the base of the cube and the marker tile. I'm still struggling with this - I'm sure I need to be using the projection matrix calculated by ARToolkit (see above). It's not working for me though and I'm almost out of ideas ...

[Chris Jones]

nice

ive read various articles on augmented reality and think its great, such as soldiers using it for displaying data about things infront of them and playing pacman on the streets

anyway, hope you get all the problems sorted out, keep us updated on your progress!

what equipment do you need for it?

Chris

[Biomass]

I've done some AR work using ARToolkit as well, but this was before I started learning Ogre. I'd be interested in following your progress.

As far as your camera calibration error goes, are you using any of ARToolkit's camera calibration code? Webcams tend to have some pretty mean distortion, so manual measurements probably will not cut it.

[futnuh]

what equipment do you need for it?

Just a webcam and a printer for making the marker tiles.

As far as your camera calibration error goes, are you using any of ARToolkit's camera calibration code?

I did the left-hand step of the two-step calibration process, http://www.hitl.washington.edu/artoolkit/documentation/usercalibration.htm Perhaps step two would remedy my problem ...

Once I've got the bugs worked out and something other than a simple cube, I'll clean up the source and put it online along with a Windows distro for people to try.

[Sylpheed]

Hi futnuh!

First of all sorry for my English, it's not very good.

I have implemented ARToolkit and modified it for my project purpouse. But now I need a powerful engine, so I've decided to use Ogre. The problem is that I don't know how to "link" OpenCV with ARToolkit video capture (while OpenCV uses IPLIMAGE, ARToolkit uses ARuint8) and I don't know exactly where to call the method that recognize the markers. Please, could you explain a little more how did you solve this problems?

Thanks a lot, and sorry again for my poor english

[futnuh]

I have implemented ARToolkit and modified it for my project purpouse. But now I need a powerful engine, so I've decided to use Ogre. The problem is that I don't know how to "link" OpenCV with ARToolkit video capture (while OpenCV uses IPLIMAGE, ARToolkit uses ARuint8) and I don't know exactly where to call the method that recognize the markers. Please, could you explain a little more how did you solve this problems?

I'll try and bundle up the source of my simple demo app today or tomorrow.
Note that I am using ARToolkitPlus. If your IPLIMAGE is RGB or RGBA, you'll need to convert it to grayscale. You can then access the raw image data (cast to an unsigned char) as follows:

Code: Select all

CV::IplImage mCvGrayscaleImage = CV::cvCreateImage(CV::cvSize(320, 240), 8, 1);
mCvGrayscaleImage->origin = mCvCaptureImage->origin;
cvCvtColor(mCvCaptureImage, mCvGrayscaleImage, CV_RGB2GRAY);
rawdata = (unsigned char*)(mCvGrayscaleImage->imageData);


Call the marker detection code whenever you get a new image frame. Since the OpenCV camera capture routines are blocking calls, you probably want to run this code in a separate thread. The Webcam plugin does this for you. More soon ...

[Sylpheed]

I'll try and bundle up the source of my simple demo app today or tomorrow.
Note that I am using ARToolkitPlus. If your IPLIMAGE is RGB or RGBA, you'll need to convert it to grayscale. You can then access the raw image data (cast to an unsigned char) as follows:

Code: Select all

CV::IplImage mCvGrayscaleImage = CV::cvCreateImage(CV::cvSize(320, 240), 8, 1);
mCvGrayscaleImage->origin = mCvCaptureImage->origin;
cvCvtColor(mCvCaptureImage, mCvGrayscaleImage, CV_RGB2GRAY);
rawdata = (unsigned char*)(mCvGrayscaleImage->imageData);


Call the marker detection code whenever you get a new image frame. Since the OpenCV camera capture routines are blocking calls, you probably want to run this code in a separate thread. The Webcam plugin does this for you. More soon ...

Thanks a lot dude I appreciate your help so much, I'm at work and I get scared a little with Ogre and OpenCV (is the first time I code in a 3D engine).

I hope I'm not bothering you with this question:

so could I call ARTollkit's marker detection with that rawdata?

What I mean is:

is rawdata the same image structure than ARUint8?

Sorry, I'm a noob as a coder Thanks a lot for your answer.

[futnuh]

is rawdata the same image structure than ARUint8?

Without looking at the source, I'm guessing yes. And as they are both single byte, the worst that will happen will be an unrecognizable image.

Many years ago I phoned tech support for an unnamed volume visualization program. Their program was crashing when importing one of our 128^3 data sets of raw byte data (no header). The tech support guy tried to claim that I had a malformed dataset: he just couldn't understand that as long as the file size was correct, I definitely had a dataset of "something".

[futnuh]

For anyone following this thread, I have been struggling over the past few days trying to understand why the ARToolkitPlus' RPP pose estimator occasionally yields zero'd modelview matrices and eventual locks up requiring me to kill the process (ctrl-c in my cygwin terminal).

I've created a little demo app that exhibits the behaviour on two of my machines, but not on the toolkit author's machine overseas. If you are an XP user and can spare a few minutes, please take a look at http://www.ogre3d.org/phpBB2/viewtopic.php?p=166974#166974.

[Sylpheed]

Thanks a lot futnuh, you are great

When I arrive at home I will try the link you've posted and I will post the result.

Regards

[Sylpheed]

Hi all! I'm keeping the hard work, but I have some problems

- OpenCV shows the image horizontal mirrored, and I don't know why! I haven't modified the original source code from OpenCV.

- I have an error when I try to open my camera_parameter.dat. I have to use the cameraparams.dat from futnuh source code! How could I build my working camera_param.dat? I used the the 2 steps to calibrate my webcam (Logitech messenger webcam), should I use only one step?

- And now the most important thing, the position and rotation of the object.

float vector[16];
for (int i = 0; i < 16; i++){
vector[i] = mTracker->getModelViewMatrix()[i];
}
Ogre::Vector3 position = Ogre::Vector3(-vector[12], vector[13], -vector[14]);
Ogre::Vector3 xr = Ogre::Vector3((vector[0]), (vector[1]), (vector[2]));
Ogre::Vector3 yr = Ogre::Vector3((vector[4]), (vector[5]), (vector[6]));
Ogre::Vector3 zr = Ogre::Vector3((vector[8]), (vector[9]), (vector[10]));
Ogre::Vector3 x = xr.normalisedCopy();
Ogre::Vector3 y = yr.normalisedCopy();
Ogre::Vector3 z = zr.normalisedCopy();
Ogre::Quaternion orientation = Ogre::Quaternion(x, y, z);
Ogre::Quaternion y180 = Ogre::Quaternion(Ogre::Radian(Ogre::Degree(180.0f)), Ogre::Vector3::UNIT_Y);
myObject->setOrientation(y180*orientation);
myObject->setPosition(position);

When I move the marker, the object doesn't move. I set the orientation and position of the object with the function getModelViewMatrix, but it doesn't work, the object doesn't move, but it rotates at least. Have I to set the orientation and position of the camera instead of the object??

Thanks a lot of you, and sorry for my bad Englush

[futnuh]

OpenCV shows the image horizontal mirrored, and I don't know why! I haven't modified the original source code from OpenCV.

Is the ARToolkit detecting your marker and returning a non-zero modelview matrix? I.e., markers are detected but the image in Ogre is showing up flipped? In this case, use a "scale -1 1" in your material file to flip the texture on the screen. If the marker isn't detected, you'll need to flip the image using the cvFlip() routine. Alternatively, your camera driver might allow you to toggle horizontal flip.

I have an error when I try to open my camera_parameter.dat. I have to use the cameraparams.dat from futnuh source code! How could I build my working camera_param.dat? I used the the 2 steps to calibrate my webcam (Logitech messenger webcam), should I use only one step?

I am using "camera_calib2.exe" from the ARToolkit.

When I move the marker, the object doesn't move. I set the orientation and position of the object with the function getModelViewMatrix, but it doesn't work, the object doesn't move, but it rotates at least. Have I to set the orientation and position of the camera instead of the object??

I'm not sure if ARToolkit's getModelView matrix function is in OpenGL format or not. Print out the raw matrix and your position vector. Does the position make sense? You might need to choose different matrix elements instead. Finally, you set the camera or the object depending on the application. For a fixed camera and moving marker, it makes more sense to move the object.

[Sylpheed]

Hi friends!!

Thanks futnuh, you are awesome!

Things are going OK, I had some problems with the position, but it was because I forgot to change the size of the pattern.

Now I'm confused because the framerate is under 10 fps. It's due to the function "calc", if I quit that line the framerate going up to 40 fps. Anybody know why??

Thanks a lot, regards

[futnuh]

Thanks futnuh, you are awesome!

I think you need to renormalize your praise. For future reference, I am a happy but middling hacker. Sinbad is awesome. As was Gandhi.

Now I'm confused because the framerate is under 10 fps. It's due to the function "calc", if I quit that line the framerate going up to 40 fps. Anybody know why??

Well, the calc routine is where all the hard work is done. There is a fair amount of iterative solving going on underneath to come up with the final modelview matrix. That said, I've read people being able to increase performance by 100% by improving the camera calibration. Also, you'll probably want to move the 'calc' call into the same thread that is capturing your camera image. That way, when Ogre gets told about a new frame, the calculation has already been done.

[Sylpheed]

You're right futnuh, it is the camera calibration . Now it works perfectly, near 50 fps. The next step is translate the object one pattern to other at a certain speed (for example, 0.1 units per frame). I implemented all of this functions in ARTooKit, but I couldn't import 3D Studio Max animations in OpenGL, so I had to use a 3D Engine (everybody agrees Ogre is the best free 3D Engine). The coding is becoming difficult due to my ignorance in OpenCV and Ogre's API.

I will inform in next days Regards

[cybereality]

Wow, nice work futnuh. I've been following the development of the ARToolkit since it was released, though I haven't got around to playing with it yet. Its nice to see that some people, like yourself, are expanding the uses of the platform and pushing the envelope. Please keep updating this project, its looking really good.

// cybereality

[futnuh]

The next step is translate the object one pattern to other at a certain speed (for example, 0.1 units per frame). I implemented all of this functions in ARTooKit, ...

Instead of making it frame based, I'd suggest using relative time. Decide how long you want it to take for the object to jump from one marker to another. Then start interpolating the position and orientation (the latter using quaternion lerp or slerp - see the Ogre API). Every time a new webcam frame comes in, you'll have to adjust your target. In between these, Ogre will move your object towards the last known target position.

[Sylpheed]

That's a good idea, but what happens if the target gets moved? The object would move faster or slower if the target go further or nearer from its original position.

I implemented the next idea when I was coding in ARToolkit:

Source --------- |Object| ----------------------> Target

Source is the marker where the object starts (its initial position). When the camera aims to a target (a marker with a higher weight than the source's weight) the object starts moving. But I have to save a reference of the world's position of the object, because if the camera or the marker moves, the object have to keep the relative position. To obtain that, I use the distance from the object to the target. It's like this:

1 - The object is positioned on the source.
2 - In the next frame, the camera aims to a target. The object starts to move. We put away the distance between Object and Target (for example, 15 units)
3 - In the next frame, the camera still aims to the target, so we have to move the object. But before that, we have to re-positionate the object, because the camera could be moved from its original position. So all we have to do is get the actual position of the target and then we force the object to keep the same distance with the target in the previous frame(in the example, the 15 units). Once the object is positioned, we can move it.

For the speed of the object I use a constant and the distance from the object to the target of the three axes.

Total distance (magnitude) = sqrt ( (dist_x ^ 2) + (dist_y ^ 2) + (dist_z ^ 2) )

Units to move in axe x = (dist_x * speed) / Total Distance
Units to move in axe y = (dist_y * speed) / Total Distance
Units to move in axe z = (dist_z * speed) / Total Distance

And that's all!

[Timme]

Hi futnuh,

nice to see that you got your project working . Good work!
May I ask you, for what will you use the ARToolkit in future? Will it replace the gyroscopic mouse and you will be able to manipulate the "sliceable head" with your hands?

@Sylpheed: Are you using the OpenGL Subsystem for rendering? I have an ATI graphic card (Radeon 9600XT) and the texture with the web cam image is corrupt since the plugin is compiled against Dagon. Do you have this problem, too? For now I couldn't solve that problem.

[Sylpheed]

Are you using the OpenGL Subsystem for rendering?

Not, I'm using DirectX Subsystem. I can see nothing with openGL subsystem, the rendering window shows the grey color for the background, so I have to use Directx.

[DWORD]

futnuh, nice project!

I've actually been wanting to do this (positioning, orienting with markers) for some months since I saw a small demo at CeBIT. I thought it would be cool to use this in Ogre, and now you've paved the way. I didn't know that ARToolkit could do this. That's pretty cool.

Keep it up.

[futnuh]

May I ask you, for what will you use the ARToolkit in future?

We are pitching a number of projects to a few potential clients . One idea, for example, would be an agricultural exhibit where children grow a virtual plant on their personal marker. By going to different stations, they can use virtual tools (that replace the AR markers) to add sunshine, nitrogen, moisture etc.

Right now I'm just trying to build a compelling demo to showcase the possibilities. Note that the ARToolkit is GPL rather than LGPL. For commercial use though, the ARToolkit people have very reasonable licensing terms - see their page for contact details.