[WIP] Webcam Plugin

[Timme]

[Edit]The second beta of the webcam plugin has been released. The plugin is based on the video plugin of pjcast.[/Edit]

Features of the plugin:
- Fully scriptable through material script
- automatic update of the texture
- error handling if webcam couldn't be initialised

I provide a binary version and the source (with VC++ 2005 project files) Both versions include the Plugin_Webcam.dll and a Demo:
http://tuan.kuranes.free.fr/Ogre.html#WebCam ([EDIT] new download link)

Note for user who wants to compile the source code: You will have to download OpenCV ( http://www.intel.com/technology/computi ... /index.htm ) and to set up the include path in the projects settings of "WebcamPlugin":
Include:
(OpenCVPath)/cv/include
(OpenCVPath)/cxcore/include
(OpenCVPath)/otherlibs/highgui
Lib:
(OpenCVPath)/lib

[CaseyB]

How far along has this plugin come? This is exacly the sort of thing that we had planned on implementing on top of the Quake3 engine for a work project! I use Ogre at home as a hobby, and the boss wanted to develop on the Quake engine, but it's looking more and more like there would be less work to add what we need to Ogre instead of adding what we need to Quake!

[liberator]

I am planning on making webcam support as a CEGUI widget gfx side with my engine framework providing the backend.

Good luck with yours.

[tuan kuranes]

Very interesting !

Questions :

- Are threads mandatory for performance when getting image ?
- I understand OpenCV is the lib that take care of geting webcam Image and configuration, is it supposed to work out of the box on other platform ? did you tried it on macosx or gnu/linux?

Sugestions :

- Flipping Texture could be done using Texture Coordinates matrix, that would be indeed faster. (video plugin of pjcast does that, no ?)

btw, I can provide File hosting, if you want.

[Timme]

@liberator: Good luck with your project, too. You can load the Ogre::Texture into an CEGUI Panel. The texture can be loaded like the render texture in the CEGUI Demo.

@CaseyB: I never used the Quake3 engine, but I think its not too difficult to implement a webcam into the Quake3 engine. You can also use OpenCV to load the webcam images.

@tuan kuranes:

tuan kuranes
I understand OpenCV is the lib that take care of geting webcam Image and configuration, is it supposed to work out of the box on other platform ? did you tried it on macosx or gnu/linux?

Yes, I think the plugin will also be compileable on linux. Thats why I choseOpenCV. (And because of the Computer Vision code) But I haven't tried yet. Maybe, I will release linux makefiles in future.

tuan kuranes
Are threads mandatory for performance when getting image ?

My webcam only delivers 15 frames per second. So I had to use a thread which pulls the data stream from the webcam. The update method checks if there is a new frame available and it will updates the texture if necessary.

tuan kuranes
Flipping Texture could be done using Texture Coordinates matrix, that would be indeed faster. (video plugin of pjcast does that, no ?)

Yeah, I will fix that in the next version.

tuan kuranes
btw, I can provide File hosting, if you want.

Would be nice

P.S.: Has anybody tried to compile the sources?

Example Materialscript:

Code: Select all

material <MaterialName>
{
   technique
   {
      pass
      {

         texture_unit
         {
            texture_source webcam_video
            {
               //Flip true //flips around the y axis
               //Webcam_Number 0 //Defines which webcam has to be use. (Could also be a TV-card. 0 is the standart device. > 0 are other devices. -1 is a random device)
            }
         }
      }
   }
}



[tuan kuranes]

My webcam only delivers 15 frames per second. So I had to use a thread which pulls the data stream from the webcam. The update method checks if there is a new frame available and it will updates the texture if necessary.

Ok. then I guess that "cvQueryFrame()" is a blocking call.

Would be nice

File Hosted There, Mail me for files update or anything

http://tuan.kuranes.free.fr/Ogre.html#WebCam

[Timme]

tuan kuranes:Ok. then I guess that "cvQueryFrame()" is a blocking call.

Yes, cvQueryFrame() is a blocking call.

[tuan kuranes]

P.S.: Has anybody tried to compile the sources?

webcam.cpp is in include directory instead of src.
Demo has unessecary include dependency with plugin (and therefore openv and ptypes) you can safely remove the

Code: Select all

#include <WebcamController.h>
#include <WebcamPluginPreReqs.h>


at top of webcamplugindemo.h
In vcproj you could use use Env Var name like $(OGRE_HOME), $(OPENCV) and even $(ConfigurationName)
and add a postbuild event like

Code: Select all

copy $(OutDir)\$(TargetFileName) $(OGRE_HOME)\Bin\$(ConfigurationName); copy $(OutDir)\*.lib $(OGRE_HOME)\lib\opt\$(ConfigurationName);copy $(OutDir)\*.pdb $(OGRE_HOME)\lib\opt\$(ConfigurationName)

for dlls and

Code: Select all

copy $(OutDir)\$(TargetFileName) $(OGRE_HOME)\Bin\$(ConfigurationName);

for exes

[Timme]

@tuan kuranes: Thank you very much

webcam.cpp has moved to src. (the ptypes folder should also be deleted in src) (Will be changed with the new update)

Code: Select all

#include <WebcamController.h>
#include <WebcamPluginPreReqs.h>

I forgot to remove that headers, because I had first to update the textures manually.

On my machine I haven't set env vars yet. To compile the source you have to create a new folder (named for example "webcamPlugin") and to copy all the files from the zip into it. The new folder has to be created in your ogre path:

<OgrePath>
-ogrenew
-videoPlugin

Did you use Ogre 1.0.6 or Dagon?

[tuan kuranes]

The good thing is that OgreSDK and Opencv installer does create those env vars automatically. Env Var let user choose where he can put projects.

[Timme]

I haven't noticed that OpenCV creates env vars. Ok, then can I set up the include pathes for OpenCV in the project settings.

Yes, the SDK creates env var, but I'm using the sources.

Maybe I can provide two project solutions: One for the SDK user and one for the Ogre users which build Ogre from source?

[tuan kuranes]

I haven't noticed that OpenCV creates env vars. Ok, then can I set up the include pathes for OpenCV in the project settings.

Sorry, I'm wrong there, OpenCV add its bin directory to system path, not its path as a env_var.

Maybe I can provide two project solutions: One for the SDK user and one for the Ogre users which build Ogre from source?

Current solution is Ok, provided user know where to put files. Multiples vcproj could be a good Idea, tough.

[Timme]

Tuan Kuranes:Flipping Texture could be done using Texture Coordinates matrix, that would be indeed faster.

Flipping the image with a Texture Coordinates matrix is not much faster then flipping it the old methods.

[Kencho]

Timme, I might be wrong, but the texture matrix multiplication is always performed when textures are involved, so there shouldn't be any performance loss How can that be as slow as traditional methods?

[Timme]

Because I use two threads (one for the webcam and one for the rendering) I have to variables which stores the image data. One variable is used in the thread with the webcam, one is used in the rendering thread. Copying the image data from one variable is not much faster then flipping the image data from the one variable into the other.

If you don't write the line "Flip true" into the material file, the program will run as fast as without the Texture Coord matrix. With the command "Flip true" the image is flipped a second time, because you can use the webcam in more than one texture unit. The image has to be flipped the first time because the ogre image has its origin left at the top. The image delivered by the webcam has its origin at the bottom.

[tuan kuranes]

Texture Matrix Idea idea is that you'll to never need to flip the image in software. So it has to be faster or at least equal.
If used in other texture units, they should also use the "flipped" texture matrix.
But It appears you still flip it each frame in software even when using texture matrix ?

[Timme]

The texture will be flipped by the matrix in future. But this has an disadvantage: The transform matrix has no effect on the progammable pipeline. User who use the webcamplugin in combination to a shader program will have to regard the transform matrix, too.
I think the next version will be a little bit faster and will save memory on the graphics card.
In the past I've loaded the image data without resizing it to 512x512 texel (in software mode with a function in OpenCV) before. Resizing the image with Ogre seemed to be slower as with OpenCV, especially if you use large images.

[Timme]

A new beta has been released (beta 2).

I've altered a few things:
- In the new version only one texture per webcam is created instead of one texture per material.
- A few methods were added.
- Fixed a memory leak.

Download link: http://tuan.kuranes.free.Fr/Ogre.html#Webcam

The source is tested with Ogre 1.0.6 build from source. People who want to compile the source with the SDK will have to change the dictories.

[bal]

Argh, d3dx9_28.dll is missing and I can't find just the dll on the net (instead of downloading the big DX SDK). Perhaps include it (although it's against the EULA).

[Timme]

You can download the DirectX December 2005 - Redistributable for Software Developers. I think its not as big as the SDK. Maybe, you will find the DLL in a binary demo released by somebody else.

[bal]

Works great now, although only in DirectX.

[futnuh]

I compiled Timme's plugin and his accompanying demo today against the Dagon SDK. It works wonderfully. Two questions if I may ...

1) How does one access the raw image data from within a frame listener? Is it as simple as getting hold of a regular texture from the enclosing material? (I want to pass this along to the ARToolkitPlus to hopefully extract the relative camera position with respect to a marker tile in the image.)

2) I spent most of today playing with the plugin under ... pyOgre. Getting it working was as simple as copying over the DLL into my pyOgre distro and updating the plugins file. One problem though, unlike the C++ demo, I'm getting a hard python crash (invalid memory reference) when exiting. Unfortunately there's nothing abnormal in the Ogre log. I wonder, is there a C++ way of *forcing* the plug-in to "clean-up" (close the camera, end the reading thread, etc.)? I'm guessing this might be the source of the problem. The pyOgre community doesn't have any ideas on this ... so we're turning to the heavy guns

Cheers,
Darran.

[Praetor]

If it is a hard crash, can you debug it? Or is this one of those things python catches, and thus it hides the true source of the error?

[Timme]

Hi futnuh,

1.) yes, you can get the raw image from the webcam. Here is an example code. The WebcamListener is a part of the plugin.

//TestWebcamListener.h

Code: Select all

#include <WebcamListener.h>

class TestWebcamListener : public WebcamListener
      {
      public:
         TestWebcamListener();
         void newFrame(Webcam* webcam);
         virtual ~TestWebcamListener();
      };


//TestWebcamListener.cpp

Code: Select all

TestWebcamListener::TestWebcamListener()
{
}

void TestWebcamListener::newFrame(Webcam* webcam)
{
CV::IplImage* lImg = webcam->getCvImage();
//Do something with the image...
}

TestWebcamListener::~TestWebcamListener()
{
}


And this piece of code is about, how you can tell the webcam that the TestWebcamListener belongs to the webcam

Code: Select all

ExternalTextureSourceManager::getSingleton().setCurrentPlugIn("webcam_video");
WebcamController* WebcamCtrl = (WebcamController*)ExternalTextureSourceManager::getSingleton().getCurrentPlugIn();

testWebcamListener = new VisionControlledObjectManager::VisionControlledObjectWebcamListener(this);
WebcamCtrl->getWebcamByNumber(0)->addWebcamListerner(testWebcamListener);


And remove the TestWebcamListener after use:

Code: Select all

WebcamCtrl->getWebcamByNumber(0)->removeWebcamListener(testWebcamListener);
delete testWebcamListener;


2.) I never have used the plugin under pyOgre, but there could be an error in the code, because the plugin is still beta. I updated the files at the first of April last time. I fixed one memory leak and added the WebcamListener. Maybe you have still the old version. If you were using the new version, it would be honest from you taking a look over the code and helping me to find the mistake. But I think that you still use the old version, because the WebcamListener.h doesn't seem to be in your archive and I got a memory fault, too, when I linked the Plugin_Webcam.lib into my project with the old plugin.

[futnuh]

Hi Timme,

1) I was hoping that I could access the texture data without needing to call a method unique to the plugin class (in your case, getCvImage()). Is it possible to get the image data using only "stock" Ogre calls? Ogre developers, if not, wouldn't it be possible for the ExternalTextureSource baseclass to have something akin to a getImage() method that all PlugIns implement?

2) On the crash issue, your C++ code snippet helped a lot. I'm now able to call ExternalTextureSourceManager.getSingleton().getCurrentPlugIn.shutDown() on program exit. The crash has disappeared and the following lines are appearing in the Ogre log:

Code: Select all

All webcams will be destroyed
Webcam 0 is shut down
Thread Cleaned
All webcam textures will be destroyed


Thanks for your help.