[WIP] Webcam Plugin

A place for users of OGRE to discuss ideas and experiences of utilitising OGRE in their games / demos / applications.

[WIP] Webcam Plugin

Postby Timme » Wed Jan 11, 2006 9:43 pm

[Edit]The second beta of the webcam plugin has been released. The plugin is based on the video plugin of pjcast.[/Edit]

Features of the plugin:
- Fully scriptable through material script
- automatic update of the texture
- error handling if webcam couldn't be initialised

I provide a binary version and the source (with VC++ 2005 project files) Both versions include the Plugin_Webcam.dll and a Demo:
http://tuan.kuranes.free.fr/Ogre.html#WebCam ([EDIT] new download link)

Note for user who wants to compile the source code: You will have to download OpenCV ( http://www.intel.com/technology/computi ... /index.htm ) and to set up the include path in the projects settings of "WebcamPlugin":
Include:
(OpenCVPath)/cv/include
(OpenCVPath)/cxcore/include
(OpenCVPath)/otherlibs/highgui
Lib:
(OpenCVPath)/lib
Last edited by Timme on Sun Jan 15, 2006 11:34 am, edited 2 times in total.
Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby CaseyB » Thu Jan 12, 2006 1:17 am

How far along has this plugin come? This is exacly the sort of thing that we had planned on implementing on top of the Quake3 engine for a work project! I use Ogre at home as a hobby, and the boss wanted to develop on the Quake engine, but it's looking more and more like there would be less work to add what we need to Ogre instead of adding what we need to Quake!
User avatar
CaseyB
OGRE Contributor
OGRE Contributor
 
Posts: 1335
Kudos: 2
Joined: 20 Nov 2005
Location: Columbus, Ohio

Postby liberator » Thu Jan 12, 2006 9:29 am

I am planning on making webcam support as a CEGUI widget gfx side with my engine framework providing the backend.

Good luck with yours.
User avatar
liberator
Greenskin
 
Posts: 117
Kudos: 0
Joined: 24 Jan 2005
Location: Sillicon Valley, California

Postby tuan kuranes » Thu Jan 12, 2006 10:21 am

Very interesting !

Questions :

- Are threads mandatory for performance when getting image ?
- I understand OpenCV is the lib that take care of geting webcam Image and configuration, is it supposed to work out of the box on other platform ? did you tried it on macosx or gnu/linux?

Sugestions :

- Flipping Texture could be done using Texture Coordinates matrix, that would be indeed faster. (video plugin of pjcast does that, no ?)

btw, I can provide File hosting, if you want.
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
 
Posts: 2653
Kudos: 4
Joined: 24 Sep 2003
Location: Haute Garonne, France

Postby Timme » Thu Jan 12, 2006 4:16 pm

@liberator: Good luck with your project, too. You can load the Ogre::Texture into an CEGUI Panel. The texture can be loaded like the render texture in the CEGUI Demo.

@CaseyB: I never used the Quake3 engine, but I think its not too difficult to implement a webcam into the Quake3 engine. You can also use OpenCV to load the webcam images.

@tuan kuranes:
tuan kuranes
I understand OpenCV is the lib that take care of geting webcam Image and configuration, is it supposed to work out of the box on other platform ? did you tried it on macosx or gnu/linux?

Yes, I think the plugin will also be compileable on linux. Thats why I choseOpenCV. (And because of the Computer Vision code) But I haven't tried yet. Maybe, I will release linux makefiles in future.

tuan kuranes
Are threads mandatory for performance when getting image ?

My webcam only delivers 15 frames per second. So I had to use a thread which pulls the data stream from the webcam. The update method checks if there is a new frame available and it will updates the texture if necessary.

tuan kuranes
Flipping Texture could be done using Texture Coordinates matrix, that would be indeed faster. (video plugin of pjcast does that, no ?)

Yeah, I will fix that in the next version.

tuan kuranes
btw, I can provide File hosting, if you want.


Would be nice ;-)

P.S.: Has anybody tried to compile the sources?

Example Materialscript:
Code: Select all
material <MaterialName>
{
   technique
   {
      pass
      {

         texture_unit
         {
            texture_source webcam_video
            {
               //Flip true //flips around the y axis
               //Webcam_Number 0 //Defines which webcam has to be use. (Could also be a TV-card. 0 is the standart device. > 0 are other devices. -1 is a random device)
            }
         }
      }
   }
}

Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby tuan kuranes » Thu Jan 12, 2006 4:38 pm

My webcam only delivers 15 frames per second. So I had to use a thread which pulls the data stream from the webcam. The update method checks if there is a new frame available and it will updates the texture if necessary.

Ok. then I guess that "cvQueryFrame()" is a blocking call.

Would be nice


File Hosted There, Mail me for files update or anything

http://tuan.kuranes.free.fr/Ogre.html#WebCam
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
 
Posts: 2653
Kudos: 4
Joined: 24 Sep 2003
Location: Haute Garonne, France

Postby Timme » Thu Jan 12, 2006 4:39 pm

tuan kuranes:Ok. then I guess that "cvQueryFrame()" is a blocking call.


Yes, cvQueryFrame() is a blocking call.
Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby tuan kuranes » Thu Jan 12, 2006 5:26 pm

P.S.: Has anybody tried to compile the sources?


webcam.cpp is in include directory instead of src.
Demo has unessecary include dependency with plugin (and therefore openv and ptypes) you can safely remove the
Code: Select all
#include <WebcamController.h>
#include <WebcamPluginPreReqs.h>

at top of webcamplugindemo.h
In vcproj you could use use Env Var name like $(OGRE_HOME), $(OPENCV) and even $(ConfigurationName)
and add a postbuild event like
Code: Select all
copy $(OutDir)\$(TargetFileName) $(OGRE_HOME)\Bin\$(ConfigurationName); copy $(OutDir)\*.lib $(OGRE_HOME)\lib\opt\$(ConfigurationName);copy $(OutDir)\*.pdb $(OGRE_HOME)\lib\opt\$(ConfigurationName)

for dlls and
Code: Select all
copy $(OutDir)\$(TargetFileName) $(OGRE_HOME)\Bin\$(ConfigurationName); 

for exes
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
 
Posts: 2653
Kudos: 4
Joined: 24 Sep 2003
Location: Haute Garonne, France

Postby Timme » Thu Jan 12, 2006 5:38 pm

@tuan kuranes: Thank you very much

webcam.cpp has moved to src. (the ptypes folder should also be deleted in src) (Will be changed with the new update)

Code: Select all
#include <WebcamController.h>
#include <WebcamPluginPreReqs.h>


I forgot to remove that headers, because I had first to update the textures manually.

On my machine I haven't set env vars yet. To compile the source you have to create a new folder (named for example "webcamPlugin") and to copy all the files from the zip into it. The new folder has to be created in your ogre path:

<OgrePath>
-ogrenew
-videoPlugin

Did you use Ogre 1.0.6 or Dagon?
Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby tuan kuranes » Thu Jan 12, 2006 5:53 pm

The good thing is that OgreSDK and Opencv installer does create those env vars automatically. Env Var let user choose where he can put projects.
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
 
Posts: 2653
Kudos: 4
Joined: 24 Sep 2003
Location: Haute Garonne, France

Postby Timme » Thu Jan 12, 2006 6:00 pm

I haven't noticed that OpenCV creates env vars. Ok, then can I set up the include pathes for OpenCV in the project settings.

Yes, the SDK creates env var, but I'm using the sources.

Maybe I can provide two project solutions: One for the SDK user and one for the Ogre users which build Ogre from source?
Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby tuan kuranes » Thu Jan 12, 2006 6:10 pm

I haven't noticed that OpenCV creates env vars. Ok, then can I set up the include pathes for OpenCV in the project settings.

Sorry, I'm wrong there, OpenCV add its bin directory to system path, not its path as a env_var.

Maybe I can provide two project solutions: One for the SDK user and one for the Ogre users which build Ogre from source?


Current solution is Ok, provided user know where to put files. Multiples vcproj could be a good Idea, tough.
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
 
Posts: 2653
Kudos: 4
Joined: 24 Sep 2003
Location: Haute Garonne, France

Postby Timme » Fri Jan 13, 2006 4:23 pm

Tuan Kuranes:Flipping Texture could be done using Texture Coordinates matrix, that would be indeed faster.


Flipping the image with a Texture Coordinates matrix is not much faster then flipping it the old methods.
Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby Kencho » Fri Jan 13, 2006 4:31 pm

Timme, I might be wrong, but the texture matrix multiplication is always performed when textures are involved, so there shouldn't be any performance loss :? How can that be as slow as traditional methods?
Image
User avatar
Kencho
OGRE Retired Moderator
OGRE Retired Moderator
 
Posts: 4011
Kudos: 1
Joined: 19 Sep 2003
Location: Burgos, Spain

Postby Timme » Fri Jan 13, 2006 4:47 pm

Because I use two threads (one for the webcam and one for the rendering) I have to variables which stores the image data. One variable is used in the thread with the webcam, one is used in the rendering thread. Copying the image data from one variable is not much faster then flipping the image data from the one variable into the other.

If you don't write the line "Flip true" into the material file, the program will run as fast as without the Texture Coord matrix. With the command "Flip true" the image is flipped a second time, because you can use the webcam in more than one texture unit. The image has to be flipped the first time because the ogre image has its origin left at the top. The image delivered by the webcam has its origin at the bottom.
Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby tuan kuranes » Fri Jan 13, 2006 5:12 pm

Texture Matrix Idea idea is that you'll to never need to flip the image in software. So it has to be faster or at least equal.
If used in other texture units, they should also use the "flipped" texture matrix.
But It appears you still flip it each frame in software even when using texture matrix ?
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
 
Posts: 2653
Kudos: 4
Joined: 24 Sep 2003
Location: Haute Garonne, France

Postby Timme » Fri Jan 13, 2006 6:08 pm

The texture will be flipped by the matrix in future. But this has an disadvantage: The transform matrix has no effect on the progammable pipeline. User who use the webcamplugin in combination to a shader program will have to regard the transform matrix, too.
I think the next version will be a little bit faster and will save memory on the graphics card.
In the past I've loaded the image data without resizing it to 512x512 texel (in software mode with a function in OpenCV) before. Resizing the image with Ogre seemed to be slower as with OpenCV, especially if you use large images.
Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby Timme » Sun Jan 15, 2006 11:44 am

A new beta has been released (beta 2).

I've altered a few things:
- In the new version only one texture per webcam is created instead of one texture per material.
- A few methods were added.
- Fixed a memory leak.

Download link: http://tuan.kuranes.free.Fr/Ogre.html#Webcam

The source is tested with Ogre 1.0.6 build from source. People who want to compile the source with the SDK will have to change the dictories.
Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby bal » Sun Jan 15, 2006 1:54 pm

Argh, d3dx9_28.dll is missing and I can't find just the dll on the net (instead of downloading the big DX SDK). Perhaps include it (although it's against the EULA).
User avatar
bal
Greenskin
 
Posts: 100
Kudos: 0
Joined: 09 Dec 2004
Location: Geluwe, Belgium

Postby Timme » Sun Jan 15, 2006 2:09 pm

You can download the DirectX December 2005 - Redistributable for Software Developers. I think its not as big as the SDK. Maybe, you will find the DLL in a binary demo released by somebody else.
Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby bal » Sun Jan 15, 2006 4:27 pm

Works great now, although only in DirectX.
User avatar
bal
Greenskin
 
Posts: 100
Kudos: 0
Joined: 09 Dec 2004
Location: Geluwe, Belgium

Postby futnuh » Tue Jul 11, 2006 9:00 am

I compiled Timme's plugin and his accompanying demo today against the Dagon SDK. It works wonderfully. Two questions if I may ...

1) How does one access the raw image data from within a frame listener? Is it as simple as getting hold of a regular texture from the enclosing material? (I want to pass this along to the ARToolkitPlus to hopefully extract the relative camera position with respect to a marker tile in the image.)

2) I spent most of today playing with the plugin under ... pyOgre. Getting it working was as simple as copying over the DLL into my pyOgre distro and updating the plugins file. One problem though, unlike the C++ demo, I'm getting a hard python crash (invalid memory reference) when exiting. Unfortunately there's nothing abnormal in the Ogre log. I wonder, is there a C++ way of *forcing* the plug-in to "clean-up" (close the camera, end the reading thread, etc.)? I'm guessing this might be the source of the problem. The pyOgre community doesn't have any ideas on this ... so we're turning to the heavy guns ;-)

Cheers,
Darran.
futnuh
Goblin
 
Posts: 225
Kudos: 1
Joined: 25 Sep 2005
Location: Calgary, Alberta

Postby Praetor » Tue Jul 11, 2006 1:26 pm

If it is a hard crash, can you debug it? Or is this one of those things python catches, and thus it hides the true source of the error?
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
 
Posts: 3335
Kudos: 3
Joined: 21 Jun 2005
Location: Rochester, New York, US

Postby Timme » Wed Jul 12, 2006 12:48 am

Hi futnuh,

1.) yes, you can get the raw image from the webcam. Here is an example code. The WebcamListener is a part of the plugin.

//TestWebcamListener.h
Code: Select all
#include <WebcamListener.h>

class TestWebcamListener : public WebcamListener
      {
      public:
         TestWebcamListener();
         void newFrame(Webcam* webcam);
         virtual ~TestWebcamListener();
      };

//TestWebcamListener.cpp
Code: Select all
TestWebcamListener::TestWebcamListener()
{
}

void TestWebcamListener::newFrame(Webcam* webcam)
{
CV::IplImage* lImg = webcam->getCvImage();
//Do something with the image...
}

TestWebcamListener::~TestWebcamListener()
{
}


And this piece of code is about, how you can tell the webcam that the TestWebcamListener belongs to the webcam

Code: Select all
ExternalTextureSourceManager::getSingleton().setCurrentPlugIn("webcam_video");
WebcamController* WebcamCtrl = (WebcamController*)ExternalTextureSourceManager::getSingleton().getCurrentPlugIn();

testWebcamListener = new VisionControlledObjectManager::VisionControlledObjectWebcamListener(this);
WebcamCtrl->getWebcamByNumber(0)->addWebcamListerner(testWebcamListener);


And remove the TestWebcamListener after use:
Code: Select all
WebcamCtrl->getWebcamByNumber(0)->removeWebcamListener(testWebcamListener);
delete testWebcamListener;


2.) I never have used the plugin under pyOgre, but there could be an error in the code, because the plugin is still beta. I updated the files at the first of April last time. I fixed one memory leak and added the WebcamListener. Maybe you have still the old version. If you were using the new version, it would be honest from you taking a look over the code and helping me to find the mistake. But I think that you still use the old version, because the WebcamListener.h doesn't seem to be in your archive and I got a memory fault, too, when I linked the Plugin_Webcam.lib into my project with the old plugin.
Timme
Greenskin
 
Posts: 122
Kudos: 0
Joined: 13 Mar 2005
Location: Germany

Postby futnuh » Wed Jul 12, 2006 4:27 am

Hi Timme,

1) I was hoping that I could access the texture data without needing to call a method unique to the plugin class (in your case, getCvImage()). Is it possible to get the image data using only "stock" Ogre calls? Ogre developers, if not, wouldn't it be possible for the ExternalTextureSource baseclass to have something akin to a getImage() method that all PlugIns implement?

2) On the crash issue, your C++ code snippet helped a lot. I'm now able to call ExternalTextureSourceManager.getSingleton().getCurrentPlugIn.shutDown() on program exit. The crash has disappeared and the following lines are appearing in the Ogre log:

Code: Select all
All webcams will be destroyed
Webcam 0 is shut down
Thread Cleaned
All webcam textures will be destroyed


Thanks for your help.
futnuh
Goblin
 
Posts: 225
Kudos: 1
Joined: 25 Sep 2005
Location: Calgary, Alberta

Next

Return to Using OGRE in practice

Who is online

Users browsing this forum: No registered users and 2 guests