Mogre with WPF GUI

I have trying to use WPF as GUI for MOGRE. I really did not like the options I had for UI My initial experiments tried to extend the D3DImage trick to blend MOGRE into WPF. But I kept feeling that It was a bad solution.

Then I switched to try getting a transparent WPF window to be rendered over the render window. My first idea was to use a WPF popup to avoid air space restrictions with the WinForm that I was using as a render window. I did have success with that until I hit the restriction of a WPF popup not taking up more than 75% of screen space. A couple of hors of experimentation game me another solution. Use Modeless Interop between WPF and WinForms to overlay a transparent borderless WPF form over the underlying WinForms. The clincher: It takes ~25 lines of code to get this up and running

Take a look at the code to launch a WPF Window: https://bitbucket.org/Hormis/windowsinput/src/8624b5cb92fa/WindowsInput/RenderWindow.cs#cl-31
the code on the WPF side to track parent form is here: https://bitbucket.org/Hormis/windowsinput/src/8624b5cb92fa/WindowsInput/ViewControls.xaml.cs#cl-29

As you can see from this, WPF can be used to provide a UI for a MOGRE engine. Since WPF can easily be styled using templates and styles, it could serve as a GUI system while you are experimenting or even for the final game!

Theoretically, you can use the Ogre's own render window using the System.Windows.Forms.NativeWindow class, but I did not do it as it requires a bit more code

PS: the repository is at: https://bitbucket.org/Hormis/windowsinput/. It is something I banged up to play with Input handling using Raw Input and DirectInput. I just dropped in the WPF based UI to give it a full workout.

Thanks to Cygon for the Mogre 1.7.3 builds I used.

Edit: Fixed links to the restored Repository.

Thank you very much for your report and that you published your source code.

I plan to create a tiny wiki page about Mogre and WPF, which contains useful links and perhaps some basic information and screenshoots.
As you said I think it's a good combination.
There are also some other interesting topics and examples about Mogre+WPF. Now there is one more.

Do you still work with Mogre or did you just try it?
If you like, you can post a screenshot of your WPF GUI.

Would this work in full screen mode?

Oh I just saw this! Thanks for posting this, hormis, but... you took the source repository down? Why?

Very sad to see that someone deleted his repository.

I suggest to write him a personal forum message.
And then hope that he will read and answer.

I found out that he deleted his repository, but still has his BitBucket account.
So I wrote him a message and asked to re-publish his code.

I got an answer from Hormis via BitBucket:
I have put it back up and updated links on the original post. It seems to have triggered a mod review.

Sorry about the mixup with the repo. I seemed to have deleted it accidentally when I was cleaning up some of my private repos.

He edited his first post of this topics.
The dead links are living again

Thanks Hormis, that's really cool! It's great to have another approach to WPF+Mogre.

I tried it but I will have to experiment with it more to compare with the D3DImage approach. Why doesn't D3DImage feel good to you? Or what do you like about this better?

It is nice you show it taking not much code, although performance is probably going to be my top priority -- do you know how performance compares?

Actually I started out planning to overlay a WPF overlay on top of Mogre's native window. This is an approach I built to figure out if it will work out. I had used WPF before and found that the FPS that I can attain seemed lower with D3DImage than when I was rendering to a Window Form object or using the Native Window.

What I really like about this is the fact that since the overlay is another window being composited on top, you can design various game screens standalone using the WPF designer.

There are a couple of drawbacks to this.
1) True Fullscreen is out. You can fake it though.
2) Some combination of elements seems to result in WPF stealing focus entirely. This means lots of testing to ensure it works correctly.

Fullscreen can be faked using the code at: https://bitbucket.org/Hormis/windowsinput/src/8624b5cb92fa/WindowsInput/RenderWindow.cs#cl-81. I did play around with the WPF Render to Texture options which are really good. It may be possible to reverse the D3DImage trick and use it to render the WPF UI to Ogre/Mogre texture and composite it using the compositor. Of course, it require you to play quite a bit around with how to route input correctly from Engine -> WPF. I probably experiment when I have some time to spare.

Right now, I am not using this trick much as my toy engine is still in the state where I can use the built-in ogre overlays. Unfortunately, I have not been able to play with my toy engine as much as I like these days.

Actually, let me just compare both techniques.

D3DImage

1. Simplicity and Predictability
   Driven by Composition events of WPF.
   Vsync Locked to about 60fps or less.
   Mogre Render Texture pushed to WPF via D3DImage[/list:u]
   
   Overlaid Window technique
   
   1. More complex to do it right (Focus stealing issues need lots of testing)
      Driven by game loop https://bitbucket.org/Hormis/windowsinput/src/8624b5cb92fa/WindowsInput/MogreRenderer.cs#cl-196 (Uses Application.DoEvents() to pump events)
      Vsync under control of Mogre
      Desktop Compositor composites the windows together.[/list:u]
      
      Neither can do true fullscreen, but it can be faked. Technically, you can call the Mogre.RenderWindow.SetFullscreen and have it work with the Overlaid Window Technique, but the WPF window cannot follow and you lose the UI.
      
      Overlay technique is far more fiddly to get right. For example, on the code in the repo, run it and click on the textblock. You will see the focus shift to the WPF window. The main title bar highlight is removed and the button gets focused. At this point if you hit enter the button will activate and the app will toggle fullscreen.
      
      As you can see, this is the reason I feel the perfect solution is to pull updates from wpf to Mogre instead of the other way around as D3DImage technique is doing. Unfortunately, the API's that Microsoft provides are a bit lacking as they allow you to create a render texture manually but not reuse an existing D3DTexture to render to.

Thanks for your good information!

Yeah thanks!
I definitely need to get this sorted out. I'm making a commercial indie game and I'm trying to determine whether D3DImage will work well enough or be a showstopper.

I'm curious about the reverse approach as well , although it sounds like it might be a fair amount of work, to get input working for example.
(and if it is a lot of work, it gets me thinking that I really like WPF but I wish there was something just as cool that was open source and cross platform and maybe I should make it .)

Here are a couple articles on the reverse approach:

http://catalog.codeproject.com/Articles/59790/Use-a-screenshot-of-a-WPF-visual-as-a-texture-in-M
http://www.grumpydev.com/2009/01/03/taking-wpf-screenshots/

I did start playing with this: I have radial HP bars in my game made by WPF, but it takes too long to update (5ms eep), so I capture them as Png data (at say 5% intervals) so I can (in the future) upload them to Mogre as a texture.

Beauty, can we start a wiki page for Mogre+WPF to at least start dumping links in? (And organize it later as we have time?)

Ok, so I made a break through in Mogre WPF integration today.

Meet Double Buffered D3DImage: https://bitbucket.org/Hormis/doublebufferedd3dimage. This is sort of a middle ground between the Overlay Window and a Pure D3DImage. You will need to grab the dependencies from https://bitbucket.org/JaredThirsk/mogreinwpf/downloads which I got from your Sig Meharin. I used the one called "Dependencies - 11.03.21 1.7-Jared-r2.7z"

Speed Comparision:

1. 
   <60FPS approx. - D3DImage technique (did not test).
   570 - 650 FPS - Double Buffered D3DImage technique. (Actual rendering is at 810 - 840 FPS. The rate given is the rate of CompositionTarget.Render calls received)
   1250 - 1275 FPS - Overlay Window Technique.
   1275 - 1305 FPS - Pure Render to Windows Forms. (Got by turning off WPF overlay on the WindowsInput code)
   [/list:u]
   
   Here is my findings. The main bottle neck in both D3DImage and Double Buffered D3DImage techniques is calling D3DImage.AddDirtyRect in the main thread. Do it over the full frame and the frame rate drops to a pitiful levels. It seems to me that WPF is trying to soft lock to 60FPS algorithmically rather than using any true VSync facilities. So it looks like it expect you to return very quickly from any CompositonTarget.Render call.
   
   I use 2 Textures, one to which I render and the other being used as the back buffer of the D3DImage. I am blitting from the one to the other using CopyTextureTo and marking the whole rect dirty outside the CompositionTarget.Render event. This means that the CompositionTarget.Render becomes mostly a constant time operation which WPF seems to like very much.
   
   The composition code is at https://bitbucket.org/Hormis/doublebufferedd3dimage/src/e8365af6722a/DoublebufferD3DImage/MogreImage.cs#cl-120. I am calling a 1 px AddDirtyRect as without it the frame rates seems to be very jittery as if it is not compositing every frame. Add that call, and the animation becomes smooth.
   
   Blitting is done at https://bitbucket.org/Hormis/doublebufferedd3dimage/src/e8365af6722a/DoublebufferD3DImage/MogreImage.cs#cl-224 which is called after the render happens in the main loop of the renderer: https://bitbucket.org/Hormis/doublebufferedd3dimage/src/e8365af6722a/DoublebufferD3DImage/MogreRenderer.cs#cl-201
   
   Some other things I tried out and results:
   
   1. 
      Swapping Render Targets instead of Blitting - Flashing screen
      Rendering directly to back buffer of D3DImage - Flashes, Incomplete rendered frames, etc.
      RenderingOneFrame called from within CompositionTarget.Render while rendering directly to back buffer - Flashes, incomplete render.
      [/list:u]
      
      There are still some amount of bugs in the code as it is not bullet proof. I have seen D3DImage lose the FrontSurface rarely, but this is fine as a starting point for experimenting. Short of multithreading, it is impossible to get the composition frame rate of WPF up to this level. Also if you are using this technique, be sure to test it when the frame rate goes low to around 60FPS or so. I am not sure how the Composition behave at those frame rates.
      
      Finally a comparison to D3DImage technique.
      
      1. 
         Simplicity and Predictability
         Driven by game loop. Composition is Driven by WPF events.
         Frame rate is not limited. There is no vsync, but the output looks almost like it is since it is similar ti triple buffered rendering. (But with lots of inefficient blitting )
         Mogre Render Texture pushed to WPF via D3DImage
         [/list:u]
         
         Note: I am using two render textures in the code as I did not find a way to get a surface ptr from the Texture object directly. Shouldn't it be there?

Awesome! I'm excited to try this out!

Are you using the Direct3D 9Ex Ogre plugin?

Have you tried WPF Perforator to see what FPS it reports? (http://msdn.microsoft.com/en-us/library/aa969767.aspx)

I don't know how to get a surface pointer from a texture. For a Mogre RenderTarget, this is the code I see:

renderTarget.GetCustomAttribute("DDBACKBUFFER", out /* IntPtr */ surface);

I used the Direct3DEx plugin. However, I am seeing a slow memory corruption which eventually causes the application to crash. So I switched to the vanilla version x64 bit build of Mogre built by Cygon and available from http://www.ogre3d.org/addonforums/viewtopic.php?f=8&t=14645. The frame rate and composition rate is slower at about 650ish and 420ish. Still working on it.

Even with Composition events firing at a very high rate, the actual FPS to screen seems to be around 34FPS 1680x1050 as reported by the WPF Perforator. I will need to look at some existing D3DImage samples to see if the performance is the same there.

I guess I would have to look in to it tomorrow.

Cygons x64 build is extremely stable under multithreading.

Meharin, do you have a Direct3D9Ex plugin build with both D3DCREATE_MULTITHREADED and D3DCREATE_FPU_PRESERVE set? I need to try the case of running game loop on its own thread. The current build has some issue that makes it unstable on my machine on longer runs.

Btw, I noticed something funny. If I try to use a flip strategy when the composition calls are coming in faster than 60FPS, the Video memory usage starts to look like a sawtooth waveform.

This shows up in my ogre.log:

01:13:20: D3D9 : RenderSystem Option: Multithreaded = Yes
01:13:20: D3D9 : RenderSystem Option: Floating-point mode = Consistent

And this is in my D3D9Ex source repo, (which is slightly out of date from my build, can't remember what exactly changed):


// Do we want to preserve the FPU mode? Might be useful for scientific apps
ConfigOptionMap& options = renderSystem->getConfigOptions();
ConfigOptionMap::iterator opti = options.find("Floating-point mode");
if (opti != options.end() && opti->second.currentValue == "Consistent")
{
LogManager::getSingleton().logMessage(" - D3D Floating-point mode: Consistent");
extraFlags |= D3DCREATE_FPU_PRESERVE;
}

// Jared added [BTW: Meharin == Jared]
opti = options.find("Multithreaded");
if (opti != options.end() && opti->second.currentValue == "true")
extraFlags |= D3DCREATE_MULTITHREADED;

#if OGRE_THREAD_SUPPORT == 1
extraFlags |= D3DCREATE_MULTITHREADED;
#endif
if(extraFlags & D3DCREATE_MULTITHREADED)
{
LogManager::getSingleton().logMessage(" - D3D thread mode: Multithreaded");
}
else
{
LogManager::getSingleton().logMessage(" - D3D thread mode: Singlethreaded");
}


My ogre.cfg:
[Direct3D9 Rendering Subsystem]
...
Floating-point mode=Fastest

What?? Based on my cfg file I would think I wouldn't have Consistent.

Oh, but I have this in my source code:


_root.RenderSystem.SetConfigOption("Multithreaded", "Yes");
_root.RenderSystem.SetConfigOption("Floating-point mode", "Consistent");


If you set those options, does it show up in your ogre.log?

That is disturbing that my build isn't stable. I have had sporadic access violations in kernel dlls so maybe I am seeing it too. My head has been in C# land for years so I am rusty on debugging C++. How are you detecting lack of stability?

Actually I remember that the original builds Cygon built for 1.7.3 also had a couple of issues like this.

So Multi-threaded is an option on your build huh? I will try with that.

I looked a bit more detailed into the WPF rendering architecture yesterday. I will give you a couple of things to try on your code as well. First the Dispatcher object on System.Windows.Application.Current has a Hooks property where you can associate with the DispatcherInactive event. This should raise the effective FPS as in your code, you can do world ticking in there in addition to the WPF Composition event, may be even rendering. That inactive event should fire more frequently than the composition event.

Second, the WPF rendering happens on a separate thread from the dispatcher thread. I think it would be better to turn on Multithreaded always with the Direct3D9Ex plugin.

Thanks, I hadn't tried the DispatcherInactive event before. I added it to my MogreInWpf project and I can check it in at some point.

I wonder if it is used for the DispatcherTimer internally? In my MogreWpf test app I have #if's to try either System.Timers.Timer or System.Windows.Threading.DispatcherTimer.

How fast do you get the Update loop to run with DispatcherInactive?

The main bottle neck seems to be WPF Lock Dirty Rect Unlock.I still have to try out from the Windows Forms angle. But pure WPF does not allow me to speed beyond 40ish FPS on the Double Buffer technique. I even tried switching to Mogre compositor framework to render to the backbuffer from the rendertarget. Not much of a speed up.

If by end of day, I cannot get a full solution, I am going to shelve the experiments on wpf for now.

In hindsight, the solution was right in front of me. It all seems to come down to the Dispatcher and the AddDirtyRect calls. As long as I remained in and use the WPF System.Windows.Application objects to manage my run loop, I was stuck with <60 FPS framerates.

I have two data points. Data point 1 is the x86 DoublebufferedD3DImage project, the latest version with Jared's (Meharin) Direct3D9Ex plugin. It runs at around 400 - 450ish FPS. The project when configured to use MogreCompositor to push data from the actual scene rendertarget to the backbuffer of D3DImage turned in those framerates. Use a blit, and the frame rate for render seemed to actually rise to around 600ish. Either option is a very very nice approach. Depending on your project requirements, you can choose either. Unfortunately, the WPF Performance tool does not recognize my code as being a WPF application so I could not verify the framerates with it.

I forked Jared's MogreInWpf repo and committed in a minor change in which switched it to run using a game loop based on System.Windows.Forms.Application.DoEvents(). The basic thing did not raise the framerate by any amount. It was still plodding along at a stately 60FPS vsynced. That was until I added code to Dirty the backbuffer rect from the game loop. When I did so, the performance climbed to 280ish on my machine with no other changes needed. There is the unfortunate thing that the framerate drops when you raise the size of the window. A reason for that may be that the render call becomes more expensive as the frame size increases and my experiments seems to confirm that WPF composition events slack off then. This can be verified by counting the composition rate (Actual Composition events/sec fired).

I guess that somehow, the WPF Dispatcher + AddDirtyRect is the bane of all high performance Mogre WPF rendering.

Note: Jared, There is a pull request with my changes. But accepting it means a lot more changes need to be done to raise the full frame speed back up the previous levels.

Edit: framerate -> frame size

BTW, a word of warning to others who want to break the 60FPS speed limit. Using the known path of WPF Application object means that you will not dodge the speed limit of 60FPS. This is because when you do so, the Lock() call takes a long time to complete (upwards of couple of ms). There seems to be a difference in how WPF hosted in WinForms behaves from a WPF on native.

I spend around a day and half beating my head bloody on the WPF dispatcher. It is a no go. I repeat. WPF Dispatcher = soft VSync 60fps regardless of what else you do. (Yes even multithreading. Been there done that).

I managed to raise the Frame rate to around 500 ish. I am kind of flying on the ragged edge of what you are supposed to do with the Ogre compositor. But still, it got me a performance boost. My first implementation worked due to a bug.

I still need to raise the scene complexity to find which is ultimately a better thing to do. Blit or a quad render using the compositor. Also the compositor approach needs a patch to ogre. It needs Ogre to support the case where in a Compositor can be invoked without triggering a scene render. The final compositor I am using does not have any input at all except a texture. So to do this properly, I need to support the case of it being valid to create a viewport with null camera as long as any associated compositor chain have techniques with first pass being specified as input none.

Beauty, can we start a wiki page for Mogre+WPF to at least start dumping links in? (And organize it later as we have time?)

Nice to see all the activity on this topic.
I created a wiki page with a basic structure.
The structure is just a suggestion. You can modify it as you want:
http://www.ogre3d.org/tikiwiki/Mogre+and+WPF

It's a good idea to insert related links and to copy&paste useful information from the forum. (e.g. the 2 ways of interaction, which were described by Hormis)

The next time I have to reduce my Ogre work, because of my diploma thesis.
Just to let you know.

Now I read the whole topic.
Thank you very much for your research and documentation (especially to Hormis).
Combining the forces and ideas of different people is good for improving the WPF integration.
I'm shure this will be useful for others, too.


switched it to run using a game loop based on System.Windows.Forms.Application.DoEvents()
In 2010 there was a discussion about game loops.
There I read that DoEvents() is not good (because other applications have much power to disturb??).
They told 2 alternatives.
One is an API call (only available for .NET 4.0) and the other one is an API call of Ogre (message pump??).
I don't remember the details. If you are interested, look to the topic Gameloop etc questions. It contains good information.

Also the compositor approach needs a patch to ogre. It needs Ogre to support the case where in a Compositor can be invoked without triggering a scene render.
I suppose it's not easy to get modifications into the official Ogre sources. Especially when there is no advantage for "common Ogre users". (I think the modification is only needed for Mogre and there only for usage with WPF.)
An other option: Add an option to the MogreBuilder, which appies the needed Ogre source modifications.
(For this maybe use a patch from a modified Ogre fork??)
The advantage would be: Everybody can build a custom Mogre (e.g. newest Ogre version with the wanted options and components).
Just an idea.

Thanks Beauty. I found the post you mentioned http://www.ogre3d.org/addonforums/viewtopic.php?f=8&t=13133&start=30#p75466. Followed the links and found that it is because that DoEvents does some allocations per call and hence at high FPS can result in faster Generation 0 Garbage Collections (aka GC Pauses). My current Toy Engine switches between DoEvents and the Ogre Message Pump automatically depending on how the window was created. I guess I can accept the inconvenience of DoEvents in my editor.

I will have to look at in detail at the current compositor implementation. It may have benefits to all users in a couple of cases where you are compositing purely from RenderTargets to RenderTarget and needs no scene input at all. The current version of the DoublebufferedD3DImage uses it in that way. For now it uses an Empty Scene and a fake camera to prevent the scene getting rendered unnecessarily in the WPF compositing call.

I also rounded off my investigation today with benchmarking all solutions with Fraps to measure the FPS. And the winners are:

1. Double Buffered D3DImage Technique: Stable 30-40 FPS. Gameloop can run much faster at around 500-ish FPS.
   Mogre In WPF: Stable 60 FPS. Gameloop Speed is limited to same FPS in current implementation. May be improved by DispatcherInactive Events. Waiting confirmation.
   Overlay Window Technique: Render speed unlimited. But potential focus issues can mar the experience.
   [/list:u]
   
   When I get some time, I will summarize and dump the results in to the wiki page you created.

Beauty, thanks for creating the wiki page!

And thanks for the stats and research, hormis!

1. Double Buffered D3DImage Technique: Stable 30-40 FPS. Gameloop can run much faster at around 500-ish FPS.
   Mogre In WPF: Stable 60 FPS. Gameloop Speed is limited to same FPS in current implementation. May be improved by DispatcherInactive Events. Waiting confirmation.
   Overlay Window Technique: Render speed unlimited. But potential focus issues can mar the experience.
   [/list:u]
   

For MogreInWpf I'm not sure what you mean by gameloop speed is limited to same FPS in current implementation. I provided another sample app that lets you choose a few timing methods and I have the loop one running at 2500 FPS. I have tried DispatcherInteractive and the most I see is around 1900-2000.

Fraps never shows it going above 62FPS. (Using 'Monitor Aero desktop (DWM)')

Also, is there any reason why you would want a gameloop with a high 200+ FPS? I'm guessing there's no point in Ogre rendering any more than a certain number of FPS (Vsync? Vsync x2?).

I still need to dive into this a little more and clean up my benchmark app and try the double buffering and the layered approach.

I see what you mean about Gameloop in MogreInWpf.

I feel that double buffering is going to be useful mainly to decouple the rendering from composition events. That would be needed only if the actual execution time of the render call in the composition event is long enough that it causes a drop in framerate. Then by using double buffering you can do a cheaper copy/composite rather than a full scene render. The bigger problem there is that I haven't really been able to get it to around 60FPS, yet. May be you will have better luck.

Due to this, I am currently looking into Miyagi as an alternative for use in Game code and WPF based editor currently. If it works out, I will just use a simple WindowsFormsHost and render to that as the airspace issues arise only when you want to place WPF objects above the WinForms. For an editor, the overlay technique is fine if I ever discover a need to put WPF controls on top of the WindowsFormsHost.

Yeah I will give double buffering a shot. I'm surprised you can't get to 60FPS. Have you simplified the scene? Reduced the resolution? If you have done those two things I would think it might be a timing/timer issue.

For the wiki, I just dumped all the links I could think of and took an initial stab at hydrating with some minimal content. Feel free to totally change anything I did.

No no - you made a good job.
I just added some links, example pictures, etc.

I feel that double buffering is going to be useful mainly to decouple the rendering from composition events. That would be needed only if the actual execution time of the render call in the composition event is long enough that it causes a drop in framerate. Then by using double buffering you can do a cheaper copy/composite rather than a full scene render. The bigger problem there is that I haven't really been able to get it to around 60FPS, yet. May be you will have better luck.

I tried out your code but the part where it calls CompositorManager.Singleton.AddCompositor, I get null as a result and nothing happens. Did you have to modify Ogre? Which version are you using?

I am using the Dirext3d9Ex plugin from the "Dependencies - 11.03.21 1.7-Jared-r2.7z' on MogreInWpf. Dump all the dlls from that straight into the Binaries folder (no need for subfolders). A robocopy on the Post build copies it into the output folders. Files in my Binaries folder:

cg.dll
Mogre.dll
OgreMain.dll
OgrePaging.dll
OgreProperty.dll
OgreRTShaderSystem.dll
OgreTerrain.dll
OIS.dll
plugins.cfg
Plugin_BSPSceneManager.dll
Plugin_CgProgramManager.dll
Plugin_OctreeSceneManager.dll
Plugin_OctreeZone.dll
Plugin_ParticleFX.dll
Plugin_PCZSceneManager.dll
RenderSystem_Direct3D9.dll
RenderSystem_Direct3D9Ex.dll
RenderSystem_GL.dll
resources.cfg


Verify that the Media\Compositor\scripts\wpfoutput.compositor and Media\materials\scripts\Identity.material is being loaded. They should appear in the Ogre.log output.

10:01:37: Parsing script Identity.material
10:01:37: Parsing script Ogre.material
10:01:37: Parsing script wpfoutput.compositor


BTW I also removed the x64 build configuration with which I was experimenting with Cygon's x64 build. I pushed the changes to the repo.

If the compositor approach still does not work, you can try with the blit approach. Include the Ogre.log from failed run so I can try to figure out what is wrong and fix it.