Random Crashing in Mogre project

JoeC

27-02-2009 10:41:09

Hello,
I've been using Mogre to develop an exhibit for a museum. It all works pretty much ok but is crashing at random intervals. So for example it ran fine for 12 days of heavy use, then it crashed. Then it crashed the next day. I have not been able to accurately reproduce or find the source of the error. When it crashes it produces an "Application had a problem" error and doesn't write anything to the log.
Currently I've tried:
- Writing to the log to find out where its crashing but all this shows is that its somewhere in the main game loop. There is no obvious behaviour from the user which is causing the problem.
- Creating a version which "plays itself" but this doesn't reproduce the crash.
- Checking for memory leaks using perfmon. Seems fine.

I have seen the error happen once on my development machine. It creates the "Application had a problem" error and gives the location as msvcr90.dll which I think is the MS Visuall C++ redistributable dll.

I'm using mogre 1.48 with these addons:
MogreNewt
OgreMagic
FSLOgreCS

This has been going on for several months and I'm really at the end of my tether. Any ideas on how to find this bug and fix it would be gratefully received.

Cheers

J

Beauty

28-02-2009 01:50:25

This is not nice ...

About an Exception called "Application had a problem" I never heard.

Here just some ideas how to find out where the problem could be.

1.
Maybe run the program from Visual Studio in Debug Mode.
Not nice, but then you could find the right place in the code (if the visitors doesn't do something after the crash.)
For this there should be no catch() that catches the exception. Because then the focus is not on the right place.

2.
Put every line of the render loop (or other suspectable parts) into a try{ ... } catch(Exception e) { <write something to logfile> } block. So you would get the right place for any exception in this blocks.

3.
Alternatively you can write an outher catch(Exception e) block that catches everything. The inner exceptions and stack information can be helpful.
For unexpected exceptions I have all of my code in such a catch block.
If there comes an exception, the following Method will be called. It collects information and return it.
This can be written to a logfile.
Maybe this helps.

    public static String InnerExeptionMoreInfo(Exception e)

    {

        String trace;

        String message = "Exeption:  " + e.Message + "\n";

        

        // in Ogre Exeptions some attributes are null

        if (e.TargetSite != null)

        {

            message += String.Format("\n    TargetSite:  {0}    ", e.TargetSite.Name);

            if (e.TargetSite.DeclaringType != null)

                message += String.Format("declared by: {0}    ", e.TargetSite.DeclaringType.FullName);

            if (e.TargetSite.ReflectedType != null)

                message += String.Format("reflected by:  {0}", e.TargetSite.ReflectedType.FullName);

        }

        if (e.Source != null)

            message += "\n  Source:  " + e.Source;

        if (e.StackTrace != null)

            trace = e.StackTrace;



        while (e.InnerException != null)

        {

            e = e.InnerException;  // eine Stufe tiefer

            

            //-- prevent problems by null references --

            String e_TargetSite_Name = "_null_";

            String e_TargetSite_DeclaringType_FullName = "_null_";

            String e_TargetSite_ReflectedType_FullName = "_null_";

            String e_Source = "_null_";

            if (e.TargetSite != null)

            {

                e_TargetSite_Name = e.TargetSite.Name;

                if (e.TargetSite.DeclaringType != null)

                    e_TargetSite_DeclaringType_FullName = e.TargetSite.DeclaringType.FullName;

                if (e.TargetSite.ReflectedType != null)

                    e_TargetSite_ReflectedType_FullName = e.TargetSite.ReflectedType.FullName;

            }

            if (e.Source != null)

                e_Source = e.Source;





            message += "\n\n--------------------"

                     + "\n  InnerExeption:  " + e.Message

                     + String.Format("\n    TargetSite:  {0}    declared by: {1}    reflected by:  {2}",

                         e_TargetSite_Name, e_TargetSite_DeclaringType_FullName, e_TargetSite_ReflectedType_FullName)

                     + "\n    Source:  " + e_Source;

            if (e.StackTrace != null)  // don't overwrite by lower levels

                trace = e.StackTrace;

        }  // while





        if (e.StackTrace != null)  // put StackTrace to end of message

            message += "\n\n  StackTrace\n  (important is first line with a line number at the end): \n" + e.StackTrace.ToString();



        return message;

    } // InnerExeptionMoreInfo()

4.
vcredist_x86.exe (2008) has a SP1. Maybe there are bugfixed things in it.
http://www.microsoft.com/downloads/deta ... layLang=en

5.
Disable shadows.
For long time I had no problem. But then I recogniced that shadows can cause a crash (when calling RenderOneFrame() ). It seems to happen only in special situations, but then reproducable.

6.
Maybe the source of problem is in the wrapper code. But this would be hard to find out (I think).

7.
If possible, maybe use an other computer in the museum.
On one computer of a friend often was an error that never happened on my 3 developing / testing computers.
Maybe this solves the problem ...

8.
Ogre 1.4.9 was released, because there were problems with a new NVidia driver.
If you use an NVidia card in the museum computer try to use older drivers (maybe after reinstalling the operation system).

9.
If you use DirectX, try to use OpenGL instead (or the other way round). It doesn't need many changes. Maybe just changing one line in a config file can do it.

JoeC

02-03-2009 17:31:28

>>
This is not nice ...

About an Exception called "Application had a problem" I never heard.
>>
Thanks for your reply and your sympathy

1) Running the exhibit within the debugger is an interesting idea. Is there anyway of automating this so that it does it automatically? I presume you could use an one of those tools which automates windows commands by recording what you do. I'll look into it but I'm concerned that as the error probably isn't in the C# code (see below) the debugger won't catch it.

2) and 3) I've already tried putting the code within a "try...catch" block and it doesn't produce anything. I think this is because the error is not within the C# code so it doesn't throw a C# exception.

4) Updating the version of the C runtime. Windows reports that the error is actually happening within this dll (MSVCRT90.dll) so this looks like a very good solution to try. I'll definitely give it a go.

5) I tried this but for some reason when I disable the shadows my sky dome doesn't reach around the full 360 degrees. Without shadows enabled there's a black gap. Still this is a good suggestion so I'll push on with it and try and fix the sky dome problem.

6) As a problem in the wrapper code would be hard to find and even harder to fix. Lets hope its not there.

7) I have seen the problem twice on my development PC but I can't replicate it. It happens when you're moving through the scene not doing anything special. Both gallery and development PCs are different models but both made by HP and both with XFX Geforce 8600 GTS graphics cards.

8 >>
Ogre 1.4.9 was released, because there were problems with a new NVidia driver.
If you use an NVidia card in the museum computer try to use older drivers (maybe after reinstalling the operation system).
>>
I was looking at this too. My drivers are version 6.14.11.7516 and this post (http://www.ogre3d.org/forums/viewtopic.php?t=41547) seems to suggest the problem only happens using openGL with versions "175.16"

9.>>
If you use DirectX, try to use OpenGL instead (or the other way round). It doesn't need many changes. Maybe just changing one line in a config file can do it.
>>
Good idea, only I'm using Ogre magic which I think is limited to DirectX. Of course the problem could be in ogre magic but I doubt it because if it was the error message would probably say it was in the ogre magic dll. As far as I know Ogre magic doesn't use the ms visual C++ dll.

The other idea I had was to change the graphics card which is currently an XFX nVidia 8600 GTS. This may be causing the problem but once again I'd expect this to cause a crash in the Direct3D dll and the error message says the crash is in the MSVCRT90.dll. What do reckon could it still be the graphics cards. I've always chosen nVidia in the past because they're more reliable but do you think its worth trying ATI?

LIke I said, many thanks for your suggestions. I'll give them a go and let you know if I'm successful.

Cheers

J

Beauty

02-03-2009 21:37:16

1)
Running the exhibit within the debugger is an interesting idea. Is there anyway of automating this so that it does it automatically? I presume you could use an one of those tools which automates windows commands by recording what you do. I'll look into it but I'm concerned that as the error probably isn't in the C# code (see below) the debugger won't catch it.

It's a little risk (because of possibly bad fingers of visitors), but I thought about starting from Visual Studio.
When my application chashes, sometimes Windows ask me, if I want to debug it. But I don't know when it asks. One time I tried. It opened VS, but more I don't remember.

There are nice script languages for automatism and little tools, also graphicals.
The names are AutoHotkeys and AutoIt. Both are similar in function, but different in syntax. AutoHotkey often confused me. (Especially the strange variable types. One is defined by "=" and an other one by ":=". For the one you have to write %varName% and for the other one without "%".). AutoIt has a syntax like C and is more committed for us. Just try it out. In the forum you'll get solutions and help.
http://en.wikipedia.org/wiki/AutoIt
http://en.wikipedia.org/wiki/AutoHotkey

4)
Updating the version of the C runtime. Windows reports that the error is actually happening within this dll (MSVCRT90.dll) so this looks like a very good solution to try. I'll definitely give it a go.

Ask in the main forum, too. Maybe it's a common problem.

5)
I tried this but for some reason when I disable the shadows my sky dome doesn't reach around the full 360 degrees. Without shadows enabled there's a black gap. Still this is a good suggestion so I'll push on with it and try and fix the sky dome problem.

I had the same problem with the black hole on the floor. Then I found out, how to change the colour of background:
Viewport.BackgroundColour = ...
Also I added fog for far distances with the same colour than the background. So there is a seemless crossing.

Instead of a sky dome maybe you can take a sky plane. With the right params it can be bend like the hut of a mushroom. These are my params:

        Plane skyplane = new Plane(Vector3.NEGATIVE_UNIT_Y, -2000.0f);

        Smgr.SetSkyPlane(true, skyplane, "Test/CloudySky", 1000, 20, true, 3.5f, 150, 150);

9)
Good idea, only I'm using Ogre magic which I think is limited to DirectX. Of course the problem could be in ogre magic but I doubt it because if it was the error message would probably say it was in the ogre magic dll.

It's not impossible. I suppose if a library uses an other library as a depency, then maybe the problem report also can point to the depency. Ok, this are only some theoretically thoughts.

The other idea I had was to change the graphics card which is currently an XFX nVidia 8600 GTS. This may be causing the problem but once again I'd expect this to cause a crash in the Direct3D dll and the error message says the crash is in the MSVCRT90.dll. What do reckon could it still be the graphics cards. I've always chosen nVidia in the past because they're more reliable but do you think its worth trying ATI?

Maybe it's related to the graphic card (and his driver). Ask in the main forum for problems related to this file. There are much more people than here in Mogre forum. It wouldn't be a mistake to try an other graphic card, but I suppose this is not the reason. Ok, I have no idea where the problem is comming from.

I would post the complete start post to the main forum.

LIke I said, many thanks for your suggestions. I'll give them a go and let you know if I'm successful.

You are welcome.
I'm happy if I can help.
Good luck!

Beauty

03-03-2009 21:18:06

There is an interesting other thread.
It seems to be possible to use PythonOgre by .NET.
So C# (or VB) applications can be modified to use PythonOgre instead of Mogre.

If you like you can read about and make a test with your application.
If so, then post it in the thread, because the "helper person" wants to get support or testing feedback from others.

Here is the thread:
viewtopic.php?p=54540#p54540

Beauty

21-04-2009 13:26:00

Instead of try / catch blocks you can write information to a logfile.
I found the logging library log4net, that is widespreaded, robust, flexible and has a good performance.
If I read right, it should write the last logs even if the main application crashes.
(Alternatively you can enable the option fileAppender.ImmediateFlush)

On Wikipedia is a summary:
http://en.wikipedia.org/wiki/Log4j

Here is the project page:
http://logging.apache.org/log4net

Most descriptions based to an XML file configuration.

Here is a nice snippet for doing it just by code:
http://www.l4ndash.com/Log4NetMailArchi ... aspx#14858

Notice:
Bekas (the father of Mogre) updated the Mogre code now. Maybe with the current Ogre version the problem is gone.
But aware - it needs some more tests. So it's better to don't use it in the museum. Only on a test computer.
viewtopic.php?f=8&t=9831

Update:
After using the linked code snippet (only initialization) you have to create a logger.

// create logger

myLogger = log4net.LogManager.GetLogger(typeof(MyApp));



// write something to the logfile

myLogger.Debug("This is a message, e.g. Problem in Line 55 of myFile.cs");