Page 1 of 4

The Ogre COLLADA plugin

Posted: Sat Sep 10, 2005 10:08 pm
by nfz
See these links for backround info on COLLADA
http://www.collada.org/public_forum/welcome.php
http://www.khronos.org/collada/presenta ... ph2005.pdf
http://www.ogre3d.org/phpBB2/viewtopic.php?t=13315

A little background as to why I chose to use COLLADA as a data interchange format between DCC tools and Ogre. For the past year I have been using my own scene format between Blender and my engine and it has worked well but during two recent demonstrations to potential clients it was strongly hinted that they would prefer to use another DCC tool other than Blender as a world editor. Yes, I was shocked too :). So I needed a data format that all the big DCC tools have agreed to use that was not proprietory, supported physics, and the programmable GPU. I had looked at COLLADA about a year ago but there was no agreement on how the physics format should be handled and there was very little talk at the time about GPU shader support. Ageia and NVidia joined the ranks of companies supporting COLLADA many months ago and the soon to be released COLLADA 1.4 specification which contains the COLLADA FX specification shows their commitment to supporting this format.

I have started work on the plugin which allows you to load scenes and all the attached resources into Ogre directly from a COLLADA compliant xml file which usually has the extension .dae. This means that you don't have to use a special Ogre mesh and scene exporter in your favourite DCC tool if it supports COLLADA export. Alias, lightwave, 3ds Max, and XSI currently have COLLADA import/exporters. I have made a basic COLLADA exporter for Blender using Reimpell's Blender dotscene exporter.

The Ogre COLLADA plugin acts as an interface between an Ogre App and the COLLADA DOM API. Currently the COLLADA DOM API is in beta testing and available only to selected delevelopers so I've sent an email requesting to be a beta tester. I haven't had much luck in the past with big organizations recognizing me as a qualified developer so in the interm I have setup my own COLLADA DOM API which uses Xerces/tinyXML until the official COLLADA DOM API is made public. For initial development I am using Xerces so I can validate incoming xml.

The purpose of the plugin is to allow the app to query for ogre resources ie meshes, materials, skeletons, animations, lights, cameras etc within a COLLADA file (.dae). The app can also query scene info so that info can be passed to a scenemanager for constructing a scene or the plugin can be instructed to build a scene automatically.

The plugin will not be tied to a specific scenemanger so maximum flexibility is maintained. To minimize app overhead you can pass a scenemanager pointer to the buildScene() method of the plugin and the plugin will instruct the scenemanager to create the required elements to build a specific scene loaded from the COLLADA file. For Physics info in the scene it will be up to your app to query the plugin for this info or if the buildScene() method is called with a physics handler callback supplied then all physics info is automatically forwarded to your callback during the scene build process. Its up to your callback to handle this physics info as required ie setting up collision objects, dynamic bodies, joints etc.

The basic structure of the parties involved:

Code: Select all

Ogre App
   |
Ogre COLLADA Plugin
   |
COLLADA DOM API
   |
COLLADA file (.dae)
Currently the COLLADA specification is at 1.3.1. I will concentrate on basic support for 1.3.1 for now which includes meshes, fixed pipeline materials, skeletons, skinning, animation, lights, cameras and of course scenes. 1.4 is supposed to come out near the end of October 2005 which is supposed to have gpu shader support ie Cg, HLSL, GLSL support along with physics(ala Novodex).

This plugin will not initially be part of the Ogre core but could be placed in OgreAddons in cvs if there is enough interest from other developers wanting to contribute. Once I get the initial ground work done and have a little demo working I will release the source/plugin/demo through a link on my website. After the plugin matures and is "usefull" we can then talk about making it an official Ogre plugin. For now its a personal project for my own needs but I am making it generic enough so it could be used in any project.

I still have some work to do on Ogre Dagon so I won't be doing full time development on the plugin until around the end of October. For now I will just devote a few hours a week to getting the frame work setup, building some tests, and a small demo.

Posted: Sat Sep 10, 2005 10:27 pm
by Olex
Wow, COLLADA is much bigger than I thought. Good news for developers in the field. Thanks for picking this up, nfz.

Posted: Sat Sep 10, 2005 10:31 pm
by sinbad
Excellent news. I can try nudging the Collada lead if you have problems getting on the beta team, they were quite keen on getting OGRE onboard in March but I was too busy to pursue it at the time.

Posted: Sun Sep 11, 2005 12:42 am
by Jezze
In the long term, does this mean bye bye to Ogres custom format?

Posted: Sun Sep 11, 2005 12:50 am
by :wumpus:
Jezze wrote:In the long term, does this mean bye bye to Ogres custom format?
No. COLLADA is an exchange format. It is great for exchanging data between modellers, fx composers, and other tools. But because it is XML it is not recommended for use as a native data format for engines. Our current mesh format is binary which is much more efficient as a realtime format.

Posted: Tue Sep 13, 2005 8:53 am
by nfz
I got the plugin up and running although it only handles very very basic stuff right now. This evening I added physics to the blender COLLADA exporter based on the xml output from other existing COLLADA exporters. It probably doesn't match the COLLADA 1.4 specification (which isn't out yet) but this is just a prototype to get my feet wet.
Image
The image at the top is the blender scene. Its comprised of some basic coder art (cube, sphere's). All the purple blocks are instanced from one mesh.

The bottom scene is running in Ogre with physics sim running (ODE + Bullet + nfzPhysics) which is why some of the objects are in a different place compared to the Blender scene. The purple lines are showing contact points (debug view). For this test the scene, meshes, physics stuff were exported all at once from Blender with a click of a button to a file COLLADATest2.dae. Everything is loaded into the Ogre App with one call to OgreCOLLADA_plugin::buildScene("COLLADATest2.dae", MOSim::physicsLoadHandler);.

Posted: Tue Sep 13, 2005 9:13 am
by BenO
congratulations

Posted: Tue Sep 13, 2005 9:17 am
by nfz
Image

I modified one of the slides from the COLLADA Siggraph 2004 presentation to show where the Ogre COLLADA plugin fits in the scheme of things. Notice how the COLLADA data store is centralized to all the tools. During development the ogre app can act as a fast path viewer by accessing resources through the COLLADA plugin.

For final Asset output the plugin uses the built in Ogre resource exporters ie meshes, skeleton, materials. For scene output I plan to just use the scene portion of the .dae file. You could use .dae for everything but they are XML format and get very large when you have everything stuffed in them so will be slow loading. My simple COLLADATest2.dae file is about 70K buts its a very simple scene with simple meshes and a little bit of physics data.

Posted: Tue Sep 13, 2005 10:03 am
by Chris Jones
this is really cool! cant wait for it to be completed, how long roughly do you think itll be till its completed or officially released?

Posted: Tue Sep 13, 2005 2:04 pm
by sinbad
Wow, you're making good progress already :shock:

Posted: Tue Sep 13, 2005 2:50 pm
by Kencho
nfz, simply great! Guys, I have a question for you: How can you get 48 hour days? For instance, you, nfz, working on Ogre and the new material system, whilke working on this plugin, while doing your everyday life... and the same for the other Ogre team members... :D

nfz, really looking forward to bitting this :)

Posted: Tue Sep 13, 2005 5:04 pm
by nfz
Sadly, 48 hour days don't exist for me either otherwise I'd be doing a lot more sailing ie more than just a few hours a day :lol: .

My progress is misleading. I leverage other people's work to get me to where I need faster. For example I used reimpell's excellent Blender dotscene exporter and Ogre XML exporter to build the COLLADA exporter for Blender. Very little of the exporter is my code, just cut and paste and rearrange some things. For the Ogre COLLADA plugin I used the OgreXMLConverter and the dot scene loader in OgreAddons cvs as references. Again a lot of cut and paste and rearranging.

Also, I have no day job or school since I'm semi-retired so I have a little more time than most to tinker and build things.

Posted: Tue Sep 13, 2005 8:19 pm
by AmitMathew
Collada seems to be more feature-rich than the FBX format. Is that true? Too bad I didn't notice Collada earlier, since I just got my basic FBX importer working...

Well, good luck with this.

Posted: Tue Sep 13, 2005 11:46 pm
by davidclifton
Will this only be supported in Dagon and beyond, or will we see something for 1.x?

Posted: Wed Sep 14, 2005 2:40 am
by nfz
davidclifton: I'm developing it with Dagon but you could back port it to Ogre 1.0.x since the source will be available when the plugin is released.

Chris Jones: first relase of the plugin + source will be Oct 31 no matter what state its in. My feature target for initial release is to have mesh loading, fixed pipeline materials, lights, camera, scene working properly.

I now have access to the COLLADA developer site and there is a plethora of test files available for conformance testing which I am starting to use now. The test files should help out with finding the bugs faster.

Posted: Wed Sep 14, 2005 5:15 am
by Praetor
This is excellent. I've tinkering with the idea of developing my own scene format, based directly of my high-level data structures. However, if this is already supported in modellers and such, I'll just wait and use it! Just make a few adaptations here and there to my data. Great work!

Posted: Wed Sep 14, 2005 8:59 am
by Slan
Great work nfz. Since I'm currently working on the scene management framework for our project (currently expanding .scene format) it could be nice to share ideas / code. Collada has always been our target as a standard file format for asset description but I only spent very little time browsing the official spec. Is there a chance I can get my hands on your code and help with debugging / implementation ?

Posted: Wed Sep 14, 2005 12:25 pm
by Chris Jones
i too was goin to make my own scene format to incorporate graphics/physics/ etc but ill probably just wait for this instead

Posted: Wed Sep 14, 2005 1:45 pm
by davidclifton
nfz wrote:davidclifton: I'm developing it with Dagon but you could back port it to Ogre 1.0.x since the source will be available when the plugin is released.
I haven't had the opportunity to take a look at Dagon, since our requirements demand we work with a stable release...you're obviously very experienced with both releases, so if possible I would appreciate your input on how big of a task you feel this port backwards would be? i.e. how much of the more advanced functionality of Dagon does this make use of?

Posted: Wed Sep 14, 2005 1:58 pm
by Praetor
So, with Collada. You said this is an exchange format, right? Looking back, it may not be perfect for scene management within a game. What I mean is, during content creation that's great, but perhaps at a later time, the whole thing should be converted to some other format? Already, the meshes and animation and materials are converted out. With a large scene, it may be best to just make an engine-specific collada converter. Collada seems great to streamline the content creation phase, it just doesn't seem up to the task of a game runtime format.

Posted: Wed Sep 14, 2005 7:56 pm
by nfz
Praetor: for scene runtime format, an optimized binery format would definately be faster to load than trying to load from an XML file. Yes, the use of COLLADA files are more suited for the development phase. For final asset production you then take the COLLADA files and export them to the optimized binary formats.

davidclifton: I'll see what I can do to make the plugin compatible with 1.0.x. Right now it won't be hard since the plugin is very basic right now.

Posted: Wed Sep 14, 2005 8:55 pm
by Praetor
Let me explain further, nfz. We all agree Collada is a great thing for asset production. Swapping between editors is easy now. But when it's time to put it into the game, would it be better to a) create a binary Collada format, where each xml construct is simply binary-ized with some converter, or b) the converter reads in the collada and translates it to some scene format designed for your game. I guess the difference is a) make your game around collada scene format or b) convert collada to game's format.

Since Collada is a resource format content creation, it may not translate well directly to the game's needs, and often these needs can't be twisted to fit. For instance, Collada has physics and graphics information built-in. An "entity" is some object in the format that has graphics and physics info. But what about per-object scripting? Can that be added in? Or would that be a seperate file altogether. And then what about an object's sound? At some point it seems that Collada should be left for content creation only, and when it's time to put it into the game it gets rolled up along with all the other info and game object needs into your own format.

I think Collada may be greatest thing for content creation in recent memory, but I'm suddenly not convinced of its usefullness beyond that point. Of course, it doesn't have to be useful beyond that point, i just want to be sure.

Posted: Wed Sep 14, 2005 9:58 pm
by sinbad
As previously pointed out, COLLADA's primary role is as an interchange format. I fully expect that people will use COLLADA direct for development and perhaps smaller projects, and precompile it in some fashion for larger, final builds.

Just having an intermediate format like this, and some good tools to quickly view it in development or in non-startup-time-critical projects, is a very good thing indeed. We have massive problems getting data out of some modellers because we can't own them - but the major modellers have precreated, and most importantly validated, exporters.

Remember our dotscene format is quite similar, except that it references the larger binaries like .mesh rather than including them in the XML. I see no reason why you couldn't extend COLLADA (which is allowed in controled fashions) to form a 'mostly optimised' format like dotscene, where instancing and other 'lighter' data is still in XML but the chunky data is pulled out and referenced instead. Your viewer could in theory read in original COLLADA and turn it into 'specialist COLLADA', for example, or better have the option to view either - that's a very powerful combination for fast DCC turnaround and optimised finals. My Kadath project is also very likely to take COLLADA as an input to the compile process, since it's a great way to get data out of modellers fast. And whilst I might compile large chunks of it, there could well be other data I want to leave alone.

All in all, good support for COLLADA in various forms is a significant boon. Sure, there are pros/cons to all systems, but being a 'hub' of data like this promotes further development in other more specialist areas. Just look at how fast other mesh tools sprung up as soon as the XMLConverter came about.

Posted: Thu Sep 22, 2005 6:18 pm
by nfz
Some NVidia presentations on how they are supporting COLLADA in their tools for developers: http://developer.nvidia.com/object/iron ... tions.html

Posted: Thu Sep 22, 2005 6:37 pm
by Olex
What is even more imporant to note is that NVidia acutally replaces some of their formats with COLLADA as being the main one.