Custom way to load resources (custom in-engine Asset System)

Discussion area about developing or extending OGRE, adding plugins for it or building applications on it. No newbie questions please, use the Help forum for that.
Post Reply
hydexon
Gremlin
Posts: 164
Joined: Sun Apr 14, 2013 8:51 pm
x 10

Custom way to load resources (custom in-engine Asset System)

Post by hydexon »

Since my engine i'm working is based in the Helium game engine https://github.com/HeliumProject/Helium/(a bit abandoned tho, but the asset system approach is kinda good, so i'm going to reuse it) but the problem i'm not truly deep about the OGRE's ResourceSystem and since the asset system has no Archive-like API and has their own background loading system, and for every resource class must inherit from a class called Resource (https://github.com/HeliumProject/Helium ... Resource.h) to access the requested data from the asset system.

I didn't find much information about the internals of the Resource System and also i don't want to use Ogre::Archive for it, already have it as workaround and was it a mess. And some examples of custom resource loaders are too old (around 2005). And a shame of the 2012 attempts to make as optional component to separate the buggy ResourceSystem

There's some way to load my data using a custom data loading system without using an Archive-like API?
frostbyte
Orc Shaman
Posts: 737
Joined: Fri May 31, 2013 2:28 am
x 65

Re: Custom way to load resources (custom in-engine Asset Sys

Post by frostbyte »

http://www.ogre3d.org/tikiwiki/tiki-ind ... e=Snippets
keyword is "loading": ctrl-f+"loading" ("Loading Meshes From Any Path " etc )
also http://www.ogre3d.org/tikiwiki/tiki-ind ... ce+Loading
the forum contain some info...
as last resort you can always dive into ogre's source code and try figuring out how its done...
the woods are lovely dark and deep
but i have promises to keep
and miles to code before i sleep
and miles to code before i sleep..

coolest videos link( two minutes paper )...
https://www.youtube.com/user/keeroyz/videos
hydexon
Gremlin
Posts: 164
Joined: Sun Apr 14, 2013 8:51 pm
x 10

Re: Custom way to load resources (custom in-engine Asset Sys

Post by hydexon »

Nice, it's a good starting point, at least i avoided to hack the nasty Resource System to cooperate with mine, but seems i need to cover HLMS Shaders, plain shaders and some textures. Regarding OGRE Scripts such old v1 materials or skeletons maybeare covered it by using those snippets, but i wonder about compositor scripts and textures.
User avatar
Zonder
Ogre Magi
Posts: 1168
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 73

Re: Custom way to load resources (custom in-engine Asset Sys

Post by Zonder »

Don't forget the GSoC as well http://ogre3d.org/forums/viewtopic.php? ... t=resource

I believe it was incomplete why it didn't get merged but it might be worth a look
There are 10 types of people in the world: Those who understand binary, and those who don't...
frostbyte
Orc Shaman
Posts: 737
Joined: Fri May 31, 2013 2:28 am
x 65

Re: Custom way to load resources (custom in-engine Asset Sys

Post by frostbyte »

sorry i did'nt realize you're using ogre2.1
textures- very easy- you can load a texture directly from image or memory( search the forums )
HLMS- that should be pretty easy since you can now import/export from/to a json file( example on spookyboo's 'Magus' editor )
i also suggest you download DERGO https://bitbucket.org/dark_sylinc/dergo-blender
DERGO uses tcp/ip for loading/updating meshes and HLMS materials in real-time from blender but you can easly adapt/use as ref Code
for compositors... just "rip" at the compositors scriptParser class implementation and use/reuse that
for HLMS specific questions( shaders etc ), i suggest you use the OGRE2.x forums thread...
if you end-up with something worthy( and independent of the helium project ) please share...
goodluck (-:
the woods are lovely dark and deep
but i have promises to keep
and miles to code before i sleep
and miles to code before i sleep..

coolest videos link( two minutes paper )...
https://www.youtube.com/user/keeroyz/videos
hydexon
Gremlin
Posts: 164
Joined: Sun Apr 14, 2013 8:51 pm
x 10

Re: Custom way to load resources (custom in-engine Asset Sys

Post by hydexon »

So after a loong research both DERGO code how to load meshes and HLMS materials in Blender could be useful but still needs the Resource Manager (but nothing impossible to make independient of it). i have to take a look for compositor script parsers and others to reach my full independency of the OGRE's ResourceManager.

It's a real shame a lot of people wanted to remove the resource manager due their flaws.
frostbyte wrote:
if you end-up with something worthy( and independent of the helium project ) please share...
goodluck (-:
What did you mean?
frostbyte
Orc Shaman
Posts: 737
Joined: Fri May 31, 2013 2:28 am
x 65

Re: Custom way to load resources (custom in-engine Asset Sys

Post by frostbyte »

download worthy - as in useful/working
independet of the helium project ~= if only the resource system can be used and not all of the helium framework mambo-jumbo
goodluck...as in... i once read an old forum thread about some guy trying do something similiar - we'll...it was'nt easy as it sounds( never is... )
share...as in share code, snipplets, knowledge whatever...
we all suffer from the current resource system (-:
hopefully, once 2.1 is released gsoc2012 work will be fixed and merged, but for now that's a low priorty feature...
the woods are lovely dark and deep
but i have promises to keep
and miles to code before i sleep
and miles to code before i sleep..

coolest videos link( two minutes paper )...
https://www.youtube.com/user/keeroyz/videos
hydexon
Gremlin
Posts: 164
Joined: Sun Apr 14, 2013 8:51 pm
x 10

Re: Custom way to load resources (custom in-engine Asset Sys

Post by hydexon »

I didn't much yet, just removed some dead code, as well i did a complete fork of the project, i did some rebranding in the namespaces tho, updated the RapidJSON from an older version (an stable release from google code) to one from GitHub (1.2.0), Removed BSON (needed an really old version of BSON Mongo driver), Killed the Premake4 build system to CMake, removed the old incomplete DX9 renderer to OGRE (in early progress) and editor to Qt 5. i did some changes to the entity system to make it friendly to the editor.

Since the Asset System and their loaders is the most complicated part of the project since has trouble regarding resources and since there's no direct way to serialize data to an asset file, so i'm into trial and error, and OGRE resources is another challenge from the Asset System.
hydexon
Gremlin
Posts: 164
Joined: Sun Apr 14, 2013 8:51 pm
x 10

Re: Custom way to load resources (custom in-engine Asset Sys

Post by hydexon »

Later i somewhat workaround some resources without relying absolutely of Resource Manager i found some issues with the Helium Asset System and oh boy what issue i got thanks to the Resource system.

* Textures: Material, Compositor scripts can reference textures by its name, those will be loaded by the Resource Manager using their registered location UNLESS you're created one previously so my Material class who inherits Helium "Resource" class needs to keep reference of those mentioned textures, creating an Texture class for the Helium asset system to create it when is loaded, and i have to create manually with a lot of parameters required by Ogre::TextureManager::createManual and thus i have to duplicate code to keep OGRE and Helium happy.

* HLMS Shaders: Partially uses the Resource Manager, such using "archives" APIs, which Helium doesn't have unfortunately, and can be worse if the shader uses "include" macros.

* Shaders: Not a major issue, mostly can be compiled from string, but can have the same issue with both legacy and HLMS materials.

and as i thought, both Helium and OGRE's Resource System it's a huge PITA, since my old and simple projects used the resource manager since they had no serious architecture, but when i saw Helium and found it really interesting as my architecture for an serious project, now i see why everyone hates the current OGRE's resource system, scrawl's move from OGRE to OSG, and he praised of OSG leaves the user of how the resources should be loaded, i saw the ManualResourceLoader class but i see almost no examples of how to use it well.

Sincerely this flaw makes me to turn away OGRE and see other alternatives such Visualization Library or OpenSceneGraph, until OGRE takes away the current Resource System. OR adapt Helium to have an Archive-like API, removing his "key-value"-like database system to access serialized data and binary resources to an archive library (minzip or StormLib), could be nice but the Helium's true asynchronous loading would be reduced to almost useless, used only to access serialized data from Reflect libraries.

I choose Helium as their base architecture, because I SUCK, really..., SUCK at creating scalable game engine architectures (not because or lack of programming knowledge, well a bit but nothing serious, mostly are by decisions-related) and Helium is the one which doesn't rely heavily of their internal rendering engine and all their code is inside into the core/engine code.


Cheers.
User avatar
Thyrion
Goblin
Posts: 224
Joined: Wed Jul 31, 2013 1:58 pm
Location: germany
x 8

Re: Custom way to load resources (custom in-engine Asset Sys

Post by Thyrion »

hydexon wrote:OGRE's Resource System it's a huge PITA
Did you look at the "Resource System Redesign"?
http://www.ogre3d.org/forums/viewtopic.php?f=13&t=77193

Would this solve your problems?
hydexon
Gremlin
Posts: 164
Joined: Sun Apr 14, 2013 8:51 pm
x 10

Re: Custom way to load resources (custom in-engine Asset Sys

Post by hydexon »

Thyrion wrote:
hydexon wrote:OGRE's Resource System it's a huge PITA
Did you look at the "Resource System Redesign"?
http://www.ogre3d.org/forums/viewtopic.php?f=13&t=77193

Would this solve your problems?
The second post of this thread said the same, shame those changes would be never in the 2.1 series, even in the 1.x, that killed a lot of hopes, the same happened with the global illumination GSoC project.
Post Reply