new in 1.7- resources without a unique name are now possible

Discussion area about developing or extending OGRE, adding plugins for it or building applications on it. No newbie questions please, use the Help forum for that.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

new in 1.7- resources without a unique name are now possible

Post by Assaf Raman »

I want to suggest a small change to the way we load resources in OGRE, I made this change to my local code and I think it can benefit other people then me.

In the application I am writing I can select a model from the disk and load it.
Each model you select is added to the scene and you can place it wherever you want.

Each time I load a model I first create resource group named the same name as the path of my model, then I add the same path with add resource location to that group, initialize the resource group and load the model.

My problem is that all the models, textures and materials are added to a global pool, and if I open a new model with the same name or that uses a texture with the same name - I see the other model\texture that was loaded before - I always see the first resource that was loaded.

The reason is that we have a global pool of resource names that disregards the resource group while loading a resource.

I am suggesting that there will be an option to load a resource and all its dependants' resources from a specific group.

Then - I will be able to create an entity that uses a model with a non-unique name - but with a unique group+name, and all its dependencies.

The change needed is very small - mainly in ResourceManager. You will still be able to load a resource from the "global" resource pool by using AUTODETECT_RESOURCE_GROUP_NAME that already exist.

What do you think?
Last edited by Assaf Raman on Sat Sep 06, 2008 2:41 pm, edited 6 times in total.
Watch out for my OGRE related tweets here.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

Resource loading issues is something I've been rolling around in my head for 1.8. I want to really smooth up the resource loading and handling mechanisms in Ogre without making it too complicated or slow.

Basically it seems like you're proposing switching from a map of names to resources, to a map of names to resource groups, which contains a map of names to resources?
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

Yes, but I want to keep the old map - as a fallback. If you won't be able to load from the current group - it will load from the global pool.
Watch out for my OGRE related tweets here.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

I can create a patch for your review... Then if you like it - we can commit.
Watch out for my OGRE related tweets here.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

I like the idea. My thinking has been running along the lines of being reactive, meaning letting people resolve resource issues when they arise. This I think is more straightforward for Ogre (simpler, less error-prone, faster). This is proactive, meaning the user avoid resource name collisions by using resource groups ahead of time. I think it elevates the concept of the resource group to be a little more concrete and perhaps even more intuitive. Now, it can be even more a discrete collection of resources, which can be loaded and exist conceptually separately from others.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

I will post a patch later today on this thread.
Watch out for my OGRE related tweets here.
User avatar
xadhoom
Minaton
Posts: 973
Joined: Fri Dec 28, 2007 4:35 pm
Location: Germany
x 1

Post by xadhoom »

Praetor wrote:Now, it can be even more a discrete collection of resources, which can be loaded and exist conceptually separately from others.
Because I thought this would be already the case I would appreciate the change. :P
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

I want to make it so you can declare for each group when you create it - whether it will go into the global pool or not.
When trying to load a resource - first - it will try to load from the group hash, if it is not found there - it will try from the global pool. This way - you can share some of the resources among groups and not share others. Am I missing a case?
Watch out for my OGRE related tweets here.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

I think this is an expansion on the current system. So everything you could do before sounds like you can do it now.
User avatar
KungFooMasta
OGRE Contributor
OGRE Contributor
Posts: 2087
Joined: Thu Mar 03, 2005 7:11 am
Location: WA, USA
x 16
Contact:

Post by KungFooMasta »

Assaf Raman wrote:I want to make it so you can declare for each group when you create it – whether it will go into the global pool or not.
When trying to load a resource – first – it will try to load from the group hash, if it is not found there – it will try from the global pool. This way – you can share some of the resources among groups and not share others. Am I missing a case?
Maybe I'm misunderstanding. In ogre you can do something like:

Code: Select all

mSceneManager->createEntity("my name","ninja.mesh");
If you have 2 different resource groups and each have "ninja.mesh" as a resource, what happens in this case? Wouldn't you need something like this?

Code: Select all

mSceneManager->createEntity("my name","ninja.mesh","my resource group 2");
Creator of QuickGUI!
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

@KungFooMasta: Try for yourself - here are 2 models with the same name - create a sample that shows them side by side.

If you can load them both and show them side by side (without changing the model name) - my code is not needed.

[EDIT] Better if you use these models[/EDIT]
Last edited by Assaf Raman on Sat Aug 02, 2008 2:41 am, edited 1 time in total.
Watch out for my OGRE related tweets here.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

Here is my patch.

You can use this sample code to test it out. Here are the models for the samples. You will only need to change the model path in the code.

I will be happy if you can review it. I worked on it for a few hours.

[EDIT] Better if you use these models[/EDIT]
Last edited by Assaf Raman on Sat Aug 02, 2008 2:41 am, edited 1 time in total.
Watch out for my OGRE related tweets here.
reptor
Ogre Magi
Posts: 1120
Joined: Wed Nov 15, 2006 7:41 pm
Location: Finland
x 5

Post by reptor »

In cases when we want to allow users of our applications to add data to the application (think about people who like to mod games) it is really a requirement that resources with the same name must be able to exist and be loaded at the same time.

Definitely we don't want to introduce some stupid rules for the users like "you must first check what resource names the game already originally uses and not use any of those" or "don't use resource names which have been used by other modders!" etc... what a nightmare that would be.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

I found a small error with the in the models - the name of the material was different between the models - so here is an updated version of the models with exact same mesh \ texture \ skeleton \ material names but different data - this way you can see that all of the different resource types are the still with the same name but loaded correctly.

SO - basically - before my patch - if you tried loading the models - you would have seen two instances of the same model.
Image
After my patch - you would see two different models.
Image
My patch doesn't affect any exiting code as I far as I can see. I recommend applying it. I will wait for other OGRE team members to confirm - then I will commit.
Watch out for my OGRE related tweets here.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

It's really late here. I'll apply the patch tomorrow and check it out.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

@Praetor: Note - I found out along the way that we don't support material names with spaces (This is unrelated to this patch).
Watch out for my OGRE related tweets here.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Sorry for not pitching in earlier, I've been without internet access for a few days.

I don't think we should be looking at this from the standpoint of placing a resource into a 'global pool'. By using a 'global pool' option you force both ends of the contract - resource loading and resource referencing - to agree, and I think this could be tricky to use in a more general context.

Taking a step back, I think the core of this requirement is allowing multiple ambiguous resource names to be present, each referring to different resources, and for definitions that are 'closer' to the referring site to be preferred, while still allowing a fallback to other groups - not just to the general group, but to other groups too. This was the intention of AUTODETECT_RESOURCE_GROUP_NAME, not just a global fallback.

So, there are 2 parts to this - 1) dealing with the naming ambiguity, and 2) dealing with priority.

I think the simplest and most robust way to manage this is to change the ResourceMap target to a hash_multimap instead of a regular hashmap. Where there is ambiguity, the resource group to which each resource belongs can be checked, and the closest one used (preference order: 1. the same group, 2. the general group, 3. any other group). Obviously on creation we must prevent the same name being used twice for the same group, and also if there is a changeResourceGroup call. While we could just nest the list under resource groups, retrieving resources in the standard case would be slower than using a multimap on name.

Obviously this is slightly more code-disruptive than your method, since iterator types change and lookup methods need to change to use equal_range() instead of find(). However, I think it's more in keeping with the intention, avoids the need for a new 'global pool' definition which I don't think is needed, and if we're changing a core paradigm (the globality of resource names) then I'd rather we adhered as closely to the new concept as possible, than to cut corners in favour of changing less code.

The final danger here is that it makes getByName and other retrieval methods non-deterministic. You could in theory call getByName multiple times and get different pointers back, if a 'closer' load with the same name has occurred in between. This is an important behavioural change and will need time to bed in to see the practical implications in people's projects.

If it wasn't already obvious, this is definitely a 1.8 change.

Let me know what you think.
Assaf Raman wrote:@Praetor: Note - I found out along the way that we don't support material names with spaces (This is unrelated to this patch).
Yeah we do - you just have to enclose them in quotes if it's in a script. There's at least one in Example.material.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

Multimap is a good idea.
Regarding the rest – I didn't understand how would you load then show side by side the two sample models (look above). Can you demonstrate this small case? I will be willing to write the code needed but I need to understand the result you want.
Watch out for my OGRE related tweets here.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

There happened to be a festival right near my house this weekend, so I found myself neglecting my computer.

The more complicated case definitely seems to get really complicated. It seems like from now on you'd have to lug around a resource name and resource group for everything. getByName would become very dangerous to use. Handling the more complicated case was what I was trying with the resource listeners. Instead of actually allowing same-named resources, we give the user an opportunity to step in and rename one of the conflicting resources (or remove the old one, or stop the new one from coming in) before the full loading takes place. But this has its own problems like once things start getting renamed it is difficult for the user to keep track of them.

I like assaf's approach for its simplicity, but there do seem to be gaps in its coverage.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

Well, my approach has the benefit of being a simple solution for this issue and it doesn’t affect any existing code. The new code is active only when you add a group that is not in the "global pool". So – my solution is very easy to integrate to exiting code bases and doesn't affect any existing code base. Moreover – I think the idea of having some global resource and some group specific resources that can only be used within the group – is a good idea.

What are the gaps in its coverage? Can you give me a clear sample of the gap?
Watch out for my OGRE related tweets here.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Well, the issue about non-determinancy with the multimap approach only becomes an issue if you create more than one resource with the same name in different - which cannot happen in the current code base anyway. Therefore this is actually no danger to existing code, because it could not have existed before. Therefore the multimap approach breaks no more active code than Assaf's way.

The reason I don't like the global pool idea is that it introduces confusion between the default resource group and the global pool - the classic 'parallel idiom' problem. I can imagine users asking what the difference between the default 'General' resource group the global pool is. You also make it possible to share resources only in one location (and retain name freedom), the single 'global pool'. I think if we're going to do this, let's do it holistically and not take the easy (in terms of less code changes) but more conceptually confusing route. There should really just be one set of common, non-overlapping rules here.

In terms of recreating your example, the short answer is that it's no different, from an API perspective. Having loaded a same-named mesh with a different group, you will have 2 entries in the multimap, each with a different group membership. When you come to create the Entity, equal_range() will pick up both of them, and disambiguation will be based on the group name - if it matches, that's the preferred one.

As an example of possible gaps, one issue with both approaches is knowing whether a 'closer' resource exists in the group resource path or not. If you've already loaded a mesh with the name 'test.mesh' from group A, but then you try to create an entity referencing that mesh name from group B, how do you know whether 'test.mesh' just doesn't exist in group B's file paths, or that it just hasn't been loaded yet? If it's the former, you'd have to go looking for it in group B's resource paths every time an Entity with that group asks for it, compared to the current system where a resource with a name either exists or it doesn't. The alternative is to put a null entry in for group B when you try to find it and fail, just so that you can efficiently determine that next time.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

I added a group parameter to getByName and createEntity - this way you know what mesh you are loading. It has a default value - the old behavior. Why is this non-deterministic?
Watch out for my OGRE related tweets here.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Post by Assaf Raman »

How do we move on? What should I do?
Watch out for my OGRE related tweets here.
User avatar
KungFooMasta
OGRE Contributor
OGRE Contributor
Posts: 2087
Joined: Thu Mar 03, 2005 7:11 am
Location: WA, USA
x 16
Contact:

Post by KungFooMasta »

Sorry for late reply, been busy lately. I didn't get to try the meshes, but I think you guys stumbled onto what I was getting at. You'll need to allow the name of the Resource Group in the API call for creating all resources, so that users will know which resource they're using.
Creator of QuickGUI!
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Assaf Raman wrote:I added a group parameter to getByName and createEntity - this way you know what mesh you are loading. It has a default value - the old behavior. Why is this non-deterministic?
It's still deterministic the way you've done it, specifically because you've added the new concept of a global pool, which is what I don't like.

The non-determinacy of the multimap approach is not that important, I was just pointing it out - it will only occur with the dynamic fallback if something is loaded in between. The same kind of non-determinacy happens if you unload, remove, & recreate a resource anyway.

As for where now, it's up to you. I don't like the new 'global pool' concept because I think it adds confusion when there's already a general resource group, and falling back on only the global pool is not what was intended by the AUTODETECT_RESOURCE_GROUP concept originally - it should be able to find a resource in any group. I think the alternative I proposed has the same ultimate effect without introducing another core resource concept and the combinational complexity that entails. There is still the issue with both techniques about knowing whether a resource at a closer scope than the 'fallback' that's already loaded exists or not, without having to go check every time. This is a performance problem with both ways and I'm not sure how you address it without creating an entry somewhere saying 'I've checked the group you asked, and there isn't a resource there, you have to use the general one'. Because you don't know whether the resource with that name that is loaded was a fallback, or the first-choice in that other group.

However if you want to do it your way for now, and I've just confused you with my alternative, by all means do it as a patch and people can try it out & comment. When I get some time I can implement it the way I'm thinking and we can compare, although I don't know when that will be yet.
Post Reply