Adding EmbeddedZipArchiveFactory to OgreZip

[Assaf Raman]

This thread is focused on a new functionality I am adding to OGRE - the option to embed zip files of OGRE resources in an exe or dll (or any type of c++ programs on other platforms).

The smallest change I was able to find to add such functionality is to use zzlib feature to replace its file io with your own implementation, I added a new file io implementation that uses char arrays in memory instead of the original io. The zip files are embedded is a smiler way to the well known file2c - meaning they are converted to from binary file to source cpp files that should be compiled with the exe\dll, the converted files has code that creates a char array with the file as its content, then add a pointer to this array to global vector of embedded files.

The functionality to embed files in the exe can be useful in many cases - like the case where you want to have a single exe that contains all its resource inside.
Moreover - I enabled the data to be also encrypted, I needed this added functionality to fool Norton - the un-encrypted embedded zip file was a false positive for a virus. You can also have a different function to decrypt the file per zip file - making things harder for the hackers(I guess?).

You can review my changes to OgreZip here.
This zip file include the modified OgreZip and OgreRoot (h+cpp), a sample zip file converted to a cpp file and the tool used for the conversion.
Review.

I think the OgreZip code changes are reasonable and adds an interesting and useful functionality to OGRE.

I only intend to commit OgreZip.h and cpp and OgreRoot changes - not the sample embedded file.
After I will clean up the code for the tool that creates cpp files from zip files - I will also add its code to our tools.

The only modification that is a small issue is that I need to add an include to zzip/plugin.h in OgreZip.cpp, this file is part of zzlib, when I created the dependencies source package for windows - I didn't copy that file to the dependencies\include folder - but the file exist in the zzlib source directory.
I will update the dependencies source package for windows - but existing ppl that use it will have to copy the file plugin.h manually from Dependencies\src\zziplib-0.13.57\zzip to Dependencies\include\zzip.

[CABAListic]

Sounds good, but we should make sure the changes don't break any other platform.

[Assaf Raman]

Shouldn't, I use only very basic c++ code, nothing complicated, this is not a big change.

[apanloco]

A few questions pop to mind,

Is only Windows PE format supported, or does this also apply to other platforms, i.e ELF/Mach-O?

How do you append Ogre resources to these executable files? Does Ogre provide utils for that? All platforms?

Also, this encryption doesn't really sound like a viable long term solution to circumvent anti-virus protection mechanism.

Thanks

[Assaf Raman]

@apanloco: There is a tool that turns zip files to arrays in cpp files, you add the cpp to your project, at runtime you can load resources from that zip file (basically a static array is the file content).
This solution is not limited to any platform - the code is not platform specific. I am not sure about the max size for cpp source files that be compiled, limits on array sizes and such - but that seems the only real limit.
Regarding the encryption - why doesn't it sound like a long term solution?

[dermont]

Not too sure which part of the code is not platform specific but the conversion tool source contains _tmain / _TCHAR which I thought was Microsoft specific.

Edit:
Ok I see, the code for the conversion tool is just an example, not the finished article.

[Assaf Raman]

Yes.
The tool is a quick fixup I created so I can work with something, the final version will be cross.
I will post it here for your review when it will be ready.

[Kojack]

The smallest change I was able to find to add such functionality is to use zzlib feature to replace its file io with your own implementation

An even smaller way is in my wiki article at http://www.ogre3d.org/tikiwiki/Merge+ex ... e=Cookbook
It embeds a zip of ogre resources in the main exe, without needing any code changes. It can even be done to existing ogre apps without recompiling.

Your way looks cool too, and is probably easier to automate into a compiler toolchain.

[Assaf Raman]

@Kojack: I am working on an OGRE NaCl port, your idea will work there - no file access.

[Assaf Raman]

I just tested if i can compile the sample browser with most resources embedded with this new way.
I wanted to create an exe that you can just copy to the computer, and you can run it - without any other files next to it.
I converted most resources to zip then cpp files, compiled OIS as a lib, created a configuration without the cg plugin (I can't static link it - and I don't want cg.dll next to my exe in this case), made some small changes in the code - so no cfg files will be required to run, linked with the old version of DX that statically link, and got this exe that runs all the samples that don't require Cg.

[Assaf Raman]

Here is a sample of two file converted from zip to cpp.
I played with the cpp formatting that the tool does so they will look nice.

[Assaf Raman]

Committed EmbeddedZipArchiveFactory and a tool to convert zip files to cpp files - to the OGRE trunk.
Review the commit here.

[boyamer]

Seams that "zzip/plugin.h" is missing into OgreZip.cpp file,i think you should upgrade Ogre Dependencies.

[Assaf Raman]

see my first post on this thread. The file is in the Dependencies just not getting copied.

you are right, I will update them today.

[Assaf Raman]

Updated, download from the first post on this thread,

[reptor]

Interesting work.


Doesn't this work in practice so that the operating system, when it loads the program, will also always load the resource files stored as arrays in the program? It may limit the use of this idea as some programs will have too much data to load all at once.

I don't know what for example Windows does with the executable file - does it load it completely always or does it page it in as required. I guess it does load all of the executable file always - but DLLs may be paged in partially? I need to learn more about this.

What about the Windows PE resources which are stored in the exe file but differently? Are they always loaded or paged in by Windows as required?

I'm also thinking that you can end up having at least two copies of the same data in RAM - one copy in the array as 'ZIP' and another copy as another array or as a different data structure. If the data stored in the array of the exe is always loaded (as I believe it to be), you can't stop this duplication of data. Furthermore, there may in fact be even more copies of the same data as some APIs may create more copies.

Just wondering! It's great to see progress being made and this is certainly a welcome addition to OGRE as far as I am concerned! But I would like to learn more about the consequences of using it.

[Assaf Raman]

All good points, I guess that you will have to limit yourself with the size there, or put the data in dlls that you load then unload.

[Kojack]

That's where my (admittedly windows only) version works well. The windows exe format only loads into memory the data it knows about. The zip appended to the end of the file is ignored during normal loading. The only time it comes into memory is when the zzip lib loads it, and should be released once the files are extracted. You can attach a gig of resources without changing the memory usage of the exe, if you don't actually use those resources.

[Wolfmanfx]

@Assaf
I recently changed a little bit code in the OgreZip impl maybe you want to incorporate it?
But first whats the problem and why i did change something:

I have a folder structure with subfolders where i put materials/programs and other stuff into it like

/myScripts/*.material
/myShaders/*.hlsl

and so on.
Then i zip the root folder and without moving everything into the root directory, so the folder structure is intact.
So what happens here when OGRE opens this zip it generates filenames which contain the folderpath at first glance it looks ok but when ogre tries to load and an *.program file or an shader it passes
the program name directly to open without the folder prefix.
And OgreZip passes the file name directly to zzip_open so this fails (because it wants the name with folder prefix).
My feeling is that this behavior is wrong at least so i patched my local version so that filenames are just filenames (not prefixed with anything) and each time on open it tries to open directly and when this fails it tries to lookup the file in zipArchive and opens it (when this file there) so everything works again.

Of course this has shortcomings too duplicate filenames and you try to open an non existing file twice but in the end i can zip a deep folder structure and everything works like i except.

There is already a thread in the forum where some guy complained about it and everybody told him that everything is working correctly only Kojack pointed out that you have to prefix filenames with the folderpath.
I dont know if anybody is interested in that behavior at all i do thats why i did

[Assaf Raman]

Well, one simple solution is to zip the file without folder paths.

[CABAListic]

Perhaps we could allow to specify folders inside the zip archive as part of the resource location. E.g. in resources.cfg:

Code: Select all

Zip=myzipfile.zip/some/sub/folder

That would be in line with how resource locations on the filesystem work. It would, however, require to split up the path by its separators, then assemble the path until you find a file on the actual filesystem, open that as the zip file and treat the rest of the path as a subpath in the zip file. This is potentially platform-specific.

[Assaf Raman]

Doesn't feel "right" to write the path with your suggested syntax.

[Wolfmanfx]

Yeah thats an simple solution but than we have an papercut again
At least we should mention in the DOCS that material parsing with shaders (and base materials) will fail when they are placed in sub folders inside the ZIP.

[Assaf Raman]

What about adding an option to the factory to control the behavior?

[Wolfmanfx]

What about reusing the option recursive in the archive? But look if you do not use folders inside your zip then you Zero overhead in the "open" function.