Does OGRE supports non english characters in file path

[shiv_jpr]

Hi,

I am new to Ogre. I tried using Ogre SampleBrowser application on my win 7 machine (japanese locale) I have ogre.cfg file placed in "C:\\Users\\客人\\IEApplication\\ogre.cfg" which i gave to Ogre:;Root().

Ogre::Root() failed to understand this path and hence could not load the configuration file placed in the user location. This issue is not observed if the system locale is english on if the path contains only english characters. but I have no choice as the username is japanese text (客人) hence %temp% path must have non english characters.

Any help is deeply appreciated as I am stuck for long and have no solution to try now. Please let me know if you need further details.

Shiv

[dark_sylinc]

Hi,

By default, unicode path support is a PITA on Windows.
You could try compiling with the Unicode setting enabled (Alt+F7 General->Character Set; set it to "Use Unicode Character Set"), though I admit I haven't tried doing that myself.

A workaround is to use OEM codepage conversion (see this snippet) which highly broadens support for non-english characters.
However, it won't always work (it won't work if the path contains characters from different codepages; like say... Cyrillic and Japanese; or Chinese characters and the character Ñ most common in Spanish and Portuguese languages; or Traditional Chinese mixed with Simplified Chinese characters).

FileSystem is overrideable and technically you could provide an implementation that supports unicode characters. Though often our users use this override for custom file formats (i.e. place all files and folders in a "paq" file, which is a custom format)

May be other users could provide you with better replies than this one. Sadly as a western I'm not very used to dealing with filename localization problems.

[Crashy]

We've just fixed a bug related to this in BOMB, with a user having a thailandese windows user name.

This was quite tricky to fix.
Ogre doesn't build when the character set is set to "unicode", so we had to do a few hacks.

First, we've changed the FilesystemLayer class to this:

Code: Select all

/*
 -----------------------------------------------------------------------------
 This source file is part of OGRE
 (Object-oriented Graphics Rendering Engine)
 For the latest info, see http://www.ogre3d.org/

 Copyright (c) 2000-2012 Torus Knot Software Ltd

 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:

 The above copyright notice and this permission notice shall be included in
 all copies or substantial portions of the Software.

 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 THE SOFTWARE.
 -----------------------------------------------------------------------------
 */
#ifndef __FileSystemLayer_H__
#define __FileSystemLayer_H__

#include "OgreString.h"
#include "OgreUTFString.h"

#ifdef WIN32
#   ifndef PlatypacEngine_EXPORTS
#       define PLAT_DLL __declspec(dllimport)
#   else
#       define PLAT_DLL __declspec(dllexport)
#   endif
#else
#   ifndef PLAT_DLL
#       define PLAT_DLL
#   endif
#endif

namespace OgreBites
{
	/** Provides methods to find out where the Ogre config files are stored
	    and where logs and settings files should be written to.
		@remarks
			In modern multi-user OS, a standard user account will often not
			have write access to the path where the SampleBrowser is stored.
			In order to still be able to store graphics settings and log
			output and for the user to overwrite the default Ogre config files,
			this class tries to create a folder inside the user's home directory.
			Specialised implementations for each individual platform must
			subclass this abstract interface.
	  */
	class PLAT_DLL FileSystemLayer
	{
	public:
#ifdef WIN32
		typedef Ogre::UTFString FSString;
#else
		typedef Ogre::String FSString;
#endif

		virtual ~FileSystemLayer() {}

		/** Search for the given config file in the user's home path. If it can't
		    be found there, the function falls back to the system-wide install
			path for Ogre config files. (Usually the same place where the
			SampleBrowser resides, or a special config path above that path.)
			@param
				The config file name (without path)
			@return
				The full path to the config file.
		 */
		virtual const FSString getConfigFilePath(FSString filename) const = 0;

		/** Find a path where the given filename can be written to. This path
			will usually be in the user's home directory. This function should
			be used for any output like logs and graphics settings.
			@param
				Name of the file.
			@return
				The full path to a writable location for the given filename.
		 */
		virtual const FSString getWritablePath(const FSString& filename) const = 0;
	};
}


#endif

As you see, we've replaced the standard Ogre::String to a typedef named FSString which is an UTFString on WIN32 (we didn't changed the unix implementation as we didn't want to break anything)

That done, here are your FileSystemLayerImpl files:
.h:

Code: Select all

/*
 -----------------------------------------------------------------------------
 This source file is part of OGRE
 (Object-oriented Graphics Rendering Engine)
 For the latest info, see http://www.ogre3d.org/

 Copyright (c) 2000-2012 Torus Knot Software Ltd

 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:

 The above copyright notice and this permission notice shall be included in
 all copies or substantial portions of the Software.

 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 THE SOFTWARE.
 -----------------------------------------------------------------------------
 */
#ifndef __FileSystemLayerImpl_H__
#define __FileSystemLayerImpl_H__

#include "FileSystemLayer.h"
#include "OgrePrerequisites.h"
#include "OgreStringVector.h"

#ifdef WIN32
#	include "OgreUTFString.h"
#	include <io.h>
#else
#   define _access access
#endif

#ifdef WIN32
#   ifndef PlatypacEngine_EXPORTS
#       define PLAT_DLL __declspec(dllimport)
#   else
#       define PLAT_DLL __declspec(dllexport)
#   endif
#else
#   ifndef PLAT_DLL
#       define PLAT_DLL
#   endif
#endif

namespace OgreBites
{
	/** Implementation for the FileSystemLayer interface. */
	class PLAT_DLL FileSystemLayerImpl : public FileSystemLayer
	{
	public:
		/** Creates a concrete platform-dependent implementation of FileSystemLayer.
		  	@param
				A subdirectory inside the user's path to distinguish between
				different Ogre releases.
		 */
		FileSystemLayerImpl(const Ogre::String& projectName, const Ogre::String& subdir)
		{
			// determine directories to search for config files
			getConfigPaths();
			// prepare write location in user directory
			prepareUserHome(projectName, subdir);
		}

		const FSString getConfigFilePath(FSString filename) const
		{
#if OGRE_DEBUG_MODE == 1 && (OGRE_PLATFORM != OGRE_PLATFORM_APPLE && OGRE_PLATFORM != OGRE_PLATFORM_APPLE_IOS)
			// add _d suffix to config files
			Ogre::String::size_type pos = filename.rfind('.');
			if (pos != Ogre::String::npos)
				filename = filename.substr(0, pos) + "_d" + filename.substr(pos);
#endif
			// look for the requested file in several locations:

			// 1. in the writable path (so user can provide custom files)
			FSString path = getWritablePath(filename);
			if (fileExists(path))
				return path;

			// 2. in the config file search paths
			for (size_t i = 0; i < mConfigPaths.size(); ++i)
			{
				path = mConfigPaths[i] + filename;
				if (fileExists(path))
					return path;
			}

			// 3. fallback to current working dir
			return filename;
		}

		const FSString getWritablePath(const FSString& filename) const
		{
			return mHomePath + filename;
		}

		void setConfigPaths(const Ogre::StringVector &paths){
			mConfigPaths = paths;
		}

		void setHomePath(const FSString &path){
			mHomePath = path;
		}

		/** Create a directory */
		bool createDirectory(const FSString& name);

	private:
		Ogre::StringVector mConfigPaths;
		FSString mHomePath;

		/** Determine config search paths. */
		void getConfigPaths();

		/** Create an Project directory and the given subdir in the user's home. */
		void prepareUserHome(const Ogre::String& projectName, const Ogre::String& subdir);

		/** Test if the given file exists. */
		bool fileExists(const FSString& path) const;
	};
}

#endif

.cpp:

Code: Select all

/*
 -----------------------------------------------------------------------------
 This source file is part of OGRE
 (Object-oriented Graphics Rendering Engine)
 For the latest info, see http://www.ogre3d.org/
 
 Copyright (c) 2000-2012 Torus Knot Software Ltd
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 
 The above copyright notice and this permission notice shall be included in
 all copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 THE SOFTWARE.
 -----------------------------------------------------------------------------
 */
#include "FileSystemLayerImpl.h"
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <shlobj.h>
#include <io.h>

namespace OgreBites
{
	void PLAT_DLL FileSystemLayerImpl::getConfigPaths()
	{
		// try to determine the application's path
		DWORD bufsize = 256;
		char* resolved = 0;
		do
		{
			char* buf = OGRE_ALLOC_T(char, bufsize, Ogre::MEMCATEGORY_GENERAL);
			DWORD retval = GetModuleFileName(NULL, buf, bufsize);
			if (retval == 0)
			{
				// failed
				OGRE_FREE(buf, Ogre::MEMCATEGORY_GENERAL);
				break;
			}

			if (retval < bufsize)
			{
				// operation was successful.
				resolved = buf;
			}
			else
			{
				// buffer was too small, grow buffer and try again
				OGRE_FREE(buf, Ogre::MEMCATEGORY_GENERAL);
				bufsize <<= 1;
			}
		} while (!resolved);

		Ogre::String appPath = resolved;
		if (resolved)
			OGRE_FREE(resolved, Ogre::MEMCATEGORY_GENERAL);
		if (!appPath.empty())
		{
			// need to strip the application filename from the path
			Ogre::String::size_type pos = appPath.rfind('\\');
			if (pos != Ogre::String::npos)
				appPath.erase(pos);
		}
		else
		{
			// fall back to current working dir
			appPath = ".";
		}

		// use application path as config search path
		mConfigPaths.push_back(appPath + '\\');
	}
    //---------------------------------------------------------------------
	void PLAT_DLL FileSystemLayerImpl::prepareUserHome(const Ogre::String& projectName, const Ogre::String& subdir)
	{
		WCHAR path[MAX_PATH];
		if (SUCCEEDED(SHGetFolderPathW(NULL, CSIDL_PERSONAL|CSIDL_FLAG_CREATE, NULL, 0, path)))
		{
			// need to convert to OEM codepage so that fstream can use
			// it properly on international systems.
			//TCHAR oemPath[MAX_PATH];
			//CharToOemW(path, oemPath);
			mHomePath = path;
			// create the project subdir
			mHomePath = mHomePath + "\\"+ Ogre::UTFString(projectName) +"\\";
			if (! CreateDirectoryW(mHomePath.asWStr_c_str(), NULL) && GetLastError() != ERROR_ALREADY_EXISTS)
			{
				// couldn't create directory, fall back to current working dir
				mHomePath.clear();
			}
			else
			{
				mHomePath = mHomePath + subdir + '\\';
				// create release subdir
				if (! CreateDirectoryW(mHomePath.asWStr_c_str(), NULL) && GetLastError() != ERROR_ALREADY_EXISTS)
				{
					// couldn't create directory, fall back to current working dir
					mHomePath.clear();
				}
			}
		}

		if (mHomePath.empty())
		{
			// couldn't create dir in home directory, fall back to cwd
			mHomePath = "";
		}
	}
    //---------------------------------------------------------------------
	bool PLAT_DLL FileSystemLayerImpl::fileExists(const FSString& path) const
	{
		return _waccess(path.asWStr_c_str(), 00) == 0;
	}
    //---------------------------------------------------------------------
	bool PLAT_DLL FileSystemLayerImpl::createDirectory(const FSString& path)
	{
		return CreateDirectoryW(path.asWStr_c_str(), NULL) != 0 || 
			GetLastError() == ERROR_ALREADY_EXISTS;
	}
}

Now, when getting the writable path, we used this code:

Code: Select all

	OgreBites::FileSystemLayer::FSString configFilePath = mFS->getWritablePath(mFileName);
#ifdef WIN32
	FILE *f = _wfopen(configFilePath.asWStr_c_str(), L"w");
#else
	FILE *f = fopen(configFilePath.c_str(), "w");
#endif

Finaly, you may need to add a function to use UTFString as path to open Ogre::ConfigFiles
We wrote this one:

Code: Select all

//-----------------------------------------------------------------------
#ifdef WIN32
	void ConfigFile::loadDirectW(const Ogre::UTFString & filename, const String& separators,
		bool trimWhitespace)
	{
        /* Open the configuration file */
		std::ifstream fp;
        // Always open in binary mode
		fp.open(filename.asWStr_c_str(), std::ios::in | std::ios::binary);
		if(!fp)
			OGRE_EXCEPT(
			Exception::ERR_FILE_NOT_FOUND, "'" + filename + "' file not found!", "ConfigFile::load" );

		// Wrap as a stream
		DataStreamPtr stream(OGRE_NEW FileStreamDataStream(filename, &fp, false));

		load(stream, separators, trimWhitespace);

	}
#endif

Hope this helps.