Google

gcc 4.6 optimization flags and SCons build

A discussion on Python-Ogre, the library which allows you to drive OGRE directly from Python

Moderators: OGRE Team, Python-Ogre Moderators

gcc 4.6 optimization flags and SCons build

Postby dermont » Wed Oct 19, 2011 7:57 pm

Has anyone else run into problems with gcc 4.6 and the placement of optimization flags?

For the following example "-O3" is ignored and default "-O2" is used; the resultant _ois_.so library = 7.1MB.

Code: Select all
g++ -o build_dir_2.7/ois_1.0/Axis.pypp.os -c `pkg-config --cflags OGRE` -I -O3 -I./ -DBOOST_PYTHON_MAX_ARITY=19 -DBOOST_PYTHON_NO_PY_SIGNATURES -I/media/sda8/Libraries/PYTHON/python-ogre_svn/generated/ois_1.0 -fPIC -I/usr/local/include/boost -I/usr/local/include/OIS -I/usr/include/python2.7 -I/usr/local/include/boost generated/ois_1.0/Axis.pypp.cpp


With the following "-O3" is picked up correctly and _ois_.so library = 1.9MB (as per gcc4.5).

Code: Select all
g++ -O3 -o build_dir_2.7/ois_1.0/Axis.pypp.os -c `pkg-config --cflags OGRE` -I  -I./ -DBOOST_PYTHON_MAX_ARITY=19 -DBOOST_PYTHON_NO_PY_SIGNATURES -I/media/sda8/Libraries/PYTHON/python-ogre_svn/generated/ois_1.0 -fPIC -I/usr/local/include/boost -I/usr/local/include/OIS -I/usr/include/python2.7 -I/usr/local/include/boost generated/ois_1.0/Axis.pypp.cpp


Probably something wrong with my system but as a temp workaround I edited the SConstruct file to ensure the optimization flag is first (not sure if this is the correct way of doing this for SCons).
Code: Select all
        _env['CXX'] = "g++ -O3"  <----------------
        ## Use custom compilier if wanted (things like ccache)
dermont
Bugbear
 
Posts: 895
Kudos: 37
Joined: 27 Sep 2005

Re: gcc 4.6 optimization flags and SCons build

Postby dermont » Tue Nov 01, 2011 7:43 am

I'll post here since it's related to speeding up the build process. This is a follow on from this post where I've been trying to emulate the Unity part of the Ogre build.

viewtopic.php?f=3&t=14338&start=15

Batching the source files into manageable chunks for me at least improves the build times for python-ogre by over 50%. The ois/ogrepaging/ogreterrain/shadersystem modules take minutes to build. The bullet module about 30 mins(nearly 2 hours previously) and the ogre module about 90 minutes(4-5 hours previous).

I know in py++ you have the option not to split the wrapper code but I'm not sure how it works in splitting into manageable chunks and I can't really be bothered looking through the code to find out.

For the meantime you will need to run the unity.py file manually on the generated code to produce the batched source files. The source files are written to generated/module_name/unity and built in build_dir_2.7/module_name/unity.

These are the changes to the configuration files, to change to unity build/revert back to your current build set module to unity = False/True in environment.py.

You can then run scons/your buidl command as normal. I've already started implementing this in CMake so won't be making any more updates to the SCons build so feel free to make any changes etc. that you want. You can of course just ignore all this if you are happy with the current build times or just don't care.

environment.py
Code: Select all
class module(object):
    ...
    unity = False

class ois(pymodule):
    unity = True


SConstruct
Code: Select all
Index: SConstruct
===================================================================
--- SConstruct   (revision 1132)
+++ SConstruct   (working copy)
@@ -84,6 +84,8 @@
 
     ## change to ensure the source file is also in the include path due to indexing suite changes
     CCFLAGS += ' -I' +cls._source + ' '
+    if (cls.unity):
+        CCFLAGS += ' -I' + os.path.split(cls._source)[0] + ' '
 
     return CCFLAGS
 
@@ -105,6 +107,9 @@
         print 'WARNING: Generate the sources in this directory: "%s"' % _dir
         raise e
     source_files.sort()
+    if (cls.unity):
+        for j,s in enumerate(source_files):
+            source_files[j] = os.path.join('unity',source_files[j])
     return source_files ## "Image.pypp.cpp" ##source_files
 
 def get_linkflags():
@@ -167,8 +172,13 @@
         log ("SCONS: Building " + name)
 
         ## setup some defaults etc
+        if cls.unity:
+            cls.generated_dir = os.path.join(cls.generated_dir,'unity')
         cls._source = cls.generated_dir
         cls._build_dir = os.path.join ( builddir, cls.dir_name)
+        if cls.unity:
+            cls._build_dir = os.path.join(cls._build_dir,'unity')
+
         cls._name = name


unity.py
Code: Select all
import commands
import os
import os.path
from glob import iglob
import shutil
import operator

def build_module_unity(module, generated_dir, single_batches=[], batch_size=150000,batch_all=False):

    filenames = []
    getfiles = [ [files, os.path.getsize(files)] for files in iglob(os.path.join(generated_dir, '*.cpp')) ]
    sorted(getfiles, key=operator.itemgetter(1))

    total_size = 0
    for f in getfiles:
        filenames.append(f[0])

    source_dict = {}

    ### create batches for classes may cause problems
    ### combined in same batch
    known_groupings = single_batches
    batch_number = -1
    for i,k in enumerate(known_groupings):
        batch_number+=1
        source_dict["%s_batch_%s" %(str(batch_number))]=[]
        for j,m in enumerate(filenames):
            if k in m:
                header_dict[k].append(m)
                filenames.pop(j)
    total_batch_size = 0
    new_batch = True
    for i,m in enumerate(filenames):
        if (new_batch):
            batch_number+=1
            source_dict["%s_batch_%s" %(module,str(batch_number))]=[]
            new_batch = False
        source_dict["%s_batch_%s" %(module,str(batch_number))].append(m)
        total_batch_size+= os.path.getsize(m)
        if total_batch_size > batch_size:
            total_batch_size = 0
            new_batch = True
        if batch_all:
            new_batch = False
        print new_batch,batch_all, total_batch_size, batch_number
    ##batch_size = 20
    ##lists = [filenames[i:i+batch_size] for i in xrange(0, len(filenames), batch_size)]


    unity_dir = '%s/unity' %(generated_dir)
    if not os.path.exists(unity_dir):
        os.makedirs(unity_dir)

    for k in source_dict.keys():
        print k
        batch_file = open('%s/%s.pypp.cpp' %(unity_dir,k), 'w')
        for f in source_dict[k]:
            batch_file.write('''#include <%s>\n''' %(f))

    #        shutil.copyfileobj(open(f, 'r'), destination)
        batch_file.close()




##OGRE example of classes to build in own batch to a avoid compile failure
single_batches = [
        'Vector3',
        'Quaternion'
        'ColourValue',
        'Vector2',
        'Vector4',
        'Matrix3'
        'main'
]
single_batches=[]

##OIS Module build single batch
#PATH = r'/media/sda8/Libraries/PYTHON/python-ogre/generated/%s' %("ois_1.3")
#build_module_unity('ois', PATH, single_batches = single_batches, batch_size=0, batch_all=True)

### bullet split, batch size may be a tad large
PATH = r'/media/sda8/Libraries/PYTHON/python-ogre/generated/%s' %("bullet_2.79")
build_module_unity('bullet', PATH, single_batches = single_batches, batch_size=250000, batch_all=False)
dermont
Bugbear
 
Posts: 895
Kudos: 37
Joined: 27 Sep 2005

Re: gcc 4.6 optimization flags and SCons build

Postby pusheax » Thu Jan 26, 2012 10:19 pm

With unity.py batching ogre module compiles definitely faster. Linking was almost instanteneous. It took ~2 hours. While patiently watching htop's cpu/memory indicators I figured out the best batch size for me - 80KB.
Although the mostrous size of _ogre_.so frightens me - 76.2MB. On windows it is ~41MB.
I did not know that compilation of one 80KB file can take up several hundreds Megs of RAM. Learning something new every day :D
pusheax
Gnoblar
 
Posts: 3
Kudos: 0
Joined: 26 Jan 2012

Re: gcc 4.6 optimization flags and SCons build

Postby dermont » Sat Jan 28, 2012 7:38 am

pusheax wrote:With unity.py batching ogre module compiles definitely faster. Linking was almost instanteneous. It took ~2 hours. While patiently watching htop's cpu/memory indicators I figured out the best batch size for me - 80KB.
Although the mostrous size of _ogre_.so frightens me - 76.2MB. On windows it is ~41MB.
I did not know that compilation of one 80KB file can take up several hundreds Megs of RAM. Learning something new every day :D


The libraries on Linux will always be larger than that on Windows. The underlying Ogre .so files are 3 to 4 times the size of the Windows lib files, therefore I think that the python-ogre libraries on Linux is only twice the size of it's Windows counterpart is pretty good.

To reduce the size of _ogre_.so you could try disabling docstrings when building. I'm not sure that this would make much difference or is worth the effort, unfortunately when using boost python the resultant library files will always be large/slow to build.

With regards to the unity build yes you have to estimate the best batch size for your particular machine. Also keep in mind this hasn't been tested thoroughly so there may be unforeseen issues using a batch build.
dermont
Bugbear
 
Posts: 895
Kudos: 37
Joined: 27 Sep 2005


Return to Python-Ogre

Who is online

Users browsing this forum: Google Adsense [Bot] and 1 guest