gcc 4.6 optimization flags and SCons build

dermont

19-10-2011 19:57:18

Has anyone else run into problems with gcc 4.6 and the placement of optimization flags?

For the following example "-O3" is ignored and default "-O2" is used; the resultant _ois_.so library = 7.1MB.


g++ -o build_dir_2.7/ois_1.0/Axis.pypp.os -c `pkg-config --cflags OGRE` -I -O3 -I./ -DBOOST_PYTHON_MAX_ARITY=19 -DBOOST_PYTHON_NO_PY_SIGNATURES -I/media/sda8/Libraries/PYTHON/python-ogre_svn/generated/ois_1.0 -fPIC -I/usr/local/include/boost -I/usr/local/include/OIS -I/usr/include/python2.7 -I/usr/local/include/boost generated/ois_1.0/Axis.pypp.cpp


With the following "-O3" is picked up correctly and _ois_.so library = 1.9MB (as per gcc4.5).


g++ -O3 -o build_dir_2.7/ois_1.0/Axis.pypp.os -c `pkg-config --cflags OGRE` -I -I./ -DBOOST_PYTHON_MAX_ARITY=19 -DBOOST_PYTHON_NO_PY_SIGNATURES -I/media/sda8/Libraries/PYTHON/python-ogre_svn/generated/ois_1.0 -fPIC -I/usr/local/include/boost -I/usr/local/include/OIS -I/usr/include/python2.7 -I/usr/local/include/boost generated/ois_1.0/Axis.pypp.cpp


Probably something wrong with my system but as a temp workaround I edited the SConstruct file to ensure the optimization flag is first (not sure if this is the correct way of doing this for SCons).

_env['CXX'] = "g++ -O3" <----------------
## Use custom compilier if wanted (things like ccache)

dermont

01-11-2011 06:43:54

I'll post here since it's related to speeding up the build process. This is a follow on from this post where I've been trying to emulate the Unity part of the Ogre build.

viewtopic.php?f=3&t=14338&start=15

Batching the source files into manageable chunks for me at least improves the build times for python-ogre by over 50%. The ois/ogrepaging/ogreterrain/shadersystem modules take minutes to build. The bullet module about 30 mins(nearly 2 hours previously) and the ogre module about 90 minutes(4-5 hours previous).

I know in py++ you have the option not to split the wrapper code but I'm not sure how it works in splitting into manageable chunks and I can't really be bothered looking through the code to find out.

For the meantime you will need to run the unity.py file manually on the generated code to produce the batched source files. The source files are written to generated/module_name/unity and built in build_dir_2.7/module_name/unity.

These are the changes to the configuration files, to change to unity build/revert back to your current build set module to unity = False/True in environment.py.

You can then run scons/your buidl command as normal. I've already started implementing this in CMake so won't be making any more updates to the SCons build so feel free to make any changes etc. that you want. You can of course just ignore all this if you are happy with the current build times or just don't care.

environment.py

class module(object):
...
unity = False

class ois(pymodule):
unity = True


SConstruct

Index: SConstruct
===================================================================
--- SConstruct (revision 1132)
+++ SConstruct (working copy)
@@ -84,6 +84,8 @@

## change to ensure the source file is also in the include path due to indexing suite changes
CCFLAGS += ' -I' +cls._source + ' '
+ if (cls.unity):
+ CCFLAGS += ' -I' + os.path.split(cls._source)[0] + ' '

return CCFLAGS

@@ -105,6 +107,9 @@
print 'WARNING: Generate the sources in this directory: "%s"' % _dir
raise e
source_files.sort()
+ if (cls.unity):
+ for j,s in enumerate(source_files):
+ source_files[j] = os.path.join('unity',source_files[j])
return source_files ## "Image.pypp.cpp" ##source_files

def get_linkflags():
@@ -167,8 +172,13 @@
log ("SCONS: Building " + name)

## setup some defaults etc
+ if cls.unity:
+ cls.generated_dir = os.path.join(cls.generated_dir,'unity')
cls._source = cls.generated_dir
cls._build_dir = os.path.join ( builddir, cls.dir_name)
+ if cls.unity:
+ cls._build_dir = os.path.join(cls._build_dir,'unity')
+
cls._name = name


unity.py

import commands
import os
import os.path
from glob import iglob
import shutil
import operator

def build_module_unity(module, generated_dir, single_batches=[], batch_size=150000,batch_all=False):

filenames = []
getfiles = [ [files, os.path.getsize(files)] for files in iglob(os.path.join(generated_dir, '*.cpp')) ]
sorted(getfiles, key=operator.itemgetter(1))

total_size = 0
for f in getfiles:
filenames.append(f[0])

source_dict = {}

### create batches for classes may cause problems
### combined in same batch
known_groupings = single_batches
batch_number = -1
for i,k in enumerate(known_groupings):
batch_number+=1
source_dict["%s_batch_%s" %(str(batch_number))]=[]
for j,m in enumerate(filenames):
if k in m:
header_dict[k].append(m)
filenames.pop(j)
total_batch_size = 0
new_batch = True
for i,m in enumerate(filenames):
if (new_batch):
batch_number+=1
source_dict["%s_batch_%s" %(module,str(batch_number))]=[]
new_batch = False
source_dict["%s_batch_%s" %(module,str(batch_number))].append(m)
total_batch_size+= os.path.getsize(m)
if total_batch_size > batch_size:
total_batch_size = 0
new_batch = True
if batch_all:
new_batch = False
print new_batch,batch_all, total_batch_size, batch_number
##batch_size = 20
##lists = [filenames[i:i+batch_size] for i in xrange(0, len(filenames), batch_size)]


unity_dir = '%s/unity' %(generated_dir)
if not os.path.exists(unity_dir):
os.makedirs(unity_dir)

for k in source_dict.keys():
print k
batch_file = open('%s/%s.pypp.cpp' %(unity_dir,k), 'w')
for f in source_dict[k]:
batch_file.write('''#include <%s>\n''' %(f))

# shutil.copyfileobj(open(f, 'r'), destination)
batch_file.close()




##OGRE example of classes to build in own batch to a avoid compile failure
single_batches = [
'Vector3',
'Quaternion'
'ColourValue',
'Vector2',
'Vector4',
'Matrix3'
'main'
]
single_batches=[]

##OIS Module build single batch
#PATH = r'/media/sda8/Libraries/PYTHON/python-ogre/generated/%s' %("ois_1.3")
#build_module_unity('ois', PATH, single_batches = single_batches, batch_size=0, batch_all=True)

### bullet split, batch size may be a tad large
PATH = r'/media/sda8/Libraries/PYTHON/python-ogre/generated/%s' %("bullet_2.79")
build_module_unity('bullet', PATH, single_batches = single_batches, batch_size=250000, batch_all=False)

pusheax

26-01-2012 21:19:52

With unity.py batching ogre module compiles definitely faster. Linking was almost instanteneous. It took ~2 hours. While patiently watching htop's cpu/memory indicators I figured out the best batch size for me - 80KB.
Although the mostrous size of _ogre_.so frightens me - 76.2MB. On windows it is ~41MB.
I did not know that compilation of one 80KB file can take up several hundreds Megs of RAM. Learning something new every day :D

dermont

28-01-2012 06:38:15

With unity.py batching ogre module compiles definitely faster. Linking was almost instanteneous. It took ~2 hours. While patiently watching htop's cpu/memory indicators I figured out the best batch size for me - 80KB.
Although the mostrous size of _ogre_.so frightens me - 76.2MB. On windows it is ~41MB.
I did not know that compilation of one 80KB file can take up several hundreds Megs of RAM. Learning something new every day :D


The libraries on Linux will always be larger than that on Windows. The underlying Ogre .so files are 3 to 4 times the size of the Windows lib files, therefore I think that the python-ogre libraries on Linux is only twice the size of it's Windows counterpart is pretty good.

To reduce the size of _ogre_.so you could try disabling docstrings when building. I'm not sure that this would make much difference or is worth the effort, unfortunately when using boost python the resultant library files will always be large/slow to build.

With regards to the unity build yes you have to estimate the best batch size for your particular machine. Also keep in mind this hasn't been tested thoroughly so there may be unforeseen issues using a batch build.