This is the fourth post in my SCons series. The topic of this post is setting up a multi-flavor C++ project using SCons, with a separate build directory. By “flavor”, I mean something like debug vs. release.

In C++ projects, it is common to build multiple variants, or flavors, of the project. A debug flavor may build more quickly, and contain debug symbols. A release flavor may perform optimizations for runtime or other metrics. The different flavors serve different purposes, and they all can co-exist. The developer may choose which flavor(s) to build and run as she pleases.

In this post, I show how to use SCons to manage multiple flavors in a C++ project. My requirements from flavor support in a SCons-powered C++ project:

1. Define flavor profiles easily. Allow to customize construction parameters per-flavor. Support common parameters that can be overridden by flavors.
2. Support flavor-specific build directory. Build outputs should reside under their flavor build directory. Multiple flavors can co-exist at the same time, without interfering with each other. Incremental builds can be done per-flavor.
3. Let the developer choose what to build. Allow choosing one flavor, all flavors, or any subset. Syntax should be simple and intuitive.
4. The developer can execute built programs at any flavor she wants.

I add multi-flavor support on top of the previous episode in the series.

The final result is available on my GitHub scons-series repository. In the rest of this post I go into the details of what I came up with.

As a reminder, the (seemingly silly) C++ project is a simple address book program. Refer to a previous post if you’re interested in more details.

Defining Flavors

Flavors are defined in the site_scons/site_config.py file. I introduce this file as a dedicated place for project-specific configuration, separated from other build-system logic in other files. I chose the location of the file under site_scons directory, because it is automatically added to the Python search path is SConstruct and SConscript files.

Lets take a look at my example of debug & release flavors definition from site_config.py, with some common settings:

# Dictionary of flavor-specific settings that should override values
#  from the base environment (using env.Replace).
# `_common` is reserved for settings that apply to the base env.
ENV_OVERRIDES = {
    '_common': dict(
        # Use clang compiler by default
        CC          = 'clang',
        CXX         = 'clang++',
    ),
    'debug': dict(
        BUILDROOT = os.path.join(_BUILD_BASE, 'debug'),
    ),
    'release': dict(
        BUILDROOT = os.path.join(_BUILD_BASE, 'release'),
    ),
}

# Dictionary of flavor-specific settings that should extend values
#  from the base environment (using env.Append).
# `_common` is reserved for settings that apply to the base env.
ENV_EXTENSIONS = {
    '_common': dict(
        # Common flags for all C++ builds
        CCFLAGS = ['-std=c++11', '-Wall', '-fvectorize', '-fslp-vectorize'],
        # Modules should be able to include relative to build root dir
        CPPPATH = ['#$BUILDROOT'],
    ),
    'debug': dict(
        # Extra flags for debug C++ builds
        CCFLAGS = ['-g', '-DDEBUG'],
    ),
    'release': dict(
        # Extra flags for release C++ builds
        CCFLAGS = ['-O2', '-DNDEBUG'],
    ),
}

I use two Python dictionaries to define flavors (highlighted). The ENV_OVERRIDES[flavor_name] dictionary contains settings that override values from the base environment, using SCons Replace function. The ENV_EXTENSIONS[flavor_name] dictionary contains settings that extend values from the base environment, using SCons Append function. The reserved “virtual” flavor name _common is used to define the base environment.

Every flavor must define a BUILDROOT entry. This entry tells the system what should be the base build directory for that flavor. An alternative approach would be to automatically use the flavor name as a sub-directory of build_base. I chose the explicit approach, to let the developer have more control and clarity.

Reviewing what’s happening in the example:

• The compiler is set by default to clang.
• Build directories are set to build_dir/$flavor_name.
• A couple of compiler flags are set by default for all flavors (use C++11 standard, maximal warning level, etc.).
• Debug flags are added for the debug flavor (-g for debug info, define DEBUG).
• Release flavor is optimized (-O2), and NDEBUG is defined.

Building Flavors

The build process flow is in the main SConstruct:

# Get the base construction environment
_BASE_ENV = get_base_env()

# Build every selected flavor
for flavor in _BASE_ENV.flavors:
    print 'scons: + Processing flavor', flavor, '...'
    # Prepare flavored environment
    flavored_env = get_flavored_env(_BASE_ENV, flavor)
    # Go over modules to build, and delegate the build to them
    for module in modules():
        process_module(flavored_env, module)
    # Support using the flavor name as a target name for its related targets
    Alias(flavor, flavored_env['BUILDROOT'])

It is pretty self-explanatory and concise. It uses functions from site_scons/site_init.py to do the dirty work. The flow:

1. Initialize a base construction environment.
2. For each flavor to build:
   1. Customize the base construction environment for the flavor.
   2. Process all modules using the flavored construction environment.
   3. Create a flavored alias target.

The modules list is the same as in the multi-module project. It was just moved to site_scons/site_config.py.

The Alias target is pretty standard, as explained in the SCons user guide. I use it here to create a target with the flavor name, so the developer is able to build a flavor with scons flavor_name. This way, it is also possible to build multiple flavors with scons flav1 flav2 ....

Flavoring the Base Construction Environment

Once the base construction environment is initialized (see next section), it is customized per-flavor using the get_flavored_env function. This function comes from site_scons/site_init.py. Here it is, for your convenience:

def get_flavored_env(base_env, flavor):
    """Customize and return a flavored construction environment."""
    flavored_env = base_env.Clone()
    # Prepare shared targets dictionary
    flavored_env['targets'] = dict()
    # Allow modules to use `env.get_targets('libname1', 'libname2', ...)` as
    #  a shortcut for adding targets from other modules to sources lists.
    flavored_env.get_targets = lambda *args, **kwargs: \
        get_targets(flavored_env, *args, **kwargs)
    # Apply flavored env overrides and customizations
    if flavor in ENV_OVERRIDES:
        flavored_env.Replace(**ENV_OVERRIDES[flavor])
    if flavor in ENV_EXTENSIONS:
        flavored_env.Append(**ENV_EXTENSIONS[flavor])
    return flavored_env

The base env is cloned. An empty targets dictionary is added, along with a get_targets method (see previous post for details on that). Then the flavor-specific customization from site_config.py are applied, using double-star dictionary unpacking magic.

Initializing the Base Construction Environment

The base construction environment is initialized using the get_base_env function. This function also comes form site_scons/site_init.py. Here it is:

def get_base_env(*args, **kwargs):
    """Initialize and return a base construction environment.

    All args received are passed transparently to SCons Environment init.
    """
    # Initialize new construction environment
    env = Environment(*args, **kwargs)  # pylint: disable=undefined-variable
    # If specific flavor target specified, skip processing other flavors
    # Otherwise, include all known flavors
    env.flavors = (set(flavors()).intersection(COMMAND_LINE_TARGETS) or flavors())
    # Perform base construction environment customizations from site_config
    if '_common' in ENV_OVERRIDES:
        env.Replace(**ENV_OVERRIDES['_common'])
    if '_common' in ENV_EXTENSIONS:
        env.Append(**ENV_EXTENSIONS['_common'])
    return env

Again, nothing thrilling. SCons Environment created, and _common customizations are applied.

The highlighted line takes care of figuring out what flavors should be processed, and storing it in env.flavors. By default, if you just run scons, all known flavors are built (as defined by flavors()). But if you run scons debug, then only the debug flavor is built, thanks to the Alias target. So what is this set(flavors()).intersection(COMMAND_LINE_TARGETS) for?

It’s an optimization.

Let me explain.

If I’d use just flavors(), then env.flavors would contain all known flavors (e.g. ['debug', 'release']). If you run scons debug, then only the debug targets would be built, but all other flavors will get processed. Knowing that these targets will not get built, I can skip the processing and save a little time.

The optimization works by examining the command line targets specified. The intersection between the known flavors and the specified command line targets gives the requested flavors, if any. This way, if a flavor name is specified as a command line target, flavors that were not specified will not be processed at all.

The flavors() function, in a similar fashion to the modules() function, needs to generate the known flavors. It is defined in site_scons/site_config.py. It can be as simple as return ['debug', 'release']. But that would mean duplication of flavor names. So I used the flavor dictionaries to get the names of the defined flavors:

def flavors():
    """Generate supported flavors.

    Each flavor is a string representing a flavor entry in the
    override / extension dictionaries above.
    Each flavor entry must define atleast "BUILDROOT" variable that
    tells the system what's the build base directory for that flavor.
    """
    # Use the keys from the env override / extension dictionaries
    for flavor in set(ENV_EXTENSIONS.keys() + ENV_OVERRIDES.keys()):
        # Skip "hidden" records
        if not flavor.startswith('_'):
            yield flavor

I iterate over a set of joined keys lists to avoid duplicate flavors. I skip entries that start with _ (like _common), to “hide” internals.

Money Time

After reviewing the parts of the build system, it’s time to see it in action.

I didn’t review the module-level SConscript files, because they remain exactly the same compared to the previous post.

itamar@legolas sconseries (episodes/03-flavors) $ scons
scons: Reading SConscript files ...
scons: + Processing flavor debug ...
scons: |- Reading module AddressBook ...
scons: |- Reading module Writer ...
scons: + Processing flavor release ...
scons: |- Reading module AddressBook ...
scons: |- Reading module Writer ...
scons: done reading SConscript files.
scons: Building targets ...
clang++ -o build/debug/AddressBook/addressbook.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -g -DDEBUG -Ibuild/debug build/debug/AddressBook/addressbook.cc
ar rc build/debug/AddressBook/libaddressbook.a build/debug/AddressBook/addressbook.o
ranlib build/debug/AddressBook/libaddressbook.a
clang++ -o build/debug/Writer/writer.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -g -DDEBUG -Ibuild/debug build/debug/Writer/writer.cc
clang++ -o build/debug/Writer/writer build/debug/Writer/writer.o build/debug/AddressBook/libaddressbook.a
clang++ -o build/release/AddressBook/addressbook.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release build/release/AddressBook/addressbook.cc
ar rc build/release/AddressBook/libaddressbook.a build/release/AddressBook/addressbook.o
ranlib build/release/AddressBook/libaddressbook.a
clang++ -o build/release/Writer/writer.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release build/release/Writer/writer.cc
clang++ -o build/release/Writer/writer build/release/Writer/writer.o build/release/AddressBook/libaddressbook.a
scons: done building targets.
[/scons]

As expected, both flavors were processed and built. Each flavor has its build outputs under the flavor build directory, so I can execute the program I want to.

Lets check out flavor selection:


itamar@legolas sconseries (episodes/03-flavors) $ scons -c
<< ... >>
itamar@legolas sconseries (episodes/03-flavors) $ scons release

scons: Reading SConscript files ...
scons: + Processing flavor release ...
scons: |- Reading module AddressBook ...
scons: |- Reading module Writer ...
scons: done reading SConscript files.
scons: Building targets ...
clang++ -o build/release/AddressBook/addressbook.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release build/release/AddressBook/addressbook.cc
ar rc build/release/AddressBook/libaddressbook.a build/release/AddressBook/addressbook.o
ranlib build/release/AddressBook/libaddressbook.a
clang++ -o build/release/Writer/writer.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release build/release/Writer/writer.cc
clang++ -o build/release/Writer/writer build/release/Writer/writer.o build/release/AddressBook/libaddressbook.a
scons: done building targets.

You can see that only release flavor was processed and built.

Contrast that to a slight variation:

itamar@legolas sconseries (episodes/03-flavors) $ scons -c
<< ... >>
scons: done cleaning targets.

itamar@legolas sconseries (episodes/03-flavors) $ scons build/release
scons: Reading SConscript files ...
scons: + Processing flavor debug ...
scons: |- Reading module AddressBook ...
scons: |- Reading module Writer ...
scons: + Processing flavor release ...
scons: |- Reading module AddressBook ...
scons: |- Reading module Writer ...
scons: done reading SConscript files.
scons: Building targets ...
clang++ -o build/release/AddressBook/addressbook.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release build/release/AddressBook/addressbook.cc
ar rc build/release/AddressBook/libaddressbook.a build/release/AddressBook/addressbook.o
ranlib build/release/AddressBook/libaddressbook.a
clang++ -o build/release/Writer/writer.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release build/release/Writer/writer.cc
clang++ -o build/release/Writer/writer build/release/Writer/writer.o build/release/AddressBook/libaddressbook.a
scons: done building targets.

Again, only release flavor was built. But, as apparent from the highlighted line, the debug flavor was processed.

The overhead of processing flavors that will not be built is minor for a small project. I did not analyze it further, but I assume that as the project grows in size and complexity, the overhead becomes significant. This demonstrates the benefit of my flavor-skipping logic, when using flavor-name aliases as command line targets.

Summary

This concludes my multi-flavor SCons extension.

My solution fulfills the requirements I described:

1. Define flavor profiles easily. Allow to customize construction parameters per-flavor. Support common parameters that can be overridden by flavors.
   • Flavors are defined in simple dictionaries in site_scons/site_config.py. A _common flavor is used for common parameters.
2. Support flavor-specific build directory. Build outputs should reside under their flavor build directory. Multiple flavors can co-exist at the same time, without interfering with each other. Incremental builds can be done per-flavor.
   • Flavors have dedicated build directories that can co-exist with no interference.
3. Let the developer choose what to build. Allow choosing one flavor, all flavors, or any subset. Syntax should be simple and intuitive.
   • Each flavor adds a command line target alias using the flavor name. Subsets can be specified easily. Zero targets is equivalent to all targets.
4. The developer can execute built programs at any flavor she wants.
   • Simply run from the relevant flavor build directory.

The final result is available on my GitHub scons-series repository. Feel free to use / fork / modify. If you do, I'd appreciate it if you share back improvements.

See the scons tag for more in my SCons series. The next post will continue from here, adding flavor-helper script to simplify things even further.