This is the fourteenth post in my SCons series. The topic of this post is adding a shortcut for the custom SCons Protoc builder from the previous episodes.

The shortcut is in line with the SConscript simplification approach described in an earlier episode. In this installment, I add a new Proto shortcut to the collection, so the address book SConscript can look like this:

"""AddressBook proto-based library SConscript script"""

Import('*')

AbProtos = ['person.proto', 'addressbook.proto']

Proto(AbProtos)
Lib('addressbook', protos=AbProtos)

The final result is available on my GitHub scons-series repository.

Adding a new Proto shortcut

Building on top of the shortcuts infrastructure, I add a new entry for Proto in the shortcuts dictionary:

shortcuts = dict(
    Lib       = self._lib_wrapper(self._env.Library, module),
    StaticLib = self._lib_wrapper(self._env.StaticLibrary, module),
    SharedLib = self._lib_wrapper(self._env.SharedLibrary, module),
    Proto     = self._proto_wrapper(),
    Prog      = nop,
)

The _proto_wrapper() method returns a wrapped custom Protoc builder:

def _proto_wrapper(self):
    """Return a wrapped Protoc builder."""
    def compile_proto(proto_sources, **kwargs):

        """Customized Protoc builder.

        Uses Protoc builder to compile proto files specified in `proto_sources`.

        Optionally pass `cpp=False` to disable C++ code generation.
        Optionally, pass `python=True` to enable Python code generation.

        Optionally pass `PROTOPATH=[...]` to override default list of
         proto search paths (default is [$BUILDROOT]).
        Optionally pass `PROTOCPPOUT=path` and `PROTOPYOUT=path` to
         override default output path for C++ / Python outputs
         (respectively), when output for this language is enabled.
         Default output paths are $BUILDROOT (so expect output files
         in the module directory under the flavor build dir).
        Tip: Don't mess with these...
        """
        if not hasattr(self._env, 'Protoc'):
            raise StopError('Protoc tool not installed.')
        # use user-specified value, or set default
        kwargs.setdefault('PROTOPATH', ['$BUILDROOT'])
        any_output = False
        for gen_flag_name, default_gen_flag, path_name, default_path in [
                ('cpp', True, 'PROTOCPPOUT', '$BUILDROOT'),
                ('python', False, 'PROTOPYOUT', '$BUILDROOT'),
            ]:
            gen_output_flag = kwargs.pop(gen_flag_name, default_gen_flag)
            if gen_output_flag:
                any_output = True
                # use user-specified value, or set default
                kwargs.setdefault(path_name, default_path)
            else:
                kwargs[path_name] = ''
        # check that at least one output language is enabled
        if any_output:
            targets = self._env.Protoc([], proto_sources, *args, **kwargs)
            for gen_node in targets:
                gen_filename = os.path.basename(gen_node.path)
                if gen_filename.endswith('.pb.cc'):
                    # Save generated .pb.cc sources in proto_cc dictionary
                    #  (without the ".pb.cc" suffix)
                    self._proto_cc[gen_filename[:-6]] = gen_node
        else:
            sprint('warning: Proto target with no output directives')
    return compile_proto

When the Proto shortcut is used in a SConscript file, it invokes the nested compile_proto() function. This custom builder expects to receive a list of .proto sources, with optional keyword arguments to customize the build. The custom builder then uses the Protoc builder (as-is from the previous episode) to do the work, passing the source list to it.

After compiling the proto files, the builder saves generated C++ nodes in a dictionary, keyed by source filename without the “.pb.cc” suffix. This is required so other targets will be able to refer to these nodes later.

By default, the custom builder sets up the PROTOPATH and PROTOCPPOUT arguments to $BUILDROOT. This makes the Protoc builder work exactly as it did in the previous episode. It executes relative to the build directory, and generates only C++ code in the same build directory.

All of the Protoc builder arguments can be directly manipulated by the caller, by passing them as additional keyword arguments. This is made possible by passing **kwargs to the underlying Protoc builder.

In addition, the shortcut builder supports special keyword arguments to control its own behavior. The cpp and python keyword arguments are boolean flags that control whether the custom builder generates C++ and Python code. The default values (cpp=True and python=False) can be overridden by passing them as keyword arguments to the Proto shortcut. Since the underlying Protoc builder doesn’t know about these flags, they are extracted from kwargs before passing it down (using kwargs.pop).

Note that passing PROTOPYOUT to set a directory for Python generated code is not sufficient. You will also need to explicitly pass python=True, or change the default flag value to True.

The shortcut prints a warning if it recognizes that all valid output languages are disabled. It also raises a StopError if the Protoc builder is missing.

Adding support for the protos keyword argument

The changes described so far are sufficient for this SConscript to work:

"""AddressBook proto-based library SConscript script"""

Import('*')

AbProtos = ['person.proto', 'addressbook.proto']

Proto(AbProtos)
Lib('addressbook', ['person.pb.cc', 'addressbook.pb.cc'])

This may be just fine, but not enough for my OCD. I don’t like it that I need to repeat the names of the proto’s, and even worse – that I need to know the naming convention of generated C++ files!

To be able to use Lib('addressbook', protos=AbProtos) instead of Lib('addressbook', ['person.pb.cc', 'addressbook.pb.cc']), I need to modify the existing wrappers that handle the other shortcuts. Since all wrappers already accept **kwargs, I can check if protos key exists, and use it to extend the sources list. I added this line to both existing wrappers (lib and prog):

sources = self._extend_proto_sources(sources, kwargs)

The _extend_proto_sources() method pops the protos member if it exists, and uses the _proto_cc dictionary to add generated .pb.cc nodes to the sources list:

def _extend_proto_sources(self, sources, kwargs_dict):
    """Return the sources list, extended with proto-cc-sources.

    @param  sources     The original list of sources
    @param kwargs_dict  The keyword argument dictionary

    The protos list, if specified, is at kwargs_dict['protos'].
    If it is specified, the sources list is extended accordingly,
     and the 'protos' key is removed (so it's safe to pass-through).
    """
    return listify(sources) + [
        self._proto_cc[re.sub(r'\.proto$', '', proto)]
        for proto in listify(kwargs_dict.pop('protos', None))]

Notice that he method allows specifying the names of the protos with or without the “.proto” suffix, by stripping it out if it’s there (using re.sub()).

Summary

Now my OCD can relax a bit, with the SConscript clean and neat, while nothing is changed in the actual compile & link commands and outputs.

The final result is available on my GitHub scons-series repository. Feel free to use / fork / modify. If you do, I’d appreciate it if you share back improvements.

See the scons tag for more in my SCons series. In the next episode I will add Python to the party, so better stay tuned!

Another thing that still bothers me, is the way I needed to specify linking with the external protobuf library. Although the AddressBook library is the one using protocol buffers, I needed to specify LIBS=['protobuf'] on the Writer and Reader program targets! I understand this from a compile/link point of view, but I still don’t like it. Ideally, I’d like to “label” the AddressBook library as “requiring” the external protobuf library, and have SCons propagate this information for me. If you feel this way too, stay tuned for a future episode that will deal with propagating required libraries.