Category Software Engineering

Software engineering, development, design, quality, and methodologies.

Shell Foo: Getting a cpplint Breakdown Report On All Project Source Files

By Wednesday, December 3, 2014 0 Software Engineering , , , , Permalink 1

You’re working on a non-trivial project, with multiple source files, in multiple languages. You want to use cpplint to get a “style report” for all C/C++ files in the project.

You don’t want to get bogged down with hundreds of style warnings – you just want the final summary of warnings count per category.

In addition, you want to run only on C/C++ files, and avoid duplicates resulting from the build-directory.

How do you accomplish that?

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Continue Reading…

Python Logging Traps

By Monday, December 1, 2014 0 Software Engineering Permalink 2

The Pythons logging framework is powerful, but gives you plenty of ways to shoot yourself in the foot. To make matters worse, logging mistakes are subtle and slip through code review easily.

The post Python Logging Traps by Simon Weber originally appeared on www.simonmweber.com

In his time at Venmo, Simon had seen logging mistakes cause everything from unneeded debugging to application crashes. Here are the most common culprits.

Continue Reading…

Shell Foo: Listing All C/C++ Files In a Project, Excluding Build Dir

By Wednesday, November 26, 2014 0 Software Engineering , , , Permalink 1

You’re working on a non-trivial project, with multiple source files, in multiple languages. You want to list all C/C++ source files in the project, recursively, but without getting duplicates from the build dir.

Assuming your C/C++ files are all .h or .cc files, and the build directory is build/ – how would you do it?

Here’s my preferred solution:

$ find . -name \*.h -or -name \*.cc | grep -vE "^\.\/build\/"

Shell-Foo credit for this one: Eyal Fink.

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Continue Reading…

Shell Foo: Printing the Nth Line of a File

By Wednesday, November 19, 2014 2 Software Engineering , , , Permalink 2

You’re running a Python script from the command line, and it throws some exception in your face. You want to take a quick look at the line that raised the exception (say, 42). How would you print that line?

My favorite way (minimal typing):

$ sed -n 42p myscript.py

Shell-Foo credit for this one: Eyal Fink.

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Continue Reading…

Sort Your Include Files!

By Monday, November 17, 2014 0 Software Engineering , , , , Permalink 1

The Google C++ Style Guide defines a guideline for names and order of includes. It occurred to me that grouping and sorting include files is tedious and error prone, and a computer can do it much better. So I wrote a script that does exactly that 🙂 .

The nitpick script is available on my GitHub cpplint fork.

If you’re comfortable with Python, you can figure out the script straight from nitpick.py source code and the accompanying unit tests. You can also read the rest of the post for some plain English review 🙂 .

Continue Reading…

Teaching cpplint About External Libraries h-Files

By Thursday, November 13, 2014 0 Software Engineering , , , Permalink 1

The Google C++ Style Guide defines a guideline for names and order of includes. Unfortunately, the cpplint tool that automates style-guide checking isn’t consistent with that guideline, as far as “other libraries h files” are concerned.

In this post, I demonstrate the claimed inconsistency, and suggest an enhancement to cpplint to fix the inconsistent behavior.

The enhancement is available on my GitHub cpplint fork (including addition unit tests).

Continue Reading…

Use cpplint to Check Your C++ Code Against Google’s Style Guide

By Wednesday, November 12, 2014 0 Software Engineering , , Permalink 1

cpplint is an automated checker for C++ code. It checks the style of an input C++ source file against Google’s C++ style guide.

If you’re writing C++ code, and trying to follow the said style guide, I strongly recommend using this tool!

Endless rants can be written on programming style and style-guides. This is not one of them 🙂 .

At the end of the day, my personal opinion is that it doesn’t really matter what conventions you choose to follow. What does matter is that if you collaborate with a team on a common codebase, it’s extremely important to obtain and sustain consistency. To that goal, as long as one consistent style guide is followed by all collaborators – the specifics of the style guide are not too important.

When choosing a style guide to follow, having an automated tool available for collaborators to check themselves has great value. Without it, code reviews end up revolving around style issues instead of the logic being reviewed (you are practicing code reviews, right?). This is the main reason that I default to Google’s C++ style guide.

One of the things I like most about Google’s cpplint tool is that it prefers false negatives over false positives. This means that, while it misses some “style issues”, it usually doesn’t mistakenly produce warnings for compliant code. From my experience, developers are quick to ditch tools that generate lots of cruft. It’s much better that the developer fixes some reported issues instead of ignoring all.

The cpplint tool is available on this official Subversion repository, as well as from my GitHub fork of it. I forked it to make some changes. They might be specific to how I work, but maybe others may benefit from them. Checkout the cpplint tag for more on my changes and other cpplint-related stuff. See also the google-styleguide project.

Shell Foo: Brace Expansion

By Wednesday, November 5, 2014 2 Software Engineering , , Permalink 1

It might be common knowledge, but personally I didn’t know that most shells expand braces to generate all possible combinations!

The scenario for this is when you’re writing a command that takes multiple similar arguments or flags. You can use brace expansion to just write the “schema”, and the shell will generate all the arguments for you!

For example, say you want to create a directory layout for a project. You might use something like mkdir proj/req proj/spec proj/design proj/impl proj/finanes proj/docs. Not too bad, but much nicer to write this instead:

mkdir proj/{req,spec,design,impl,finances,docs}

How about printing all 2-digit hexadecimal numbers? Piece of cake!

$ echo 0x{{0..9},{A..F}}{{0..9},{A..F}}
0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0A 0x0B 0x0C 0x0D 0x0E 0x0F 0x10 0x11 0x12 0x13 0x14 0x15 0x16 0x17 0x18 0x19 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x20 0x21 0x22 0x23 0x24 0x25 0x26 0x27 0x28 0x29 0x2A 0x2B 0x2C 0x2D 0x2E 0x2F 0x30 0x31 0x32 0x33 0x34 0x35 0x36 0x37 0x38 0x39 0x3A 0x3B 0x3C 0x3D 0x3E 0x3F 0x40 0x41 0x42 0x43 0x44 0x45 0x46 0x47 0x48 0x49 0x4A 0x4B 0x4C 0x4D 0x4E 0x4F 0x50 0x51 0x52 0x53 0x54 0x55 0x56 0x57 0x58 0x59 0x5A 0x5B 0x5C 0x5D 0x5E 0x5F 0x60 0x61 0x62 0x63 0x64 0x65 0x66 0x67 0x68 0x69 0x6A 0x6B 0x6C 0x6D 0x6E 0x6F 0x70 0x71 0x72 0x73 0x74 0x75 0x76 0x77 0x78 0x79 0x7A 0x7B 0x7C 0x7D 0x7E 0x7F 0x80 0x81 0x82 0x83 0x84 0x85 0x86 0x87 0x88 0x89 0x8A 0x8B 0x8C 0x8D 0x8E 0x8F 0x90 0x91 0x92 0x93 0x94 0x95 0x96 0x97 0x98 0x99 0x9A 0x9B 0x9C 0x9D 0x9E 0x9F 0xA0 0xA1 0xA2 0xA3 0xA4 0xA5 0xA6 0xA7 0xA8 0xA9 0xAA 0xAB 0xAC 0xAD 0xAE 0xAF 0xB0 0xB1 0xB2 0xB3 0xB4 0xB5 0xB6 0xB7 0xB8 0xB9 0xBA 0xBB 0xBC 0xBD 0xBE 0xBF 0xC0 0xC1 0xC2 0xC3 0xC4 0xC5 0xC6 0xC7 0xC8 0xC9 0xCA 0xCB 0xCC 0xCD 0xCE 0xCF 0xD0 0xD1 0xD2 0xD3 0xD4 0xD5 0xD6 0xD7 0xD8 0xD9 0xDA 0xDB 0xDC 0xDD 0xDE 0xDF 0xE0 0xE1 0xE2 0xE3 0xE4 0xE5 0xE6 0xE7 0xE8 0xE9 0xEA 0xEB 0xEC 0xED 0xEE 0xEF 0xF0 0xF1 0xF2 0xF3 0xF4 0xF5 0xF6 0xF7 0xF8 0xF9 0xFA 0xFB 0xFC 0xFD 0xFE 0xFF

Isn’t it cool? 🙂

I believe this feature is available in many shells, if not all. The examples above were tested on bash on OS X 10.10 and Ubuntu Linux 14.0.4.1.

Shell-Foo credit for this one: Eyal Fink.

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Continue Reading…

Shell Foo: Random Log Sampling

By Wednesday, October 29, 2014 0 Software Engineering , , , Permalink 0

Say you’re executing some long-running program (maybe a service) multiple times. Each execution creates a new log file in /tmp/. Log files are named something like myprog.log.timestamp.pid. Each log entry is a line of text in the file, containing the log level (e.g. “ERROR”, “INFO”, etc.).

How would you choose a random sample of 10 error messages from the latest execution?

Shell-Foo credit for this one: Eyal Fink.

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Continue Reading…

Getting Rid of Redundant Import In SConscripts

By Thursday, October 23, 2014 0 Software Engineering , , Permalink 1

This is the seventh post in my SCons series. The topic of this post is getting rid of the last bit of overhead in SConscript files – the Import('*') line.

According the the SCons user guide, a SConscript files needs to Import('...') shared symbols. It’s possible to import all exported symbols with Import('*'). This is the method I used in previous episodes to make shortcuts available in module-level SConscript files

Perhaps you’re fine with this little remaining overhead in every SConscript file. After all, it’s not such a big deal. With the wildcard syntax, you will never need to go over old SConscript files and update their import list when you add a new shortcut.

If you prefer your SConscript files as minimal as possible, here’s a dirty little hack I use. Instead of passing the shortcuts dictionary to a delegated SConscript file using the exports argument, I modify the _SConscript.GlobalDictdirectly before invoking the SConscript() function.

Using my example project, you may recall from the previous episode how the delegation looks like:

def process_module(self, module):
    print 'scons: |- Reading module', module, '...'
    # Execute the SConscript file, with variant_dir set to the
    #  module dir under the project flavored build dir.
    self._env.SConscript(
        sconscript_path,
        variant_dir=os.path.join(self._env['BUILDROOT'], module),
        exports=shortcuts)

The hack is a minor modification of this snippet:

def process_module(self, module):
    print 'scons: |- Reading module', module, '...'
    # Execute the SConscript file, with variant_dir set to the
    #  module dir under the project flavored build dir.
    SCons.Script._SConscript.GlobalDict.update(shortcuts)
    self._env.SConscript(
        sconscript_path,
        variant_dir=os.path.join(self._env['BUILDROOT'], module))

The result of the hack is that the shortcuts are injected directly into the global dictionary. This means that these symbols are available in SConscript files, without any call to Import(..)!

As always, the entire project is available on my GitHub scons-series repository. Feel free to use / fork / modify. If you do, I’d appreciate it if you share back improvements.

See the scons tag for more in my SCons series.

Continue Reading…