Switch is a holdover from lower-level languages (mostly, various assembly languages) where the destination of a jmp instruction could be computed or looked up in a table. In modern languages, there are usually more appropriate structures for determining which piece of code to jump to: that’s what virtual dispatch does, for example.

Unfortunately, it’s not always obvious how to “fix” code that makes heavy use of switch.

I normally try to avoid this, but for the sake of clarity I’ve omitted a couple of good style points. Online presentation makes certain Java conventions difficult to present; just because the following examples use a single compilation unit with default-visibility symbols doesn’t mean you should, too.

Starting Points

Let’s take a simple example that uses a switch statement to control the state of some service. The surrounding context involves a simple control protocol for a light dimmer: it can either be turned fully off, turned fully on, or set to a “dim” state halfway between the two.

class States {
    public final int OFF = 0;
    public final int DIM = 1;
    public final int ON = 2;
}
 
class DimmerController {
    public void setIntensity(int intensity) {
        doStuffBeforeChangingIntensity();
        switch (intensity) {
            case OFF:
                disable();
                break;
            case DIM:
                beginDimCycle();
                break;
            case ON:
                beginFullCycle();
                break;
        }
        doStuffAfterChangingIntensity();
    }
 
    private void disable() {
        /* ... */
    }
 
    private void beginDimCycle() {
        /* ... */
    }
 
    private void beginFullCycle() {
        /* ... */
    }
}

Eminent Domain

There are a few obvious problems with the version above. For starters, there’s nothing preventing someone from passing the wrong value, either from the wrong set of constants or by hard-coding the literal values instead of using the constants. We could add a default clause that raises an IllegalArgumentException, but we’d still only find out about the problem at runtime. We’d really like to find out about errors in our programs as early as possible: compile time, or even as we type them. Most languages provide a tool for doing that; since we’re using Java, we’ll use an enum.

enum State {
    OFF,
    DIM,
    ON
}
 
class DimmerController {
    public void setIntensity(State intensity) {
        doStuffBeforeChangingIntensity();
        switch (intensity) {
            case OFF:
                disable();
                break;
            case DIM:
                beginDimCycle();
                break;
            case ON:
                beginFullCycle();
                break;
        }
        doStuffAfterChangingIntensity();
    }
 
    private void disable() {
        /* ... */
    }
 
    private void beginDimCycle() {
        /* ... */
    }
 
    private void beginFullCycle() {
        /* ... */
    }
}

On Your Best Behaviour

That’s better, but it still has most of the fundamental problems switch statements bring with them. While I’m only showing one switch, apps that have them tend to have multiple switches over the same set of constants, which leads to code duplication throughout the app and makes it hard to determine what a given constant will cause the system to do. In an object-oriented system, we normally try to keep behaviour with the data it works with, and as a side effect of refactoring to using an enumerated type, we now have real objects to use. Let’s add some methods to our enumerated type to handle the per-case behaviour.

enum State {
    OFF() {
        public void apply() {
            disable();
        }
 
        private void disable() {
            /* ... */
        }
    },
    DIM() {
        public void apply() {
            beginDimCycle();
        }
 
        private void beginDimCycle() {
            /* ... */
        }
    },
    ON() {
        public void apply() {
            beginFullCycle();
        }
 
        private void beginFullCycle() {
            /* ... */
        }
    };
 
    public abstract void apply();
}
 
class DimmerController {
    public void setIntensity(State intensity) {
        doStuffBeforeChangingIntensity();
        intensity.apply();
        doStuffAfterChangingIntensity();
    }
}

Opening the Set

You can stop here and have a reasonably well-structured program. The specifics of what each constant means are stored with the constant, and it’s now impossible to pass an illegal value into setIntensity. However, we’ve still embedded into DimmerController the assumption that we will only ever be setting the intensity of a three-state light - but we don’t actually use that assumption anywhere. We can refactor further to eliminate this dependency by realizing that State’s public interface really represents any arbitrary dimmer state callback:

 
interface State {
    public void apply();
}
 
enum DimmerState implements State {
    OFF() {
        public void apply() {
            disable();
        }
 
        private void disable() {
            /* ... */
        }
    },
    DIM() {
        public void apply() {
            beginDimCycle();
        }
 
        private void beginDimCycle() {
            /* ... */
        }
    },
    ON() {
        public void apply() {
            beginFullCycle();
        }
 
        private void beginFullCycle() {
            /* ... */
        }
    };
 
    public abstract void apply();
}
 
class DimmerController {
    public void setIntensity(State intensity) {
        doStuffBeforeChangingIntensity();
        intensity.apply();
        doStuffAfterChangingIntensity();
    }
}

Everything In Its Place

On the other hand, sometimes the heavy lifting (in our case, the work of actually changing the intensity of a light) actually belongs outside the enumeration. This is true most of the time: enumerations can’t hold per-use state, and creating a new State instance (from the interface-based example above) for each light isn’t always appropriate. We can pull those methods back out of the enumeration by introducing a third class to act as a callback for the apply method:

enum State {
    OFF() {
        public void applyTo(DimmerCallback dimmerCallback) {
            dimmerCallback.disable();
        }
    },
    DIM() {
        public void applyTo(DimmerCallback dimmerCallback) {
            dimmerCallback.beginDimCycle();
        }
    },
    ON() {
        public void applyTo(DimmerCallback dimmerCallback) {
            dimmerCallback.beginFullCycle();
        }
    };
 
    public abstract void applyTo(DimmerCallback dimmerCallback);
}
 
 
class /* or interface */ DimmerCallback {
    private void disable() {
        /* ... */
    }
 
    private void beginDimCycle() {
        /* ... */
    }
 
    private void beginFullCycle() {
        /* ... */
    }
}
 
class DimmerController {
    // Or from dependency injection, or...
    private final DimmerCallback dimmerCallback = new DimmerCallback();
 
    public void setIntensity(State intensity) {
        doStuffBeforeChangingIntensity();
        intensty.applyTo(dimmerCallback);
        doStuffAfterChangingIntensity();
    }
}

Ok, But I Have This Integer

In most of the cases where people feel compelled to use magic ints or an enumerated type, there’s an external force at work. The most common scenario involves file formats or network protocols where an integer or short known string needs to map directly to program behaviour. With the behaviour separated out into a real type with testable features, you lose the ability to near-blindly pass the protocol value directly into your logic.

Remember when I said that switch statements are really a giant lookup table? That should give you a hint. You can still use a lookup table to convert from an external representation to an internal one; depending on how far down the chain of refactorings you’ve gone, you may even get one for free. Consider the following, assuming we still have a State enumeration:

public State parseRequestedState(int state) throws ProtocolException {
    try {
        return State.values()[state];
    } catch (IndexOutOfBoundsException ioobe) {
        throw new UnexpectedValueException(ioobe);
   }
}

Java’s enumeration support gives us the lookup table we need for free. We can do the same thing with strings, if they happen to match the enum constants’ names; alternately, we can use an explicit lookup table stored in an array or a HashMap. There are also plenty of strategies for automatically populating a lookup table using mechanisms like classpath scanning, ServiceLoader, annotation processing tools, and so on.

There are lots of places to go from here. For example, enum-based implementations of the state object pattern begin with a very similar refactoring. In languages without strong enums, the structure outlined here can be implemented using normal classes with a fixed pool of instances, instead (Java’s enum support does exactly that, under the hood), skipping directly to the interface-based version near the end.

1
2
3
4
5

<?php
while ($read < $n && (false !== ($buf = fread($this->sock, $n - $read)))) {
    /* ... */
}
?>

contains a subtle bug. Go ahead, read the relevant function documentation and see if you can spot it.

PHP’s fread function directly maps the C library’s fread function. As TBlue explains, the C fread function returns 0 when reading at the end of a stream. PHP does the simplest possible thing with this and returns an empty string; to determine that the stream is at end of file, programs must check with feof as well.

Most modern languages realize that explicitly checking for errors is a fairly error-prone convention. Java’s InputStream.read(byte[]) methods return -1 on EOF (and only on EOF) and raise exceptions on errors. Python’s File-like objects convention supports mechanisms other than raw byte reads (primarily, iteration) that make stopping at end of file implicit, on top of only returning an empty string at end of input (as with Java, Python raises exceptions when a stream read fails). Not PHP, though! In PHP, the loop above has to look like

1
2
3
4
5
6

<?php
while ($read < $n && !feof($this->sock) && (false !== ($buf = fread($this->sock, $n - 
$read)))) {
    /* ... */
}
?>

Encountered in the wild: php-amqplib bug #12.

The Singleton pattern specifically refers to the object creational pattern where a class manages a single instance of itself and, usually, forbids other instances from existing. There’s also the “singleton” scope in Spring, which means something more limited: Spring will only create one instance of the bean. Don’t confuse the arguments that Singleton classes are a bad idea for arguments that objects you only have one of are a bad idea.

I mention this because Spring scopes come up in the following examples, and the default Spring scope is also called singleton. Now, with that out of the way, on with the show.

Moving a Singleton creation dependency to Spring

Injecting a singleton into its clients involves replacing code like this:

public class Mailer {
  private static final Mailer instance = new Mailer ();
 
  public static Mailer getInstance () { return instance; }
 
  private Mailer () { }
 
  public void sendMail (String recipient, String message) {
    System.out.printf ("I am sending %s to %s!\n",
      message, recipient);
  }
}
 
/* Using a Struts2 Action as an example "client" */
public class SendMailAction {
  public void setRecipient (String recipient) {
    this.recipient = recipient;
  }
 
  public void setMessage (String message) {
    this.message = message;
  }
 
  public void execute () {
    Mailer.getInstance ().sendMail (recipient, message);
  }
 
  private String message;
  private String recipient;
}

with code like this:

public class SendMailAction {
  public SendMailAction (Mailer mailer) {
    this.mailer = mailer;
  }
 
  public void setRecipient (String recipient) {
    this.recipient = recipient;
  }
 
  public void setMessage (String message) {
    this.message = message;
  }
 
  public void execute () {
    mailer.sendMail (recipient, message);
  }
 
  private final Mailer mailer;
  private String message;
  private String recipient;
}

The corresponding Spring configuration file looks like

<beans xmlns="http://www.springframework.org/schema/beans">
  <bean id="mailer"
        class="ca.grimoire.examples.Mailer"
        factory-method="getInstance" />
 
  <bean id="action"
        scope="request"
        class="ca.grimoire.examples.SendMailAction">
    <constructor-arg ref="mailer" />
  </bean>
 
  <!-- more definitions -->
</beans>

If getInstance, for some reason, needs parameters, they can be passed as elements nested inside its bean definition.

You can now write isolated tests for SendMailAction:

// Using JMock rather than a hand-crafted alternate implementation.
@RunWith(JMock.class)
public class SendMailTest {
  // This requires some extension to work with class mocks
  // See http://www.jmock.org/mocking-classes.html
  private final Mockery mockery = new JUnit4Mockery() {{
    setImposteriser(ClassImposteriser.INSTANCE);
  }};
 
  @Test
  public void sendsExpectedMessage () {
    final Mailer mailer = mockery.mock (Mailer.class);
 
    mockery.checking (new Expectations () {{
      oneOf (mailer).sendMail ("Recipient", "Hello World");
    }});
 
    SendMailAction action = new SendMailAction (mailer);
    action.setRecipient ("Recipient");
    action.setMessage ("Hello World");
 
    action.execute ();
  }
}

Prior to this change, writing a test for SendMailAction that did not also involve Mailer would’ve been difficult and error-prone.

Injecting dependencies into Singletons

You can also inject dependencies, including other Singleton instances, into a Singleton instance, which is helpful for breaking Singleton graphs. Let’s give Mailer another Singleton it depends on:

public class SmtpConnection {
  private static final SmtpConnection instance = new SmtpConnection ();
 
  public static SmtpConnection getInstance () { return instance; }
 
  public PrintStream getSmtpStream () {
    return System.out;
  }
}
 
public class Mailer {
  private static final Mailer instance = new Mailer ();
 
  public static Mailer getInstance () { return instance; }
 
  private Mailer () { }
 
  public void sendMail (String recipient, String message) {
    SmtpConnection connection = SmtpConnection.getInstance ();
    connection.getSmtpStream ().printf (
      "I am sending %s to %s!\n", message, recipient);
  }
}

You can move this dependency into Spring if you change Mailer:

public class Mailer {
  private static final Mailer instance = new Mailer ();
 
  public static Mailer getInstance () { return instance; }
 
  private Mailer () { }
 
  public void setConnection (SmtpConnection connection) {
    this.connection = connection;
  }
 
  public void sendMail (String recipient, String message) {
    connection.getSmtpStream ().printf (
      "I am sending %s to %s!\n", message, recipient);
  }
 
  private SmtpConnection connection;
}

The corresponding Spring configuration looks like:

<beans xmlns="http://www.springframework.org/schema/beans">
 
  <bean id="connection"
        class="ca.grimoire.examples.SmtpConnection"
        factory-method="getInstance" />
 
  <bean id="mailer"
        class="ca.grimoire.examples.Mailer"
        factory-method="getInstance">
    <property name="connection" ref="connection" />
  </bean>
 
  <!-- more definitions -->
</beans>

Even clients who aren’t using Spring to inject the Mailer Singleton can be used with this, so long as the Spring injection happens before anything tries to use the Mailer.

Without an interface limiting the operations, though, clients of Mailer may decide to call setConnection - don’t do this because it becomes a form of mutable global state. If you’re going to follow this route, either force clients to use a narrower interface without the setters or be very careful about code review.

The Singleton pattern is one of the common solutions identified in Design Patterns. At its core, a Singleton is a class that only allows one instance. All clients of the class will operate against the same instance. The implementation outlined in Design Patterns provides a class-level function replacing the class’ constructor in its public API which manages the existence of the singleton instance.

A fairly stock Java Singleton implementation for, for example, sending mail runs like this:

public class Mailer {
  private static final Mailer instance = new Mailer ();
 
  public static Mailer getInstance () { return instance; }
 
  /* Note the access specifier: no other class can access this
     constructor. */
  private Mailer () {
    /* ... */
  }
 
  public void sendMail (String recipient, String message) {
    /* ... */
  }
}

From an implementor’s point of view, Singleton classes cost almost nothing, either in terms of extra code to maintain or in terms of assumptions to test. In the average application, each class providing some service to the application only has one instance, so constraining the class to only one instance isn’t a huge leap. These kinds of classes tend to be stateless, or almost stateless, so clients don’t necessarily need to be aware that they’re sharing instances with other clients as there are no extra synchronization or garbage collection hazards to handle. There are even times when Singletons are the right thing: either they accurately model some aspect of the problem where only a single representative can exist (for example, Java’s Runtime class, which exposes services provided by the JVM running the program), or the resulting API is compellingly simple and easy to use (for example, Log4J’s Logger, which is effectively backed by a Singleton registry of Logger instances).

Implementing a Singleton correctly can easily depend on the dark corners of the language’s object model. The most common source of Singleton implementation bugs are caused by language or deployment models that scope “global” state to only part of the system. Networked systems can easily have an instance on every node, and applications that have multiple AppDomains or ClassLoaders can easily create multiple Singleton instances in a single address space. In multithreaded apps (that is, almost anything written this decade), Singletons can also be threading hazards: if they manage access to an external resource, then it’s very easy to build a synchronization bottleneck into your entire application in the name of keeping access to that resource thread-safe.

Singletons cause most of the problems associated with them on the consumer side. The most common implementation pattern for Singleton clients is very simple: every occurrence of

Mailer m = new Mailer (/* ... */);

gets replaced with

Mailer m = Mailer.getInstance ();

This wins nothing: the client is tightly bound to a number of assumptions about the identity and lifecycle of the provider. In fact, it loses a little: instead of giving each client its own isolated state, now there’s a guarantee that some state is shared between client instances, via the Singleton instance. There are also knock-on effects: because the client controls exactly which provider it uses, it’s difficult to isolate only the client class for testing. It also becomes difficult, if not impossible, to insert behaviour between the client and the singleton provider - no AOP, no proxies, and no remoting, without making invasive changes to the client or the singleton.

Over the long term, as software evolves, patterns tend to propagate. Inline getInstance() calls for Singleton clients tend to get repeated in new code. In a sufficiently evolved codebase that uses Singletons extensively, a badly-placed getInstance() call can easily cause half the objects in the system to spring into being and initialize themselves. Even when this doesn’t cause performance issues on some code paths, it can easily cause startup and initialization order to become unpredictable during apparently-safe changes. Remembering which code paths are “supposed” to initialize singletons becomes an extra form of friction for developers working on the codebase.

Singletons aren’t popular by accident, though. In most modern object-oriented systems, there is a constellation of core classes which, in the final system, only need to exist once. The Mailer example I’ve been using is actually a good example: if it manages connections to the underlying mail system itself in a sane way, then there probably only needs to be a single instance of Mailer in the deployed app. If Mailer handles any connection pooling, then ensuring the creation of new instances is carefully controlled becomes almost mandatory. Singletons provide a really simple way to provide a single instance to an application.

Most development time, thought, and effort goes into the code paths that get used in the final product, so there’s an easy conceptual leap to make from “the system I’m building only needs one instance” to “this object should only allow one instance.” Developers without fairly extensive experience with mocking and other unit-testing techniques, or who don’t value the ability to insert new behaviour without changing existing code, can easily see no problem with using Singletons to manage lifetime and to provide access to the instance, since it’s convenient.

There are alternatives to Singletons for managing service objects: the standard Java examples are EJBs, which delegates control over object lifecycles to the app container, and Spring, which takes responsibility for lifecycle management and wiring between providers and consumers within the app. Other languages have similar tools available, either in the language or as a library. However, these tools have a higher barrier to entry than “just another Singleton”: even the simplest inversion of control implementation is more lines of code than a Singleton.

The kinds of problems Singletons cause only really become problems in medium-sized and larger codebases. Not coincidentally, medium-sized and larger codebases tend to be the product of long evolution from a small, relatively clean initial product, which means there’s been lots of time for people to forget the program’s overall structure. Refactoring to clean up the problems isn’t difficult, but surprisingly little has been written about it.

For most Singletons, where obtaining an instance is a parameter-less call to a getInstance()-like method, refactoring that call out to an injected dependency via a constructor or method parameter is a low-risk change that can be made and tested for one class at a time. Most, if not all, getInstance() calls follow a very simple, predictable code path, which means there are no side effects to take care of, and no changes at all have to happen in the provider. The only potential “gotcha” is the first call, which may do initialization work. This kind of change also has high rewards: injected dependencies can be replaced with alternate implementations, which makes testing much easier and lets you augment or replace the implementation without altering the clients.

There’s more risk in changing the Singleton itself first. Refactoring the lifecycle management code out of the Singleton and into a higher layer does help, but while clients are calling getInstance() directly, any change to the lifecycle management means changing every client at the same time, which in turn means you need to re-test all of those clients. Once the clients have been refactored, though, it’s much safer and easier to change the lifecycle managment.

Singletons are not the Great Evil they’re sometimes depicted as. However, they’re used in a lot of situations as a crutch to avoid writing real inter-object dependency management, to the detriment of software using them. Their convenience can outweigh the problems, especially in small systems or prototypes, but you need to have a way out when they start to cause problems.

Next time: implementation examples, for Spring.

The next time you think to reach for the “reformat source code” button in your IDE, think twice before you press it. Automatic reformatting is a great way to turn a small merge hazard contained to a single method into a huge merge hazard affecting an entire file. Merge tools are relatively stupid: in general, they’re not aware of the syntactic structure of a source file, just the text by lines; as a result, it’s very easy to fool them into thinking a one-line change and a reformat are a conflict covering the entire file.

If you’re using automatic formatting tools, please check with the other developers on your project and make sure that your code formatter settings agree with theirs. If you can’t make that happen, then don’t reformat code unless everyone’s nobody but you is touching those files.

Please, think of the merges.

Yes, there is only one user. Yes, I still think it’s worthwhile.

Now raise your hand if your compiler or build tool can tell you about them.

That’s been a personal itch of mine for a while, so I took the chance to scratch it and learn the Java annotation processing API at the same time. The result: TODO Annotations, a set of annotations intended to replace // TODO comments with something easier to check. Now I can have my build server complain and die when FIXMEs go un-fixed without imposing the same stricture on my development environment.

You can grab the source and bang on it - it provides a decent “hello world” example of working with the compiler. For more information on what the API can do, check the annotation processing API, as well as the language model API, and the Annotation Processing Tool docs. Patches, bug reports, and feedback are always welcome. Binaries are also available in the Maven repository.

Or, you can use the Maven taglist plugin - but then you’re stuck with Maven.

make understanding

Make’s basic approach to building software is based on walking directed graphs. Each node of a Makefile’s graph represents a file, and the arrows point to other nodes representing dependencies. Make begins at the nodes named on the command line and follows the arrows to determine which other nodes to build first.

Make reads the graph, and the actions required to build any files it decides it needs to, are read from a text file called a Makefile with a very simple structure. Lines like

alice: bob gerald

specify that a file named alice depends on files named bob and gerald. Dependency rules can also include shell commands to perform when building alice:

alice: bob gerald
        cat bob gerald > alice

Makefiles are an elegant way to describe the proces of building native programs, but on their own they’re insufficient and inefficient for larger projects.

make hit-and-miss

Make grew out of the Unix culture. Make itself is designed to run from a shell, which makes it very easy to automate; Make editing modes and execution plugins exist for every worthwhile Unix text editor and Make itself forms the backbone for a number of Unix IDEs. In turn, Make uses a list of shell commands to execute at each step, which makes it very easy to extend—in fact, Make has no way to compile or package code on its own, relying on a toolchain like GCC for the finer details.

The same culture also popularized the C language, which requires both compilation into intermediate object files and linking into the final program. Because compiling source code can take a while for large projects, Make has features to support incremental recompilation. When Make encounters a node whose file is newer than the files of all dependency nodes, it skips the build stage for that file on the assumption that it’s already up to date. This means that running the same Make command twice will usually do different, but equivalent, things: the first one will build the entire dependency graph from source, and the second one will notice that everything’s up to date and do nothing.

On the surface, this is a safe optimization: if your source files haven’t changed, and your compile process is deterministic, then you don’t need to rebuild the intermediate or final products. However, specificying the project’s dependency graph correctly is not as easy as it appears.

Consider the following C program, composed of two source files and a header:

logging.h

#ifndef LOGGING
#define LOGGING

void log_message (const char *message);

#endif

logging.c

#include 
#include "logging.h"

void log_message (const char *message) {
    printf ("Log message: %s\n", message);
}

main.c

#include "logging.h"

int main () {
    log_message ("Hello, world.");
    return 0;
}

The final product, an executable named example-1, has a dependency graph shaped like this:

Both objects depend on the header file and one source file.

Unfortuntely, Make’s code-agnostic nature makes is unable to spot the dependencies on logging.h without help, giving this graph:

Make's stock rules are unaware of included files.

Adding a new feature to the “logging” library by modifying the source and header will recompile both logging.o and the final program, but not main.o. Depending on the kind of changes involved, this could be harmless, or it could lead to some fairly strange bugs—bugs that magically disappear if you build from a clean environment.

For a trivial project, like this one, it was easy enough to specify them manually in the Makefile I used to test this example:

example-1: main.o logging.o
	$(CC) $(LDFLAGS) $^ -o $@

main.o: main.c logging.h

logging.o: logging.c logging.h

In a real project, however, there can be tens or hundreds of header files, and manually maintaining the list of header files in two places (in the source itself, and in the Makefile) is tedious and error-prone. There is no good, general-case solution for this, but there are language-specific tools that can parse a source tree and spit out a list of include files in a format Make can understand. Using these tools is one more piece of complexity in the build and one more obstacle to maintenance, compounded by the Unix philosophy of gluing tools together.

GNU Make’s manual suggests this snippet for dependency analysis of C and C++ projects:

%.d: %.c
        @set -e; rm -f $@; \
         $(CC) -M $(CPPFLAGS) $< > $@.$$$$; \
         sed 's,\($*\)\.o[ :]*,\1.o $@ : ,g' < $@.$$$$ > $@; \
         rm -f $@.$$$$

include *.d

…clear as mud.

Various tools have grown up around Make that automate this and other kinds of configuration, from various dependency scanning tools to the GNU Autotools suite, used by almost every recent Unix-and-C project. These tools generate Makefiles that are about as comprehensible as that last snippet and degrade both the portability and the simplicity of the build system.

Make’s own tools for streamlining builds aren’t much better: “pattern” rules, which are made out of a combination of a wildcard matching language and special variables in the Makefile, can bridge gaps in dependency trees and allow Make to infer intermediate steps from the sources available and the target to be built. Patterns can also standardize builds by packaging up common steps; the pattern rule Make provides for compiling C source files takes advantage of this:

%.o: %.c
        $(CC) -c $(CPPFLAGS) $(CFLAGS) $^ -o $@

In a large project that makes heavy use of patterns, though, it rapidly becomes impossible to determine which source files will be rebuilt until you actually go to run the build.

make finished

Designing your project to play to Make’s strengths can create simple, readable, fast builds. However, even with the voluminous documentation on how to use Make available on the web, it can be hard to keep your build within Make’s outdated build model in the face of complex projects. If you plan to use Make, find someone who’s already experienced with it before setting out; it’s not simple enough for an inexperienced team to use effectively, and by the time they can use it to its fullest, they’ll be experts and your project will be late.

I’ve started with Make because it was one of the first tools to formalize the idea of a build process. While it hasn’t aged particularly well, and its support for comfort features and chrome is weak compared to modern offerings, Make is still a competent tool and does well at three of the fundamentals of a build system: automatability, standardization, and extensibility.

Edited Nov. 18, 2008: corrected the name of the header in #include directives in the example. Thanks, Plouj!

Curiousity piqued, I cooked up a script that reproduces the problem and performs the merge from various directions to examine the results. Subversion, sadly, performed dismally: none of the merge scenarios tested retained content changes when merging structural changes to the same files.

The Preferred Outcome

Both changes survive the merge.

The diagram above shows a very simple source tree with one directory, dir-a, containing one file with two lines in it. On one branch, the file is modified to have a third line; on another branch, the directory is renamed to dir-b. Then, both branches are merged, and the resulting tree contains both sets of changes: the file has three lines, and the directory has a new name.

This is the preferred outcome, as no changes are lost or require manual merging.

Subversion

Subversion loses the content change.

There are two merge scenarios in this diagram, with almost the same outcome. On the left, a working copy of the branch where the file’s content changed is checked out, then the changes from the branch where the structure changed are merged in. On the right, a working copy of the branch where the structure changed is checked out, then the changes from the branch where the content changed are merged in. In both cases, the result of the merge has the new directory name, and the original file contents. In one case, the merge triggers a rather opaque warning about a “missing file”; in the other, the merge silently ignores the content changes.

This is a consequence of the way Subversion implements renames and copies. When Subversion assembles a changeset for committing to the repository, it comes up with a list of primitive operations that reproduce the change. There is no primitive that says “this object was moved,” only primitives which say “this object was deleted” or “this object was added, as a copy of that object.” When you move a file in Subversion, those two operations are scheduled. Later, when Subversion goes to merge content changes to the original file, all it sees is that the file has been deleted; it’s completely unaware that there is a new name for the same file.

This would be fairly easy to remedy by adding a “this object was moved to that object” primitive to the changeset language, and a bug report for just such a feature was filed in 2002. However, by that time Subversion’s repository and changeset formats had essentially frozen, as Subversion was approaching a 1.0 release and more important bugs without workarounds were a priority.

There is some work going on in Subversion 1.6 to handle tree conflicts (the kind of conflicts that come from this kind of structural change) more sensibly, which will cause the two merges above to generate a Conflict result, which is not as good as automatically merging it but far better than silently ignoring changes.

Mercurial

Mercurial preserves the content change.

Interestingly, there are tools which get this merge scenario right: the diagram above shows how Mercurial handles the same two tests. Since its changeset language does include an “object moved” primitive, it’s able to take a content change for dir-a/file and apply it to dir-b/file if appropriate.

Further Resources

If you feel like reproducing this yourself, or want to adapt my test scripts to work with your favourite version control system, the scripts are available from Mercurial or as a zip. Patches and suggestions welcome.

Build systems are critical to software development.

They’re also one of the most common avoidable engineering failures.

A reliable, comfortable build system has measurable benefits for software development. Being able to build a testable, deployable system at any point during development lets the team test more frequently. Frequent testing isolates bugs and integration problems earlier, reducing their impact. Simple, working builds allow new team members to ramp up more quickly on a project: once they understand how one piece of the system is constructed, they can apply that knowledge to the entire system and move on to doing useful work. If releases, the points where code is made available outside the development team, are done using the same build system that developers use in daily life, there will be fewer surprises during releases as the “release” build process will be well-understood from development.

Builds Have Needs, Too

In 1947, Abraham Maslow described a hierarchy of needs for a person’s physical and mental well-being on the premise that all the items at the lowest level of the hierarchy must be met before a person will be able to focus usefully on higher-level needs. Maslow’s hierarchy begins with a set of needs that, without which, you do not have a person (for long)—physiological needs like “breathing,” “food,” and “water.” At the peak, there are extremely high-level needs that are about being a happy and enlightened person—”creativity,” “morality,” “curiosity,” and so on.

Builds, and software engineering as a whole, can be described the same way: at the top of the hierarchy is a working system that solves a problem, and at the bottom are the things you need to have software at all. If you don’t meet needs at a given level, you will eventually be forced to stop what you’re doing at a higher level and face them.

Before a build is a build, there are five key needs to meet:

• It must be repeatable. Every time you start your build on a given source tree, it must build exactly the same products without any further intervention. Without this, you can’t reliably decide whether a given build is “good,” and can easily wind up with a build that needs to be run several times, or a build that relies on running several commands in the right order, to produce a build.
• It must be automatable. Build systems are used by developers sitting at their desks, but they’re also used by automatic build systems for nightly builds and continuous integration, and they can be made into parts of other builds. A build system that can only be run by having someone sit down at a keyboard and mouse and kicking it off can’t be integrated into anything else.
• It must be standardized. If you have multiple projects that build similar things—for example, several Java libraries—all of them must be built the same way. Without this, it’s difficult for a developer to apply knowledge from one project to another, and it’s difficult to debug problems with individual builds.
• It must be extensible. Not all builds are created equal. Where one build compiles a set of source files, another needs five libraries and a WSDL descriptor before it can compile anything. There must be affordances within the standard build that allow developers to describe the ways their build is different. Without this, you have to write what amounts to a second build tool to ensure that all the “extra” steps for certain projects happen.
• Someone must understand it. A build nobody understands is a time bomb: when it finally breaks (and it will), your project will be crippled until someone fixes it or, more likely, hacks around it.

If you have these five things, you have a working build. The next step is to make it comfortable. Comfortable builds can be used daily for development work, demonstrations, and tests as well as during releases; builds that are used constantly don’t get a chance to “rust” as developers ignore them until a release or a demo and don’t hide surprises for launch day.

• It must be simple. When a complicated build breaks, you need someone who understands it to fix it for you. Simple builds mean more people can understand it and fewer things can break.
• It must be fast. A slow build will be hacked around or ignored entirely. Ideally, someone creating a local build for a small change should have a build ready in seconds.
• It must be part of the product. The team responsible for developing a project must be in control of and responsible for its build. Changes to it and bugs against it must be treated as changes to the product or bugs in the product.
• It must run unit tests. Unit tests, which are completely isolated tests written by and for developers, can catch a large number of bugs, but they’re only useful if they get run. The build must run the unit test suite for the product it’s building every build.
• It must build the same thing in any environment. A build is no good if developers can only get a working build from a specific machine, or where a build from one developer’s machine is useless anywhere else. If the build is uniform on any environment, any developer can cook up a build for a test or demo at any time.

Finally, there are “chrome” features that take a build from effective to excellent. These vary widely from project to project and from organization to organization. Here are some common chrome needs:

• It should integrate with your IDEs. This goes both directions: it should be possible to run the build without leaving your IDE or editor suite, and it should be possible to translate the build system into IDE-specific configurations to reduce duplication between IDE settings and the build configuration.
• It should generate metrics. If you gather metrics for test coverage, common bugs, complexity analysis, or generate reports or documentation, the build system should be responsible for it. This keeps all the common administrative actions for the project in the same place as the rest of the configuration, and provides the same consistency that the system gives the rest of the build.
• It should support multiple processors. For medium-sized builds that aren’t yet large enough to merit breaking down into libraries, being able to perform independent build steps in parallel can be a major time-saver. This can extend to distributed build systems, where idle CPU time can be donated to other peoples’ builds.
• It should run integration and acceptance tests. Taking manual work from the quality control phase of a project and running it automatically during builds amplifies the benefits of early testing and, if your acceptance tests are good, when your project is done.
• It should not need repeating. Once you declare a particular set of build products “done”, you should be able to use those products as-is any time you need them. Without this, you will eventually find yourself rebuilding the same code from the same release over and over again.

What Doesn’t Work

Builds, like any other part of software development, have antipatterns—recurring techniques for solving a problem that introduce more problems.

• One Source Tree, Many Products. Many small software projects that survive to grow into large, monolithic projects are eventually broken up into components. It’s easy to do this by taking the existing source tree and building parts of it, and it’s also wrong. Builds that slice up a single source tree require too much discipline to maintain and too much mental effort to understand. Break your build into separate projects that are built separately, and have each build produce one product.
• The Build And Deploy System. Applications that have a server component often choose to automate deployment and setup using the same build system that builds the project. Too often, the extra build steps that set up a working system from the built project are tacked onto the end of an existing build. This breaks standardization, making that build harder to understand, and means that that one build is producing more than one thing—it’s producing the actual project, and a working system around the project.
• The Build Button. IDEs are really good at editing code. Most of them will produce a build for you, too. Don’t rely on IDE builds for your build system, and don’t let the IDE reconfigure the build process. Most IDEs don’t differentiate between settings that apply to the project and settings that apply to the local environment, leading to builds that rely on libraries or other projects being in specific places and on specific IDE settings that are often buried in complex settings dialogs.
• Manual Steps. Anything that gets done by hand will eventually be done wrong. Automate every step.

What Does Work

Similarly, there are patterns—solutions that recur naturally and can be applied to many problems.

• Do One Thing Well. The UNIX philosophy of small, cohesive tools works for build systems, too: if you need to build a package, and then install it on a server, write three builds: one that builds the package, one that takes a package and installs it, and a third that runs the first two builds in order. The individual builds will be small enough to easily understand and easy to standardize, and the package ends up installed on the server when the main build finishes.
• Dependency Repositories. After a build is done, make the built product available to other builds and to the user for reuse rather than rebuilding it every time you need it. Similarly, libraries and other inward dependencies for a build can be shared between builds, reducing duplication between projects.
• Convention Over Extension. While it’s great that your build system is extensible, think hard about whether you really need to extend your build. Each extension makes that project’s build that much harder to understand and adds one more point of failure.

Pick A Tool, Any Tool

Nothing here is new. The value of build systems has been discussed in great detail elsewhere. Much of the accumulated build wisdom of the software industry has already been incorporated to one degree or another into build tools. What matters is that you pick one, then use it with the discipline needed to get repeatable results without thinking.