TestMain—What is it Good For?

As stated in the release notes, Go 1.4 focuses primarily on implementation work, but it also provides a few new tools for developers. The testing package provides one of these tools. Test code may now contain a TestMain function that provides more control over running tests than was available in prior releases of Go.

Justinas Stankevičius and Cory Jacobsen have written excellent posts about TestMain recently. Both posts demonstrate how a package can perform global setup and shutdown steps when running tests, but there are other uses for TestMain.

First a quick review of the mechanics. When we run go test the go tool generates a main program that runs the test functions for our package. If the go tool finds a TestMain function it instead generates code to call TestMain. A typical TestMain follows.

func TestMain(m *testing.M) {
    setup()
    code := m.Run() 
    shutdown()
    os.Exit(code)
}

From the above code we can see that writing a TestMain function allows us to control four aspects of test execution.

  1. Setup.
  2. How and when to run the tests.
  3. Shutdown.
  4. Exit behavior.

With appropriate use of these four items, we can satisfy several different use cases that were not well served prior to Go 1.4.

Use Cases

The following sections describe four uses for TestMain that I think are worth understanding. The first two cases motivated the Go team to add support for TestMain. The third case is drawn from my experience with a project at my day job. The last use case is the sole example of a TestMain in the Go 1.4 standard library.

Global Startup and Shutdown Hooks

As already mentioned, adding a TestMain to a package allows it to run arbitrary code before and after tests run. But only the second part is really new. Pre-test setup has always been possible by defining an init() function in a test file. The challenge—as described in issue #8159—has been bookending that with corresponding shutdown code when all tests have completed.

Testing from the Main OS Thread

Many graphics libraries are particular about which OS thread calls their API. Gustavo Niemeyer encountered this while working on his qml package for Go. In issue #8202 Russ Cox demonstrated how TestMain can address this need:

func init() {
    runtime.LockOSThread()
}

func TestMain(m *testing.Main) {
    go func() {
        os.Exit(m.Run())
    }()
    runGraphics()
}

The above code ensures that runGraphics() runs in the main goroutine locked to the main OS thread while the package tests run elsewhere. Presumably the tests exercise APIs that communicate with the runGraphics() goroutine.

Subprocess Tests

Sometimes we need to test the behavior of a process, rather than a function. For example, one of my projects follows the crash-only software design philosophy. We have tests that deliberately crash and restart pieces of the system in different combinations and permutations between key events.

Initially we wrote a separate package main that produced an executable the crash tests launched. This approach worked, but it was fragile. When we forgot to rebuild the test executable before running go test the behavior we just tried to fix would still appear broken.

We recently adopted a better approach demonstrated by Andrew Gerrand in the presentation on testing techniques he gave at Google I/O 2014. On slide 23, he showed how to reuse the test binary produced by go test as the child process. This technique eliminates the need for a separate build and ensures the testing and tested code are always in sync.

TestMain was not available at the time Andrew gave his presentation. In Go 1.3 our only choice was for the child process to call the tested code from within a TestXxx() function. This constraint came with some baggage.

  • The command line arguments passed to the child process must be compatible with the testing package.
  • The test framework will, at a minimum, write either ‘PASS’ or ‘FAIL’ to stdout after the test function returns. The child process must make sure to call os.Exit() rather than return from the TestXxx() function to avoid the extra output.
  • We end up writing pseudo-tests such as this example from the os/exec package:

    // TestHelperProcess isn't a real test. It's used
    // as a helper process for TestParameterRun.
    func TestHelperProcess(*testing.T) {
        if os.Getenv("GO_WANT_HELPER_PROCESS") != "1" {
            return
        }
        defer os.Exit(0)
        ...
    }
    

These issues are admittedly not major. They do not—by themselves—make a convincing argument for adding the TestMain feature. But TestMain gives us a potentially less finicky solution. Adapting Andrew’s example:

func Crasher() {
    fmt.Println("Going down in flames!")
    os.Exit(1)
}
func TestMain(m *testing.M) {
    switch os.Getenv("TEST_MAIN") {
    case "crasher":
        Crasher()
    default:
        os.Exit(m.Run())
    }
}

func TestCrasher(t *testing.T) {
    cmd := exec.Command(os.Args[0])
    cmd.Env = append(os.Environ(), "TEST_MAIN=crasher")
    err := cmd.Run()
    if e, ok := err.(*exec.ExitError); ok && !e.Success() {
        return
    }
    t.Fatalf("process err %v, want exit status 1", err)
}

Global Resource Checks

Go programmers often look to the standard library for examples of good, idiomatic Go code. I found one use of TestMain in the Go 1.4 standard library. The net/http package contains the following interesting use of TestMain.

func TestMain(m *testing.M) {
    v := m.Run()
    if v == 0 && goroutineLeaked() {
        os.Exit(1)
    }
    os.Exit(v)
}

// Verify the other tests didn't leave
// any goroutines running.
func goroutineLeaked() bool {
    ...
}

As you can see, if all the tests succeed TestMain will still exit with a failure code if it finds any leaking goroutines. Ideally each test would check for leaked resources, but occasionally the approach taken above is the best way.

Conclusion

The TestMain feature added to Go’s testing framework in the latest release is a simple solution for several testing use cases. TestMain provides a global hook to perform setup and shutdown, control the testing environment, run different code in a child process, or check for resources leaked by test code. Most packages will not need a TestMain, but it is a welcome addition for those times when it is needed.

Go Ahead

What’s Past is Prologue

I’ve been programming computers since the days of the Commodore 64. I guess that means my programming experience is properly measured in decades. In the three decades I’ve been instructing computers to do my bidding I’ve done so primarily in a handful of mainstream languages.

If you squint your eyes and don’t worry about a year or two here or there the decades break down as follows.

  • Decade one started with BASIC and ended with C, with a bit of Pascal in the middle.
  • Decade two was mostly spent programming in C++.
  • Decade three was dominated by Java.

Once More Unto the Breach

I am now at the beginning of my fourth decade of writing software and—once more—my primary language seems to be changing. Since the beginning of this year I have been coding almost exclusively in Go.

Although I have been using Go professionally for only a few months, my adoption of Go took place gradually over the last four years.

  • 2010—First became aware of Go.
  • 2011—Ported part of a Java program to Go to get a feel for the language and wrote an entry for ants.aichallenge.org.
  • 2012—Wrote a few small tools for hobby projects.
  • 2013—Redesigned a desktop MP3 player originally written in Java as a localhost web application written in Go.

In late 2013 we made the decision (at work) to rewrite an old C++ application. It is a distributed application that must deal with a lot of concurrency and run on both Windows and Linux. The choice of language was between Java and Go. We knew we could do it in Java, but I thought Go was a better fit. After some proof of concept work my managers were persuaded. Eight months later I am glad we chose Go.

I Like This Place and Willingly Could Waste My Time in It

In his essay “Most Software Stinks!” Charles Connell wrote about seven qualities that he felt all beautiful software possesses. He suggested that of those seven qualities simplicity is the most important. He also suggested that the simplicity of a program tells us something about the ability of its authors.

Junior programmers create simple solutions to simple problems. Senior programmers create complex solutions to complex problems. Great programmers find simple solutions to complex problems. — Charles Connell

The idea that simplicity is a worthy design goal is not new. I was first told to “keep it simple stupid” in the late ‘80s, but the KISS principle has been around since at least 1960. Although the idea is not new, we must all learn it in our own way and our own time.

Good decisions come from experience, and experience comes from bad decisions. — Unknown

It seems to me that the the Go designers are great programmers who have learned from plenty of bad decisions in the past. They have kept Go simple without sacrificing expressiveness. They have wisely chosen the path of including only what is good for the language as a whole.

Perhaps more importantly, however, they are leading by example. The community that is growing around Go appears to value simplicity. Maybe that is self- selection bias at work, or maybe the language and standard library act as models that educate the community about the value of simplicity. Regardless, I have found the language and community invigorating.

Hello, world!

For a little while now I have had the itch to blog about what I am doing with the Go Programming Language. I have a domain and a hosting company, so what’s been stopping me? The problem has been lack of a good tool to write it with.

Trying out Hugo

I’ve heard good things about Hugo on multiple occasions. The latest example being Nate Finch’s glowing review, “Hugo Is Friggin’ Awesome”. I learned from Nate’s post that Hugo is written by Steve Francia, who I had the pleasure of meeting and hanging out with one evening at GopherCon 2014.

No excuses left. Just do it!