Go Package Management

A summary of dependency management in Go.

First published on July 25, 2013. Last revised on April 17, 2014.

Gophers with parcels
Just another day at the Go package sorting facility.

Remote Packages

The go get command installs remote packages directly from version control (GitHub, Bitbucket, Google Code, Launchpad). Building on top of DVCS is a brilliant move. There is a lovely immediacy to pulling in a dependency. Import and go.

Package managers like Bundler, Pip and NPM all have some mechanism to bypass their central repositories and pull code directly from source control. The go tool makes this first-class.

When I want to contribute to an upstream package, those other tools force me to reconfigure my project. With the go tool, I switch to the package’s directory, make a change and test it out. Sending a pull request is just a few steps away. Open source collaboration made simple.

For hobbyists and those starting out, the go tool is all you need.

Resources:

The simplicity of the go tool does come with its trade-offs.

“I want to have reproducible builds on different computers at different times. This is currently problematic; let’s make things better.” - David Hinkes

Go get does not have an explicit concept of versions. Members of a team may be importing different revisions (version skew). Dependencies may introduce incompatibilities or security vulnerabilities, or even change licenses or disappear. For these reasons, it’s desirable to have more control over the upgrade process when building a real-world application.

A plate of packages.
I had some packages for lunch and now they are part of me.

Vendor Your Dependencies

Large companies like Google and Facebook do most development in a single giant repository. This enables massive refactors in a single commit, such as changing an API and every use of it.

When it comes to third-party dependencies, the Go Team recommends copying them into the source tree:

“If you’re using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.) Store the copy under a new import path that identifies it as a local copy. For example, you might copy original.com/pkg to you.com/external/original.com/pkg.” - Go FAQ

There is no need for a central archive of every version of every Go library ever released. Dependencies may move or disappear in the world outside your project, but your project has a copy of the dependencies it needs at any given point in history. Brad Fitzpatrick describes this as a “hermetic view of the world”.

How is travelling back through the history of your project useful? Martin Fowler describes some benefits in ReproducibleBuild:

  • To reproduce a bug on the release branch that doesn’t happen on master.
  • Using bisect to isolate the commit that introduced a bug.

It’s possible to upgrade dependencies in a controlled and tested manner, resolving all the stated issues with go get.

So what are the trade-offs?

  • Compared to go get, it is a little more complicated, likely requiring an additional tool or script to copy, upgrade and remove dependencies.
  • It’s more difficult to contribute upstream (particularly if import rewriting is one-way). If upstream contributions are frequent, this option may be less appealing.
  • This solution applies well to applications, but it’s not clear how to better manage dependencies of a library.

Resources:

Gophers with pliers and packing tape.
Not all these tools are appropriate for the job.

The Right Tool

Most language ecosystems have settled on one tool for versioning packages. Yet it’s worth noting that Yukihiro “Matz” Matsumoto isn’t the driving force behind Bundler, nor is Guido van Rossum the BDFL for Pip. Members of the community built these tools, and in time they became established as the standard.

Go itself doesn’t mandate a one-size-fits-all solution:

“Versioning is a source of significant complexity, especially in large code bases, and we are unaware of any approach that works well at scale in a large enough variety of situations to be appropriate to force on all Go users.” - Go FAQ

Using go get or copying dependencies into a vendor folder are the approaches endorsed by the Go Team.

Others have simply forked all their dependencies to control when upgrades occur. Some have built tools to checkout dependencies with SHA revision identifiers. Still others utilize DVCS features like submodules or subtree merging. Experiment, find what works for you.

The go tool provides a place for newcomers to start and some conventions to build upon.

“One of the virtues of the Go tool is that it uses Git, Mercurial, Subversion, Bazaar…” - Andrew Gerrand, Go Fireside Chat 2013

By convention, go get will create the folder $GOPATH/src/github.com/howeyc/fsnotify for the code retrieved from github.com/howeyc/fsnotify.

The compiler actually doesn’t care where the source code came from. As long as the package can be found at $GOPATH/src/github.com/howeyc/fsnotify, it’s absolutely fine if the code actually came from a fork at github.com/mycompany/fsnotify or a mirror at code.google.com/p/go.exp/fsnotify.

Adding remotes, checking out tags, all the usual DVCS operations are fair game.

The Next Big Thing

So you have an idea that will revolutionize how we go get packages?

First, join the GoPM mailing list, share your passion, and find out what others are working on. Next, evaluate what’s out there. We don’t need 20 tools that do the same thing (NIH). We want to:

“…channel the energy of all the people involved in productive directions, such that we’ll actually improve the ecosystem as whole, rather than people going off their separate ways and just creating more fractured communities.” - Nick Coghlan, Nobody Expects the Python Packaging Authority

Think Different

Take off those Node.js or Ruby-coloured glasses. Recognize that Go is different.

No NPM

NPM has a unique strategy for handling multiple versions. Each module requires the versions it wants, which may result in multiple versions of the same module being used in an application. This works well for JavaScript, but Go is not JavaScript.

Beyond bloating up executables, the issues will vary from package to package. Different versions may compete for exclusive access to the same file, same port or other resources. Values of interface types may be compatible across both versions of the package, while values of other types won’t be implicitly interchangeable. The init() functions will be called for both packages, and there will be distinct versions of top-level variables and all global state. Thanks to Kyle Lemons and John Waycott for making the issues concrete.

No Bundler

Those coming from the Ruby on Rails community may remember copying third-party plugins into the vendor folder in the days before Bundler. Surely Go has something more “sophisticated” than that?

As you read above, the approach used by Google actually has a number of strengths. It won’t be suitable for every situation, but the apparent simplicity and increased repository size shouldn’t deter you from trying it.

Let’s remember that Bundler’s approach isn’t without flaws:

  • The story of Sprockets 2.2 backport is a case where one gem took control away from developers, requiring a hacky workaround.
  • Central repositories go down, get hacked and are costly to run.
  • Different permissions for those who can commit and those who can publish a gem have resulted in gems without a recent release.

More importantly, such a tool my never gain sufficient traction if existing Go packages are expected to contain version metadata for it to work. Even if it is technically feasible, “Bundler for Go” may not be culturally suitable. When in Go do as the gophers do.

Keep It Simple

Handling dependencies is incredibly important to the Go Team, even more important than DRY:

“Through the design of the standard library, great effort was spent on controlling dependencies. It can be better to copy a little code than to pull in a big library for one function. Dependency hygiene trumps code reuse.” - Go at Google

When releasing open source libraries, aim to minimize the number of dependencies. Learn and use the standard library. Be go gettable. Take a look at gopkg.in for your next major version.

When building an app, try existing tools to vendor your dependencies. Contribute improvements. If need be, try another approach. Share what works for you. Let’s make things better.

Comment on reddit or the GoPM mailing list.

Related Articles:

Nathan Youngman

Software Developer and Author