Go Package Management
go get command installs remote packages directly from version control (GitHub, Bitbucket, Google Code, Launchpad). Building on top of DVCS is a brilliant move. There is a lovely immediacy to pulling in a dependency. Import and go.
When I want to contribute to an upstream package, those other tools force me to reconfigure my project. With the go tool, I switch to the package's directory, make a change and test it out. Sending a pull request is just a few steps away. Open source collaboration made simple.
For hobbyists and those starting out, the go tool is all you need.
The simplicity of the go tool does come with its trade-offs.
"I want to have reproducible builds on different computers at different times. This is currently problematic; let's make things better." - David Hinkes, Go+
Go get does not have an explicit concept of versions. Members of a team may be importing different revisions (version skew). Dependencies may introduce incompatibilities or security vulnerabilities, or even change licenses or disappear. For these reasons, it's desirable to have more control over the upgrade process when building a real-world application.
Vendor Your Dependencies
Large companies like Google and Facebook do most development in a single giant repository. This enables massive refactors in a single commit, such as changing an API and every use of it.
When it comes to third-party dependencies, the Go Team recommends copying them into the source tree:
"If you're using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.) Store the copy under a new import path that identifies it as a local copy. For example, you might copy
you.com/external/original.com/pkg." - Go FAQ
There is no need for a central archive of every version of every Go library ever released. Dependencies may move or disappear in the world outside your project, but your project has a copy of the dependencies it needs at any given point in history. Brad Fitzpatrick describes this as a "hermetic view of the world".
How is travelling back through the history of your project useful? Martin Fowler describes some benefits in ReproducibleBuild:
- To reproduce a bug on the release branch that doesn't happen on master.
bisectto isolate the commit that introduced a bug.
It's possible to upgrade dependencies in a controlled and tested manner, resolving all the stated issues with
So what are the trade-offs?
- Compared to
go get, it is a little more complicated, likely requiring an additional tool or script to copy, upgrade and remove dependencies.
- It's more difficult to contribute upstream (particularly if import rewriting is one-way). If upstream contributions are frequent, this option may be less appealing.
- This solution applies well to applications, but it's not clear how to better manage dependencies of a library.
- Manage Dependencies With Godep by William Kennedy
The Right Tool
Most language ecosystems have settled on one tool for versioning packages. Yet it's worth noting that Yukihiro "Matz" Matsumoto isn't the driving force behind Bundler, nor is Guido van Rossum the BDFL for Pip. Members of the community built these tools, and in time they became established as the standard.
Go itself doesn't mandate a one-size-fits-all solution:
"Versioning is a source of significant complexity, especially in large code bases, and we are unaware of any approach that works well at scale in a large enough variety of situations to be appropriate to force on all Go users." - Go FAQ
go get or copying dependencies into a
vendor folder are the approaches endorsed by the Go Team.
Others have simply forked all their dependencies to control when upgrades occur. Some have built tools to checkout dependencies with SHA revision identifiers. Still others utilize DVCS features like submodules or subtree merging. Experiment, find what works for you.
go tool provides a place for newcomers to start and some conventions to build upon.
"One of the virtues of the Go tool is that it uses Git, Mercurial, Subversion, Bazaar..." - Andrew Gerrand, Go Fireside Chat 2013
go get will create the folder
$GOPATH/src/github.com/howeyc/fsnotify for the code retrieved from
The compiler actually doesn't care where the source code came from. As long as the package can be found at
$GOPATH/src/github.com/howeyc/fsnotify, it's absolutely fine if the code actually came from a fork at
github.com/mycompany/fsnotify or a mirror at
Adding remotes, checking out tags, all the usual DVCS operations are fair game.
The Next Big Thing
So you have an idea that will revolutionize how we
go get packages?
"...channel the energy of all the people involved in productive directions, such that we'll actually improve the ecosystem as whole, rather than people going off their separate ways and just creating more fractured communities." - Nick Coghlan, Nobody Expects the Python Packaging Authority
Take off those Node.js or Ruby-coloured glasses. Recognize that Go is different.
Beyond bloating up executables, the issues will vary from package to package. Different versions may compete for exclusive access to the same file, same port or other resources. Values of
interface types may be compatible across both versions of the package, while values of other types won't be implicitly interchangeable. The
init() functions will be called for both packages, and there will be distinct versions of top-level variables and all global state. Thanks to Kyle Lemons and John Waycott for making the issues concrete.
Those coming from the Ruby on Rails community may remember copying third-party plugins into the
vendor folder in the days before Bundler. Surely Go has something more "sophisticated" than that?
As you read above, the approach used by Google actually has a number of strengths. It won't be suitable for every situation, but the apparent simplicity and increased repository size shouldn't deter you from trying it.
Let's remember that Bundler's approach isn't without flaws:
- The story of Sprockets 2.2 backport is a case where one gem took control away from developers, requiring a hacky workaround.
- Central repositories go down, get hacked and are costly to run.
- Different permissions for those who can commit and those who can publish a gem have resulted in gems without a recent release.
More importantly, such a tool my never gain sufficient traction if existing Go packages are expected to contain version metadata for it to work. Even if it is technically feasible, "Bundler for Go" may not be culturally suitable. When in Go do as the gophers do.
Keep It Simple
Handling dependencies is incredibly important to the Go Team, even more important than DRY:
"Through the design of the standard library, great effort was spent on controlling dependencies. It can be better to copy a little code than to pull in a big library for one function. Dependency hygiene trumps code reuse." - Go at Google
When building an app, try existing tools to vendor your dependencies. Contribute improvements. If need be, try another approach. Share what works for you. Let's make things better.
- Managing Dependencies by Nathan Youngman
- Go and Package Versioning by Joshua Poehls
- Maybe Go without dependency management can be a good thing by Matthew Holt
- Why I think Go package management is important and Thoughts on Go package management six months on by Dave Cheney
- Go Package Management For 2014 by William Kennedy
- Rx: My prescription for your Go dependency headaches by Kyle Lemons
- 100% Reproducible Builds in Go (with Johnny Deps) by Baron Schwartz
- Dependency Hell by Chris Kowalik
- A journey in golang package manager by Bruno Michel