Go Package Management
The Go tool-chain includes the
go get command to install third-party packages directly from version control systems.
"One of the virtues of the Go tool is that it uses Git, Mercurial, Subversion, Bazaar..." - Andrew Gerrand, Go Fireside Chat 2013
I think building on top of DVCS is a brilliant move. While
go get isn't a full-featured package manager like Bundler (Ruby) or Pip (Python), we can use it to bootstrap more capable tools built by the Go Community.
"Everything else is built on this flexible foundation (FTP) and has grown over time." - Steffen Mueller, Python people want CPAN and how the latter came about
A number of third-party tools exist. The ones I'm aware of, in alphabetical order are: Dondur, Envie, Go Nuts, Go Pin, Go Pack, Go Package Manager, Gob, godep, godeps, goem, gom, gondler, gopack, Gope, goproj, Goven, gpm (a fork of Johnny Deps), GVM, Pak, Rx and vendorize.
Newcomers to the Python community were in a similar situation, but it's getting better, now that there is BDFL-delegate overseeing packaging:
"That basically gives us the power to say 'Yes'. And lets us channel the energy of all the people involved in productive directions, such that we'll actually improve the ecosystem as whole, rather than people going off their separate ways and just creating more fractured communities." - Nick Coghlan, Nobody Expects the Python Packaging Authority
Let's Make Things Better
On June 23rd, David Hinks posted a Google Doc with the following message:
"I want to have reproducible builds on different computers at different times. This is currently problematic; let's make things better." - David Hinkes, Go+
After contributing to David's document, I put together my own proposal based loosely on Bundler's
Gemfile.lock. Dave Cheney pointed out some flaws, and Kyle Lemons observed similarities between my proposal and Rx cabinets.
This was before I realized the myriad of available options. Git provides submodules and subtree merging. Third-party tools like Rx and Go Pin can track references, or Goven can copy dependencies into your project.
The problem is, we don't have a "Go Packaging Authority" providing guidance, so every newcomer is left to find their own way. Update: The Go Team recommends copying third-party dependencies into an application's source tree. This approach provides the benefits that Martin Fowler outlines in his article "ReproducibleBuild", what Brad Fitzpatrick describes as a hermetic view of the world.
I thought I should take a step back, do some research, and really think about this more. Not because I'm any sort of expert, but because I would like Go to have a great packaging story. Over the month of July, I used a Google Doc as my outlet, and invited others to contribute.
While I don't have a grand solution, I believe there are a few fundamentals to building a great packaging story.
1. Reliable Builds
Chris Howey's fsnotify package has a legitimate reason to move. First to
code.google.com/p/go.exp/fsnotify and eventually into the standard library.
What would happen if a project were to import both
code.google.com/p/go.exp/fsnotify? Each version is in a separate namespace, so it should compile.
Beyond bloating up executables, the issues will vary from package to package. Different versions may compete for exclusive access to the same file, same port or other resources. Values of interface types may be compatible across both versions of the package, while values of other types won't be implicitly interchangeable. The
init() functions will be called for both packages, and there will be distinct versions of top-level variables and all global state. Thanks to Kyle Lemons and John Waycott for making the issues concrete.
It's unlikely that one package would depend on fsnotify from two locations, but it may depend on a package which imports
code.google.com/p/go.exp/fsnotify while another dependency uses
github.com/howeyc/fsnotify. We run into trouble as the dependency tree grows.
One Import Path
To avoid this scenario, every package must have one – and only one – import path. The entire Go ecosystem must use the same import path for fsnotify.
go get will create the folder
$GOPATH/src/github.com/howeyc/fsnotify for the code retrieved from
The compiler doesn't care where the source code came from. As long as the package can be found at
$GOPATH/src/github.com/howeyc/fsnotify, it's absolutely fine if the code came from a fork at
github.com/mycompany/fsnotify or a fictional mirror at
If the Go Community as a whole decided to use
import "fsnotify", we could move the code to
$GOPATH/src/fsnotify, and the compiler would be happy.
Avoid Versions in Paths
A project should only ever import one version of a given package. What would happen if your project were to import both
labix.org/v1/mgo? It's the same problem all over again.
While this example may sound far-fetched, it is easy to see this becoming an issue as packages mature and dependency trees get deeper.
2. Encourage Upstream Contributions
One of the great things about
go get is how easy it is to switch into a third-party package, perform a patch and pull request, and get back to work without reconfiguring the project.
gofmt helps ensure that a pull request only has the necessary changes, without any superfluous formatting differences.
Avoid Rewriting Imports
Anything that makes upstream contributions more work feels like a step backwards. It doesn't take much to rewrite import paths locally, but it makes it just a little bit harder to test and submit a patch.
Update: Upon further consideration, a tool that could rewrite import paths bidirectionally, or otherwise enable merging changes upstream, would relieve my concerns.
Don't Try To Gofix Others
Gofix is an incredibly intriguing tool, but it isn't a silver bullet.
It's not to the point where package authors can easily ship gofix modules to update dependent code. As it turns out, designing an simple AST-altering API is a bit of a challenge.
Even with such a wonderful tool, I have my doubts on how well it would scale to the larger package ecosystem. It's one thing to gofix my own code, but gofixing third-party packages would leave my entire workspace with uncommitted changes. Then what?
Don't Forget Our History
Any package management solution that does away with the hidden .git, .hg, .bzr folder removes the ease of contribution.
3. Work Together
Package management tools may be technically interesting to develop, but the real challenge is a people problem. Gophers hail from *NIX and Windows, with backgrounds in C, Python, Ruby, PHP and .NET. We are a diverse group with differing views on what package management should look like.
While embracing DVCS is a great idea, I believe it can be taken too far.
DVCS-specific solutions like Git submodules make it more difficult to use and contribute to Bazaar packages like goamz, gocheck and mgo. I don't want to see the Go community fractured by DVCS selection.
Similarly, package management tools should be cross-platform, not catering to users of a particular OS.
As a package maintainer, I'm not about to say "you must install Go Nuts to use this package" or "I recommend Rx to get new versions of this package."
We as a community need to agree on a set of tools and a strategy for managing dependencies. If multiple package management tools are to exist, they need to interoperate.
It would be great if the Go Team specified and built a full-featured package manager for Go. It's not a requirement though. It doesn't need to be part of the
go tool to be what we all use.
- Releasing Open Source
- How to Write Go Code
- Organizing Code to Support Go Get by William Kennedy
- Johnny Deps: 100% Reproducible Builds in Go by Baron Schwartz
- Rx: My prescription for your Go dependency headaches by Kyle Lemons
- Dependency Hell by Chris Kowalik
- A journey in golang package manager by Bruno Michel