Skip to main content

Godep for End User Go Projects

Every language has a problem with dependencies. In C there are problems with bumping minor shared library versions and having unexpected bugs appear in applications. Ruby and Python have problems with applications having conflicting dependencies requiring the use of bundle or virtualenv to create private library stores.

Go has a distinct advantage because it is a statically compiled language. In other words, a Go program always carries its runtime dependencies with it; no possibilities of library versions changing. However, creating a developer environment in which to compile this static binary is a bit more difficult and some tools are required.

While working on etcd, a project written in Go, we took an evolutionary path towards a dependency system that satisfied a few basic goals we had:

  • Reproducible builds: given the same git hash and version of the Go compiler we wanted an identical binary every time.
  • Zero dependencies: developers should be able to fork on GitHub, make a change, build, test and send a PR without having anything more than a working Go compiler installed.
  • Cross platform: compile and run on OSX, Linux and Windows. Bonus points for cross-compilation.

tl;dr We arrived at using godep but the rest of the post below has some insights on how we arrived here.

Checked in GOPATH

To satisfy the need for reproducible builds and zero dependencies we checked in a copy of the GOPATH to third_party/src. However, this revealed several problems over time:

  1. go get github.com/coreos/etcd was broken since downstream dependencies would change master and go get would setup a GOPATH that looked different than our checked in version.
  2. Windows developers had to have a working bash. Soon, we were maintaining a copy of our build script written in Powershell.

We felt that go get wasn't a useful tool for installing etcd since it was just a project built in Go and go get is primarily useful for easily grabbing libraries when you are hacking on something. However, we were overwhelmed every week with bug reports from users who wanted to simply go get github.com/coreos/etcd.

To solve the Windows problem I created third_party.go which ported the GOPATH management shell script to Go and gave us the ability to drop the Powershell.

third_party.go

third_party.go worked well for a few weeks and we could remove the duplicate build logic in the Powershell scripts. The basic usage was simple:

# Bump the raft dependency in the custom GOPATH
go run third_party.go bump github.com/coreos/go-etcd
# Use third_party.go to set GOPATH to third_party/src and build
go run third_party.go build github.com/coreos/etcd

But, there was a fatal flaw with this setup: it broke cross compilation via GOOS and GOARCH.

GOOS=linux go run third_party.go build github.com/coreos/etcd
fork/exec /var/folders/nq/jrsys0j926z9q3cjp1yfbhqr0000gn/T/go-build584136562/command-line-arguments/_obj/exe/third_party: exec format error

The reason is that GOOS and GOARCH get used internally by go run. Meaning it literally tries to build third_party.go as a Linux binary and runs it. Running a Linux binary on an OSX machine doesn't work.

Furthermore by using third_party.go didn't get us any closer to being "go gettable". With the continued weekly emails and bugs for this I started looking around for better solutions and found goven.

goven and goven-bump

goven achieves all of the desirable traits: reproducible builds, zero dependencies to start developing, cross compilation, and as a bonus go install github.com/coreos/etcd works!

The basic theory of operation is it checks all dependencies into subpackages of your project. Instead of importing code.google.com/p/goprotobuf you import github.com/coreos/etcd/third_party/code.google.com/p/goprotobuf. It makes the imports uglier but it is automated by goven.

Along the way I wrote some helper tools to assist in bumping dependencies which can be found on GitHub at philips/goven-bump. The scripts goven-bump and goven-bump-commit grab the hg revision or git hash of the dependency along with running goven. This makes bumping a dependency and getting a basic commit message as easy as:

cd ${GOPATH}/github.com/coreos/etcd
goven-bump-commit code.google.com/p/goprotobuf
git commit -m 'bump(code.google.com/p/goprotobuf): 074202958b0a25b4d1e194fb8defe5d69c300774'

godep save

The way that goven re-wrote the package paths turned out to be a very successful model for us and saved the developers of etcd a ton of headaches. However, the goven-bump scripts were rather annoying and it forced the developer to use git log to find the hash of the upstream dependency.

After emailing and chatting with Keith Rarick, author of goven and godep, he told me that he was planning on adding a feature to his godep project that rewrote import paths just like goven but had the advantages of a static dependency manifest file. After a few months of nagging he merged the godep save -r feature and I could finally start converting projects to stop using goven-bump. Woo!

godep adds some additional complexity for the etcd team. But, the simplicity it presents to regular contributors and users used to go get make it worth the additional effort. For all of the CoreOS non-library projects that are written in Go we now use godep and have achieved all of our goals:

  • Reproducible builds
  • Zero dependencies
  • Cross platform
  • Easy to introspect

If you maintain a Go project, I highly recommend you go install github.com/tools/godep.