Manage Go CLI tools via Go modules and tools.go¶

| golang |

Table of Contents

Problem definition
Initial thoughts
tools.go in action
Summary
References

Problem definition¶

Every my Go project has at least two Go CLI tools: mockery and golangci-lint.
These CLI tools might have different version in different repository. I can’t use the latest version for any of this tool because it is error prone (i.e. mockery generates artifacts differently in a new version; or golangci-lint has some new check(s) which fails on my not up to date repos). It means I need to install Go CLI tools globally (see GOBIN) and update every projects repo I’ve been working with locally to use tools from GOBIN.
It’s just nightmare to work in such environment and each Go CLI tool update is a PITA.
The ideal solution is to define Go CLI tool per repository like it was done in Node.js, Java (Maven, Gradle), Scala (SBT) and so on. Unfortunately, Go doesn’t have built-in solution for dev dependencies. But, we can introduce some conventions that help us to reach our goal.

Initial thoughts¶

Basic high-level idea has been described in How can I track tool dependencies for a module?. Also, I have collected some other tools.go variations in References section.

Unfortunately, I don’t like all of them except uber’s - really impressive and huge Makefile. So, I took it and adjusted it to my own need.

`tools.go` in action¶

Expectations¶

clone Go repo
run test
all required dependencies will be downloaded and installed automatically on repo level. Go CLI tools can’t collide with any globally installed tools. Repo tools have higher priority then global tools while running them locally from repo’s root folder.

Sample project¶

You can check demo project here halyph/demo-tools-go.

$ tree .
.
├── LICENSE
├── Makefile
├── README.md
├── .bin
│   ├── golangci-lint
│   └── mockery
├── build
│   └── demo
├── cmd
│   └── demo
│       └── main.go
├── go.mod
├── go.sum
├── pkg
│   └── magic
│       ├── magic.go
│       ├── magic_test.go
│       └── mocks
│           └── my_foo.go
└── tools
    ├── go.mod
    ├── go.sum
    └── tools.go

.bin/ - make downloads and installs Go CLI tools in this folder
build/ - make builds Go application into this folder
tools/ - tools.go and CLI-related go.mod
pkg/ - location of all Go sources

All other folders and files are pretty standard.

Makefile¶

I show here only essential to tools.go make targets.

Makefile has several important parts:

VERSION          := snapshot
NAME             := demo

GIT_HEAD         := $(shell git rev-parse HEAD)
PACKAGES         := $(shell find . -name *.go | grep -v -E "vendor|tools" | xargs -n1 dirname | sort -u)
MAIN_DIR         := ./cmd/$(NAME)
TEST_FLAGS       := -race -count=1 -mod=readonly -cover -coverprofile coverprofile.txt
LINK_FLAGS       := -X main.Version=$(VERSION) -X main.GitHead=$(GIT_HEAD)
BUILD_FLAGS      := -mod=readonly -v

.PHONY: download
download:
    @echo Download go.mod dependencies
    @go mod download

# usually unnecessary to clean, and may require downloads to restore, so this folder is not automatically cleaned
BIN   := $(shell pwd)/.bin
TOOLS := $(shell pwd)/tools

# helper for executing bins, just `$(BIN_PATH) the_command ...`
BIN_PATH := PATH="$(abspath $(BIN)):$$PATH"

.PHONY: install
install: download ## Install useful CLI tools
    @echo Installing tools from $(TOOLS)/tools.go
    @cd $(TOOLS) && cat tools.go | grep _ | awk -F'"' '{print $$2}' | GOBIN=$(BIN) xargs -tI % go install %

.PHONY: default
default: build

.PHONY: generate
generate:
    $(BIN_PATH) go generate $(PACKAGES)

.PHONY: test-generate
test-generate: install generate test

.PHONY: lint
lint: run-lint

.PHONY: test
test: run-lint run-test

.PHONY: build
build:
    CGO_ENABLED=0 go build $(BUILD_FLAGS) -ldflags="$(LINK_FLAGS)" -o build/$(NAME) $(MAIN_DIR)
    @echo build complete

.PHONY: clean
clean:
    rm -rvf pkg/mocks build coverprofile.txt

.PHONY: run-lint
run-lint:
    $(BIN_PATH) golangci-lint --version
    $(BIN_PATH) golangci-lint run $(PACKAGES)

.PHONY: run-test
run-test:
    go test $(TEST_FLAGS) $(PACKAGES)

The most interesting parts are:

GOBIN=$(BIN) xargs -tI % go install % - installs Go CLI tools into .bin folder (see “install” Makefile target)
BIN_PATH := PATH="$(abspath $(BIN)):$$PATH" - context-base custom “PATH” variable
$(BIN_PATH) is substituted to PATH (see above)
- $(BIN_PATH) go generate $(PACKAGES) - use CLI tools from .bin folder
- $(BIN_PATH) golangci-lint run $(PACKAGES) - use golangci-lint from .bin folder

`tools/tools.go`¶

//go:build tools

package tools

import (
    _ "github.com/golangci/golangci-lint/cmd/golangci-lint"
    _ "github.com/vektra/mockery/v2"
)

`tools/go.mod`¶

Go to tools/ folder and run go mod tidy command to update go.mod and go.sum files.

module example.com/tools

go 1.20

require (
    github.com/golangci/golangci-lint v1.54.2
    github.com/vektra/mockery/v2 v2.35.3
)

require (
    // indirect dependencies
    ...
)

`pkg/magic/magic.go`¶

Sample mockery go:generate annotation:

package magic

//go:generate mockery --exported --all --dir ./ --case=snake --outpkg=mocks
type MyFoo interface {
    Process(input int) int
}

type Foo struct {
}

func (Foo) Process(input int) int {
    return input * 10
}

User flow¶

Normal¶

git clone - clone the repo
make test - downloads/updates Go CLI tools, generates mocks, runs linter, runs tests
make build - builds application
make clean - remove all generated artifacts

Update CLI tools¶

Let’s update golangci-lint version:

repo version is: v1.54.2
the latest is: v1.55.2

Steps:

verify currently installed golangci-lint version

➜ .bin/golangci-lint --version
golangci-lint has version v1.54.2 built with go1.20.4 from (unknown, mod sum: "h1:oR9zxfWYxt7hFqk6+fw6Enr+E7F0SN2nqHhJYyIb0yo=") on (unknown)

go to tools/ folder and update golangci-lint version in go.mod file

diff --git a/tools/go.mod b/tools/go.mod
index 645c3ee..cb1abfa 100644
--- a/tools/go.mod
+++ b/tools/go.mod
@@ -3,7 +3,7 @@ module example.com/tools
 go 1.20

 require (
-       github.com/golangci/golangci-lint v1.54.2
+       github.com/golangci/golangci-lint v1.55.2
        github.com/vektra/mockery/v2 v2.35.3
 )

run go mod tidy to update go.mod and go.sum files and to download the latest Go CLI dependencies into local cache

➜ go mod tidy
go: downloading github.com/golangci/golangci-lint v1.55.2
go: downloading github.com/Antonboom/testifylint v0.2.3
go: downloading github.com/alecthomas/go-check-sumtype v0.1.3
go: downloading github.com/catenacyber/perfsprint v0.2.0
go: downloading github.com/ghostiam/protogetter v0.2.3
...

go back to project’s root folder cd ..
run make install
verify just updated golangci-lint version

➜ .bin/golangci-lint --version
golangci-lint has version v1.55.2 built with go1.20.4 from (unknown, mod sum: "h1:yllEIsSJ7MtlDBwDJ9IMBkyEUz2fYE0b5B8IUgO1oP8=") on (unknown)

Summary¶

Such setup is very flexible and detaches me from the global Go CLI tools.
For some people it’s might be not enough, that’s why I encourage you to check uber’s Makefile.