cmd/go: go list has too many (more than zero) side effects

[DisposaBoy]

<rant>

This is quite honestly becoming (literally) rage-inducing so I'll keep it short...

Since the introduction of the package cache and modules, go list has gained some (IMO) nasty side-effects like downloading things from the internet and compiling (CGO) packages when all I did was ask it to print the import path of the current package.

Additionally, package querying shouldn't result in updating any files. This coupled with the fact that GOPATH/mod/... is readonly means that if you're in a package inside the mod cache and run go list it might fail because it can't write to go.mod (why is it updating the file?!?!?!).

This script that creates a go.mod file with an extra empty line at the end demonstrates the latter issue:

$ bash -c 'cd $(mktemp -d) && echo "package app" > app.go && echo -e "module app\n" > go.mod; go list -mod=readonly'
go: updates to go.mod needed, disabled by -mod=readonly

To make matters worse, go/build suddenly started calling go list so a simple operation like .Import(...FindOnly) that used to take no more than a couple milliseconds, now takes several seconds for no good reason... all because the go tool decided it was going download things from the internet, compile things and god knows what else... all manner of surprises I didn't ask for.

Usually I'd just code my way around it with the power of NIH, but the behavior of go/build and package lookups and querying in general is un(der)-documented and I don't want to have to keep track of whatever new magic it mightwill gain in the future.

I doubt any of this is ever going to be fixed, so it'd be nice if these things were documented so I could answer questions like "given an import path, how do I go about finding it in GOPATH, vendor, module cache, build cache, etc.?" without having to rely on some broken black box.

[ALTree]

Thanks for the report.

At least some of the slowness you are observing may be imputable to outstanding go list bugs (see for example #29427 (comment)).

cc @heschik

[myitcv]

@DisposaBoy (in a modules world) you need to move to use go/packages. go/packages replaces go/build.

cc @matloob / @ianthehat - perhaps we should add some (temporary) documentation to go/packages to explain how it is a replacement for go/build et al?

[myitcv]

cc @bcmills - should we also add an advisory to go/build in Go 1.12 pointing (temporarily) to go/packages?

[DisposaBoy]

@myitcv I don't see how go/packages solve any of these issues. Last time I looked at it (a couple weeks ago) it still takes ~200ms to answer the same simple query in GOPATH mode and correct me if I'm wrong, but it also calls go list which means it doesn't solve any of the issues I mentioned.

[myitcv]

@DisposaBoy previously, multiple calls to go/build were effectively zero-cost. In the new world, these are replaced by a single call to go/packages.Load.

If you continue to use go/build, in certain usage patterns you will end up making multiple calls to go list, which, even for relatively small projects, can become costly.

go/packages has come into existence to provide an abstraction layer atop various drivers. There is a driver for the go command, just as there is for build systems like Bazel, Blaze and others. All efforts for optimisation are therefore directed via go/packages.

So the first step is moving away from go/build to go/packages (which can do everything from simply resolving package patterns to loading fully type and syntax information (see https://godoc.org/golang.org/x/tools/go/packages#LoadMode).

If you are still seeing issues after moving to go/packages, then we can certainly help to diagnose further. There are a number of things you might be running into, but narrowing this down to a single go/packages.Load call will help.

Issues that spring to mind include the aforementioned #29427, #28739. The latter is hopefully going to be addressed by an upcoming CL that works by caching directory/file operations where possible.

[matloob]

cc @bcmills @ianthehat

@myitcv @DisposaBoy As much as I'd like to see everyone move to go/packages, I don't think doing so will help with these problems. The workings of modules, the build/list cache, etc., are complicated enough that we need go list as a black box to own the logic.

Even if we had documentation about all this stuff (which would certainly be a good thing), it's possible that changes across versions of Go might break you, and having the go list logic reimplemented I think would just lead to bugs and incompatibilities.

It doesn't seem likely to me that Go list will change to do less work, but I don't know enough about it and the workings of modules to say why. I'll leave that to @bcmills.

Of course we'll work to get go/packages to use go list as efficiently as it can, but we'll be limited by the behavior of go list (which is in turn limited by the requirements of go modules).

[bcmills]

It doesn't seem likely to me that Go list will change to do less work,

To the contrary, I expect that it will change to do substantially less work, especially when the -e flag is present. (That's a discussion I need to have with @jayconrod and @rsc for Go 1.13.)

[bcmills]

See also #28692.

[bcmills]

At any rate, it is probably true that go list has too many side effects today, but zero is probably too few.

For example, if go list attempts to write the go.mod file to fix formatting or to remove redundant declarations, and it fails to do so because the directory is read-only, perhaps that just shouldn't be an error. That doesn't mean that it shouldn't try, though.

[josharian]

Why should go list fix formatting or remove redundant declarations? Zero really does seem like the right number of side-effects for go list.