cmd/go: allow proxies to supply only some modules

[arschles]

This is a proposal for extending the vgo download API to add a mechanism to allow proxies to redirect vgo to a VCS. This functionality would be useful if a proxy has only a subset of packages in a go.mod file.

Currently, if someone adds two modules "a" v1.0.0 and "b" v1.0.0 to their go.mod file and they then run GOPROXY=myprox.com vgo install, everything works as expected if the proxy has both modules at the given versions. If not, the command fails.

It would be helpful to allow the proxy to tell vgo to fetch one or both of the modules from the VCS if it doesn't have them in its cache. This would be useful for proxy implementations where the proxy will not/cannot cache the module in its own storage or can but doesn't have that module/version in its cache. The Athens project is a present day use case for the latter - it fills its caches asynchronously.

One implementation possibility is adding a $GOPROXY/a/@v/v1.0.0?go-get=1 network request that expects the same output as already-existing ?go-get=1 requests define. This request could be made before starting the download protocol as it currently exists. The new mechanism would allow the proxy to choose to do one of the following for the given module and version identifier:

1. Send vgo to the standard download API via the meta tag
2. Send vgo to the VCS via the meta tag
3. Return a 404

[arschles]

Update after slack discussion with @myitcv and @zeebo:

1. we could certainly do cache fills synchronously
2. there could be some "gotchas" with hosting the proxy, like timeouts when doing a cache fill for a large module (i.e. github.com/kubernetes/kubernetes)
3. assuming the proxy synchronously fills caches, the user experience would be about the same as if vgo get downloaded from VCS
4. if the proxy has a partial failure and cannot serve code, GOPROXY=myprox.com vgo get will always fail with the current behavior

Another concern that was brought up is that with GOPROXY set, the current behavior of GOPROXY=myprox.com vgo get github.com/kubernetes/kubernetes doesn't let the proxy redirect vgo to another location (i.e. a CDN) it still has to serve metadata and the zipped source code itself.

[marwan-at-work]

The proxy can redirect vgo to another URL as long as that URL implements the download protocol. That's because the net/http client handles redirect status codes internally (up to 10 times by default)

The big question here is: can the GOPROXY tell Vgo (during a build) to go use a VCS source instead of the proxy itself? Which is a little different than a simple redirect to another Download Protocol enabled URL.

[marwan-at-work]

@rsc How do you envision the flow of the Proxy telling Vgo to switch to VCS?

I'm trying to follow the vgo code and it seems that on an initial build, the initial contact with the proxy's download protocol can be any of these endpoints: /@latest or /list or /@v/{certainRevision}.info.

This means that the proxy would need to return a consistent code (maybe a 404) on each of these endpoints to signal vgo to reconstruct the modfetch.Repo interface from a vcs source instead of the *proxyRepo one. Vgo would then have to back up a few steps and retry again.

Vgo can potentially always hit /list as the first entry point to the download protocol, and if the list is empty then switch to vcs. Or it can always hit @latest first and if it returns 404 then switch to vcs.

Another solution, is that the Download Protocol could implement a @probe call with a couple of parameters (module path and revision) and then the Proxy can early on tell vgo to just go for the VCS source for this particular module.

For efficiency, vgo can potentially send the entire list of modules it wants to probe to the proxy.

I haven't fully understood the last 2 days worth of changes to vgo so apologies if I'm a bit off.

[rsc]

I don't think it makes sense for a proxy to tell the go command "go to this VCS instead". We're trying to migrate to proxy by default and while VCS will probably always be with us, I'd rather not mix the two.

I do think it would probably be OK to let GOPROXY be preference list and to also allow some setting like GOPROXY=direct as an explicit name for what the default behavior is. So you could say GOPROXY=https://myproxy/,direct and just let myproxy return a 404 for the things it doesn't know about. Then the proxy isn't in charge of the actual redirect; it's only in charge of "it's not me".

[marwan-at-work]

@rsc that's about what I had in mind. The proxy shouldn't say "go to this vcs", it can just say "I don't have this module"

Does that mean the /@probe endpoint makes sense? Since that means Go can just ask the proxy whether it can work with a specific module before it even asks for @latest or /list.

I've altered the Go code if you'd like to look at a reference of what I'm suggesting: marwan-at-work@3767be8