cmd/go: support GOPROXY fallback on unexpected errors

[heschi]

In #26334, particularly #26334 (comment), we decided that the go command should only try the the next proxy in GOPROXY if it received a deliberate not found (404/410) response. This prevents unwanted leakage of private module paths in case a private proxy has an outage -- if the go command tried the next proxy on 500s, it would leak the request to proxy.golang.org or any other public proxy in the chain.

For public proxies like proxy.golang.org, this argument doesn't apply. Falling back from a public proxy, particularly to direct, should be much less risky. The only information leaked is that a particular IP address wants a public module, which must by definition have been public to be served by the public proxy in the first place. And that's not a new risk -- public proxies are free to serve a 404 whenever they want.

Therefore, allowing fallback on all errors would improve reliability of the ecosystem with only minimal costs. As a strawman, we could support | delimiters, which would be used like:

GOPROXY=goproxy.corp,proxy.golang.org|direct

meaning to require an affirmative response from goproxy.corp, then try proxy.golang.org and fall back to direct on any failure, expected or otherwise. Precisely, | after an entry means "if the prior entry fails in any way, continue to the next entry".

The default value of GOPROXY would presumably then change to GOPROXY=proxy.golang.org|direct.

@FiloSottile just in case there are security implications, but I'm pretty sure sum.golang.org covers this same as it does anything else.

cc @jayconrod @bcmills @katiehockman @hyangah

[FiloSottile]

This is not a security no-op, although it might be worth the tradeoff. Is it actually common for proxy.golang.org to fail while sum.golang.org is available?

The difference is that a network adversary becomes capable of causing a fallback to direct. While you're right that this does not impact the privacy of the module names themselves, nor the integrity guarantees of the checksum database, it makes it possible for the code host to observe who is using what. Moreover, list and latest results are not protected by the sumdb, and with this change an attacker would get to hide versions without causing errors by colluding with the host.

It also removes any fallback security provided by the TLS connection to the proxy should the checksum database fail to provide its intended security properties, or should it be disabled. For example, I think we disabled the sumdb in a fetch made by a test, because we are ok with trusting the proxy. If the fallback strategy changes, that would be a very different decision.

[bcmills]

Looking at my notes, I think @rsc and I discussed this sort of fallback back in January and settled on essentially the same design.

We observed that, semantically, , indicates a “definitive” proxy (that is, one that should control all module paths), while | indicates a "non-definitive" proxy (that is, one that does not control module paths per se).

From that perspective, the last entry in the list lacks a token because it is always definitive: if the last entry does not or cannot provide the requested path, then that path is de facto unavailable.

[jayconrod]

Should we fall back after a timeout if a proxy doesn't respond? That seems like another important failure mode.

Also, should we fall back after non-HTTP errors, like if we fail to connect or if the proxy returns an invalid response? Assuming yes.

[bcmills]

I would prefer not to hard-code any timeouts in the fallback algorithm proper. We can't in general assume that the user is on a fast, reliable network connection, and the decision about when a network connection has actually failed is best left to a lower level (such as the default net/http configuration, which may reasonably initialize itself from some system-specific configuration).

[rsc]

It seems clear we should do this, because we need to allow CI and other automated systems to fall back if the proxy or the network to the proxy is unavailable for some reason.

[rsc]

re @FiloSottile:

This is not a security no-op, although it might be worth the tradeoff. Is it actually common for proxy.golang.org to fail while sum.golang.org is available?

It is common to need to talk to proxy.golang.org (you just checked out a repo and are trying to build it, or you are a CI system) without needing to talk to sum.golang.org (the repo has an up-to-date go.sum file).

[rsc]

Also, re fallback strategy, it is important to note that there are two steps here:

1. Have a syntax for "fallback", namely using a | after the name instead of a ,
2. Change the default GOPROXY to do a fallback on proxy unavailability.

There are not changes in security posture from (1), just new possibilities (with security implications, but again, only if you use it).

The possible change in security possible is from (2). The argument is that reliability is more important than security here, because go.sum is providing security. (If it's not, we should fix that!)

[rsc]

re @jayconrod:

Should we fall back after a timeout if a proxy doesn't respond? That seems like another important failure mode.

http.DefaultClient has no timeout at all. That's clearly wrong - if the server is blackholed somehow and packets are being dropped, we don't want to sit. I think we probably should have some timeout, but it can be a long one, like 60 seconds.

Also, should we fall back after non-HTTP errors, like if we fail to connect or if the proxy returns an invalid response? Assuming yes.

Yes. Even some 500 errors probably should fall back. 502 Bad Gateway for example. We fall back today (for comma) on 404/410. It seems like probably any HTTP error at all should fall back for pipe.

[FiloSottile]

The possible change in security possible is from (2). The argument is that reliability is more important than security here, because go.sum is providing security. (If it's not, we should fix that!)

There are at least two parts to the security of fetching modules: integrity, which is indeed handled by the checksum database and go.sum, and privacy.

The tradeoff in this case is a privacy tradeoff: a code host that can cause a network failure (easy) can get information it would not have had otherwise on who is using what.

Also, list and latest are not protected by sumdb and go.sum, so this makes it possible to keep a client on an old version for an attacker that has control over network (easy) and code host (hard). Unlike publishing a fake version, this is undetectable externally.

I'm not saying the tradeoff is not worth it, but it's not within the scope of go.sum.