x/build,cmd/go: frequent connection timeouts to github.com since 2022-03-04

[bcmills]

#!watchflakes
post <- pkg ~ `cmd/go` && `github.com.*(Connection timed out|TLS connection)` && date < "2023-07-01"

# cd $WORK/tmp/d2/src/github.com/myitcv; git clone -- https://github.com/myitcv/vgo_example_compat $WORK/tmp/d2/src/github.com/myitcv/vgo_example_compat
            Cloning into '$WORK/tmp/d2/src/github.com/myitcv/vgo_example_compat'...
            fatal: unable to access 'https://github.com/myitcv/vgo_example_compat/': Failed to connect to github.com port 443: Connection timed out

greplogs --dashboard -md -l -e 'github.com.* Connection timed out' --since=2021-01-01

2022-04-22T19:02:29-808d40d/linux-386-longtest
2022-04-22T15:44:57-1899472/linux-386-longtest
2022-04-22T04:42:23-1e59876/linux-386-longtest
2022-04-22T00:25:08-c9031a4/linux-amd64-longtest
2022-04-22T00:05:54-c510cd9/linux-386-longtest
2022-04-22T00:05:54-c510cd9/linux-amd64-longtest
2022-04-19T15:07:49-caa4631/linux-amd64-longtest
2022-04-15T19:02:54-df08c9a/linux-386-longtest
2022-04-14T18:00:13-dd97871/linux-386-longtest
2022-04-14T18:00:13-dd97871/linux-amd64-longtest
2022-04-13T17:48:12-517781b/linux-386-longtest
2022-04-13T01:15:22-b55a2fb/linux-386-longtest
2022-04-12T22:35:20-fc1d4c1/linux-amd64-longtest
2022-04-12T22:32:01-6183920/linux-386-longtest
2022-04-12T05:46:57-2b31abc/linux-amd64-longtest
2022-04-11T16:31:45-e4e033a/linux-386-longtest
2022-04-11T16:31:43-036b615/linux-amd64-longtest
2022-04-11T16:31:33-494b79f/linux-amd64-longtest
2022-04-08T18:30:53-3a19102/linux-386-longtest
2022-03-15T17:04:57-9b112ce/linux-amd64-longtest
2022-03-07T13:47:51-0e2f1ab/linux-386-longtest
2022-03-04T20:02:41-2b8aa2b/linux-386-longtest
[Note the 5-month gap here! 🤔]
2021-10-08T14:08:12-5b9206f/linux-386-longtest
2021-10-08T14:08:12-5b9206f/linux-amd64-longtest
2021-09-07T21:39:06-dcf3545/linux-386-longtest
2021-09-07T20:37:05-8078355/linux-386-longtest
2021-09-07T20:27:30-23f4f0d/linux-386-longtest
2021-09-07T19:39:04-d92101f/linux-386-longtest
2021-06-22T02:44:43-197a5ee/linux-amd64-longtest
2021-04-02T05:24:14-aebc0b4/linux-amd64-longtest
2021-03-18T14:43:33-e726e2a/linux-amd64-longtest
2021-03-17T17:50:50-8628bf9/linux-amd64-longtest
2021-03-17T17:13:50-0bd308f/linux-amd64-longtest
2021-03-17T16:53:00-70d54df/linux-386-longtest
2021-03-17T16:53:00-70d54df/linux-amd64-longtest
2021-03-17T16:19:21-2f3db22/linux-386-longtest
2021-03-05T18:46:36-51d8d35/linux-amd64-longtest

@golang/release: were there changes to the linux-.*-longtest image or network configuration around 2022-03-04 that might explain the increase in timeouts at that point?

[heschi]

Not that I'm aware of.

[bcmills]

As with the other network-related failures, this failure mode is much more common on the linux-386-longtest builder.

greplogs -l -e 'github.com.* Connection timed out' --since=2022-04-23
2022-05-02T21:34:37-99f1bf5/linux-amd64-longtest
2022-05-02T18:38:26-af99c20/linux-386-longtest
2022-04-27T17:24:48-89044b6/linux-386-longtest
2022-04-27T14:23:28-f0c0e0f/linux-386-longtest
2022-04-27T14:22:53-83ff0b0/linux-386-longtest
2022-04-27T00:09:45-68b655f/linux-386-longtest
2022-04-26T22:09:34-13b18ec/linux-386-longtest
2022-04-26T22:08:42-e1b5f34/linux-386-longtest
2022-04-26T17:50:49-06b0a65/linux-386-longtest
2022-04-26T14:49:07-7b31af0/linux-386-longtest
2022-04-25T23:10:54-35f2aba/linux-386-longtest
2022-04-25T18:12:08-60178e6/linux-386-longtest

[bcmills]

greplogs -l -e 'github.com.* Connection timed out' --since=2022-05-03
2022-05-04T00:51:08-8b389eb/linux-amd64-longtest
2022-05-03T21:36:22-64b6e44/linux-386-longtest
2022-05-03T14:51:51-920b9ab/linux-386-longtest

[bcmills]

greplogs -l -e 'github.com.* Connection timed out' --since=2022-05-04
2022-05-05T01:33:01-ed462a6/linux-amd64-longtest
2022-05-04T15:38:23-c6688b5/linux-amd64-longtest
2022-05-04T00:51:08-8b389eb/linux-amd64-longtest

[bcmills]

greplogs -l -e 'github.com.* Connection timed out' --since=2022-05-05T01:34:00
2022-05-06T04:22:04-3ea3bc0/linux-386-longtest
2022-05-05T21:52:34-50b1add/linux-386-longtest
2022-05-05T18:48:01-53647a1/linux-amd64-longtest
2022-05-05T14:44:31-485a572/linux-386-longtest

[bcmills]

This is causing a huge amount of noise in the longtest builders, and affects the most comprehensive builders for two first-class ports (linux/386 and linux/amd64).

At this failure rate, I suspect that if this issue affected end users we'd be hearing about it by now. So my hypothesis is that this is some kind of GCE networking issue that affects the builder hosts.

Marking as release-blocker for Go 1.19. We need to figure out what's happening with the network and either fix it (if it's a GCE or builder problem) or figure out a workaround (if it's a GitHub problem).

(Note that most of these failures come from the git binary itself; #46693 may explain why we're not seeing this on the Windows builders.)

[bcmills]

When these timeouts occur, the tests are consistently at 120s or longer of running time.
That suggests that raising the connection timeout would not be helpful.

[bcmills]

Here's a theory, inspired by https://askubuntu.com/questions/218728/connection-timeout-when-accessing-github.

Maybe this timeout is some kind of rate-limiting countermeasure on the GitHub side, spurred by bursts of traffic from the Go builders running more SlowBots and post-commit CLs.

Maybe the 5-month gap from October 2021 to March 2022 is an artifact of the Go development cycle: the Go release freeze started 2021-11-01 and ended 2022-02-28. The last failure before the gap was a couple of weeks before the freeze began, and the first failure after the gap was about a week after it ended. During the freeze, the CL rate is naturally much lower, which might keep the builders below whatever threshold GitHub is using for rate-limiting.