x/build: build cross-compiled releases for secondary ports

[ianlancetaylor]

Per the discussion in #53383 we should develop a way to build a cross-compiled Go release, without requiring a C cross-compiler, but such that the release will itself default to using cgo as usual. This means that the first time that this new release is used, it will invoke the system C compiler to build the packages that require cgo: the net, os/user, and runtime/cgo packages. Those packages, built with cgo, will thereafter live in the build cache.

We will then be able to use this mechanism as part of our release process to automatically, easily, and safely build releases for a range for second ports. For first-class ports I expect that we will continue to build the releases on the systems themselves with full cgo support included. But for secondary ports we don't want to rely on having a builder for the approach system. A cross-compiled release will be easy for our users to download and use.

A disadvantage of this is that the cmd/go program included in the cross-compiled binary release will be built without cgo, meaning that it will only use the Go DNS resolver. This is mostly fine, and for cases where it is not fine people can fix the problem by running go install cmd/go. However, this will require documentation.

The actual implementation of this may be very similar to the existing bootstrap.bash script, but with some tweaks to ensure that the generated release will default to using cgo.

Of course for secondary ports that do not support cgo (currently linux/ppc64, js/wasm, plan9/*, windows/arm) the generated binary release should not expect to use cgo by default.

[rsc]

Looked into this. It works already. Just cross-compile a toolchain with CGO_ENABLED="" ./make.bash.

Specifically, this script worked fine for cross-compiling from my Mac to build a Linux distribution.
When I ran 'go env' in the Linux distribution, it properly defaulted to using cgo.

hostos=darwin
hostarch=amd64
targetos=linux
targetarch=amd64

set -e
rm -rf bin pkg
cd src
CGO_ENABLED="" GOOS=$targetos GOARCH=$targetarch ./make.bash
cd ..
mv bin/${targetos}_${targetarch}/* bin  # move target binaries to right place
rmdir bin/${targetos}_${targetarch}  # remove old place
rm -rf pkg/${targetos}_${targetarch}  # remove target archives (they will be rebuilt)
rm -rf pkg/tool/${hostos}_${hostarch}  # remove host tools

x/build/internal/task/buildrelease will have to be updated to know how to do these steps for cross-compiled toolchains, but there are no changes necessary in cmd/dist to enable that.

Leaving for @heschi and @golang/release to possibly repurpose this issue for updating buildrelease.

[rsc]

To be clear, the cross-compiled Linux distribution in the example above differs from the standard one in that it requires a C compiler to build programs using the standard library (for example package net) when cgo is enabled. So we shouldn't, say, cross-compile all our ports from macOS machines. But cross-compiling all the secondary ports from linux/amd64 should be fine.

[rsc]

Refining my previous comment, today the Windows port does not use cgo for any of std (except runtime/cgo, which is the dependency implicitly added to all cgo packages):

% CGO_ENABLED=1 GOOS=windows go list -f '{{if .CgoFiles}}{{.ImportPath}}{{end}}' std
runtime/cgo
% 

So even the first-class Windows ports could reasonably be cross-compiled from a linux/amd64 system. If we had a Linux program to generate the .msi files for us too, then we would not need Windows machines anymore for the actual generation of releases, which reduces the number of distinct trusted machines to juggle. (We'd probably want to arrange a test that copied the built release over to a Windows machine and ran ./run.bash --no-rebuild, but that doesn't require the machine to be as trusted as building the release.)

For the first-class macOS ports, there are still a few cgo packages remaining even though net is gone:

% CGO_ENABLED=1 GOOS=darwin go list -f '{{if .CgoFiles}}{{.ImportPath}}{{end}}' std
os/signal/internal/pty
os/user
plugin
runtime/cgo
% 

We could plausibly remove the macOS use of cgo in os/signal/internal/pty, os/user, and plugin too. Then the macOS distributions could also be built from linux/amd64, assuming a Linux program to generate the .pkg files for us.

This issue is only about building secondary port distributions, but we should keep in mind the possibility of using that same mechanism for building non-Linux primary port distributions.