x/build/cmd/relui: reduce Go download sizes

[bradfitz]

Go1's source download was 11 MB.
Go 1.11rc1's source download is 20MB.
Go 1.4.3's Linux download was 58 MB.
Go 1.11rc1's Linux download was 169 MB.

Clearly 169 MB is pretty ridiculous.

https://go-review.googlesource.com/c/build/+/129955 removed some useless *.a files to save 20 MB.

https://go-review.googlesource.com/c/build/+/130575 removed the blog to save 34MB.

We're now down to 116 MB for what will be the Go 1.11rc2 Linux download. Better, but not great.

This is a tracking bug for further improvements.

/cc @ianlancetaylor @andybons @dmitshur

[bradfitz]

Untarring the release and gzip-compressing each file before doing du stats gives a reasonable way to measure the cost of removing bits:

$ find . -type f -not -name '*.gz' -exec gzip {} \;

Here are the top directories:

127731439       .
127727343       ./go
83486614        ./go/pkg
54847292        ./go/pkg/tool
54843196        ./go/pkg/tool/linux_amd64
21066018        ./go/src
16095151        ./go/bin
15354919        ./go/pkg/linux_amd64_race
13273773        ./go/pkg/linux_amd64
8094311 ./go/src/cmd
3127907 ./go/src/cmd/vendor
2732975 ./go/doc
2138255 ./go/src/runtime
2102037 ./go/doc/gopher
2051880 ./go/pkg/linux_amd64_race/net
1933218 ./go/pkg/linux_amd64_race/go
1773043 ./go/misc
1752589 ./go/src/cmd/vendor/golang.org
1750311 ./go/pkg/linux_amd64/net
1748493 ./go/src/cmd/vendor/golang.org/x

• It turns out that go/pkg/linux_amd64_race directory is not required. Go 1.11 will rebuild that stuff as needed. I verified with a Dockerfile (https://play.golang.org/p/O-IW_vVnol_g). That'll save us 15 MB.

• Do we really need 2MB of Gopher images? We need some for the local godoc perhaps.

Biggest binaries:

$ du -b -a | sort -n -r | grep gz$ | head -20
8881040 ./go/pkg/tool/linux_amd64/compile.gz
7957541 ./go/bin/godoc.gz
6467215 ./go/pkg/tool/linux_amd64/pprof.gz
6435168 ./go/bin/go.gz
5961018 ./go/pkg/tool/linux_amd64/tour.gz
5183556 ./go/pkg/tool/linux_amd64/trace.gz
3756217 ./go/pkg/tool/linux_amd64/vet.gz
2739853 ./go/pkg/tool/linux_amd64/link.gz
2451059 ./go/pkg/tool/linux_amd64/cover.gz
2257606 ./go/pkg/tool/linux_amd64/cgo.gz
2187914 ./go/pkg/tool/linux_amd64/doc.gz
2145237 ./go/pkg/tool/linux_amd64/asm.gz
2060780 ./go/pkg/tool/linux_amd64/objdump.gz
1863347 ./go/pkg/tool/linux_amd64/addr2line.gz
1842210 ./go/pkg/tool/linux_amd64/nm.gz
1739594 ./go/pkg/tool/linux_amd64/dist.gz
1698346 ./go/bin/gofmt.gz
1608565 ./go/pkg/tool/linux_amd64/fix.gz
1324476 ./go/pkg/tool/linux_amd64/test2json.gz
1295801 ./go/pkg/tool/linux_amd64/buildid.gz

• Do we really need to include the tour?

• Can any binaries be combined into one binary and have different behavior based on invocation name, busybox style?

We have 6.6 MB of testdata:

$ du -b -c $(find -name testdata) | tail -1
6664279 total

• Do we need tests to pass for people who download binary distributions?

• Perhaps we should have a full download and a minimal download? (Minimal could lack godoc, gopher images, tests, test data, prebuilt *.a files, etc)

[bradfitz]

We ship three copies of Mark Twain's Tom Sawyer?

$ ls -l $(find . | grep -i twain)
-rw-r--r-- 1 bradfitz bradfitz 118509 Nov 22  2017 ./compress/bzip2/testdata/Mark.Twain-Tom.Sawyer.txt.bz2
-rw-r--r-- 1 bradfitz bradfitz 387851 Nov 22  2017 ./compress/testdata/Mark.Twain-Tom.Sawyer.txt
-rw-r--r-- 1 bradfitz bradfitz 387851 Nov 22  2017 ./net/testdata/Mark.Twain-Tom.Sawyer.txt

[FiloSottile]

About the pkg files that get rebuilt, a common issue in the past has been users not having write permissions on the system GOROOT, hence breaking or rebuilding every time. Will that be a problem?

[bradfitz]

@FiloSottile, that hasn't been a problem since at least Go 1.10. It writes to the build cache instead. You can experiment with the Dockerfile linked above.

[ALTree]

Do we really need to include the tour?

Currently we ship the tour so one can do go tool tour to run it locally, but alas that doesn't work if you built from source or you installed Go using the OS package manager, so we also have to explain in the welcome that if go tool tour doesn't work you have to go get it from the x repo.

This is all quite convoluted and I've seen reports of people confused by this. If we remove the tour from the binary releases we will also be able to simplify our instructions for running it locally: it's always "go get it, and then run the gotour command".

[bradfitz]

In that case I would like to remove the tour, maybe even for Go 1.11.

[andybons]

/cc @katiehockman

[gopherbot]

Change https://golang.org/cl/131156 mentions this issue: cmd/release: remove the tour from the releases