cmd/go: Add a command to zip the current project and push it to a hosted repository

[j-s-3]

Problem

With go.mod and GOPROXY the go tooling now has a good story for proxying open source packages but doesn't have a clear story for how best to manage and consume private packages. This is especially problematic for Repository Managers (such as Nexus Repository Manager) and makes it difficult for Go developers to share their private packages within their organizations.

Goal

Add a new command the Go CLI that creates a zip in a format that can be consumed via GOPROXY and push/POST that zip to a HTTP(s) endpoint.

This would allow Repository Managers to implement this endpoint and consume Go packages, making those packages available to Go developers via the download APIs. Go developers could then use this for sharing their private packages within their organizations.

[triztian]

I think this can be achieved this by this zipping up $GOPATH/pkg/mod/cache/download and then extracting it elsewhere, after extracting the zip archive you can use the GOPROXY env var with an absolute path pointing to the extracted location:

Something like:

zip -r gomods.zip $GOPATH/pkg/cache/download
curl -X POST -H "Content-Type: application/zip" --data @gomods.zip
https://example.org/gomods/download

And then elswhere:

GOPROXY=file:///path/to/extracted/contents go mod verify

Unless I'm missing something?

[j-s-3]

@triztian I’m referring to zipping your current project rather than its dependencies and then publishing that to a remote server such as Nexus Repository Manager. This would make for a better user experience for companies that are developing their own private libraries for use by other developers or teams.

[bcmills]

This is closely related to #28835.

[bcmills]

@triztian, there are some constraints on the file paths within the zip files: they must be slash-separated and have a “<module>@<version>/” prefix.

They also need to omit any directory subtrees containing a go.mod file.

[bcmills]

CC @jayconrod

[bcmills]

Perhaps we should start with a function in golang.org/x/mod or standalone binary in golang.org/x/tools (CC @ianthehat).

[triztian]

@jlstephens89 Got it that makes sense, then yes, it's definitively missing such feature specially with the limitations that @bcmills mentioned. After looking at the linked issue #28835 it seems that the same could be achieved with a combination of go mod edit and downloading the remote source from the repository manager (i.e. Nexus Repository Manager).

For example one could still zip up the private source and then download it as a zip file and extract it somewhere on disk.

After extracting one could use go mod edit to replace the imports to point to the version on disk, for example, suppose that the package is provider.com/some/library and that we are able to download and extract the source of such package to /path/to/extracted/sources in our dev machine, then one could do something like this (assuming that the sources have been zipped as-is and pushed to repsmanager.org):

wget https://repsmanager.org/provider.com/some/library.zip # downloads library.zip
mkdir -p /path/to/extracted/sources
unzip library.zip -d /path/to/extracted/sources

cd myproject # the root of the project that depends on provider.com/some/library package
go mod edit -replace=provider.com/some/library=/path/to/extracted/sources
go build -mod=readonly .

Doing this avoids having to pre-populate the $GOPATH/pkg/mod/cache/download with the deps of the project, on the publishers side all that she/he has to do is zip the sources and make them available for download. The Repository Managers would also have access to the sources because they arere plain zip files so as long as the zip file is pushed to the RM no special command is needed. Also by using replace statements one no longer has to deal with the setup of GOPROXY:

• Replace Docs: https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive

I just did a sample test of the go mod edit, before the edit:

λ ui ~> cat go.mod
module gioui.org/ui

go 1.12

require (
	golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9
	golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
)

Performing the edit:

 λ ui ~> go mod edit -replace=golang.org/x/image=/opt/thirdparty/go/sources/x/image

After the edit:

 λ ui ~> cat go.mod
module gioui.org/ui

go 1.12

require (
	golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9
	golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
)

replace golang.org/x/image => /opt/thirdparty/go/sources/x/image

Based on the replace docs I think one could script to replace all deps to point to the Repository Manager.

[j-s-3]

@triztian. GOPROXY and the download API make for a very natural user experience when working with a repository manager.

If we went the route of adding a command that created a zip with the correct file paths then any developer that has set GOPROXY to point to a “Group” repository, a repository type that aggregates public and private sources (or alternatively list the URLs comma separated), would have access to both internal and external dependencies with no additional effort. This would be a one time configuration for all developers in an organisation and from then on would allow centralised tooling teams to manage remotes/sources in a single place.

To publish a private dependency developers could do something like “go push http://repourl” and that dependency would then be available to an entire organisation.

[jayconrod]

At the moment, it would be hard to implement this in an external tool or in x/mod: most of the logic for this is in cmd/go/internal/modfetch and cmd/go/internal/modfetch/codehost. Those packages should eventually be moved to x/mod, but at the moment, they are tightly integrated with many other packages in the go command.

[bcmills]

Actually, I think all of the logic for actually constructing the zip file is in one location, at the end of (*modfetch.codeRepo).Zip.

I think it would be pretty straightforward to extract that body out to a function.

[triztian]

So basically it'd be a tool that performs the following HTTP (or similar) requests for the given module path?:

POST $GOPROXY/<module>/@v/list 
<version>

Append the version of this module to the ist of all known versions of the given module, one per line.

PUT $GOPROXY/<module>/@v/<version>.info

Set JSON-formatted metadata about that version of the given module.

PUT $GOPROXY/<module>/@v/<version>.mod 

Set the go.mod file for that version of the given module.

PUT $GOPROXY/<module>/@v/<version>.zip

Set the zip archive for that version of the given module.