x/vgo: not working with GitHub Enterprise

[richardwilkes]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10 darwin/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rich/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/rich/code/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/s9/9s3fk2mx5f936dkcp66ltwsc0000gn/T/go-build062786983=/tmp/go-build -gno-record-gcc-switches -fno-common"
VGOMODROOT=""

What did you do?

Attempted to build some code that had dependencies on one or more packages that come from our internal GitHub repo as well as from the public github.com using the experimental vgo tool. As this refers to an internal repo (and that is the basis of the problem), there is no way for me to give you code that reproduces the problem that you can actually try. Suffice to say, though, that even the simplest import causes the issue.

What did you expect to see?

I was hoping that GitHub Enterprise would work as well as normal GitHub.

What did you see instead?

vgo: resolving import "jaxf-github.fanatics.corp/forge/furnace"
findRepo: XML syntax error on line 25: unescaped < inside quoted string
vgo: import "jaxf-github.fanatics.corp/forge/ratelimiter" ->
import "jaxf-github.fanatics.corp/forge/furnace": unknown VCS, Repo: git,
https://jaxf-github.fanatics.corp/forge/furnace.git

[ALTree]

go get has the same issue: #17898

[richardwilkes]

Yes, I just realized that as well. This is still workable when using the normal go build, though, because we can (and do) manually checkout our internal repos into our $GOPATH/src tree. That doesn't seem to be an option with vgo, unless I've missed something.

[rsc]

I don't know much about GitHub Enterprise. Does it have similar APIs to public GitHub? It is possible this could work better with vgo than old go get. If the tag said something like githubenterprise to tell us it was that kind of server, we could use those APIs.

[richardwilkes]

Can you give me an example request? I'm happy to make it against something in our repo and give you the resulting output.

[mikkeloscar]

The APIs are the same (except for the base path) but vgo would have to know that something like https://github.mycompany.com is a Github Enterprise installation.

It would be great to think about this case from the beginning!

I opened a similar issue when dep was released: golang/dep#174 but this is unfortunately still an issue preventing us from adopting dep.

[amnonbc]

This is an issue which has caused me a fair amount of pain.

See https://go-review.googlesource.com/c/go/+/37176 for a simple fix for the go get case in go-1.10

[sdboyer]

FWIW, i have been collecting a group of company GHE customers to try to influence GitHub to actually fix this problem. (it exists in GitHub itself, as well, but it's masked by having static rules that cover it)

[amnonbc]

Hi @sdboyer
I work for a GHE customer, and raised the issue a couple of months ago with them.
This is what they told me:

"[Thaddeus Cortez] and his team will work with Engineering to establish contact with Google's Go team to help them provide a solution on their end.
We've discussed various ways of fixing this on GitHub's side, but none of the potential solutions are ideal. The meta tag approach comes with the problem of information leakage which we consider a serious security risk. It's a trade-off we are not willing to make. And, a toggleable option in the web UI would require us to turn our static error pages into dynamic ones which would come with several different issues, most notably performance and UX implications as well as added complexity in terms of code maintenance.

We really think, the Go team needs to integrate support for other sources like GitHub Enterprise (or one of our competitor's products), and not just GitHub.com."

[cjs]

Does vgo have the same reliance on the meta tag returned by the ?go-get=1 query string that go get has?

[jameschensmith]

@amnonbc & @sdboyer,
I work for a GHE customer as well, and this was their response to us:

The "go-import" meta tag is used for installing Go applications from a URL, and is indeed supported on GitHub Enterprise, there is a caveat however:
 
To avoid leaking information about repositories when the appliance is in private mode, GitHub Enterprise doesn't return the meta tag unless you're authenticated. The ideal solution is for Go to be modified to allow authentication options, but for now, you can work around this by adding .git to your repository paths. For example:
import "[ghe-hostname]/bevns/HelloWorld.git"

Or for go get:
go get -v [ghe-hostname]/bevns/HelloWorld.git

Alternatively, another workaround is to manually clone the repository into your $GOPATH and then run go-install. For example:

git clone https://[ghe-hostname]/bevns/HelloWorld $GOPATH/src/[ghe-hostname]/bevns/HelloWorld
go install [ghe-hostname]/bevns/HelloWorld

Some other customers using Go have implemented a workaround where Go import statements refer to an internal server whose sole purpose is to direct go get to the proper repository URLs. The proxy server would respond to both HTTP and HTTPS and can return a meta tag that will point to your GitHub Enterprise instance.

For example, you could have an import statement of the form import my.ghe.domain/user/project, which should result in the following HTTP request:
https://my.ghe.domain/user/project?go-get=1

This should then return a meta tag that looks something like this:
<meta name="go-import" content="my.ghe.domain git https://my.ghe.domain/user/project.git">

That should then result in the correct repository being cloned.

They also added this:

Please also keep in mind that this proxy server would need to be a standalone server and should not be configured on your GitHub Enterprise server.
 
The best solution will come when the Go team accepts and merges #17898. That will detect the headers sent from both the GitHub Enterprise appliance and GitHub.com, and use that for verification that a Git repository is being requested.
 
I would also like to mention that some alternative Go package managers such as https://glide.sh/ work with the Go toolchain and support private Git servers and repositories.

[amnonbc]

@james-r-smith yes we use the former hack as a workaround at the moment.
The problem is that it forces the users of your library to modify their import paths in a way that is not obvious (i.e. appending .git). This adds unnecessary friction, especially to new users.

[jameschensmith]

@amnonbc, I wholeheartedly agree.

[cjs]

With the GitHub Enterprise 2.13 release, on-premises GitHub Enterprise instances will now send the <meta> tags expected by go get for private instances and all repositories.

For instances that are in private mode, you will still need to manage authentication for the git pull phase of go get, but there are easy workarounds such as putting an authentication token in your .netrc or the insteadOf technique to force SSH as mentioned above.

x/vgo, from what I can tell currently only supports GitHub.com, but that's to be expected considering it is still in proposal status.