x/vgo: vgo removed -insecure flag

[saibing]

What version of Go are you using (go version)?

go version go1.10.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/baas/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/baas/git/go"
GORACE=""
GOROOT="/home/baas/my/go"
GOTMPDIR=""
GOTOOLDIR="/home/baas/my/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build608113082=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ curl -I http://rnd-github.huawei.com/db/parser/sql?go-get=1
HTTP/1.1 404 Not Found
Via: 1.1 172.18.32.117 (McAfee Web Gateway 7.6.2.14.0.23766)
Date: Thu, 17 May 2018 03:47:45 GMT
Server: GitHub.com
Status: 404 Not Found
X-Runtime: 0.006361
Content-Type: text/html; charset=utf-8
X-Request-Id: 715f56762358f8dbcf8968484163ceb3
Content-Length: 129318
X-Frame-Options: deny
Proxy-Connection: Keep-Alive
X-XSS-Protection: 1; mode=block
X-GitHub-Request-Id: 0f75ee93-d875-48d1-8dc7-8bcfa529656f
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src 'self'; connect-src 'self' http://rnd-github.huawei.com https://status.github.com rnd-github.huawei.com; font-src 'self'; form-action 'self' rnd-github.huawei.com; frame-ancestors 'none'; frame-src 'self'; img-src * data:; media-src 'none'; script-src 'self'; style-src 'unsafe-inline' 'self'
Connection: close

$ vgo get -insecure rnd-github.huawei.com/db/parser/sql

What did you expect to see?

get the rnd-github.huawei.com/db/parser/sql from github enterprise

What did you see instead?

flag provided but not defined: -insecure
usage: get [build flags] [packages]
Run 'go help get' for details.

[kardianos]

@rsc Was the removal of the -insecure flag intentional? Should it stay removed?

[kardianos]

Insecure was probably removed because many commands can now hit remote servers. Adding insecure flags to all such commands is probably not great idea.

Perhaps if this remains a requirement we could introduce a new environment variable that takes a list of host prefixes to whitelist.

GO_WHITELIST_ALLOW_INSECURE=internal.mycorp.local/vcs:vcs.test.local

[as]

I would prefer if this was a flag. I would like to see it with the ps command as that is easier to audit. Environment variables are significantly harder to trace in this manner.

[rsc]

This is the first report I've seen of GitHub Enterprise running over HTTP instead of HTTPS. Is it really not available under HTTPS in your environment?

One workaround is to put something in your ~/.gitconfig to tell git to rewrite https://your.server to http://your.server.

[saibing]

@rsc

Thank you for your answer very much.

However, this method does not solve the problem, because vgo does not read .gitconfig file, it still access https://rnd-github.huawei.com/db/parser?go-get=1.

baas@SZX1000428520:~/git/go/src/sql$ git config --get url.http://rnd-github.com/.insteadof
https://rnd-github.com/
baas@SZX1000428520:~/git/go/src/sql$ vgo get rnd-github.huawei.com/db/parser
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo get rnd-github.huawei.com/db/parser: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: finding rnd-github.huawei.com/db/parser v0.0.0-20180125184044-316ea1d6f45f
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/db/parser: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: finding rnd-github.huawei.com/go/otto v1.0.0
FindRepo: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/go/otto: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo: finding rnd-github.huawei.com/go/otto v1.0.0
FindRepo: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/go/otto: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo get: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
baas@SZX1000428520:~/git/go/src/sql$

[myitcv]

It looks like your git config isn't quite right (missing a colon). Here's an example:

git config url.http://localhost:8080/:.insteadof https://github.com/

Notice the : before the .insteadof.

Also, you'll need to ensure that you don't have old configs that are potentially clashing:

git config -l | grep -i insteadof

But like @rsc asked, do you really not have https available?

[saibing]

@myitcv

Thank you very much.

I tested it, this does not solve the problem

baas@SZX1000428520:~/git/go/src/sql$ git config -l | grep -i insteadof
url.http://rnd-github.com/:.insteadof=https://rnd-github.com/
baas@SZX1000428520:~/git/go/src/sql$ vgo build
vgo: finding rnd-github.huawei.com/db/parser v0.0.0-20180125184044-316ea1d6f45f
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/db/parser: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect

For the follow problem,　I have used nginx agent to solve this problem:）

But like @rsc asked, do you really not have https available?

[saibing]

@rsc @myitcv

Thank you for your support, I have use a nginx proxy to support https

[myitcv]

Just for the record, it looks like the insteadof config didn't work because you were using the incorrect domain. vgo was trying to resolve rnd-github.huawei.com yet your config was for rnd-github.com.

So you would need to have the following:

git config url.http://rnd-github.huawei.com/:.insteadof https://rnd-github.huawei.com/

I have use a nginx proxy to support https

Glad you have a solution.

[saibing]

@myitcv

I feel sorry for my mistake

I tested it again, but it still can not solve the problem

baas@SZX1000428520:~/git/go/src/sql$ git config -l | grep .insteadof
url.http://rnd-github.huawei.com/:.insteadof=https://rnd-github.huawei.com/
baas@SZX1000428520:~/git/go/src/sql$ vgo build
vgo: finding rnd-github.huawei.com/db/parser v0.0.0-20180125184044-316ea1d6f45f
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect

I found that vgo does not use .gitconfig's url map：

func lookupCustomDomain(path string) (Repo, error) {
	var body io.ReadCloser
	err := web.Get(
		"https://"+path+"?go-get=1",
		web.Non200OK(),
		web.Body(&body),
	)
	if body != nil {
		defer body.Close()
	}
        //...
}

[myitcv]

@saibing are you using the latest commit of vgo?

Looks like you are using an old version, pre #24915

Can you check:

go get -u golang.org/x/vgo
(command cd $(go list -f "{{.Dir}}" golang.org/x/vgo); git rev-parse HEAD)

outputs:

6a94eb3b5ccc04453d2fb45c23641e5993118068

for me.