cmd/go: go get fails with private bitbucket repositories

[savaki]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOOS="linux"
GOARCH="amd64"

What did you do?

I attempted to go get from a private bitbucket repository.

1. add the following to ~/.gitconfig

[url "git@bitbucket.org:"]
	insteadOf = https://bitbucket.org/

2. add the following to ~/.ssh/config

Host bitbucket.org
  HostName bitbucket.org
  User git
  IdentityFile ~/.ssh/id_bitbucket
  StrictHostKeyChecking no

3. ensure that a valid ssh key exists for bitbucket, ~/.ssh/id_bitbucket

4. use go get on a private bitbucket repo with modules enabled

What did you expect to see?

I expected the private repository to be pulled to the appropriate ${GOPATH}/pkg/mod
directory.

What did you see instead?

go: bitbucket.org/myprivaterepo/projectname@v0.0.0-20180614180234-a79b05b2ef79: https://api.bitbucket.org/2.0/repositories/myprivaterepo/projectname?fields=scm: 403 Forbidden

In the golang-nuts group, Russ had previous posted that this was an issue and could be resolved by hand. With the advent of modules, manually checking out the correct version and putting it into the correct location becomes way more problematic.

[guybrand]

I had Similar issues in more than one laptop with Ubunto, MacOs and windows, and instead of blind troubleshooting (or reinstall git that sometimes helped mainly on windows), I resolved/worked around with these (temp/troubleshooting) changes/steps :

1. edit /usr/local/go/src/cmd/go/internal/get/vcs.go, I added (in bold)
   ...
   func bitbucketVCS(...
   for _, vcs := range []string{"git", "hg"} {
   fmt.Println("root:",root,"vcs",vcs)
2. still editing /usr/local/go/src/cmd/go/internal/get/vcs.go
   func (v *vcsCmd) runVerboseOnly(dir string, cmd string, keyval ...string) error {
   _, err := v.run1(dir, cmd, keyval, true) //was previously false
   save
3. terminal -> cd /usr/local/go/src/cmd/go
   sudo go build
4. (you may want to backup your previous bin/go...)
   sudo mv ./go /usr/local/go/bin/go
5. cd /go-projects-dir/:
   env GIT_TERMINAL_PROMPT=1 go get

that would:

1. should you the resolution for each repo (just a helper, in case you added "url" to git config etc) per vcs
2. would turn the silent bash script that pings bitbucket to interactive mode .
   Once you do that, you can see the actual issue which may vary from wrong ssh key, credential cash missing or invalid etc, and even allow you to interact with the command (e.g. if its a missing PW, you can type the password in)

In other words - this will not SOLVE your problem but has a good success rate of showing where the issue is.

tips:
If you are not actively working using mutiple bitbucket accounts make sure you have config --global git name, email, and url (not as you stated):
git config --global user.name "myBKuser"
git config --global user.email "me@myEmail.com"
git config --global url."https://MyCompany@bitbucket.org/".insteadOf "https://bitbucket.org/"
in one case - the latter resolved the issue for me, adding a credential.helper or alternatively ssh key

and last ... return your original bin/go :)

[guybrand]

Reason and suggested resolution:
The above behavior is a result of missing global git config/credentials, and going through the code - there can be many other similar issues - unless ALL parts are in place.

When go get command tries to pull a bitbucket repo, it first tries to get the repo details as a public repo, resolving in the "403 Forbidden" reply.

Then it tries to check if a private repo is git or hg (as per issue #5375) , first using git, that will also resolve in 403 - if global git config does not include user.name , user.email.
Yet, this would not be enough without cached credentials on cred manager OR ssh key configured for your bitbucket user.
Last, to make sure the git command goes to the right private repo , git global config should either identify this private repo group/initiator whenever you ping bitbucket, or at least when you ping the specific subdomain, for this you need to either
git config --global url."https://MyCompany@bitbucket.org/".insteadOf "https://bitbucket.org/"
or
git config --global url."https://MyCompany@bitbucket.org/".insteadOf "https://bitbucket.org/MyCompany"

vcs.go main problem is not reporting right statuses that will aid the user to resolve the configuration by himself.
A fix would need to:

1. Replace the "403 Forbidden" reply that looks like an error (but is actually just a ping for a public repo) in : "cant access public repo 'https://api.bitbucket.org/2.0/repositories/my-repo' - trying to connect to a private repo
2. check if git global config parameters includes user.name , user.email, if not, check hg connection, if that does not work - echo the user with a friendly error that says : "cant access public repo 'https://api.bitbucket.org/2.0/repositories/my-repo', if this is a private repo, please run:

git config --global user.name "your-bitbucket-user"
git config --global user.email "your-bitbucket-email"
git config --global url."https://your-bitbucket-domain@bitbucket.org/".insteadOf "https://bitbucket.org/domain"

3. if no ssh key is defined/not defined well, the git command will try to prompt for a git password (go get will not support that, but rather through an error), when this fails, vcs.go is currently discarding the error and treats this as a "general error with pinging the repo", in this case, user error should display a friendly error "in order to allow access to a private repo either your git password should be cached in a credential manager, or an ssh key must be supplied" (perhaps with extra details on how to do that on according to the running OS) and the error from the request to bitbucket displayed alongside the friendly error, for cases like the case above (@savaki says he has set up an ssh key, probably something wrong there, and the ping to bitbucket probably returned some error there, my guess is the last git config I suggested using domain@ would have solved it, displaying a friendly error can save the time for guessing) .

Comments:
all the above "warnings/errors" should be stored during the process, and only display if hg fails as well.
github repo's may have similar issues, but lesser, since no need to determine if the repo is git or hg.

If no comments, I can probably set some of the above logic coded in place (vcs.go).

[bcmills]

A fix would need to: […]

For (1), we shouldn't print anything that looks like an error until we're sure it's actually an error. If we get all the way through the chain of attempts and it's still failing, then we should display a helpful message.

For (2), that seems too specific (to git and to BitBucket). Perhaps a URL (such as https://golang.org/doc/faq#git_https or a Wiki page) would be more helpful? That could collect configuration steps for various providers and tools without baking it into the go tool itself: that way, if we discover a better way to configure private repos we can publish it without updating the tool.

For (3), we should probably pass through the explicit error from the tool. (That's #25982.)

[guybrand]

1. Agreed then, I have a good concept to allow that.
2. Good idea, I would need access to edit the faq in order to do that, I was given go/wiki edit permission, but I dont see an option to edit the faq, any idea who's authorizing that ?
3. The git error in this case is quite generic (403 Forbidden) adding some local checks (e.g git.global exists) can give a better indication, however we can concatenate the errors to cover both the tool error and local so the user reading https://golang.org/doc/faq#git_https would go to the right section (e.g. "please read the git.global section on https://golang.org/doc/faq#git_https) - I do agree its specific (for git, not for bitbucket), but its not that we have too many vcs's, and git IS the most common.

[dsymonds]

This looks to be a duplicate of #16315.