Support for private git repositories

[nhhagen]

When building from private git repos and the master branch is not yet tracked or up to date the builds fails with HTTP 401 (not authenticated) when trying to fetch the branch (TC only fetches the branch it is going to build)

For quick solution user/pass could be read from TC parameters before passing to libgit2sharp

Hopefully libgit2sharp can get propper ssh support and this will not be a problem.

[nhhagen]

Here is the build log for a private build over https

[20:31:07]Checking for changes
[20:31:09]Publishing internal artifacts
[20:31:09][Publishing internal artifacts] Sending build.start.properties.gz file
[20:31:09]Clearing temporary directory: D:\TeamCity\buildAgent\temp\buildTmp
[20:31:09]Checkout directory: D:\TeamCity\buildAgent\work\1fb6f465af877b6d
[20:31:09]Updating sources: agent side checkout (4s)
[20:31:09][Updating sources] Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[20:31:09][Updating sources] Cleaning D:\TeamCity\buildAgent\work\1fb6f465af877b6d
[20:31:09][Updating sources] VCS Root: git@bitbucket.org:comperio/git-flow-test.git (4s)
[20:31:09][VCS Root: git@bitbucket.org:comperio/git-flow-test.git] revision: 4e3405146e9e577271b6d81753fcf330a5ef49e7
[20:31:09][VCS Root: git@bitbucket.org:comperio/git-flow-test.git] Cleaning D:\TeamCity\buildAgent\work\1fb6f465af877b6d
[20:31:09][VCS Root: git@bitbucket.org:comperio/git-flow-test.git] The .git directory is missing in 'D:\TeamCity\buildAgent\work\1fb6f465af877b6d'. Running 'git init'...
[20:31:09][VCS Root: git@bitbucket.org:comperio/git-flow-test.git] Commit '4e3405146e9e577271b6d81753fcf330a5ef49e7' is not found in repository. Running 'git fetch'...
[20:31:13][VCS Root: git@bitbucket.org:comperio/git-flow-test.git] Checking out branch refs/heads/develop in git@bitbucket.org:comperio/git-flow-test.git in D:\TeamCity\buildAgent\work\1fb6f465af877b6d with revision 4e3405146e9e577271b6d81753fcf330a5ef49e7
[20:31:13][VCS Root: git@bitbucket.org:comperio/git-flow-test.git] Cleaning git@bitbucket.org:comperio/git-flow-test.git in D:\TeamCity\buildAgent\work\1fb6f465af877b6d the file set ALL_UNTRACKED
[20:31:13]Step 1/3: NuGet Installer
[20:31:13][Step 1/3] scan: Searching for nuget.config files
[20:31:13][scan] Found packages folder: D:\TeamCity\buildAgent\work\1fb6f465af877b6d\packages
[20:31:13][scan] Found list of packages.config files: D:\TeamCity\buildAgent\work\1fb6f465af877b6d\packages\repositories.config
[20:31:13][scan] Scanning projects in solution file: D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test.sln
[20:31:13][scan] Found project D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj
[20:31:13][Step 1/3] install: Installing NuGet packages for git-flow-test\packages.config
[20:31:13][install] NuGet command: D:\TeamCity\buildAgent\tools\NuGet.CommandLine.DEFAULT.nupkg\tools\NuGet.exe install D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\packages.config -OutputDirectory D:\TeamCity\buildAgent\work\1fb6f465af877b6d\packages -Source https://nuget.org/api/v2/ -Source https://comperio.myget.org/F/comperio/
[20:31:13][install] Starting: D:\TeamCity\buildAgent\temp\agentTmp\custom_script8657142140777516706.cmd
[20:31:13][install] in directory: D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test
[20:31:14][install] Installing 'GitFlowVersionTask 0.4.0'.
[20:31:14][install] Successfully installed 'GitFlowVersionTask 0.4.0'.
[20:31:14][install] Process exited with code 0
[20:31:14]Step 2/3: Visual Studio (sln) (2s)
[20:31:14][Step 2/3] Starting: D:\TeamCity\buildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.MsBuildBootstrap.exe /workdir:D:\TeamCity\buildAgent\work\1fb6f465af877b6d /msbuildPath:C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe
[20:31:14][Step 2/3] in directory: D:\TeamCity\buildAgent\work\1fb6f465af877b6d
[20:31:15][Step 2/3] git-flow-test\git-flow-test.csproj.teamcity: Build target: Rebuild
[20:31:15][git-flow-test\git-flow-test.csproj.teamcity] CoreClean
[20:31:15][CoreClean] MakeDir
[20:31:15][MakeDir] Creating directory "obj\Debug\".
[20:31:15][git-flow-test\git-flow-test.csproj.teamcity] PrepareForBuild
[20:31:15][PrepareForBuild] MakeDir
[20:31:15][MakeDir] Creating directory "bin\Debug\".
[20:31:15][git-flow-test\git-flow-test.csproj.teamcity] UpdateAssemblyInfo
[20:31:15][UpdateAssemblyInfo] GitFlowVersionTask.UpdateAssemblyInfo
[20:31:16][GitFlowVersionTask.UpdateAssemblyInfo] Version not in cache. Calculating version.
[20:31:16][GitFlowVersionTask.UpdateAssemblyInfo] No local branch with name master found, going to try on the remote origin(https://nhhagen@bitbucket.org/comperio/git-flow-test.git)
[20:31:16]
[GitFlowVersionTask.UpdateAssemblyInfo] Error occurred: LibGit2Sharp.LibGit2SharpException: An error was raised by libgit2. Category = Net (Error).
Request failed with status code: 401
   at LibGit2Sharp.Core.Ensure.HandleError(Int32 result)
   at LibGit2Sharp.Core.Proxy.git_remote_connect(RemoteSafeHandle remote, GitDirection direction)
   at LibGit2Sharp.Network.Fetch(Remote remote, Nullable`1 tagFetchMode, ProgressHandler onProgress, CompletionHandler onCompletion, UpdateTipsHandler onUpdateTips, TransferProgressHandler onTransferProgress, Credentials credentials)
   at GitFlowVersion.LibGitExtensions.GetBranch(IRepository repository, String name) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\LibGitExtensions.cs:line 129
   at GitFlowVersion.VersionOnMasterFinder.<VersionsOnMaster>d__3.MoveNext() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionOnMasterFinder.cs:line 71
   at GitFlowVersion.VersionOnMasterFinder.Execute() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionOnMasterFinder.cs:line 37
   at GitFlowVersion.DevelopVersionFinder.GetSemanticVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\BranchFinders\DevelopVersionFinder.cs:line 36
   at GitFlowVersion.DevelopVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\BranchFinders\DevelopVersionFinder.cs:line 13
   at GitFlowVersion.GitFlowVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\GitFlowVersionFinder.cs:line 44
   at GitFlowVersion.VersionForRepositoryFinder.GetVersion(Repository repository) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionForDirectoryFinder.cs:line 18
   at GitFlowVersion.VersionCache.GetVersion(String gitDirectory) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionCache.cs:line 39
   at GitFlowVersionTask.UpdateAssemblyInfo.Execute() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersionTask\UpdateAssemblyInfo.cs:line 57
[20:31:16][git-flow-test\git-flow-test.csproj.teamcity] Project git-flow-test\git-flow-test.csproj.teamcity failed.
[20:31:16][Step 2/3] Process exited with code 1
[20:31:16][Step 2/3] MSBuild output
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.DevelopVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\BranchFinders\DevelopVersionFinder.cs:line 13 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.GitFlowVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\GitFlowVersionFinder.cs:line 44 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionForRepositoryFinder.GetVersion(Repository repository) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionForDirectoryFinder.cs:line 18 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionCache.GetVersion(String gitDirectory) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionCache.cs:line 39 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersionTask.UpdateAssemblyInfo.Execute() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersionTask\UpdateAssemblyInfo.cs:line 57 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] Done Building Project "D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity" (Rebuild target(s)) -- FAILED.
[20:31:16][MSBuild output] 
[20:31:16][MSBuild output] Build FAILED.
[20:31:16][MSBuild output] 
[20:31:16][MSBuild output] "D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity" (Rebuild target) (1) ->
[20:31:16][MSBuild output] (UpdateAssemblyInfo target) -> 
[20:31:16][MSBuild output]   MSBUILD : error : Error occurred: LibGit2Sharp.LibGit2SharpException: An error was raised by libgit2. Category = Net (Error). [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error : Request failed with status code: 401 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at LibGit2Sharp.Core.Ensure.HandleError(Int32 result) [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at LibGit2Sharp.Core.Proxy.git_remote_connect(RemoteSafeHandle remote, GitDirection direction) [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at LibGit2Sharp.Network.Fetch(Remote remote, Nullable`1 tagFetchMode, ProgressHandler onProgress, CompletionHandler onCompletion, UpdateTipsHandler onUpdateTips, TransferProgressHandler onTransferProgress, Credentials credentials) [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.LibGitExtensions.GetBranch(IRepository repository, String name) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\LibGitExtensions.cs:line 129 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionOnMasterFinder.<VersionsOnMaster>d__3.MoveNext() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionOnMasterFinder.cs:line 71 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionOnMasterFinder.Execute() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionOnMasterFinder.cs:line 37 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.DevelopVersionFinder.GetSemanticVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\BranchFinders\DevelopVersionFinder.cs:line 36 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.DevelopVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\BranchFinders\DevelopVersionFinder.cs:line 13 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.GitFlowVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\GitFlowVersionFinder.cs:line 44 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionForRepositoryFinder.GetVersion(Repository repository) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionForDirectoryFinder.cs:line 18 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionCache.GetVersion(String gitDirectory) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionCache.cs:line 39 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersionTask.UpdateAssemblyInfo.Execute() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersionTask\UpdateAssemblyInfo.cs:line 57 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] 
[20:31:16][MSBuild output]     0 Warning(s)
[20:31:16][MSBuild output]     1 Error(s)
[20:31:16][MSBuild output] 
[20:31:16][MSBuild output] Time Elapsed 00:00:00.68
[20:31:16][Step 2/3] Step Visual Studio (sln) failed
[20:31:16]Step 3/3: NuGet Pack
[20:31:16][Step 3/3] Build step NuGet Pack is skipped because the previous step has failed
[20:31:16]Publishing artifacts
[20:31:16][Publishing artifacts] Collecting files to publish: [D:\TeamCity\buildAgent\temp\buildTmp\nuget24305130608310186packages\nuget.xml => .teamcity/nuget]
[20:31:16][Publishing artifacts] Sending D:/TeamCity/buildAgent/temp/buildTmp/nuget24305130608310186packages/nuget.xml
[20:31:16]Publishing internal artifacts
[20:31:16][Publishing internal artifacts] Sending build.finish.properties.gz file
[20:31:16]Build finished

[JustinThirkell]

i've come across exactly the same error.
@SimonCropp, @andreasohlund , how are you handling this on the Particular TC server?

for the moment i'm hacking in a post-versioncontrol, pre-build step to ensure the master branch exists for a develop branch build.
This basically does the following (i'm yet to handle case where the master branch already exists on the build server from a previous build):

• call git remote set-branches origin --add master
• call git fetch origin
• call git branch --track master origin/master
  (we've got a private github enterprise and using ssh exclusively to get to it from our TC)

i'm guessing since you guys (Particular) are using this for your develop branch builds you must have found a way to ensure master branch is available at build time...?

thanks

[andreasohlund]

We automatically pull down the master branch and that works on TC as well (you need agent checkout enabled)

[JustinThirkell]

thanks for quick reply.

yep, i've got agent checkout enabled. (also found out how to disable shallow fetch for a repo which is our TC default!!)
from the logs it appears TC is issuing the following:
"C:\Program Files (x86)\Git\bin\git.exe" fetch --progress origin +refs/heads/develop:refs/heads/develop

how do you ensure master branch is auto pulled down too?

btw, i'm loving gitflowversion - very very cool idea.

[andreasohlund]

We auto pull master as well

On Tue, Oct 22, 2013 at 8:36 AM, Justin Thirkell
notifications@github.comwrote:

thanks for quick reply.

yep, i've got agent checkout enabled. (also found out how to disable
shallow fetch for a repo which is our TC default!!)
from the logs it appears TC is issuing the following:
"C:\Program Files (x86)\Git\bin\git.exe" fetch --progress origin
+refs/heads/develop:refs/heads/develop

how do you ensure master branch is auto pulled down too?

btw, i'm loving gitflowversion - very very cool idea.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-26780255
.

[JustinThirkell]

sorry, being dense here.
I understand you auto pull master branch down onto the build server (and how that makes it all work)

what i don't understand is how exactly you're doing that.
are you using a custom git step to checkout or using the builtin jetbrains.git vcs?

our setup is that we tell TC to checkout develop branch - can't see anywhere to tell TC to auto pull master as well...

we're using the jetbrains.git vcs, and fwiw, our vcs settings are:
fetch url = git@github.mycompany.com:mycompany/MyRepo.git
default branch = %branch% (which is set as a per-buildconfig parameter)
branch specification = [currently empty]

[andreasohlund]

This is how we do it

https://github.com/Particular/GitFlowVersion/blob/4a879843194ad858e76f9bb0502517e90a9d675d/GitFlowVersion/LibGitExtensions.cs#L43

So you don't need anything extra.

fetch url = git@github.mycompany.com:mycompany/MyRepo.git
Think this might be the issue, can you try https instead?

On Tue, Oct 22, 2013 at 8:45 AM, Justin Thirkell
notifications@github.comwrote:

sorry, being dense here.
I understand you auto pull master branch down onto the build server (and
how that makes it all work)

what i don't understand is how exactly you're doing that.
are you using a custom git step to checkout or using the builtin
jetbrains.git vcs?

our setup is that we tell TC to checkout develop branch - can't see
anywhere to tell TC to auto pull master as well...

we're using the jetbrains.git vcs, and fwiw, our vcs settings are:
fetch url = git@github.mycompany.com:mycompany/MyRepo.git
default branch = %branch% (which is set as a per-buildconfig parameter)
branch specification = [currently empty]

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-26780624
.

[JustinThirkell]

right, ok, that explains it.
yep, this in previous build logs:

[GitFlowVersionTask.UpdateAssemblyInfo] Error occurred: GitFlowVersion.MissingBranchException: Could not fetch from 'git@github.dev.xero.com:Xero/Bus.git' since LibGit2 does not support the transport. You have most likely cloned using SSH. If there is a remote branch named 'master' then fetch it manually, otherwise please create a local branch named 'master'. ---> LibGit2Sharp.LibGit2SharpException: An error was raised by libgit2. Category = Net (Error).
This transport isn't implemented. Sorry

currently we're not allowed to use https - it's all ssh here. sigh.
above error message is why i'm having to manually pull down master branch via ssh.

so the answer is that Particular are using https to enable fetching master via LibGitExtensions.

think I'll be asking Ops to turn on https tomorrow. ;)
thanks for your help

[JustinThirkell]

oh, and hunting down ssh status for libgit2, look who turns up in the PR comments...
libgit2/libgit2#1558 (comment)
😄

[SimonCropp]

I was just about to post that link.

So you need to ask the libgit2sharp guys about ssh on windows. Since we (particular) are happy using https we are not going to push it too hard.

[nhhagen]

https has not been a problem for me either, but a password protected https repo (e.g. private github/bitbucket repo) does not work on version 0.4.0 (as far as I've gotten).

I still use ssh and are using this workaround for now.

Add a new PowerShell step (the is a plugin for TC)

add this script:

$branch = . git symbolic-ref --short -q HEAD;. git pull; foreach ($remote in . git branch -r) { . git checkout $remote.Trim().Replace("origin/", ""); . git pull } . git checkout $branch

It will track and update all branches.

My set-up is currently

TC with 6 agents and private bitbucket repos accessed over ssh.

[JustinThirkell]

that's really helpful - thanks very much!

[JakeGinnivan]

The command line version supports username and password being passed.

Are we happy to close this issue, also do we need an issue to track adding username/password support in the MSBuild task?
@andreasohlund @SimonCropp