Git checkout of large repositories is very slow

[kaidokert]

Windows Build Number

10.0.18363.0

Processor Architecture

AMD64

Memory

200 Gb

Storage Type, free / capacity

SSD 200GB/ 1TB

Relevant apps installed

git version 2.31.1.windows.1

Traces collected via Feedback Hub

N/A

Isssue description

Checking out large repos even from a local mirror is slow, compared to Linux / Mac.

Even more importantly, so is switching branches / tags.

Steps to reproduce

Let's download a sample well-known repo, about 23 Gb

git clone --mirror https://github.com/chromium/chromium.git chromium-mirror

Now let's check out a source tree from local mirror:

Powershell, NTFS drive:

PS> Measure-Command { git clone chromium-mirror chromium }                                                                                                                                                                                        
Cloning into 'chromium'...                                                                                                                                                                                                                                                
done.                                                                                                                                                                                                                                                                     
Updating files: 100% (362149/362149), done.                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                          
TotalMinutes      : 12.370800555                                                                                                                                                                                                                                          

Just over 12 minutes.

On Linux, ext4, similar hardware

$ time git clone chromium-mirror chrome
Cloning into 'chrome'...
done.
Updating files: 100% (362149/362149), done.
real	0m23.937s

About 24 seconds.

Now, let's check out a bit older tag:
Powershell:

PS> Measure-Command { git checkout 60.0.3072.0 }                                                                                                                                                                                         
Updating files: 100% (539180/539180), done.                                                                                                                                                                                                                               

TotalMinutes      : 15.750022275                                                                                                                                                                                                                                          

15 minutes to switch a tag.

On Linux:

time git checkout 60.0.3072.0
Updating files: 100% (539180/539180), done.
Note: switching to '60.0.3072.0'.
..
real	0m22.045s

Again, about 22 seconds

Finally, let's delete these experiment directories:

Powershell:

PS> Measure-Command { rm -r chromium }
...
TotalMinutes      : 7.40425179833333                                                                                                                                                                                                                                      

7 minutes

Linux:

$ time rm -rf chrome
real	0m5.580s

5 seconds

Expected Behavior

Would expect checkout speed on similar disks to be at least on the same order of magnitude.

Actual Behavior

The operations in this example are about 25-30x slower, on almost identical hardware.

Of course the problem doesn't seem inherent to Git, it's a similar I/O problem when working with large directory trees with many files, i.e. Node node_modules issues ( #21 ) and others ( #17 #27 ), as evidenced by the fact that rm -r took over 60x longer.

[AvriMSFT]

Thanks for filing! Good news - We're actively investigating the performance of Git on Windows and are working with internal teams here at Microsoft and external teams at Github to diagnose the problem and find solutions for reducing the performance gaps between Windows and Mac/Linux. I will update this thread as fixes are identified and complete. In the meantime, can you repro the issue on Windows and capture a trace via Feedback Hub? We can use this data to better understand how Git performs in the wild and fuel our investigations.

[itoleck]

My testing cloning Chromium locally between 2 SSDs (Intel 660p & WD SN550).

Windows Terminal PowerShell Core window
D:\Temp  Measure-Command { git clone c:\Users\chad\source\repos\chromium-mirror d:\temp\chromium }
Cloning into 'd:\temp\chromium'...
done.
Checking out files: 100% (363000/363000), done.

Days : 0
Hours : 0
Minutes : 4
Seconds : 38
Milliseconds : 220
Ticks : 2782200516
TotalDays : 0.00322013948611111
TotalHours : 0.0772833476666667
TotalMinutes : 4.63700086
TotalSeconds : 278.2200516
TotalMilliseconds : 278220.0516

Windows Terminal Ubuntu WSL
/mnt/c/Users/chad/source/repos$ time git clone /mnt/c/Users/chad/source/repos/chromium-mirror/ /mnt/d/Temp/
Cloning into '/mnt/d/Temp'...
done.
Updating files: 100% (363000/363000), done.

real 82m14.742s
user 1m30.634s
sys 9m18.089s

Seems like WSL does NOT help the Git performance in Windows.

[sabbyX]

@itoleck are you using WSL2? if so cross-os filesystem performance is slow, https://docs.microsoft.com/en-us/windows/wsl/compare-versions#performance-across-os-file-systems

[kaidokert]

@AvriMSFT posted feedback hub capture at https://aka.ms/AAcdkct. Although not sure how useful that is, as it's just even rm -r directory command that is slow.

[kaidokert]

Ping, the label here still says "Needs Author Feedback", but i'm not sure what other information could i provide.

[matheustavares]

Hi @kaidokert and @itoleck

This is a bit tangent to the actual issue, but: the next Git release (2.32.0) will come with the "parallel checkout" feature, which allows git checkout, git clone, and other checkout-related commands to parallelize the creation of files in the working tree.

I haven't tested it on WSL, but I'm getting around 2x speedup when cloning the linux repository with Git for Windows on a SSD.

If you would be interested in testing it out before the release, you'd need to compile Git on your machine. The code for this feature has already landed in the master branch of the upstream Git repository and also on the main branch of the Git-for-Windows repository. Then, to activate parallel-checkout, you must set the checkout.workers config to the desired number of parallel processes (one means sequential).

As one of the developers of this feature, I'd be very interesting in any feedback you have about it. So if you do run the tests with parallel checkout or have any question/suggestion about it, please let me know :)

[kaidokert]

It does seem like majority of the slowness here doesn't actually come from file creation but from file deletion, which of course happens a lot when switching branches.

[asklar]

Hey @kaidokert I didn't see this called out in the template but do you have a Windows Defender exclusion enabled for this folder? It would be interesting to see how the cloning timing might vary with defender out of the picture.

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[Eli-Black-Work]

@asklar Does Windows Defender normally examine files when they're being permanently deleted? Just curious; not too familiar with how antivirus works under the hood 🙂