sparse-checkout does not appear to work on windows. "error: invalid path "

[wdtj]

When I attempt to checkout a repository from github I get the error:

error: invalid path 'configs/perl-modules/DIST.64/perl-HTML-Tree-1:5.03-1.el6.noarch.rpm'

I suspect the issue is that the path contains a : which is illegal on Windows.

After researching the error, I've found 2 possible answers:
1) Change the path on the repository file. Unfortunately, this is is a team resource and can not be fixed in the foreseeable future.
2) Use sparse-checkout. I've tried this with no effect as evidenced in the following:

$ git clone -n git@github.com:XXXXXX/deploy.git          [[[Name obscured because it is a private repo.]]]
Cloning into 'deploy'... 
    remote: Enumerating objects: 57, done.
    remote: Counting objects: 100% (57/57), done.
    remote: Compressing objects: 100% (49/49), done.
    remote: Total 86457 (delta 10), reused 22 (delta 8), pack-reused 86400
    Receiving objects: 100% (86457/86457), 1.50 GiB | 4.73 MiB/s, done.
    Resolving deltas: 100% (59779/59779), done.
    $ cd deploy/ 
    $ git config core.sparsecheckout true
    $ echo www >> .git/info/sparse-checkout
    $ git checkout centos6
    error: invalid path 'configs/perl-modules/DIST.64/perl-HTML-Tree-1:5.03-1.el6.noarch.rpm'
    error: invalid path 'configs/perlbrew/perls/perl-5.24.1/man/man3/App::Cpan.3'
    ... (repeat for many files in the above path)

This was done with Git for Windows "git version 2.28.0.windows.1". I have also tried both types of line endings and using various version of .git/info/sparse-checkout such as:

    *
    !/configs/perl-modules/
    !/configs/perlbrew/perls/perl-5.24.1/man/man3

Checkout works fine on Linux, MacOS and WSL, only problem is that my IDEs don't work in those environments. Why isn't sparse-checkout working on Windows. Is there any other possibilities? Given the number of hits I have encountered searching with Google, maybe the there should be an enhancement to git or fir-for-windows to allow filepath mapping.

Further dialog is ocuring at https://stackoverflow.com/questions/63727594/github-git-checkout-returns-error-invalid-path-on-windows

[dscho]

This has come up before; the path names are checked when the corresponding files enter the Git index, even if the entries are not written out. See #2777 for more information (of which I think this here ticket is a duplicate).

[PhilipOakley]

issue is that the path contains a : which is illegal on Windows.

As an aside, I wondered if the clean and smudge style filters may offer a route to at least creating a valid local filename/path that could at least allow users to view the files. Maybe the file in the filesystem gets some sort of a symref/gitlink line to show it's 'real' path (and act as a pseudo flag for the smudge/clean filter). 'real' path being the repository path, rather than the local FS path.

The 'problem' at the moment is that Git retains the filename/pathname, which makes it a no-go, but it could be sold to some in the linux community as a mechanism for avoiding having Word.doc files pollute their pristine FS, being replaced by a panadoc variant 😉 .

just a thought. probably too many obstacles for the path mapping.

[wdtj]

@dscho was correct that#2777 was my issue and offers a bypass to the problem. At stated in the other issue: I've spent about 2 days trying to resolve the issue. I whole hardheartedly suggest that something be changed to prevent this in the future. Either a warning to set core.protectNTFS when sparse-checkout is specified or similar an enhancement to remap bad filepaths.

With the frequency of hit's I have seen suggesting core.sparse-checkout to fix the error, you would be well advised to accept this issue or reopen #2777 .

[dscho]

With the frequency of hit's I have seen suggesting core.sparse-checkout to fix the error, you would be well advised to accept this issue or reopen #2777 .

I will leave this ticket open all right, at the same time I encourage you to think about ways how you could bring the change about that you want to see.

To give you a couple of pointers:

• https://github.com/git/git/blob/v2.28.0/read-cache.c#L967 is the location of the call to is_valid_path().

• That is_valid_path() is actually a macro that points to is_valid_win32_path() on Windows: https://github.com/git/git/blob/v2.28.0/compat/mingw.h#L474

• The is_valid_win32_path() function skips any validation when core.protectNTFS is set to false: https://github.com/git/git/blob/v2.28.0/compat/mingw.c#L2602

• The verify_path() function is actually called in quite a few places: https://github.com/git/git/search?q=verify_path&unscoped_q=verify_path

• The index entries corresponding to the files that are excluded from the sparse checkout are marked with the CE_SKIP_WORKTREE bit: https://github.com/search?q=%22CE_SKIP_WORKTREE%22+repo%3Agit%2Fgit&type=Code&ref=advsearch&l=&l=

• If I read the code correctly, it is the apply_sparse_checkout() function (https://github.com/git/git/blob/v2.28.0/unpack-trees.c#L492-L554) that ultimately determines whether an index entry (data type struct cache_entry, for hysterical raisins) is to be skipped from the worktree or not.

Note: the index entries are created only if their corresponding paths pass the verify_path() test, and only quite a bit later do we actually know whether they have the CE_SKIP_WORKTEE bit set.

Also note: the verify_path() test is completely bypassed when reading an index file from disk. In other words, the test is only performed when a file is added to the index.

... is to figure out how to delay the call to is_valid_path() safely in the make_cache_entry()/make_transient_cache_entry() code paths (and only from those code paths, not from, say, apply.c's check_unsafe_path() code path!) to the time when the file is to be written (i.e. when a file is not marked with CE_SKIP_WORKTREE nor CE_NEW_SKIP_WORKTREE at all, or when it was marked with CE_SKIP_WORKTREE but now is no longer marked with that bit).

This, of course, has to be done with great care, as any regression in the safeguards would open users up for vulnerabilities that are no longer unknown (because they have been fixed as part of Git v2.24.1).

[kaovilai]

@dscho What about using WSL git directly?
https://docs.microsoft.com/en-us/windows/wsl/interop#run-linux-tools-from-a-windows-command-line
Like replacing use of git.exe [param] with wsl.exe git [param]
This would resolve the issue for my use case. I just need to learn a way to do it.
I asked a question on Superuser about this.

[kaovilai]

I can imagine Git-for-Windows becoming a proxy to git in WSL. It would resolve this issue. All interoperability between different file systems will then be delegated to WSL team. Maybe as an experimental flag.

[dscho]

What about using WSL git directly?

What about it? If it works for you, that's good. In general, it is not really a solution for this here ticket.

[ExE-Boss]

Git for Windows should probably use the U+F000 – U+F0FF range to escape the reserved characters, like what WSL does¹²³.

Footnotes

1. Enable filenames with colon (:) on drvfs microsoft/WSL#1514 ↩

2. When browsing a WSL folder in Explorer, there's no way to name a file/folder with some characters microsoft/WSL#5515 ↩

3. When browsing a WSL directory, some characters are not properly displayed microsoft/WSL#5516 ↩

[dscho]

Git for Windows should probably use the U+F000 – U+F0FF range to escape the reserved characters, like what WSL does​microsoft/WSL#1514​microsoft/WSL#5515​microsoft/WSL#5516.

And how would that help? Git would now understand those, but everything else, most notably the build system, would not.

Which means that Git would all of a sudden pretend that it succeeded, but the users would now be faced with a totally broken checkout that none of their tooling handles (if they used Cygwin for their tooling, they would most likely use Cygwin's git.exe, not Git for Windows').