npm 5.0.1 can't install a specific git dependency

[mistydemeo]

I'm opening this issue because:

• npm is crashing.
• npm is producing an incorrect install.
• npm is doing something I don't understand.
• Other (see below for feature requests):

What's going wrong?

When I try to install a package which uses a particular git repository at a specific commit, npm tries to git clone with the wrong branch and the clone fails. The specific spec is https://github.com/github/hubot-slack.git#87f012a8112d52261e1572e05c545b5ae59ababd (or its shorthand github/hubot-slack#87f012a8112d52261e1572e05c545b5ae59ababd). Instead of trying to clone at that commit, npm runs the following command:

/usr/local/bin/git clone --depth=1 -q -b 11/head https://github.com/github/hubot-slack.git

e.g., it's trying to clone branch 11/head. I'm not sure where it's getting that from. npm 3 does fetch this repository/reference correctly.

Sorry if this is reported in the wrong project. :) I stepped a module or two down the dependency tree but couldn't quite place what code is responsible for doing the clone and preparing its arguments.

How can the CLI team reproduce the problem?

The following package.json, with a single dependency, reproduces the problem:

{
  "name": "repro",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "hubot-slack": "https://github.com/github/hubot-slack.git#87f012a8112d52261e1572e05c545b5ae59ababd"
  }
}

The debug log is here: https://gist.github.com/283f0f4484d3a8dd7b0cf31acda81b40

supporting information:

• npm -v prints: 5.0.1
• node -v prints: v8.0.0
• npm config get registry prints: https://registry.npmjs.org/
• Windows, OS X/macOS, or Linux?: macOS 10.12
• Network issues:
  • Geographic location where npm was run: Vancouver, Canada
  • I use a proxy to connect to the npm registry.
  • I use a proxy to connect to the web.
  • I use a proxy when downloading Git repos.
  • I access the npm registry via a VPN
  • I don't use a proxy, but have limited or unreliable internet access.
• Container:
  • I develop using Vagrant on Windows.
  • I develop using Vagrant on OS X or Linux.
  • I develop / deploy using Docker.
  • I deploy to a PaaS (Triton, Heroku).

[zkat]

bah humbug. This one requires some explanation... this is all in pacote itself.

So! There's some cleverness that npm5 added in order to maximize how often it does shallow clones rather than full clones. I researched all the various restrictions on different kinds of clone to see what could be done, and one thing that stood out was that if we have an actual named ref, then we can (probably?) do a git clone of that ref after getting a remote rev list and seeing if the hash that the user requested is in that list.

More succintly, the algo is something like:

1. User requests user/repo#deadbeef.
2. pacote does an ls-remote on user/repo and collects all the refs
3. it looks for deadbeef in the list of refs, and picks the ref that refers to the hash, if it finds a match
4. If it finds a named ref, it will then try to do a shallow clone of the repo targeting that ref as a branch -- which means the clone only needs to fetch the one commit and its objects
5. if the clone succeeds, it packs it up in a tarball and the rest of the install continues.

I'm not sure off the top of my head why you're getting 11/head specifically, or why that doesn't work. I think there's two things that can both be done pacote-side to fix this without throwing away the shallow clone trick:

1. if a shallow clone fails, pacote should just fall back and retry the full clone without using the ref, and switch to the commit by hash before packing

2. we should figure out why 11/head in particular was picked as The Chosen Ref, and then why it didn't actually work.

Both of these changes would happen pacote-side, but I'll keep this issue open to track it. I think the bulk of the changes needed to do this could be done in pacote's git fetcher, with possibly some tweaks to pacote's git utility depending on the findings.

If anyone wants to pick this patch up, I'll help however I can and accept a PR that fixes this over there.

@mistydemeo thanks for this report! I'm really glad we found a corner case with my Cunning Plan™. I clearly did not think it all the way through 😂

[mistydemeo]

Looks like the output of git ls-remote is being used to get the ref:

$ git ls-remote https://github.com/github/hubot-slack.git
a23e19405f33da0de68c123f7dad2ef01fd572fd	HEAD
0f20def5919a7fede5ec5b250ac22c0568d80505	refs/heads/adapter-send-attachments
ed214feb5e64ae815761c08ba1bca68911edf7a2	refs/heads/always-use-chat-postMessage
ea562aeadab3f9f58e8db6ee5e86a4a41509db6a	refs/heads/better-formatting-removal
fa5085a506e80359b6429963988973731230c77b	refs/heads/bhuga-patch-1
9c670bc6349c4cb3fe5392d8d6bfa8b1454b4719	refs/heads/custom-response
e2e197c3c1b67f0929efa675127ac24c89a0a646	refs/heads/fallback-on-attachments
e37132d9975076ad61330e6f728a5fc08137bedc	refs/heads/last-message-timestamp
a23e19405f33da0de68c123f7dad2ef01fd572fd	refs/heads/master
4fba9145c414cfe37b0ea60ef020b4a3bfb23af6	refs/heads/no_attachment_archive_links
421c44ebec4de6f588490fa540267c9d9acfd316	refs/heads/not-an-attachment
9674b7875ff1601bde13b436c7ab47aa0ee60972	refs/heads/persist-more-slack-user-data
3d80813c6927f3b1ca841c1cd2a8e549e3e7982b	refs/heads/private-messages
ecf9db9fb3bb1b905fe83d0a86e9fc7165afd625	refs/heads/remember-less-slack-info
ad634a5b19ba454375c3b491a2048e2c7a45a637	refs/heads/restore_rawText
b2617e001ec0082e72ff27ff3779dcd90f3a9d2a	refs/heads/send-multiline-as-attachments
f6a7c715ab2273cb3495c5de3d659d8152fada24	refs/heads/travis-fix
8a8cd0904d76f17495f856c73dab5edd85765808	refs/heads/v2
df7d2f7fc4148de782da56a82e1a4b1229466ebc	refs/pull/1/head
cf658d8a68e04d5eb3287cfe402891881bbf31b9	refs/pull/10/head
87f012a8112d52261e1572e05c545b5ae59ababd	refs/pull/11/head
b1bbd94d9cadf61500c892d1b84e0748c6083b6a	refs/pull/3/head
e2e197c3c1b67f0929efa675127ac24c89a0a646	refs/pull/4/head
e2e197c3c1b67f0929efa675127ac24c89a0a646	refs/pull/5/head
fa5085a506e80359b6429963988973731230c77b	refs/pull/6/head
504fa95f091f216755f4d0a1faf2497b45686987	refs/pull/7/head
ee90948bf52c22ea98c52b996d96474ee6e18561	refs/pull/8/head
4fba9145c414cfe37b0ea60ef020b4a3bfb23af6	refs/pull/9/head
df3e9cc20fb54ae28125268add6cbdac48aec858	refs/pull/9/merge
bbc23aa87580f38c2093a24ec52669dcb7645a81	refs/tags/1.5.0
e70a4a258fa6108e4dd5086483f5d72373b15966	refs/tags/1.5.1
a0d0a43336676368d7d1bffdb9b5d815bda8aeb3	refs/tags/1.5.1^{}
c502bfb0429484209c31bce10771adcc6900b326	refs/tags/1.5.2
dc09cc183f535a5749c1f9afe19be9634ee3ed8a	refs/tags/1.5.2^{}
1c4ab632b72e4f58ee474678c3baf43801567bed	refs/tags/1.5.3
16741d0fc2e8fddee780629f03c875959e8aea10	refs/tags/1.5.3^{}
ed623d540b5b39659c487808798c24fe70d6c3da	refs/tags/2.0.0
2febcf550065e4a26d871fbe994f12baf9cbffe3	refs/tags/2.0.0^{}
926a87de8a5221dea977e062a77cb32a2a6a4deb	refs/tags/2.0.1
9ce5ddcd93b4571fae989a677329246766c07230	refs/tags/2.0.1^{}
e60e034f10f726f882b4b94fa0de6571094f140b	refs/tags/2.0.2
fca11022dd0ba3d09b0ff31719b9d2aae8704d8b	refs/tags/2.0.2^{}
e63a298234af5c164c600b248a655d86b5544da5	refs/tags/2.0.3
f985af87bd42b2c96c184b6f4c642db4ce1a9cf1	refs/tags/2.0.3^{}
cb035f781f93717115de4f2dbeb279c8c64c84b2	refs/tags/2.0.4
c07240c231326ba2b612b99ab38d4a224bec8646	refs/tags/2.0.4^{}
1f865e2381a8de7abf11d507c882b73bcc3c8b8a	refs/tags/2.1.0
a7d6b5c7cfd1ea6aa3b4b048a116a0796ba43310	refs/tags/2.1.0^{}
64d1566d91c6fc37ce52bc92179ebb67aaadc523	refs/tags/2.1.1
091b598152239ac9567129121be9ff617e7a8e14	refs/tags/2.1.1^{}
9a7cadd87cd7bfb90c685ccacec8f777a6840b97	refs/tags/2.1.2
335e694207d77fe16d7bc21079998645bd30804f	refs/tags/2.1.2^{}
db7837977db5d547bc608d6cb2506350db2446be	refs/tags/2.1.3
8a5d7dcc273e5c62ff929143e945b16f2b26894b	refs/tags/2.1.3^{}
010905c9512d34939845b3f056bd9cee9b0c6b6a	refs/tags/2.2.0
c90ba26076b9ac4321c9baf9d239cafe7945e223	refs/tags/2.2.0^{}
f43deba2b53a73c01aaf1a046dc49ccf12291261	refs/tags/3.0.0
dd265c8ab8895e24b136515635eec714602d5036	refs/tags/3.0.0^{}
a1bf28281b3aa7135fb4c0473efaa6e9787b488e	refs/tags/3.1.0
6afbc1ba1213006e75975c3bfeb0eb9c9d8993e8	refs/tags/3.1.0^{}
aeb96c85f705188b7f7288244fad1e899b8770dd	refs/tags/3.2.0
d8c23424f96ce59e3f2e89ee0daf204ddf4dc4fe	refs/tags/3.2.0^{}
ba9fffb9e572ff890a190bf28adeb0e196eeef1a	refs/tags/3.2.1
f3a288b44b7df5eb688c84a1843b9e10c0c6efeb	refs/tags/3.2.1^{}
a2890d12104cf45cbd9487f9a4c7c8ba47921fb1	refs/tags/3.3.0
af9001bc14c2af358aaec301272b2ec95cb8d61c	refs/tags/3.3.0^{}

This line is what's transforming refs/pull/11/head into 11/head. The chosen ref here is neither a tag nor a branch, and may not be clonable.

[zkat]

I bet the fix is just to change that command back to ls-remote -h -t *.

[mistydemeo]

Yeah, I think so:

$ git ls-remote -h -t https://github.com/github/hubot-slack.git '*'
0f20def5919a7fede5ec5b250ac22c0568d80505	refs/heads/adapter-send-attachments
ed214feb5e64ae815761c08ba1bca68911edf7a2	refs/heads/always-use-chat-postMessage
ea562aeadab3f9f58e8db6ee5e86a4a41509db6a	refs/heads/better-formatting-removal
fa5085a506e80359b6429963988973731230c77b	refs/heads/bhuga-patch-1
9c670bc6349c4cb3fe5392d8d6bfa8b1454b4719	refs/heads/custom-response
e2e197c3c1b67f0929efa675127ac24c89a0a646	refs/heads/fallback-on-attachments
e37132d9975076ad61330e6f728a5fc08137bedc	refs/heads/last-message-timestamp
a23e19405f33da0de68c123f7dad2ef01fd572fd	refs/heads/master
4fba9145c414cfe37b0ea60ef020b4a3bfb23af6	refs/heads/no_attachment_archive_links
421c44ebec4de6f588490fa540267c9d9acfd316	refs/heads/not-an-attachment
9674b7875ff1601bde13b436c7ab47aa0ee60972	refs/heads/persist-more-slack-user-data
3d80813c6927f3b1ca841c1cd2a8e549e3e7982b	refs/heads/private-messages
ecf9db9fb3bb1b905fe83d0a86e9fc7165afd625	refs/heads/remember-less-slack-info
ad634a5b19ba454375c3b491a2048e2c7a45a637	refs/heads/restore_rawText
b2617e001ec0082e72ff27ff3779dcd90f3a9d2a	refs/heads/send-multiline-as-attachments
f6a7c715ab2273cb3495c5de3d659d8152fada24	refs/heads/travis-fix
8a8cd0904d76f17495f856c73dab5edd85765808	refs/heads/v2
bbc23aa87580f38c2093a24ec52669dcb7645a81	refs/tags/1.5.0
e70a4a258fa6108e4dd5086483f5d72373b15966	refs/tags/1.5.1
a0d0a43336676368d7d1bffdb9b5d815bda8aeb3	refs/tags/1.5.1^{}
c502bfb0429484209c31bce10771adcc6900b326	refs/tags/1.5.2
dc09cc183f535a5749c1f9afe19be9634ee3ed8a	refs/tags/1.5.2^{}
1c4ab632b72e4f58ee474678c3baf43801567bed	refs/tags/1.5.3
16741d0fc2e8fddee780629f03c875959e8aea10	refs/tags/1.5.3^{}
ed623d540b5b39659c487808798c24fe70d6c3da	refs/tags/2.0.0
2febcf550065e4a26d871fbe994f12baf9cbffe3	refs/tags/2.0.0^{}
926a87de8a5221dea977e062a77cb32a2a6a4deb	refs/tags/2.0.1
9ce5ddcd93b4571fae989a677329246766c07230	refs/tags/2.0.1^{}
e60e034f10f726f882b4b94fa0de6571094f140b	refs/tags/2.0.2
fca11022dd0ba3d09b0ff31719b9d2aae8704d8b	refs/tags/2.0.2^{}
e63a298234af5c164c600b248a655d86b5544da5	refs/tags/2.0.3
f985af87bd42b2c96c184b6f4c642db4ce1a9cf1	refs/tags/2.0.3^{}
cb035f781f93717115de4f2dbeb279c8c64c84b2	refs/tags/2.0.4
c07240c231326ba2b612b99ab38d4a224bec8646	refs/tags/2.0.4^{}
1f865e2381a8de7abf11d507c882b73bcc3c8b8a	refs/tags/2.1.0
a7d6b5c7cfd1ea6aa3b4b048a116a0796ba43310	refs/tags/2.1.0^{}
64d1566d91c6fc37ce52bc92179ebb67aaadc523	refs/tags/2.1.1
091b598152239ac9567129121be9ff617e7a8e14	refs/tags/2.1.1^{}
9a7cadd87cd7bfb90c685ccacec8f777a6840b97	refs/tags/2.1.2
335e694207d77fe16d7bc21079998645bd30804f	refs/tags/2.1.2^{}
db7837977db5d547bc608d6cb2506350db2446be	refs/tags/2.1.3
8a5d7dcc273e5c62ff929143e945b16f2b26894b	refs/tags/2.1.3^{}
010905c9512d34939845b3f056bd9cee9b0c6b6a	refs/tags/2.2.0
c90ba26076b9ac4321c9baf9d239cafe7945e223	refs/tags/2.2.0^{}
f43deba2b53a73c01aaf1a046dc49ccf12291261	refs/tags/3.0.0
dd265c8ab8895e24b136515635eec714602d5036	refs/tags/3.0.0^{}
a1bf28281b3aa7135fb4c0473efaa6e9787b488e	refs/tags/3.1.0
6afbc1ba1213006e75975c3bfeb0eb9c9d8993e8	refs/tags/3.1.0^{}
aeb96c85f705188b7f7288244fad1e899b8770dd	refs/tags/3.2.0
d8c23424f96ce59e3f2e89ee0daf204ddf4dc4fe	refs/tags/3.2.0^{}
ba9fffb9e572ff890a190bf28adeb0e196eeef1a	refs/tags/3.2.1
f3a288b44b7df5eb688c84a1843b9e10c0c6efeb	refs/tags/3.2.1^{}
a2890d12104cf45cbd9487f9a4c7c8ba47921fb1	refs/tags/3.3.0
af9001bc14c2af358aaec301272b2ec95cb8d61c	refs/tags/3.3.0^{}

[zkat]

@mistydemeo would you like to do a PR against pacote, or should I? I definitely lean more towards "MOAR CONTRIBZ" if I can <3

[mistydemeo]

I'll do a PR!

[ljharb]

Please note that github PR refs are actually quite cloneable (origin/pr/123) and it'll be important to retain the ability to npm install a pull request by ref :-)