[BUG] Ineffective caching of git dependencies

[kristoffer-zliide]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

After upgrading from Node 14 to Node 16.16, we're seeing an extreme increase in npm install times, in one case going from taking seconds to taking 17 minutes! We have a handful of git dependencies in those projects, built with TypeScript in their prepare script. I believe what made this process fast, and ultimately extremely useful, in Node 14 as that the packages were cached after the prepare step, which seems, in some cases, not to be the case after upgrading. In other cases, the cache is populated but seems to not be used, and sometime an update is not even properly performed.

Expected Behavior

npm install should be correct and fast by caching prepared git packages.

Steps To Reproduce

I have created a minimal project with a 30 second prepare script here and here, and a project that depends on it and can write the version of the dependency to the console here. The dependency there was was created with npm install git+ssh://git@github.com/kristoffer-zliide/npm-git-dep-test.git#9e32a70222842ce2033fb876d4af8182f369f22b with the output of "changed 1 package, and audited 2 packages in 3m", so even though the prepare step takes 30 seconds, installing the package takes more than four times that. Interestingly, running npm install without arguments changes git+ssh: to github:, but seemingly does nothing else, which is possibly also a bug. The project prints out "Version 1" as expected. Now

1. Change the git hash from 9e32a70222842ce2033fb876d4af8182f369f22b (Version 1) to 42146c03ce8b0d80de39d099baee5a2f1c979f89 (Version 2) in package.json
2. Run npm install - again this takes more than four times the expected 30 seconds
3. node index.js confirms that we're on Version 2
4. Change the git hash from 42146c03ce8b0d80de39d099baee5a2f1c979f89 back to 9e32a70222842ce2033fb876d4af8182f369f22b in package.json
5. Run npm install - this takes more than 2 minutes and it used to be almost instant on Node 14.

In the above scenario, an integrity property is added to the package-lock.json file with value sha512-iWdHaexWcOA0KPZtqFHI89eCkA34AB3wQCBJ8sP4Umcxqt0jWzL1N3o0KPB2vZD6+AqW03q/hn1i31Y3hmgvHQ==, and the prepared package is indeed to be found in npm-cache\_cacache\content-v2\sha512\89\67\ (confirmed with tar -xf 476...).

Clearly, there's some special handling of github URLs at play, so also (without the steps above) try

1. Uninstalling the package with npm uninstall npm-git-dep-test
2. Install the package from the other repo (git.zliide.com is a CNAME for bitbucket.org. The repo is public, but you need your ssh key to be added to a bitbucket account): npm install git+ssh://git@git.zliide.com/kristoffer_zliide/npm-git-dep-test.git#9e32a70222842ce2033fb876d4af8182f369f22b. This takes about one minute, now "only" twice the 30 seconds of the prepare script. Note that there's no integrity property, and that node index.js prints out Version 1
3. Change the git hash from 9e32a70222842ce2033fb876d4af8182f369f22b to 42146c03ce8b0d80de39d099baee5a2f1c979f89 in package.json
4. Run npm install - this takes no time, but also does nothing!!! node index.js still prints out Version 1. Is this a critical bug, or am I assuming too much of npm install?
5. Run npm update - this fixes it, again in about 1 minute; node index.js prints out Version 2. Still no integrity property. Instead of editing package.json and running npm install, doing a npm install git+ssh://git@git.zliide.com/kristoffer_zliide/npm-git-dep-test.git#42146c03ce8b0d80de39d099baee5a2f1c979f89, gets you here as well, without that scary Step 4.
6. Run npm install git+ssh://git@git.zliide.com/kristoffer_zliide/npm-git-dep-test.git#9e32a70222842ce2033fb876d4af8182f369f22b to downgrade to Version 1. This again takes about a minute and used to be almost instant on Node 14.

Environment

• npm: 8.11.0
• Node.js: 16.16.0
• OS Name: Windows 11
• npm config:

; "builtin" config from C:\Program Files\nodejs\node_modules\npm\npmrc

prefix = "C:\\Users\\...\\AppData\\Roaming\\npm"

; node bin location = C:\Program Files\nodejs\node.exe
; node version = v16.16.0
; npm local prefix = C:\Users\...\Source\temp\npm-git-test
; npm version = 8.11.0
; cwd = C:\Users\...\Source\temp\npm-git-test
; HOME = C:\Users\...

[nlf]

the caching of git dependencies is something we know we need to address, you're definitely right it's ineffective as it is now

[kristoffer-zliide]

I guess that's one way of putting it. Another would be that it's unusable and dangerously broken; TLDR:

1. if we add a fairly benign git dependency the npm i command takes more than 17 minutes to run,
2. if we change the version of a git dependency in package.json, and run npm i runs without any errors, but has not updated the dependency

This is not an issue with Node 14, but is blocking us from upgrading to Node 16.

@lukekarrys: sorry to drag you into this, but do you think this could have anything to do with it?

[olegade]

Any updates on this issue?

[Mxde59]

Hi team,

This has been bothering our dev team for over a year now ...
Making a new git branch to overcome a cache problem is too time-consuming !
Does anyone have the skills to resolve this?

Thanks.

[olegade]

Hi.
This issue is really a blocker for our team. Is anyone able to look into it?

Thanks

[AndreVirtimo]

This is still an issue and is slowing down our builds by minutes :-(