Because this isn't true. Just because you're experience this effect (which blows), doesn't mean the tool and related tooling are somehow broken. These Regex issues should be fixed, libraries should update to safe versions, things should advance and any incentive we have we should use to make this happen.
Might be nice to be able to tag libraries that should be ignored in audits. Perhaps there is such a function, not really a NPM expert. But if your projects has too many of these "high severity" problems, you probably stop doing them.
Still, I think the availability of such audits from the package manager is quite neat. As an embedded dev I think these are certainly luxury problems.
I've been in the node scene since 0.10. That's around 10 years. My packages have billions of downloads annually. My viewpoint here carries the weight of hours of debug time and frustration and confused users of my code, as well as meeting and knowing the npm staff at the time quite personally, and knowing under which circumstances package lock files were implemented.
They are broken.
> These Regex issues should be fixed
They do, pretty much immediately after they're reported.
> libraries should update to safe versions
I check all the version ranges of dependent libraries when I push a patch with vuln fixes. They get pulled just fine without needing to update every single package. This is what version ranges are for.
> things should advance
Yes but this is nebulous and vague and aside from the point.
> and any incentive we have we should use to make this happen.
I don't see where the disagreement is. This is exactly what happens all the time, nothing is the problem here. I don't get your point.
---
Package lock files were designed in a few short days and pushed out prematurely without much review by a single Npm employee (at the time) since they promised it for the v5 release. They were on a time crunch because they were trying to keep with Node.js's next major release timeline, which operates independently of npm's (at least, that's how it was conveyed to me).
So this change got pushed out, had an absolute mountain of bugs that took ages to fix (e.g. at one point adding a new dependency would delete your entire node_modules folder), and promised added security when in reality they do nothing of the sort.
Instead, they cause subtle caching-related bugs, they add an artifact to source control (which is always code smell in my book), crap up diffs/PRs, cause headaches across platforms, and do very little to help... anything, really.
They're super, super broken by design. Yet npm tells you you need them ("please commit this to your repository") and refuses to do basic security things without them (npm audit).
So why were they added? IIRC it was because the version resolution was a massive strain on npm's servers, so lockfiles removed the need to fetch tons of version information each time you added another dependency.
Oh, and don't even begin to whine about them on Twitter (at the time), lest you be yelled at by the implementor for being ignorant or something.
It was a shit show. They add absolutely nothing to the industry.
Unless we're not talking about the same reproducibility property. Also I don't know what "hermetic" means in this context but I doubt it's anything that npm solves correctly.
So when you say "npm staff at the time", do you mean at the time of node 0.10?
> and knowing under which circumstances package lock files were implemented.
> Package lock files were designed in a few short days and pushed out prematurely without much review by a single Npm employee (at the time)
The amusing thing about your comment here is the parts which are accidentally correct.
`package-lock.json` files use the same file format as `npm-shrinkwrap.json` files. Always have, although of course the format of this file has changed significantly over the years, most dramatically with npm 7.
The "design" of the shrinkwrap/package-lock file was done rather quickly, since it was initially just a JSON dump of (most of) the data structure that npm was already using for dependency tree building. However, as far as I know, the days were the standard length of 24 hours, so while that may be "short", certainly shorter than I'd often prefer, they were (as far as I know) no shorter than any other days.
This was indeed shipped without any review by even a single "npm employee", which should not surprising, as "npm" was not at that time a legal entity capable of hiring employees. The initial work was done by Dave Pacheco, and reviewed by npm's author (at that time its sole committer and entire development staff), both of whom were Joyent employees at the time.
The use of a shrinkwrap as a non-published normal-use way to snapshot the tree at build time and produce reproducible builds across machines and time was not implemented by default until npm v5, but there wasn't really much to rush, on that particular feature. You could argue that npm 5 itself was rushed, and that's probably a fair claim, since there was some urgency to ship it along with node version 8, so as not to wait a year or more to go out with node v10.
> So this change got pushed out, had an absolute mountain of bugs that took ages to fix...
Idk, I think calling it a "mountain" is relative, actually ;)
> They're super, super broken by design.
I know you're using this phrase "broken by design" in the same sense as the author of the OP means it, but... has language just changed on me here, and I didn't notice?
As I've always heard the term used, something is "broken by design" when the actual intent is for a system to fail in some way, to achieve some goal. For example, a legislative or administrative process that is intentionally slow-moving and unable to accomplish its goals in a reasonable time frame, with the hope that this leaves room for independent innovation. Or a product that requires some minor upgrade or repair to continue working, so that the seller can keep tabs on their customers more easily. That kind of thing.
I think what you mean is not that it's "broken by design", but rather it's "a broken design". Unless this is like "begging the question", and I should just accept that I'm gradually coming to speak a language of the past, while the future moves on. It's certainly not intended to cause problems, as far as I'm aware.
If you really do mean "broken by design" (in the sense of a tail light that goes out after 50k miles so that you will visit the dealership and they can sell you more stuff), I'm super curious what you think npm gets out of it.
> Yet npm tells you you need them ("please commit this to your repository") and refuses to do basic security things without them (npm audit).
As of npm v7, there's no longer any practical reason why it can only audit the lockfile, rather than the actual tree on disk. Just haven't gotten around to implementing that functionality. If you want it changed, I suggest posting an issue https://github.com/npm/cli/issues. There's some question as to whether to prioritize the virtual tree or the actual tree, since prioritizing the actual tree would be a breaking change, but no reason why it can't fall back to that if there's no lockfile present.
But even approaching build reproducibility is impossible without lockfiles. If a new version of a transitive dependency is published between my install and yours, we'll get different package trees. If we both install from the same lockfile, we'll get the same package tree. (Not necessarily the same bytes on disk, since install scripts can change things, but at least we'll fetch the same bytes, or the build will fail.)
> So why were they added? IIRC it was because the version resolution was a massive strain on npm's servers, so lockfiles removed the need to fetch tons of version information each time you added another dependency.
You do not recall correctly, sorry. (Or maybe you correctly recall an incorrect explanation?) The answer is reproducible builds. Using a lockfile does reduce network utilization in builds, but not very significantly.
> Oh, and don't even begin to whine about them on Twitter (at the time), lest you be yelled at by the implementor for being ignorant or something.
I hope my tone is civil and playful enough in this message to not consider my response "yelling".
