That is such a bad response to this.
The problem isn't that "well-meaning members of the community" decided to upload packages. The problem is that when their system decides that a package shouldn't be up it completely removes the package, as if it never existed, and allows the namespace to be reused immediately. Those "well-meaning members" should not even be able to hijack packages this way, as it means the people who aren't "well-meaning" can also do it.
What should happen is that they block downloads of the package while they investigate. That way people who attempt to download the packages get a meaningful error and people are unable to hijack the package name.
Hasn't this been an ongoing issue with npm since pretty much its inception? I remember reading articles about this vulnerability and the hijacking of packages that were taken down temporarily years ago.
How has this not been dealt with systematically yet?
Afterwords NPM came out with a blog post[2] where they went out of their way to take as little blame as possible and basically said it was the developers fault. They said they "stand by [their] package name dispute resolution policy, and the decision to which it led"- basically ensuring that no developer should ever trust their repository in the long term, as they'll happily hand over any package name to a corporate entity if that entity asks for it.
The weird thing is that they claimed to make it impossible to "unpublish" packages, so that developers could no longer rage quit their site, but apparently they didn't extend that new requirement to their own "security" systems.
[1] http://azer.bike/journal/i-ve-just-liberated-my-modules/ [2] http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm
I think it's shocking how NPM tries to subtly shift fault to "well-meaning members of the npm community" for costing NPM "additional steps and time". Shame on npm. This shouldn't have even been possible after the azer/kik issue a few years back.
I used to really care about trying to harden the Node ecosystem, and last year it was one of my main goals. I tried to send multiple vulnerability reports, do mass static analysis of npm packages, and wanted to contribute more to the ecosystem, but the consistent ambivalent reactions of much of the community that I talked to turned me off of the project entirely. If npm wants to continue to be a security dumpster fire, let it burn. Node is a waste of security researchers' time and an honest goldmine for black hats looking to compromise relatively powerful novice webdev hardware.
I don't see it changing anytime soon. npm is a business that isn't focused on security. These things keep coming up, and yet npm install metrics I'm sure aren't decreasing. Until they face meaningful competition and/or the rest of the Node community begins to give even half a care to security outside of this forum, there will be no incentive for anyone to do anything about it. It's easier to play PR, give a little lip service to it and dodge the problem than it is to add any friction to their potential growth.
For example, people often insist in the Python world that PyPI should support package signing. But it already does -- you can generate a signature for a package and upload the signature with the package. Django does this, and has been doing it for years. You can also get package download/install tools that will check the signature. But then what?
What people really mean when they say there should be "signed packages" is that there should be a whole bunch of invisible infrastructure (set up by... who, exactly? Maintained by... who, exactly?) to decide which PGP keys are authorized to sign releases of which packages. And that's close to an intractable problem for an anyone-can-contribute community repository like npm or PyPI.
I think the community should fork npm repository. Anything of value is free and open source anyways. Why does node continue to support npm people?
Now, it is easy enough to lob an aphorism but an event like this means that lessons previously learnt have been ignored (for whatever reason).
If you are going to build something that replicates or advances on previous systems the least you can do is study those systems to see what they did right and what they did wrong. In my opinion the Javascript ecosystem has produced some wonderful new systems that do things that have never been done before but in doing so they have wilfully ignored the past and didn't learn some basic lessons. In doing so they muddy the waters and in the end bring down the reputation of the whole ecosystem. I guess then it is not a huge surprise that a sizeable number of people feel that an unusually large number, as a percentage, of Javascript projects come and go and keep reinventing the wheel.
> In yesterday’s case, we got it wrong, which prevented a publisher’s legitimate code from being distributed to developers whose projects depend on it. We identified the error within five minutes and followed defined processes to reverse this block. Unfortunately, the process was complicated by well-meaning members of the npm community who believed that a malicious actor or security breach was to blame and independently attempted to publish their own replacements for these packages.
No. Assuming everything in that excerpt is true (and I happen to know it's not, but that's not even relevant here), that wasn't the problem.
The problem is that NPM allowed packages to be re-uploaded by new authors after the initial versions had been spam filtered. Especially since allowing packages to be re-uploaded by new authors was the core issue of the left-pad debacle, and the one thing NPM said they'd fixed in response.
Let's summarise here:
1. NPM has a big issue
2. They claimed they had fixed it
3. They had not
4. In their post mortem they're pretending the issue doesn't exist
This guts any remaining trust I had in npm. Even if I wanted to trust them, they're not even admitting the problem exists; how am I meant to believe they're finally going to fix it? They've stopped even promising to fix this, and moved on to lies and denial.
Unacceptable. Literally. This is pushing me away from the node ecosystem because I am not prepared to accept this sort of weaponized incompetence from the primary package repo for node.
The cold reality I'm realizing is: what are you going to do about it? If you have tens of thousands of lines of javascript written using dependencies from NPM, what is the next step?
Of course, NPM's response to the kik/left-pad problem was also pretty awful. Make it so users can't delete packages. Cool. For those of you using NPM's private offerings, this also applies to you, so hope you don't care about cleaning up your private NPM registry content.
Getting pretty tired of this. Their open source operation seems to suffer from poor handling of community and technical issues, at least from a high level. Their private registry operation is very lean on features, and also suffers from very confusing limitations. I'm surprised at how long it took there to be read-only API keys. Until last year, you literally had to give your CI keys that could publish to your organization if you wanted access to private packages. And you had to pay for an extra user for the privilege of doing it wrong. It's fixed now but it still blows my mind that it took so long. Aside from faster installation, it's actually a lot better to just use private Git URLs instead of NPM's private offerings.
NPM will never properly take the fall. They will take just enough blame to seem responsible and then shift the majority of the poor decision making onto other actors in the community in their explanations. Third time in recent memory.
Maybe not in the incident itself, but the sheer fact that many of the packages were replaced by other people constitutes a jeopardization of applications that depend on NPM. The only reason why some big package didn't get replaced with code that exfiltrated data from production or subtly backdoored it is sheer luck.
https://news.ycombinator.com/item?id=16087079
this is probably a good indication of how a much more serious security event would be treated by the organization.. just sayin
npm is full of fundamental bugs in their software, workflows, architecture, and community. We learnt this in March 2016 when one developers got a copyright claim and had to remove their package. [2] Why is it even possible to remove a package? Why is it possible to create new package with the same name? Almost no software published there have unit tests, asking developers how they tested it in a GitHub issue leaves the issue open with no response. Everything is centralized on GitHub, which was proven to fail many times when GitHub was down in a region or globally due to a DDoS. Why is Node allowed to exist? We're not in 1998 anymore.
[1] https://news.ycombinator.com/item?id=16092584 (comment above)
[2] https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/
The earlier problem you mentioned didn't seem to be about copyright. But if it had been, the answer to why it's even possible to remove a package is: because if you don't and you knowingly continue to host copyright-infringing content you can expect a very large penalty in court.
I believe this was one of the cases of a community member stepping up to publish a package quickly so that malicious code wasn't published.
Presumably they make the claim that none of the code was malicious because they checked.
> the integrity of these 106 packages were never jeopardized.
are we operating with different definitions of jeopardy here? 106 packages were absolutely at risk of harm during this window. The fact that some community members stepped up is irrelevant, a bad actor could have done a lot of damage here. I think this blog post is completely disingenuous, and doesn't make me trust npm.
1) A system that detects "spam" and then allows for the complete removal of packages as if they never existed, allowing anyone to replace them, should never be described in the neutral terms used in this post. This system appears to be an existential threat to the company and project, and one of its largest mistakes. It won't take a long investigation to figure that out; it should be obvious today.
2) The claim that the security and integrity of these hijacked npm package names was not jeopardized appears to be 100% pants-on-fire false. If it is not false, I think npm's users are owed an explanation today of why it's false, rather than a bare assertion.
I know it's hard to be in the hot seat. No animosity to any of the humans involved.
So they learnt zero from it.
Then, WHEN ARE THEY GOING TO SIGN THOSE PACKAGES? Zero, nothing. They didn't learn from mistakes, they don't listen. NPM is still open to all sorts of malicious use.
Besides NPM problems, yesterday many packages won't work because package "pinkie-promise" wasn't available. This is the full effective source code of pinkie-promise:
module.exports = typeof Promise === 'function' ? Promise : require('pinkie');
This was a security breach. Their anti-spam system should block repos by freezing the module name and returning blank files, not by deleting the entire module and subsequently allowing anyone to upload new modules. This is leftpad all over again.
Update: Woah, so I was checking out my old NPM namespaces and apparently someone took control over https://www.npmjs.com/package/filesaver.js
EDIT: it may be worth debating why npm (a binary produced by npm, Inc) is shipped with node.js at all, but that's upto the node foundation
Anyone know why they did it that way instead of giving each developer a subspace of the namespace that only that developer can publish under?
I'm thinking of the way it works on Java, where my package would be net.tzs.order_pizza, or GitHub (and numerous others) where it would be tzs/order_pizza.
Nobody would've cared it the package was "tzs/kik". For example, the Elm package system does this right.
Another example of this failure is the Reddit subreddit system where a subreddit like /r/bitcoin makes it seem like the authority when it's a carefully moderated and biased subreddit. And nobody can really compete with it without getting a worse subreddit name like truebitcoin and bitcoin2 or something. Would be much more clear if it was /r/@theymos/bitcoin and /r/@spez/politics.
Root causes wise, my impression is that software language engineering has a knowledge integration and communication problem. Lambda the Ultimate doesn't have a wiki. :) Academic papers provide a foundation, but a lot of practical professional knowledge is scattered, siloed in language communities, and time-consuming to access. Gathering best practices for language design and implementation isn't something we do well. So language creation projects often look like team and/or community "learn by doing" education exercises. Even with good people at leading companies... I had a conversation at redacted once, "why did you make (disappointing) design decision X?", "mumble mumble", "<facepalm> that's a (widespread) misconception, a misinterpretation of paper P (which admittedly could have been clearer), that's been kicking around for several decades now".
My fuzzy impression is the npm repo was not originally a mission focus, but rather a "don't be python" add on goal. Absent organized best practices, low-priority design decisions are often not made well. Perhaps Ruby, Perl5, and/or Python were copied, without attending to those communities experiences, and the implications for an even more fine-grained code sharing community.
Relevant excerpt:
> I proposed required namespaces (@user/package) as a requirement in 2013 and the npm developers acted like I had 2 heads.
Namespaced packages/repos have their own problems (figuring out which is the "good" one after some time and a few forks happen)
Most if not all of these packages are hosted somewhere like Github. It's not hard to tell which repo has the most stars, fewest issues or most recent commit, that work's already done for you.
But, we're talking about Node packages here - many of which seem to be trivial, or single line functions. How many forks of left-pad could one expect to find?
Nope.
Would you expect Google's security team to provide a detailed list of all of the security procedures in place to prevent access to their network?
Rejecting "security through obscurity" just means you shouldn't rely on obscurity alone. Obscurity + good security > good security alone, since it increases the threshold of time and ability that any attacker would need to bring to an attack.
After left-pad I think we'd all like to see a strong, well documented methodology to keep things like this from happening again. The broad strokes can't be any more clever than what the rest of us would expect, so why not at least provide a basic idea of what you're doing and then we could trust the system a bit more.
Neither of those are security systems. That's like asking for their password and claiming you've proven security through obscurity.
The idea behind dismissing obscurity is that if everything but private tokens were exposed, your system should still remain secure.
If my hat is black, I'm writing a daemon which monitors relevant npm modules and uploads subtlely modified versions if and when the possibility to do so occurs again. Particularly, but not only, targeting crypto currencies.
I started sketching out one yesterday, and then thought "What am I going to do with this? Probably just get in trouble" so I stopped.
Additionally, there's the problem of npm's fine-grained dependencies (that have become a laughing stock in itself), in my Java large projects, while I don't know every single dependency, I know all the "vendors", i.e. trusted upstream namespaces I use, and their transitive dependencies. There's the occasional non-Eclipse, non-Google, non-Apache dependency (say, XStream), but I am aware of it and fix it to a specific version. npm with its thousands of dependencies by hundreds of pseudonymous developers for simple projects is an abomination.
For typosquatting: I don't think that problem can be easily solved by package registries. They could warn you if there is a package with a similar name (and more votes, downloads, what ever) but aside from that, the developer should double check the dependency names and that's it.
Even if a package is removed as malware, a user should never unexpectedly download the work of a totally new author that they haven't vetted.
Some of the protections should be done on the user side, but that's no reason for NPM to have dangerous policies.
Oversight != 'dangerous policies'. Seems like they thought they had this fixed.
Can't understand why it hasn't happened still.
https://github.com/npm/npm/pull/4016 - I read this as "it´s hard, and we don´t care." took them forever to look at and reject a patch that would have gotten the ball rolling
Who needs to sign the packages? The developers of the packages, right? Otherwise developer credential compromise subverts the entire point.
Which keys are authorized to sign for which packages? How to prevent credential compromise from affecting those authorizations?
What's the difference between a signed legitimate package and a signed malicious package?
If they introduced package signing, would packages would adopt it fast enough for users of packages to only use signed packages?
What happens when signing keys are lost or compromised? Do we need to use countersignatures from timestamping services as with other forms of code signing, so that CI systems do not break if a key is pulled?
I think this is very much not straightforward. npm+pgp may be well intentioned but seems grossly inadequate.
Literally the second sentence is:
> Early this coming week, we will share a full analysis and technical explanation of the incident.
I'm looking forward to reading the final post-mortem, but in the mean time I really appreciate their update.
EDIT: or better yet, don’t allow people to download the update, just keep users on the old, ostensibly safe version until everything gets sorted?
It absolutely should, just like any dependency on any other third party code or servers. Especially when they regularly have incidents like this.
> Simple fact is you should not be relying on ANY package registry at the time of deployment.
This is true, but doesn't prove the previous claim.
Automated spam filters help to avoid dodgy packages. Spam filtering operates on heuristics so it’s sensible to not publicise how they work.
The automated spam filter kills dodgy uploads, as these mostly happen on previously unused names a decision is made to not have the spam packages’ name remain taken. Among other things this stops the spammers from leaving all the good names blocked.
The spam code gets a false positive and the above logic kicks in, leaving previously used names now available. This is noticed and corrected within a few hours.
The npm team will likely improve their spam filtering heuristics and also ensure that formerly good packages that get spam flagged do not release their names - they have indicated roughly this on Twitter.
This only just happened, and it’s the weekend, so I’d expect a full write up will be released during the week when they’ve had time to do a post-mortem and work through the salient points.
Yes, but you need to keep in mind:
1. This stuff is hideously important because getting it wrong is a incredibly serious security hole. Letting people publish a package with a name that other people are already requiring is a defcon 1 emergency.
2. They already had a major issue with this and said they had fixed it. You get, at most, one time when you can say "whups, we didn't realise how big an issue this was!".
3. It's their job to get it right. This is, literally, npm inc. It's not some hobbyists, or a service being provided by people in their spare time.
So when you say:
> The automated spam filter kills dodgy uploads, as these mostly happen on previously unused names a decision is made to not have the spam packages’ name remain taken.
I have to disagree. They needed to check how many times the flagged package had been downloaded, and after the left-pad debacle, they knew (or should have known) that.
They're not taking this seriously.
from what it appears, the reason someone else was allowed to register the same package name was because their database didn't pull in the entries for that package. an error? yes. an issue? yes. but it is not a repeat of the same incidence of left-pad.
i'm not going to pass judgement until i read the full report. i think at the very least, whatever mechanism they were using to copy/backup the database seems to have an issue with deleting entries and needs to be addressed
I am a little shocked at the brevity of the procedure. I am guessing by the blog post pointing out the brevity that I am meant to be shocked, but if I turned out that 310 word procedure for downtime I'd be pretty confident it was not good enough. I'm not confident that I would be able to read that document and understand what to do if I was the one on pager duty. Maybe it would be different if I worked there and perhaps the procedures are more granular and each of the stages well explained in other procedures. Maybe.
I'm finding it hard to understand what npm does that is so different to other languages' package managers that it can have so many problems and such bad public relations. Linux distros have repositories that handle packages with far greater sizes and quite a few are purely voluntary efforts. They also manage to include package signing as well. I've known of a few hiccups, but none that been handled quite so bad as to completely undermine my trust in the tool altogether.
Is there any compelling technical reason why we couldn't have a tool that works more like go get for node? For those that haven't used it, you just point it at a git repository.
[0] http://ceejbot.tumblr.com/post/169045932039/how-to-handle-an...
[0] https://github.com/svanderburg/node2nix
[1] https://nixos.org/nix/about.html
[2] https://github.com/svanderburg/node2nix/blob/22d0c19575ecd19...
Edit: fix link formatting
Centralized trusted authorities cannot be trusted if they are private companies who don't answer to you. If you aren't paying them and you don't have a management position in said company, things like this will happen.
Does this have any real impact on the NPM package adoption though? There is no alternative node module package directory, is there?
It seems like any action they could have taken would have had exactly the same effect (well, maybe some prospective products don't adopt JS backends, but you have all webpack projects locked in pretty tight, right?)
Very little about how I use npm requires it to be centralized; I don't ever search npm looking for a package to install; I find a package I want and then look to install it.
The one issue I've seen over the years is it seems npm Inc does not officially sanction on site npm repos (presumably to help monetize their private hosted repos), and finding info on how to set up a private repo is lacking.
That way the official clients would support it but you could easily ignore it if you wanted to.
Isn't that the opposite of good practice? You shouldn't rely on obscurity. It's better to have the security processes out in the open so that it can be audited and flaws pointed out.
There's a lot of questions open that are non-specific to that though, so I'd agree that they could be sharing more.
> We don’t discuss all of our security processes and technologies [with each other internally, so dumb stuff keeps happening]