The thing is that these sites (as with many things) grow from a core of early tastemakers. In some circles these people know the hottest bands, the best new books, wear the coolest clothes, or know about the new restaurants in town before everyone else. In tech they know the best sites, and where they go the rest will follow. "People reading HN" might be a small minority in the grand scheme of things, but I would be willing to bet that they collectively have a huge influence on the web at large. They were the natural drivers of Google+, and they didn't bite. Meanwhile things have moved on from Facebook to Twitter, Instagram, Snapchat, etc leaving Google+ in the dust. Do not ignore vocal and influential groups of users, even if they are small in number.
This is what happened as far as I could tell: the techie world (friends of Googlers) joined, then crashed head first into the Real Name Policy and told their friends to stay the hell away. It never recovered.