Nodejitsu can handle hacker news traffic without even the slightest blink, with one dyno - I had a post up on the homepage for hours with over 200 concurrent visitors and like 13,000 total for the day and never had a single issue. Again, $3/month for that.
No need to over-complicate things : )
It's an overkill all you need is static content.
My blog is a Django app running on a 512MB Rackspace VM with a basic Apache, mod_wsgi, and Postgres setup. I have a dozen low traffic applications all running on the same server alongside it. I have zero caching set up so every hit is doing the full process of querying the database and pushing the results out through a template.
I was on the front page of HN last year and the server load was negligable. I have graphs: http://thraxil.org/users/anders/posts/2012/06/02/How-Django-...
On newer deployments, I'm still using Apache+mod_wsgi to manage the Django processes (it's a setup that has been reliable for me and that I know how to debug), but I'm sticking an nginx proxy in front as a matter of course. Not for performance reasons, but just to leave myself a hook for handling later changes. I can trivially spin up additional application backend servers and load balance across them, or quickly flick a switch to have nginx cache a particular resource in an emergency.
If he'd had W3 total cache, or varnish (my preferred option) in front of it, he wouldn't of noticed the impact.
There's a bigger question of why Wordpress still doesn't ship with sensible caching enabled by default, backwards compatibility with some plugins just doesn't cut it for me.
Now i'm wondering if i'm not both overoptimizing and missing the point entirely.
And yes I do realize 'custom php framework' is missing the point of frameworks and that php is an abomination. Getting Laravel to do what I wanted (awesome as it is, and it is) started to seem like trying to tweak a cannon into a flyswatter.
It's still fun, but I dread it ever getting posted here. Assuming it gets finished.
There is a very nice hack I applied to my WordPress installs, but it's not for the faint hearted: hack the WP index.php to serve pages directly from Memcache for all non-authenticated GET requests. The beauty of this is that for most requests, the WP environment (which is horribly bloated) does not even get loaded.
It works about an order of magnitude better than all those cache plugins, but the downside is of course that software updates require some special care.
That way you wouldn't need to worry about software updates.
After that I wrote a cronjob that runs a script which automatically increases my virtual machines RAM. When I have little or no visits it uses 1GB RAM the most it went up to was 9GB RAM (increasing with 1GB at a time).
It also sends me an e-mail when increasing/decreasing the RAM so I can ensure that it doesn't crash.
I also use Wordpress with caching enabled and the above helps me keep the costs down, don't want the server to run on 8GB RAM all the time nor 1GB RAM.
(I guess the answer is "No, only if you've misconfigured", but still...)
Probably the biggest lesson we've learned from collecting data on latency and timeouts from our backend is that there are a LOT of very poor hosts out there. Trying to strike a balance between intermittent failures, and actual "Your site is down, you need to do something" is pretty difficult, unless you are willing to simply wait long enough. We'll regularly see timeouts from places like Tumblr and Github Pages--they never go down, but things can get spotty. We can also tell when certain hosting providers are doing their weekly backups (or something - just guessing) because things will just go to pot at the same time every weekend for about an hour.
I can second the S3/CloudFront recommendation. Pretty rock solid (modulo downtime in specific regions).
For anyone curious about the app: http://vigil-app.com
Forget any other setup you've ever used, go with this for high performance WordPress
1) Nginx [set so that request to static content to completely bypass PHP]
2) PHP-FPM
3) APC [Object & Database cache]
4) W3 Total Cache or WP Super Cache
5) Varnish
There many other high performance WordPress setup you could find out there but in term of simplicity and manageability, this is the one I fancy most. The bottleneck in this setup is RAM, not PHP, WordPress or MySQL read/write operation as I've seen in many server setup.
Disclaimer : I run small operation of managed WordPress hosting called KittySensei[2]
Additionaly, S3 and Cloudfront are extremely cheap themselves. And you get a few nice bonusses (like low latency, everywhere).
My otherwise poorly maintained blog didn't have a problem with a simultaneous HN/Slashdot hit, and I'm not even sure I'm using all the settings correctly. But the basic page cache seems to do what it promises.
Edit: oops, I see the OP is using Super Cache as well but still flopped. Then he is definitely doing someone wrong. My blog is hosted on the cheapest hosting plan that Dreamhost allows.
http://www.gigpeppers.com/post-mortem-of-a-failed-hackernews...
Server cut through it like butter :) Just lots of caching, optimization, sprites, etc. That traffic isn't even that bad, but I was happy my rails code performed nicely.
As an aside, I found that being on Reddit can give much, much more traffic than HN.
For now what I will do is use a very lightweight Wordpress theme, and cloudflare.
Is that enough?
http://www.ewanleith.com/blog/900/10-million-hits-a-day-with...
It works, has been on HN front page a couple of times, and other similar sites, sometimes getting > 50k hits per day without an issue.
Also, I like to sometimes fiddle with stuff, and not let my "webmastering" get too rusty.
It is good to sometimes ssh to something that you own and wreck it at your will :)
To me, part of the fun of setting up a blog, is not only writing, but also taking care of it. I don't know really why (I have no intention of doing that for other people)
