Yeah sorry I didn't want it to end up sounding like a threaded argument - and I was sort of brain dumping as I go.

Hardware wise - We use standard servers (super micro), packed with several tiers of SSDs (Intel for the high end, SanDisk for the lower end).

Software wise, again all off the shelf, well understood tools: Debian Linux, DRBD, iSCSI, LACP, LVM, Puppet.

Our compute servers are blades with Debian VMs running Docker containers Of our applications.

Edit: something we've gained greatly from that isn't off the shelf is that we moved to running very modern Linux Kernels - we have CI builds triggered as new stable versions are released and they are stock standard except that we do patch them with GRSecurity and ensure SELinux is enforcing.

All this doesn't cost much time to manage at all - we don't even have a storage admin and to be honest - if we needed one we'd be doing something wrong - apart from physical failure (which is very rare these days) there really isn't anything to do with storage - it's almost boring!

> most ppl will never need more than 100k IOPs, let alone 500k+

These types of statements are always false. If there is anything the computing industry has taught us is that people always need more resources. Always.

FYI - there are laws around where data is allowed to be hosted and what country that parent company is located in. Even if there weren't laws on this - I think it would be pretty irresponsible to trust all your data and servers to one large off shore corporation, especially one that has a fragile political climate (mind you, want countries don't!)

Amazon's bandwidth rates are more than 10 times what we can get locally, and power is included in our colo rental fees. I'm assuming that will pretty much be the situation for mrmondo too.

In terms of monthly costs, all the gear I deal with is lease to own: We pay less per month when the servers are new, and 3 years down the line our bills drop. There's no reason to have large capital expenditures just because you want your own gear.

As for security, it's not really that simple. Amazon's physical security may be top notch, and their patching for Xen and network security may be just fine, but beyond that you're pretty much on your own with Amazon just as you are with your own gear. You still need to understand how to configure firewall settings, and understand how to keep your VMs secure. Amazon's security needs to be top notch because it adds an additional layer that you don't have direct control over, but that does not provide any additional security that you would not have in most reasonable colo facilities where the physical network devices past the service providers network drop is totally in your control, in a locked environment.

> My experience is that companies rarely need expensive network storage gear

The thing is, this gear isn't expensive. For about $250/month I can lease to own a 2TB PCIe SSD delivering 2.8GB/s read, 1.9GB/s write, 450k read and 150k write IOPS. That's in the UK, with 20% VAT, and without shopping around. Or if I want something with the performance profile of Amazons new offering, I can pay $25/month. Amazons cheapest EBS offerings, which are nowhere near what this article about, costs $200/month for 2TB space. Go for provisioned IOPS and the EBS cost skyrockets.

AWS is the expensive network storage option, not leasing your own.

I can lease servers to put it in to get me "free" compute capacity for the difference in cost of the raw storage and still have money left over after spares and hosting/power.

Factor in bandwidth and it gets downright comical - Amazons bandwidth prices are so totally out of whack that where for managed/colo setups CDN is an expense, for AWS setups a good CDN can save you vast amounts of money by cutting your bandwidth charges. And that's without discounted rates. Start putting decent volumes through and host at a carrier neutral facility and paying even 1/20'th for bandwidth vs. AWS is well within reach with peering arrangements and a good mix of transit providers.