If there's one thing my gut says has changed for the worse since I joined, it's the cant away from supporting people who build things to tearing them down.
I asked in a "how do we improve HN comments" thread awhile back if Paul Graham could just add this to the guidelines, but it got drowned out by all the nerdy feature requests and didn't get much discussion.
On the bright side, looking to the quality comments here is a great way to learn constructive criticism (as well as learn about a variety of other topics too).
I have some sympathy; I've seen a Linux server randomly corrupt its file cache, no idea why. Google's study found 8% of DIMMs experienced at least one memory error a year. If you can't trust your RAM, what can you trust? http://research.google.com/pubs/pub35162.html
Building & maintaining a website is always a learning lesson because there are so many different areas to study.
Now the files (and when streaming directly, the packets) have a header containing some metadata and the actual WAL log entries have a fixed formatting, so it's likely that the slave would have detected this corruption (unless you were really unlucky which would then easily replicate the corruption over to the slave).
But that would just lead to the slave stopping to actually replicate. Unless you watch your clients whether they are still ok, streaming from the master and the replication lag is reasonably low, you would not notice the replication stopping. When you fail over, you get to the state which the database was in when the first corrupted packet arrived.
So either you check your slaves, or you use two-phase commit, ensuring that your data has reached the slaves, but that has some serious performance costs.
BTW: I would assume this was far more likely an issue with their storage, not with RAM.
The real question is, why on earth would they use non-ECC memory on their database server.
Perhaps because the database server is a cloud server from someone like Amazon, Mediatemple or Linode where you have no control on the underlying hardware.
