undefined | Better HN

0 pointsswiftcoder5mo ago0 comments

Yeah, and that default strategy tends to become very, very painful the first time you encounter non-trivial database corruption.

For example, one of my employers routinely tested DB restore by wiping an entire table in stage, and then having the on call restore from backup. This is trivial because you know it happened recently, you have low traffic in this instance, and you can cleanly copy over the missing table.

But the last actual production DB incident they had was a subtle data corruption bug that went unnoticed for several weeks - at which point restoring meant a painful merge of 10s of thousands of records, involving several related tables.

0 comments

matt-p5mo ago

Yeah, but automating a solution for all possible "one off subtle data corruption bugs" is a lot of energy and effort to be honest.

swiftcoderOP5mo ago

For sure. It's more about having a pipeline for pulling data from multiple sources - rather than spin up a whole new DB cluster, you usually want to pull the data into new tables in your existing DB, so that you can run queries across old & new data simultaneously

j / k navigate · click thread line to collapse

0 pointsswiftcoder5mo ago0 comments

Yeah, and that default strategy tends to become very, very painful the first time you encounter non-trivial database corruption.

0 comments

matt-p5mo ago

Yeah, but automating a solution for all possible "one off subtle data corruption bugs" is a lot of energy and effort to be honest.

swiftcoderOP5mo ago

j / k navigate · click thread line to collapse