undefined | Better HN

0 pointsw8whut1y ago0 comments

daily test restore is infeasible for anything but toy projects. You should periodically test your restore procedures, but its incredibly costly and time consuming for sizeable platforms. Its just not that easy to restore a 10+TB backup for example, and thats a _tiny_ backup size for a b2c product.

they can easily go into the hundreds of TB, depending on your platform.

and i might add: i vividly remember gitlabs article how they have had automated backups and test restores for years, but when they actually needed them... it turned out some data wasn't part of it after all. just because youre testing your restore procedure doesnt mean you've actually accomplished anything.

0 comments

KingOfCoders1y ago

Have two backups, the most recent data and everything else. Archive data to different databases, e.g. only have the most recent 6 months in a production OLTP database.

"daily test restore is infeasible for anything but toy projects. "

It probably depends on what you call "toy" project. If you work for Google, yes I think everything is a toy project, and you're right. I only worked for ~$200M ARR/1M DAU businesses and restoring was no problem. From your point working for a FAANG business it's a toy project I can see that. But there are many more "toy projects" of this kind than FAANG companies.

"10+TB backup for example, and thats a _tiny_ backup size for a b2c product."

Sure.

rodgerd1y ago

> but its incredibly costly and time consuming for sizeable platforms.

If your restores are too time consuming to test regularly, they sure as shit aren't going to be useful in a disaster.

usrnm1y ago

Why is that? Hours or even days of downtime are still better than just losing all data. It's a simple cost-benefit analysis, and it's ok to pick different trade-offs depending on your use case

rodgerd1y ago

I have spent most of my career in newspaper publishing and banking.

A newspaper that doesn't publish for a few days might recover. A bank that drops off the Swift network for days isn't a bank any more.

cyberax1y ago

Russia and Iran would like to have a few words with you.

Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though.

Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.

3 more replies

prmoustache1y ago

It depends on the accepted downtime.

j / k navigate · click thread line to collapse

0 comments

KingOfCoders1y ago

Have two backups, the most recent data and everything else. Archive data to different databases, e.g. only have the most recent 6 months in a production OLTP database.

"daily test restore is infeasible for anything but toy projects. "

"10+TB backup for example, and thats a _tiny_ backup size for a b2c product."

Sure.

rodgerd1y ago

> but its incredibly costly and time consuming for sizeable platforms.

If your restores are too time consuming to test regularly, they sure as shit aren't going to be useful in a disaster.

usrnm1y ago

Why is that? Hours or even days of downtime are still better than just losing all data. It's a simple cost-benefit analysis, and it's ok to pick different trade-offs depending on your use case

rodgerd1y ago

I have spent most of my career in newspaper publishing and banking.

A newspaper that doesn't publish for a few days might recover. A bank that drops off the Swift network for days isn't a bank any more.

cyberax1y ago

Russia and Iran would like to have a few words with you.

Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though.

Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.

3 more replies

prmoustache1y ago

It depends on the accepted downtime.

j / k navigate · click thread line to collapse