Ask HN: Startup Disaster Recovery

I run a credit card company on AWS, our website runs on a single, small aws instance, but the POS/Financial servers are setup this way:

- 8 multi-az oracle rds instances, 4 of them are slaves. It means we have 16 servers where 8 are manually standing by and 8 are on rds multi-az system. The 4 slaves are set to be used to recover from a data tampering event, not aws outages.

- Some dozens of app servers on the same region of each database. We use a custom tcp/ip server I built in java almost a decade ago. We are thinking about migrating to apache Mina.

My personal tips:

- Why are you paying the heroku premium? Build you own aws servers, put nginx and set cheap instances behind it. Let capistrano be your heroku.

- Use some dynamic dns service, I would recommend dynect. Avoid amazon Route 53.

- Use multi-az, set up the same database in another region (continent)

- Get out of us-east

Build the very same setting into another region, create scripts to load the slave country's database if possible.

If your database is too fancy or large:

Shard or distribute.

(Edit) Almost forgot: ALWAYS have fresh as possible, offline backups of everything. Oracle is god when it comes to making this easy.

- Don't worry about code backups, assuming you have the following: 1. Local copy 2. Local backups 3. GitHub 4. GitHub's own backups 5. Copy on Heroku 6. Heroku's backups (?) 7. Any server backups

That's a hell of a lot of backups.

- Make sure you can time how fast you can provision a new server and restore backups. When you actually have to, it'll give you some peace of mind knowing that your backups can be restored, and that you can reliably guess how long it'll take.

- Make sure your domains are "Locked" in NetworkSolutions, which prevents transfer-out requests. I'm still mixed on whois privacy, but you're probably okay using it with NetworkSolutions.

- I've also struggled with figuring out what to do if I can't run the business suddenly. For your domain, you can register it for several years or set it up on auto-renew, so that's not really an issue. As far as GitHub/Heroku, ... that's a tough question.