How Stack Overflow plans to survive the next DNS attack (opens in new tab)

Author here. CloudFlare's DNS was measurably, and consistently, faster than almost every provider we tested. So I can concur with you in that regards. It was ultimately faster than Route 53 and Google Cloud (the path we ended up going).

However, when our relationship with CloudFlare ended and we moved to Fastly, one of the reasons we did so was unreliable lag with DNS updates at CloudFlare. Sometimes it would take minutes or hours for DNS changes to be affected in their system, and they could never provide a satisfactory reason why.

+1 - I am also very interested why they chose to switch out from CloudFlare for both their DNS and their CDN and over to Fastly. Nick Craver did a write up where they specifically mentioned [1] Cloudflare for both their DNS and CDN.

Do you think after the Dyn outage everyones sysadmins are running round adding redundancy, too worried to trust the uptime of their site in the hands of just CloudFlare?

[1] http://nickcraver.com/blog/2016/02/17/stack-overflow-the-arc...

One major issue with their dns was unpredictable lag while changing records. On most occasions we can change a record and it would propigate immediately. In some rare cases, it could take fifteen minutes, and in a few instances up to eight hours before the change would show up on their nameservers.

Cloudflare was unable to diagnose this or give us any comfort that it would improve.

Can you maybe share how you did the sync between them? Also there are already some tools pulling zone data from Dyn and adding it to Route53? Can you share why did you choose an own sync. (I'm planning to do the same and I'm interested in other's opinions)

Author here. One thing I didn't cover in the post was about how to have Google and AWS honour their SLA's is that you have to use _all 4_ of the nameservers they provide. Because we're not doing that (only using half), we have to balance the chance of an outage versus the impact of an outage.

You are correct in saying that more empirical data could be used here. We might even end up changing our minds. I'm not much of a numbers person but I might pass this onto some of the people in our company who love solving problems like this.

Author here. I agree. There are built-in mechanisms for doing this - AXFR and IXFR. However, these mechanisms were not really designed with this sort of scale in mind. You have to keep an up to date whitelist of all the servers that can talk to each other, and they would need to talk to each other on a non-anycasted address (otherwise the notify packet would go to just a single anycasted node).

Managing whitelists between multiple 3rd party DNS providers is likely to break frequently as servers move around, are added, removed, etc.

Interestingly, Hurricane Electric would have been one of our top choices if they had a first class API and a commercial SLA. Their ability to support zone transfers is admirable and did not go un-noticed. DNS Made Easy also supports zone transfers.

This'd be awesome, and there are very few providers which offer this. When looking into this, I could only find HE and BuddyNS.

Everyone seems to be inventing their own custom API for this, which I guess is the 'modern developer friendly' approach, but it results in a bit of a mess. Example: Caddy's implementation of the Let's Encrypt / ACME dns-01 challenge has all these plugins: https://caddyserver.com/download

We ended up running our own authoritative nameservers, which is not ideal. But at least cloud offerings allow you to spread across regions.

Sadly most of the bigger providers require you to code to their API - which makes migrations a little more complex - and that goes double if you get locked into using special DNS-records (rather than common types such as A/AAAA/MX/etc).

That's one of the reasons why the DNS hosting I support, which uses git-hooks to trigger updates, only currently pushes the DNS data to Amazon's route53 infrastructure.

At the time of the most recent Dyn outage I looked at allowing users to support multiple back-ends, to abstract away the pain of redundancy, but it seemed there was surprisingly little interest.

So DNSMadeEasy made it all the way through the barrage of tests. I even wrote a library for their API so that we could integrate it into our DNS software (https://github.com/mhenderson-so/godnsmadeeasy), but at the end of the day their performance in certain regions was not good enough. In some countries they were measurably faster than R53, but in others they were measurably much slower.

EdgeCast were dropped due to pricing, and that there's talk of Verizon selling the EdgeCast services again.

DNSimple didn't make it to performance testing because they only had 5 POPs, as opposed to 20+ of other providers.

CloudFlare's DNS was consistently one of the fastest, you are correct about that. If you read my responses to other comments here, you'll find that we decided not use their DNS service because of some fairly pervasive API issues we had with it.

We have a pretty cool tool for managing this at Stack. Changing from a single provider to two can literally be done by changing one line of config. We should be open sourcing it very soon, and a blog post on it sometime thereafter.

Denominator is not actively developed: https://github.com/Netflix/denominator/issues/374

Last commit of substance was in Sept 2015.

I've thought about doing something like this, the biggest issue I found is feature parity between DNS providers.

If you could have a unified API that would create the records on multiple providers that would be money, it's just that you'd lose out on some things like Route 53 health checking, etc.

Once you have configured your zone with multiple providers, it's simply a matter of adding NS entries for each provider's authoritative servers to your registrar. The harder part is ensuring that the zones are kept in sync and that you don't rely on features (such as GSLB stuff or ALIAS records) that aren't available with all providers.

It's up to the client resolver to handle failover, so it's not perfect in terms of availability, but better than nothing.

For example:

  $ dig ns amazon.com
  amazon.com.		3599	IN	NS	ns4.p31.dynect.net.
  amazon.com.		3599	IN	NS	ns1.p31.dynect.net.
  amazon.com.		3599	IN	NS	ns3.p31.dynect.net.
  amazon.com.		3599	IN	NS	ns2.p31.dynect.net.
  amazon.com.		3599	IN	NS	pdns1.ultradns.net.
  amazon.com.		3599	IN	NS	pdns6.ultradns.co.uk.

(note that this is also TLD redundant, since there's a .co.uk included)

Author here. Google and AWS are anycasted services. We were not interested in running our own anycasted DNS due to the management overhead of doing so, and because the cost of anycasting our own services would be orders of magnitude more expensive than outsourcing that to an established DNS provider.