undefined | Better HN

0 pointsstepri5mo ago0 comments

“Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery.”

It’s always DNS.

0 comments

Nextgrid5mo ago

I wonder how much of this is "DNS resolution" vs "underlying config/datastore of the DNS server is broken". I'd expect the latter.

babarjaana5mo ago

Dumb question but what's the difference between the two? If the underlying config is broken then DNS resolution would fail, and that's basically the only way resolution fails, no?

mrktf5mo ago

My speculation: 1st one - it just DNS fails and you can repeat later. second one - you need working DNS to update your DNS servers with new configuration endpoints where DynamoDB fetches its config (classical case of circular dependencies - i even managed get similar problem with two small dns servers...)

vidarh5mo ago

DNS is trivial to distribute if your backing storage is accessible and/or local to each resolver, so it's a reasonable distinction to make: It suggests someone has preferred consistency at a level where DNS doesn't really provide consistency (due to caching in resolvers along the path) anyway, over a system with fewer failure points.

wdfx5mo ago

... wonders if the dns config store is in fact dynamodb ...

kjsingh5mo ago

DNS is managed by Route53 which has no dependency on Dynamodb for data plane

ej_campbell5mo ago

Background on the service: https://aws.amazon.com/builders-library/reliability-and-cons...

CaptainOfCoit5mo ago

I feel like even Amazon/AWS wouldn't be that dim, they surely have professionals who know how to build somewhat resilient distributed systems when DNS is involved :)

1 more reply

paulddraper5mo ago

Those aren't really that different.

That's a major way your DNS stops working.

huflungdung5mo ago

I don’t think it is DNS. The DNS A records were 2h before they announced it was DNS but _after_ reporting it was a DNS issue.

koliber5mo ago

It's always US-EAST-1 :)

shamil0xff5mo ago

Might just be BGP dressed as DNS

bayindirh5mo ago

Even when it's not DNS, it's DNS.

JoBrad5mo ago

Sometimes it’s BGP.

nijave5mo ago

I don't think that's necessarily true. The outage updates later identified failing network load balancers as the cause--I think DNS was just a symptom of the root cause

I suppose it's possible DNS broke health checks but it seems more likely to be the other way around imo

lkjdsklf5mo ago

I don’t work for aws, but a different cloud provider so this is not a description of this incident, but an example of the kind of thing that can happen

One particular “dns” issue that caused an outage was actually a bug in software that monitors healthchecks.

It would actively monitor all servers for a particular service (by updating itself based on what was deployed) and update dns based on those checks.

So when the health check monitors failed, servers would get removed from dns within a few milliseconds.

Bug gets deployed to health check service. All of a sudden users can’t resolve dns names because everything is marked as unhealthy and removed from dns.

So not really a “dns” issue, but it looks like one to users

oneeyedpigeon5mo ago

Downtime Never Stops!

commandersaki5mo ago

Someone probably failed to lint the zone file.

DrewADesign5mo ago

DNS strikes me as the kind of solution someone designed thinking “eh, this is good enough for now. We can work out some of the clunkiness when more organizations start using the Internet.” But it just ended up being pretty much the best approach indefinitely.

movpasd5mo ago

Seems like an example of "worse is better". The worse solution has better survival characteristics (on account of getting actually made).

1 more reply

us0r5mo ago

Or expired domains which I suppose is related?

dexterdog5mo ago

That's why they wrote the haiku

indycliff5mo ago

the answer is always DNS

j / k navigate · click thread line to collapse

0 comments

Nextgrid5mo ago

I wonder how much of this is "DNS resolution" vs "underlying config/datastore of the DNS server is broken". I'd expect the latter.

babarjaana5mo ago

Dumb question but what's the difference between the two? If the underlying config is broken then DNS resolution would fail, and that's basically the only way resolution fails, no?

mrktf5mo ago

vidarh5mo ago

wdfx5mo ago

... wonders if the dns config store is in fact dynamodb ...

kjsingh5mo ago

DNS is managed by Route53 which has no dependency on Dynamodb for data plane

ej_campbell5mo ago

Background on the service: https://aws.amazon.com/builders-library/reliability-and-cons...

CaptainOfCoit5mo ago

I feel like even Amazon/AWS wouldn't be that dim, they surely have professionals who know how to build somewhat resilient distributed systems when DNS is involved :)

1 more reply

paulddraper5mo ago

Those aren't really that different.

That's a major way your DNS stops working.

huflungdung5mo ago

I don’t think it is DNS. The DNS A records were 2h before they announced it was DNS but _after_ reporting it was a DNS issue.

koliber5mo ago

It's always US-EAST-1 :)

shamil0xff5mo ago

Might just be BGP dressed as DNS

bayindirh5mo ago

Even when it's not DNS, it's DNS.

JoBrad5mo ago

Sometimes it’s BGP.

nijave5mo ago

I don't think that's necessarily true. The outage updates later identified failing network load balancers as the cause--I think DNS was just a symptom of the root cause

I suppose it's possible DNS broke health checks but it seems more likely to be the other way around imo

lkjdsklf5mo ago

I don’t work for aws, but a different cloud provider so this is not a description of this incident, but an example of the kind of thing that can happen

One particular “dns” issue that caused an outage was actually a bug in software that monitors healthchecks.

It would actively monitor all servers for a particular service (by updating itself based on what was deployed) and update dns based on those checks.

So when the health check monitors failed, servers would get removed from dns within a few milliseconds.

Bug gets deployed to health check service. All of a sudden users can’t resolve dns names because everything is marked as unhealthy and removed from dns.

So not really a “dns” issue, but it looks like one to users

oneeyedpigeon5mo ago

Downtime Never Stops!

commandersaki5mo ago

Someone probably failed to lint the zone file.

DrewADesign5mo ago

movpasd5mo ago

Seems like an example of "worse is better". The worse solution has better survival characteristics (on account of getting actually made).

1 more reply

us0r5mo ago

Or expired domains which I suppose is related?

dexterdog5mo ago

That's why they wrote the haiku

indycliff5mo ago

the answer is always DNS

j / k navigate · click thread line to collapse