Hello admins of EU AWS systems – DNS outage in eu-west-1

9 pointsdanielhunt9y ago16 comments

https://status.aws.amazon.com

Looks like there's a DNS outage in AWS Dublin

All DNS request coming from `eu-west-1b` fail completely if you're using the default AWS DNS resolution itself - if you switch to `8.8.8.8` it's all totally fine, but that's of no use if you have private services that need internal AWS resolution

`1a` and `1c` are also affected, but to a much lower extent from what I can tell

16 comments

aidos9y ago

Question for the more experienced in the group - is there a way of caching the DNS locally within you VPC in case something like this happens again? Or does that just cause more issues than it solves?

When I discovered it I tried switching my resolv.conf to 8.8.8.8 but of course none of my internal stuff worked because of how my security group / subnet / ip restrictions are setup internally.

jlgaddis9y ago

I'm not a big user of AWS but I don't know that there's anything stopping you from running your own recursive resolver (in your VPC) that your other machines use (instead of using an external resolver or whatever Amazon provides).

You could, for instance, stand up an instance or two running Unbound, forward to Amazon's own resolvers for internal stuff, and forward to external resolvers for external resolution (or just handle it yourself instead of using forwarders).

tomekit9y ago

The main reason to use AWS is so you don't need to do anything except calculate profit from your app, at least in theory. If you need to set up own resolvers or anything else within the AWS there is no reason to use AWS in the first place.

chatmasta9y ago

Yes, you can specify DNS server to use in the DHCP options set of the VPC. I've done this to point to internal DNS servers.

fern4lvarez9y ago

It's weird, I see actually most of my issues coming from instances located in `eu-west-1a`, whereas `eu-west-1b` and `eu-west-1c` look fine.

aidos9y ago

I recall reading somewhere many years ago that AWS randomise a, b, c etc per account. Otherwise people have a tendency to pick A and be done with it.

danielhuntOP9y ago

That's true - they loadbalance instance creation unless you specifically ask for an instance in a zone

That doesn't apply here though. Instances that are already up are experiencing DNS resolution issues because the Amazon-provided DNS service (which is distinct from Route53) is failing

1 more reply

danielhuntOP9y ago

Interesting difference in experience there

Complicated networks are complicated, I suppose

danielhuntOP9y ago

Seeing an increase in errors in `1a` and `1c` now myself too

neo20019y ago

6:29 PM PST We have identified the root cause of the DNS resolution issues in the EU-WEST-1 Region and continue working towards resolution.

paugay9y ago

same here :) let's see how long it takes

neo20019y ago

sigh

j / k navigate · click thread line to collapse

16 comments

aidos9y ago

When I discovered it I tried switching my resolv.conf to 8.8.8.8 but of course none of my internal stuff worked because of how my security group / subnet / ip restrictions are setup internally.

jlgaddis9y ago

tomekit9y ago

chatmasta9y ago

Yes, you can specify DNS server to use in the DHCP options set of the VPC. I've done this to point to internal DNS servers.

fern4lvarez9y ago

It's weird, I see actually most of my issues coming from instances located in `eu-west-1a`, whereas `eu-west-1b` and `eu-west-1c` look fine.

aidos9y ago

I recall reading somewhere many years ago that AWS randomise a, b, c etc per account. Otherwise people have a tendency to pick A and be done with it.

danielhuntOP9y ago

That's true - they loadbalance instance creation unless you specifically ask for an instance in a zone

That doesn't apply here though. Instances that are already up are experiencing DNS resolution issues because the Amazon-provided DNS service (which is distinct from Route53) is failing

1 more reply

danielhuntOP9y ago

Interesting difference in experience there

Complicated networks are complicated, I suppose

danielhuntOP9y ago

Seeing an increase in errors in `1a` and `1c` now myself too

neo20019y ago

6:29 PM PST We have identified the root cause of the DNS resolution issues in the EU-WEST-1 Region and continue working towards resolution.

paugay9y ago

same here :) let's see how long it takes

neo20019y ago

sigh

j / k navigate · click thread line to collapse