Ask HN: Is Apple down?

253 pointscrgt4y ago106 comments

https://developer.apple.com doesn't work App Store doesn't work iMessage doesn't work. Not just me - coworkers also struggling.

Any idea what's going on?

106 comments

Animats4y ago

    nslookup
    > server a.ns.apple.com
    Default server: a.ns.apple.com
    Address: 2620:149:ae0::53#53
    Default server: a.ns.apple.com
    Address: 17.253.200.1#53
    > developer.apple.com
    Server:  a.ns.apple.com
    Address: 2620:149:ae0::53#53

    developer.apple.com canonical name = developer-cdn.apple.com.akadns.net.
    ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

Ah. So Apple's own DNS servers are redirecting developer.apple.com to something on "akadns.net", which is operated by Akamai. But Apple's own DNS servers refuse to resolve that, probably because it's not in the apple.com zone.

    nslookup
    > developer-cdn.apple.com.akadns.net
    Server:  127.0.0.53
    Address: 127.0.0.53#53

    Non-authoritative answer:
    developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
    world-gen.g.aaplimg.com canonical name = apple-c.g.aaplimg.com.
    apple-c.g.aaplimg.com canonical name = apple-cf.g.aaplimg.com.
    apple-cf.g.aaplimg.com canonical name = apple-lr.g.aaplimg.com.
    > server a.ns.apple.com
    Default server: a.ns.apple.com
    Address: 2620:149:ae0::53#53
    Default server: a.ns.apple.com
    Address: 17.253.200.1#53
    > developer-cdn.apple.com.akadns.net
    Server:  a.ns.apple.com
    Address: 2620:149:ae0::53#53

    ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

It's clearly a botched DNS configuration. Not clear what the intent was. Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server? Or am I misreading that?

It's generally considered bad form to have all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.

Anyway, this looks like an attempt to outsource something to Akamai that went badly wrong.

lima4y ago

> Or am I misreading that.

Yes:

    developer.apple.com. 73 IN CNAME developer-cdn.apple.com.akadns.net.
    developer-cdn.apple.com.akadns.net. 73 IN CNAME world-gen.g.aaplimg.com.
    world-gen.g.aaplimg.com. 13 IN CNAME apple-c.g.aaplimg.com.
    apple-c.g.aaplimg.com. 8 IN CNAME apple-cf.g.aaplimg.com.
    apple-cf.g.aaplimg.com. 8 IN CNAME apple-lr.g.aaplimg.com.
    apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com.
    apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com.

The Akamai CNAME just points to a series of aaplimg.com CNAME (eventually ending up with apple-lr.g.aaplimg.com), which is Apple's own CDN domain. The CDN's resolvers (a.gslb.aaplimg.com and b.gslb.aaplimg.com) refused to serve A records for apple-lr.g.aaplimg.com.

They fixed that and now it's back up.

This kind of setup is typically done for flexibility reasons (geographical DNS load balancing or similar, where the Akamai DNS servers serve as the geo LB).

> It's generally considered bad form to have the all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.

Not necessarily - this is what glue records[1] are for. Many large companies host their authoritative DNS on the same domain, it's not a bad practice when done carefully.

[1]: https://ns1.com/blog/glue-records-and-dedicated-dns

silisili4y ago

> Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server.

It's just a CNAME, meaning go look that up. It does not indicate that developer-cdn.apple.com.akadns.net is a DNS server.

The above seems to indicate that somewhere in the chain of resolving developer-cdn.apple.com.akadns.net, a DNS server refused the query. A dig +trace should indicate which.

frays4y ago

Works with other DNS servers.

  $ nslookup developer-cdn.apple.com.akadns.net a.ns.apple.com
  Server:  a.ns.apple.com
  Address: 17.253.200.1#53

  ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

  $ nslookup developer-cdn.apple.com.akadns.net 1.1.1.1
  Server:  1.1.1.1
  Address: 1.1.1.1#53

  Non-authoritative answer:
  developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
  Name: world-gen.g.aaplimg.com
  Address: 17.253.121.201
  Name: world-gen.g.aaplimg.com
  Address: 17.253.121.202

pul4y ago

Now also works on their authoritative DNS servers again: https://www.nslookup.io/dns-records/apple.com#authoritative

jonfw4y ago

This looks like an Akamai DNS load balancing solution. It will route a user to an endpoint based on a bunch of statistics (think location, availability, latency, and/or load), and will often handle caching and DDOS protection as well

variant4y ago

I think something with DNSSEC:

https://puck.nether.net/pipermail/outages-discussion/2022-Ma...

joveian4y ago

I noticed a few weeks ago that developer.apple.com was failing DNSSEC and that this had been going on for a while (follow the "previous analysis" links to see earlier errors as well):

https://dnsviz.net/d/developer.apple.com/Yidc2Q/dnssec/

It doesn't seem like many people have noticed or cared, so I doubt many people use DNSSEC at all and the whole system could (and should) be scrapped one day with barely anyone noticing.

lima has an anaylsis of the issue causing trouble:

https://news.ycombinator.com/item?id=30757487

1 more reply

tptacek4y ago

AAPLIMG.COM isn't DNSSEC-signed either.

1 more reply

mnd9994y ago

Can we refer to this as “Doing a Facebook?”

Karrot_Kream4y ago

This has nothing to do with the BGP failures that FB had earlier. This is a DNS configuration problem. It's much simpler to fix.

tshaddox4y ago

Yep.

Wife: My Apple Maps isn't working.

Me: Hmm, it's not working for me either. They must be having server problems. You should use Google Maps for now.

Wife: I can't download Google Maps either, the App Store doesn't seem to be working.

zionic4y ago

Yeah I posted about that here (this just bit me) https://news.ycombinator.com/item?id=30757193 and I was flagged to oblivion.

Looks like I really need to keep a 3rd party nav app installed just in case!

8ytecoder4y ago

Always. Here maps is a good backup solution. It allows you to download pretty much the entire world - if you have the space in your phone.

1 more reply

BrightOne4y ago

Try Organic Maps - offline-first OpenStreetMap app. It's really good!

moepstar4y ago

Maps, App Store, iMessage on macOS works

They work on iOS as well - so it seems to be a regional thing?

(Location: Germany)

Twisell4y ago

On my side in France apple Map only partially work. Basemap are displaying correctly but query and routing function are unreachables. "Domain name not found" (translated from french). So it could be a DNS meltdown?

Usually basemap because they are heavy are served through a separate CDN.

RandallBrown4y ago

Everything in the App Store was working for me except actually downloading apps. Seems to be (mostly) resolved now.

1 more reply

hamaluik4y ago

Maps and iMessage are working for me in Canada, but not music.

traceroute664y ago

I agree. Regional.

Guessing the issues are centered on North America.

donarb4y ago

Both Apple Maps and Google Maps work in the browser, no need for an app.

rileymat24y ago

I would not say need, but the connection from the device to carplay is really nice.

tshaddox4y ago

Probably true, but we wanted it for driving directions via CarPlay and were in a bit of a rush. The car's built-in navigation (which we otherwise never use) ended up working fine, but the browser versions probably would have been my next attempt.

zionic4y ago

For turn by turn?

divbzero4y ago

Same with my Apple Maps over the course of an hour this morning.

Rough order of events:

1. Not working (could not find server)

2. Not working (request timeout)

3. Restart app

4. Working

Perhaps DNS was broken for awhile and restarting the app cleared the DNS cache and forced a fresh IP lookup?

asvitkine4y ago

Can you use the web version of Google maps?

jandorn4y ago

Yes. I have been doing that for quite some time.

donohoe4y ago

You wouldn't think it if you went by this:

https://www.apple.com/support/systemstatus/

adwi4y ago

I’m sure it wasn’t when you posted 10 minutes prior, but FWIW currently listing 11 outages:

> App Store - Outage Today, 12:32 PM - ongoing Some users are affected Users may be experiencing intermittent issues with this service.

Apple Arcade - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Apple Music - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Apple TV+ - Outage Today, 12:32 PM - ongoing Some users are affected Users may be experiencing a problem with Apple TV+. We are investigating this issue.

iTunes Store - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Podcasts - Outage Today, 12:32 PM - ongoing Some users are affected Users are experiencing a problem with this service. We are investigating and will update the status as more information becomes available.

Radio - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Apple Business Manager - Outage Today, 1:14 PM - ongoing Some users are affected Users may be unable to sign in.

Apple School Manager - Outage Today, 1:14 PM - ongoing Some users are affected Users may be unable to sign in.

Device Enrollment Program - Outage Today, 1:14 PM - ongoing Some users are affected Users are experiencing a problem with this service. We are investigating this issue.

Schoolwork - Outage Today, 1:14 PM - ongoing Some users are affected This service may be slow or unavailable.

mwnivek4y ago

At the bottom of that status page, it says: Looking for developer system status? Find it here: https://developer.apple.com/system-status/

The link is currently not working...

ninju4y ago

Looks like that link is working now...and shows a page of all GREEN :-)

1 more reply

oxplot4y ago

A lot of system status pages are updated by humans who will verify issues before reporting them. Main reason is to avoid overly surface every minor and transitory issue to public view.

_joel4y ago

Quite easy to verify if the entire developer site is down though, non?

2 more replies

nneonneo4y ago

Looks like it’s been updated. Currently showing 11 services down, some of which have been down for over an hour.

ransom15384y ago

That is just a static github page with html. These are just green dots on a screen.

Twisell4y ago

Well make sense to me to host your status page outside your main infrastructure.

perihelions4y ago

Here's a lot of crowd-sourced anecdata points:

- "Multiple Apple services are down such as: (Will be updating this list)"

https://old.reddit.com/r/apple/comments/tjg8tz/megathread_ap... ("[Megathread] Apple Outages")

samwillis4y ago

I chose the perfect time to restore a repaired iPhone, don’t seem to be able to fully login to iCloud, it’s hanging on the login screen…

Edit: It’s also refusing to download any apps, doesn’t even show the progress circle. Just a download icon next to the app name on the Home Screen and errors out when you click it.

Edit: Login and app downloads now working as of 6.00GMT

spansoa4y ago

It's times like this that force us to remind ourselves how reliant we are on critical services like these. On one hand, we can celebrate (Internet snow-day!) but on the other we are forced to shop around for alternatives too.

I often wondered how medieval the world would become if there was a huge sun flare ejection that breached the magnetic field and destroyed a bunch of data-centers. Think of the mess we'd be in!

1 more reply

jcims4y ago

I’m sure it’s happens more than I’m aware but i have to say that i can’t recall an App Store outage since i got back in the platform 3-4 years ago. Not bad!

rvieira4y ago

I picked a terrific time to lose my temper and do a `rm -rf /Library/Developer/CommandLineTools ; xcode-select --install` /facepalm

lima4y ago

Looks like their DNS servers are responsive, but refuse to serve records:

    $ dig -t NS developer.apple.com
    [...]
    apple-lr.g.aaplimg.com.     14400   IN      NS      b.gslb.aaplimg.com.
    apple-lr.g.aaplimg.com.     14400   IN      NS      a.gslb.aaplimg.com.

    $ dig @a.gslb.aaplimg.com developer.apple.com
    [...]
    ;; ->>HEADER<<- opcode: QUERY, status: REFUSED
    ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
    ;; WARNING: recursion requested but not available
    ;; WARNING: recursion requested but not available

Most likely a configuration mistake that'll be undone as soon as they figured out how to re-deploy their DNS servers while DNS is down.

Unlikely to be BGP shenanigans as some people on Twitter claim. My network has direct peerings to Apple's AS714.

chewmieser4y ago

Definitely. Downdetector shows a bunch of reports too (e.g. https://downdetector.com/status/apple-music/). I noticed issues with Music and News, seems like a ton of their services are down

callalex4y ago

Downdetector has predicted about 50 of the last 3 outages, and linking to them here just makes the self-fueling cycle even worse.

cmg4y ago

They're fine for knowing that something is going on, but not great for knowing exactly what the cause is.

For example, when Facebook's services went down in October, people were reporting that AT&T and other cell carriers were down because they couldn't open the apps. As far as I know there wasn't an outage with any of the carriers that day.

chewmieser4y ago

I think they’re about as useful as any anecdotal data out there. Unusually high numbers of reports when you’re seeing issues yourself is about as good as it gets until a status page is updated (which it thankfully has been finally).

synaesthesisx4y ago

Yes. Even developer.apple.com won't load at all for me. Who wants to take bets on DNS as the culprit?

gjsman-10004y ago

MacRumors says Apple is down.

https://www.macrumors.com/2022/03/21/icloud-and-apple-servic...

Big outage... is it some stupid DNS issue again?

andremedeiros4y ago

It could be DNS. I had to disable DNSSEC for the `.apple.com` zone to even work.

chuinard4y ago

My app update was rejected because my Upgrade screen was unable to fetch prices from their servers and instead showed an infinite spinner.

thih94y ago

Would an infinite spinner also show up if the server was up but the connection was problematic? If yes, this would be about not handling network errors, which sounds like a decent rejection reason to me.

donatj4y ago

My Apple Music stopped working mid song and is being weird now. Everything seems to be working fine for my wife. Weirdly spotty.

oxplot4y ago

iCloud Private Relay is shown as affected as well. This is an interesting case when it comes to failure behavior. From security perspective, you want your connection to stop working instead of falling back to insecure. Is this the case? Can anyone confirm?

mathieuh4y ago

It fell back to insecure for me, for about 30 seconds (maybe longer before I noticed) I couldn’t connect to the Internet from my iPhone, then I got a notification saying private relay was unavailable and I was able to connect again.

A few minutes later it gave me another notification saying private relay was working again.

nyuszika7h4y ago

iCloud Private Relay is not designed to be a full-fledged VPN anyway. HTTPS traffic in apps (other than browsers) bypasses it AFAIK.

ChrisMarshallNY4y ago

They seem to have been having a bit of a lie-down, today. I can't submit TestFlight builds, but now, it is taking longer, before the server throws a nutty, so I guess the fix is on its way.

Maursault4y ago

MobleMe is still down.

https://www.mac.com/

https://www.me.com/

SlimyHog4y ago

Yeah, I'm seeing anecdotal reports of a bunch of services out

rateofclimb4y ago

App Store Connect was down for me but appears to be up again now.

selimthegrim4y ago

I haven’t been able to cancel subscriptions lately. I filed a refund request and complaint to Apple, maybe it didn’t get through because of this?

mariojv4y ago

It's a partial outage for me. I was just able to send an iMessage, but directions on Maps are not working. I live in central Texas.

leviathan4y ago

I've been struggling with a DNS downtime at Mediatemple all day. Is there a possible more global DNS issue?

SalimoS4y ago

Yes, got a notification that Apple private relay is unavailable

And another notification that it’s back online 40min later

antupis4y ago

Yeah had to close private reley because websites didn’t load.

teeray4y ago

I had abnormal trouble pulling video I uploaded to iCloud yesterday. Something is up.

wanderer_4y ago

I noticed a blip in iMessage earlier, but it sorted itself out before too long.

brown9-24y ago

the domain name developer.apple.com resolves through a series of CNAMEs to Apple's CDN (applimg.com), which if it was down would explain other things like iMessage also being unavailable.

1023bytes4y ago

Yeah, for me the CNAME chain ends with apple-lr.g.aaplimg.com, which doesn't resolve to anything

variant4y ago

Some reports that there were DNSSEC validation issues w/ proxy.safebrowsing.apple which CNAMEs to aaplimg.com.

sys_647384y ago

AAPL is down too, today.

hit8run4y ago

For me in Germany: iMessage up App Store up Developer site down

ComputerGuru4y ago

It’s coinciding with an AWS outage. Probably not unrelated.

camhart4y ago

https://www.the-sun.com/tech/4944089/apple-maps-down-icloud-...

gunzor4y ago

Can't upload an ipa to App Store Connect for an hour.

jhgb4y ago

> Any idea what's going on?

Must be gravity. (Sorry, I had to.)

alopes4y ago

Had a few issues with the App Store with OS 12.3

selimthegrim4y ago

Me too.

traceroute664y ago

Must be regionalised. Nothing wrong here.

tiahura4y ago

Down in French Polynesia.

spike0214y ago

iMessage texts are working fine for me but an image I sent to a friend is stuck. Music is also down for me.

novateg4y ago

Down for me via CloudFlare WARP

fargle4y ago

It's always DNS.

vaxman4y ago

gotECC?

https://earthsky.org/sun/sun-activity-solar-flares-cme-week-...

LPDDR5 in the SoC doesn’t.

vaxman4y ago

If I had just dropped $2K-$12K on a media-centric computer with the intent of running encrypted backups, spreadsheets, databases and other inappropriate tasks for non-ECC memory (looking at you Leo), I’d downvote too!

Melatonic4y ago

It is always DNS ;-D

ragnot4y ago

Down for me.

windex4y ago

fedora is down.

barkingcat4y ago

maybe their dns expired?

j / k navigate · click thread line to collapse

106 comments

Animats4y ago

    nslookup
    > server a.ns.apple.com
    Default server: a.ns.apple.com
    Address: 2620:149:ae0::53#53
    Default server: a.ns.apple.com
    Address: 17.253.200.1#53
    > developer.apple.com
    Server:  a.ns.apple.com
    Address: 2620:149:ae0::53#53

    developer.apple.com canonical name = developer-cdn.apple.com.akadns.net.
    ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

    nslookup
    > developer-cdn.apple.com.akadns.net
    Server:  127.0.0.53
    Address: 127.0.0.53#53

    Non-authoritative answer:
    developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
    world-gen.g.aaplimg.com canonical name = apple-c.g.aaplimg.com.
    apple-c.g.aaplimg.com canonical name = apple-cf.g.aaplimg.com.
    apple-cf.g.aaplimg.com canonical name = apple-lr.g.aaplimg.com.
    > server a.ns.apple.com
    Default server: a.ns.apple.com
    Address: 2620:149:ae0::53#53
    Default server: a.ns.apple.com
    Address: 17.253.200.1#53
    > developer-cdn.apple.com.akadns.net
    Server:  a.ns.apple.com
    Address: 2620:149:ae0::53#53

    ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

Anyway, this looks like an attempt to outsource something to Akamai that went badly wrong.

lima4y ago

> Or am I misreading that.

Yes:

    developer.apple.com. 73 IN CNAME developer-cdn.apple.com.akadns.net.
    developer-cdn.apple.com.akadns.net. 73 IN CNAME world-gen.g.aaplimg.com.
    world-gen.g.aaplimg.com. 13 IN CNAME apple-c.g.aaplimg.com.
    apple-c.g.aaplimg.com. 8 IN CNAME apple-cf.g.aaplimg.com.
    apple-cf.g.aaplimg.com. 8 IN CNAME apple-lr.g.aaplimg.com.
    apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com.
    apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com.

They fixed that and now it's back up.

This kind of setup is typically done for flexibility reasons (geographical DNS load balancing or similar, where the Akamai DNS servers serve as the geo LB).

Not necessarily - this is what glue records[1] are for. Many large companies host their authoritative DNS on the same domain, it's not a bad practice when done carefully.

[1]: https://ns1.com/blog/glue-records-and-dedicated-dns

silisili4y ago

> Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server.

It's just a CNAME, meaning go look that up. It does not indicate that developer-cdn.apple.com.akadns.net is a DNS server.

The above seems to indicate that somewhere in the chain of resolving developer-cdn.apple.com.akadns.net, a DNS server refused the query. A dig +trace should indicate which.

frays4y ago

Works with other DNS servers.

  $ nslookup developer-cdn.apple.com.akadns.net a.ns.apple.com
  Server:  a.ns.apple.com
  Address: 17.253.200.1#53

  ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

  $ nslookup developer-cdn.apple.com.akadns.net 1.1.1.1
  Server:  1.1.1.1
  Address: 1.1.1.1#53

  Non-authoritative answer:
  developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
  Name: world-gen.g.aaplimg.com
  Address: 17.253.121.201
  Name: world-gen.g.aaplimg.com
  Address: 17.253.121.202