Any idea what's going on?
nslookup
> server a.ns.apple.com
Default server: a.ns.apple.com
Address: 2620:149:ae0::53#53
Default server: a.ns.apple.com
Address: 17.253.200.1#53
> developer.apple.com
Server: a.ns.apple.com
Address: 2620:149:ae0::53#53
developer.apple.com canonical name = developer-cdn.apple.com.akadns.net.
** server can't find developer-cdn.apple.com.akadns.net: REFUSED
Ah. So Apple's own DNS servers are redirecting developer.apple.com to something on "akadns.net",
which is operated by Akamai. But Apple's own DNS servers refuse to resolve that, probably because it's not in the apple.com zone.More:
nslookup
> developer-cdn.apple.com.akadns.net
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
world-gen.g.aaplimg.com canonical name = apple-c.g.aaplimg.com.
apple-c.g.aaplimg.com canonical name = apple-cf.g.aaplimg.com.
apple-cf.g.aaplimg.com canonical name = apple-lr.g.aaplimg.com.
> server a.ns.apple.com
Default server: a.ns.apple.com
Address: 2620:149:ae0::53#53
Default server: a.ns.apple.com
Address: 17.253.200.1#53
> developer-cdn.apple.com.akadns.net
Server: a.ns.apple.com
Address: 2620:149:ae0::53#53
** server can't find developer-cdn.apple.com.akadns.net: REFUSED
It's clearly a botched DNS configuration. Not clear what the intent was. Did they really want to point
"developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server? Or am I misreading that?It's generally considered bad form to have all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.
Anyway, this looks like an attempt to outsource something to Akamai that went badly wrong.
Yes:
developer.apple.com. 73 IN CNAME developer-cdn.apple.com.akadns.net.
developer-cdn.apple.com.akadns.net. 73 IN CNAME world-gen.g.aaplimg.com.
world-gen.g.aaplimg.com. 13 IN CNAME apple-c.g.aaplimg.com.
apple-c.g.aaplimg.com. 8 IN CNAME apple-cf.g.aaplimg.com.
apple-cf.g.aaplimg.com. 8 IN CNAME apple-lr.g.aaplimg.com.
apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com.
apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com.
The Akamai CNAME just points to a series of aaplimg.com CNAME (eventually ending up with apple-lr.g.aaplimg.com), which is Apple's own CDN domain. The CDN's resolvers (a.gslb.aaplimg.com and b.gslb.aaplimg.com) refused to serve A records for apple-lr.g.aaplimg.com.They fixed that and now it's back up.
This kind of setup is typically done for flexibility reasons (geographical DNS load balancing or similar, where the Akamai DNS servers serve as the geo LB).
> It's generally considered bad form to have the all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.
Not necessarily - this is what glue records[1] are for. Many large companies host their authoritative DNS on the same domain, it's not a bad practice when done carefully.
It's just a CNAME, meaning go look that up. It does not indicate that developer-cdn.apple.com.akadns.net is a DNS server.
The above seems to indicate that somewhere in the chain of resolving developer-cdn.apple.com.akadns.net, a DNS server refused the query. A dig +trace should indicate which.
$ nslookup developer-cdn.apple.com.akadns.net a.ns.apple.com
Server: a.ns.apple.com
Address: 17.253.200.1#53
** server can't find developer-cdn.apple.com.akadns.net: REFUSED
$ nslookup developer-cdn.apple.com.akadns.net 1.1.1.1
Server: 1.1.1.1
Address: 1.1.1.1#53
Non-authoritative answer:
developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
Name: world-gen.g.aaplimg.com
Address: 17.253.121.201
Name: world-gen.g.aaplimg.com
Address: 17.253.121.202https://puck.nether.net/pipermail/outages-discussion/2022-Ma...
https://dnsviz.net/d/developer.apple.com/Yidc2Q/dnssec/
It doesn't seem like many people have noticed or cared, so I doubt many people use DNSSEC at all and the whole system could (and should) be scrapped one day with barely anyone noticing.
lima has an anaylsis of the issue causing trouble:
Wife: My Apple Maps isn't working.
Me: Hmm, it's not working for me either. They must be having server problems. You should use Google Maps for now.
Wife: I can't download Google Maps either, the App Store doesn't seem to be working.
Looks like I really need to keep a 3rd party nav app installed just in case!
They work on iOS as well - so it seems to be a regional thing?
(Location: Germany)
Usually basemap because they are heavy are served through a separate CDN.
Guessing the issues are centered on North America.
Rough order of events:
1. Not working (could not find server)
2. Not working (request timeout)
3. Restart app
4. Working
Perhaps DNS was broken for awhile and restarting the app cleared the DNS cache and forced a fresh IP lookup?
> App Store - Outage Today, 12:32 PM - ongoing Some users are affected Users may be experiencing intermittent issues with this service.
Apple Arcade - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.
Apple Music - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.
Apple TV+ - Outage Today, 12:32 PM - ongoing Some users are affected Users may be experiencing a problem with Apple TV+. We are investigating this issue.
iTunes Store - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.
Podcasts - Outage Today, 12:32 PM - ongoing Some users are affected Users are experiencing a problem with this service. We are investigating and will update the status as more information becomes available.
Radio - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.
Apple Business Manager - Outage Today, 1:14 PM - ongoing Some users are affected Users may be unable to sign in.
Apple School Manager - Outage Today, 1:14 PM - ongoing Some users are affected Users may be unable to sign in.
Device Enrollment Program - Outage Today, 1:14 PM - ongoing Some users are affected Users are experiencing a problem with this service. We are investigating this issue.
Schoolwork - Outage Today, 1:14 PM - ongoing Some users are affected This service may be slow or unavailable.
The link is currently not working...
- "Multiple Apple services are down such as: (Will be updating this list)"
https://old.reddit.com/r/apple/comments/tjg8tz/megathread_ap... ("[Megathread] Apple Outages")
Edit: It’s also refusing to download any apps, doesn’t even show the progress circle. Just a download icon next to the app name on the Home Screen and errors out when you click it.
Edit: Login and app downloads now working as of 6.00GMT
I often wondered how medieval the world would become if there was a huge sun flare ejection that breached the magnetic field and destroyed a bunch of data-centers. Think of the mess we'd be in!
$ dig -t NS developer.apple.com
[...]
apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com.
apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com.
$ dig @a.gslb.aaplimg.com developer.apple.com
[...]
;; ->>HEADER<<- opcode: QUERY, status: REFUSED
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; WARNING: recursion requested but not available
Most likely a configuration mistake that'll be undone as soon as they figured out how to re-deploy their DNS servers while DNS is down.Unlikely to be BGP shenanigans as some people on Twitter claim. My network has direct peerings to Apple's AS714.
For example, when Facebook's services went down in October, people were reporting that AT&T and other cell carriers were down because they couldn't open the apps. As far as I know there wasn't an outage with any of the carriers that day.
https://www.macrumors.com/2022/03/21/icloud-and-apple-servic...
Big outage... is it some stupid DNS issue again?
A few minutes later it gave me another notification saying private relay was working again.
And another notification that it’s back online 40min later
Must be gravity. (Sorry, I had to.)
https://earthsky.org/sun/sun-activity-solar-flares-cme-week-...
LPDDR5 in the SoC doesn’t.