We know how critical Docker Hub and services are to millions of developers, and we’re sorry for the pain this is causing. Thank you for your patience as we work to resolve this incident. We’ll publish a post-mortem in the next few days once this incident is fully resolved and we have a remediation plan.
Isn't it everyone using multiple cloud providers nowadays? Why are they affected by single cloud provider outage?
True multi-tenancy is not only very rare, it's an absolute pain to manage as soon as people start using any vendor-specific functionality.
On the other hand, it's pretty embarrassing at this point for something as fundamental as Docker to be in a single region. Most cloud providers make inter-region failover reasonably achievable.
Being multi-cloud does not come for free: time, engineers, knowledge and ultimately money.
No? I very much doubt anyone is doing that.
Oh yes. All of them, in fact, especially if you count what key vendors host on.
> Why are they affected by single cloud provider outage?
Every workload is only on one cloud. Nb this doesn’t mean every workflow is on only one cloud. Important distinction since that would be more stable.
Thankfully, AWS provides a docker.io mirror for those who can't wait:
FROM public.ecr.aws/docker/library/{image_name}
In the error logs, the issue was mostly related to the authentication endpoint:▪ https://auth.docker.io → "No server is available to handle this request"
After switching to the AWS mirror, everything built successfully without any issues.
Also, quay.io - another image hoster, from red hat - has been read-only all day today.
If you're going to have docker/container image dependencies it's best to establish a solid hosting solution instead of riding whatever bus shows up
Just had to change
FROM {image_name}
to FROM mirror.gcr.io/{image_name}
Hope this helps![0]: https://cloud.google.com/artifact-registry/docs/pull-cached-...
FROM mirror.gcr.io/{image_name}
We received failed to resolve source metadata for mirror.gcr.io/
So it looks like these services may not be true mirrors, and just functioning as a library proxy with a cache.If you're image is not cached on one of these then you may be SOL.
Wonder how many builds or redeployments this will break. Personally, nothing against Docker or Docker Hub of course, I find them to be useful.
Just engineering hygiene IMO.
That doesn’t make sense unless you have some oddball setup where k8s is building the images you’re running on the fly. Theres no such thing as “base image” for tasks running in k8s. There is just the image itself and its layers which may come from some other image.
But it’s not built by k8s. Its be built in whatever is building your images and storing I. Your registers. That’s where you need your true base image caching.
It's not just about reducing your exposure to third parties who you (presumably) don't have a contract with, it's also good mitigation against potential supply chain attacks - especially if you go as far as building the base images from scratch.
Asside; seems Signal is also having issues. Damn.
So not agile!
Depends on the implementation, of course: I'm speaking to 'distribution/distribution', the reference. Harbor or whatever else may behave differently, I have no idea.
It is a huge deal if I can start investigating and deploying such a solution as a techie right away, compared to having to go through all the internal hoops for a software purchase.
Kuik: https://github.com/enix/kube-image-keeper?tab=readme-ov-file...
Also it looks kuik uses CRDs to store information about where images are cached, while Spegel uses its own p2p solution to do the routing of traffic between nodes.
If you are running k3s in your homelab you can enable Spegel with a flag as it is an embedded feature.
docker got requests to allow you to configure a private registry, but they selfishly denied the ability to do that:
https://stackoverflow.com/questions/33054369/how-to-change-t...
redhat created docker-compatible podman and lets you close that hole
/etc/config/docker: BLOCK_REGISTRY='--block-registry=all' ADD_REGISTRY='--add-registry=registry.access.redhat.com'
For example, if you're on a team and you have documentation containing commands, but your docker config is outdated, you can accidentally pull from docker's global public registry.
A welcome change IMO would be removing global registries entirely, since it just makes it easier to tell where your image is coming from (but I severely doubt docker would ever consider this since it makes it fractionally easier to use their services)
Even if you could configure a default registry to point at something besides docker.io a lot of people, I'd say the vast majority, wouldn't have bothered. So they'd still be in the same spot.
And it's not hard to just tag images. I don't have a single image pulling from docker.io at work. Takes two seconds to slap <company-repo>/ at the front of the image name.
Ex: `docker pull ghcr.io/linuxcontainers/debian-slim:latest`
Google Container Registry provides a pull-through mirror, though, just prefix `mirror.gcr.io` and use `library` as the user for the Docker Official Images. For example `mirror.gcr.io/library/redis` for https://hub.docker.com/_/redis.
> [Monitoring] We are seeing error rates recovering across our SaaS services. We continue to monitor as we process our backlog.
- AWS
- Vercel
- Atlassian
- Cloudflare
- Docker
- Google (see downdetector)
- Microsoft (see downdetector)
What's going on?