Docker Systems Status: Full Service Disruption (opens in new tab)

I think more often than not, companies are using a single cloud provider, and even when multiple are used, it's either different projects with different legacy decisions or a conscious migration.

True multi-tenancy is not only very rare, it's an absolute pain to manage as soon as people start using any vendor-specific functionality.

No, that's pretty rare, and generally means you can't count on any features more sophisticated than VMs and object storage.

On the other hand, it's pretty embarrassing at this point for something as fundamental as Docker to be in a single region. Most cloud providers make inter-region failover reasonably achievable.

You can be multi-cloud in the sense that you aren't dependent on any single provider, or in the sense that you are dependent on all of them.

Not only they are not using multiple cloud providers, they are not using multiple cloud locations.

Because it's hard enough to distribute a service across multiple machines in the same DC, let alone across multiple DCs and multiple providers.

Because even if service A is using multiple cloud providers not all the external services they use are doing the same thing, especially the smallest one or the cheapest ones. At least one of them is on AWS East-1, fails and degrades service A or takes it down.

Being multi-cloud does not come for free: time, engineers, knowledge and ultimately money.

Multi cloud is not nearly as trivial as often implied to implement for real world complex projects. Things get challenging the second your application steps off the happy path

> Isn't it everyone using multiple cloud providers nowadays? Why are they affected by single cloud provider outage?

No? I very much doubt anyone is doing that.

> Isn't it everyone using multiple cloud providers nowadays?

Oh yes. All of them, in fact, especially if you count what key vendors host on.

> Why are they affected by single cloud provider outage?

Every workload is only on one cloud. Nb this doesn’t mean every workflow is on only one cloud. Important distinction since that would be more stable.

they are using multiple cloud providers, but judging by the cloudflare r2 outage affecting them earlier this year I guess all of them are on the critical path?

Looking at the landscape around me, no. Everyone is in crisis cost-cutting, "gotta show that same growth the C-suite saw during Covid" mode. So being multi-provider, and even in some cases, being multi-regional, is now off the table. It's sad because the product really suffers. But hey, "growth".

Rate limits are primarily applied to unauthenticated users, open source projects and business accounts have none/much higher tresholds

based on the solution, it seems like it is quite straightforward to switchover

We tried this initially

  FROM mirror.gcr.io/{image_name}

We received

  failed to resolve source metadata for mirror.gcr.io/

So it looks like these services may not be true mirrors, and just functioning as a library proxy with a cache.

If you're image is not cached on one of these then you may be SOL.

> You never know if an upstream image is purged randomly from docker, and your K8s node gets replaced, and now can't pull the base image for your service.

That doesn’t make sense unless you have some oddball setup where k8s is building the images you’re running on the fly. Theres no such thing as “base image” for tasks running in k8s. There is just the image itself and its layers which may come from some other image.

But it’s not built by k8s. Its be built in whatever is building your images and storing I. Your registers. That’s where you need your true base image caching.

My advice: document the issue, and use it to help justify spending time on removing those vestigial dependencies on Docker asap.

It's not just about reducing your exposure to third parties who you (presumably) don't have a contract with, it's also good mitigation against potential supply chain attacks - especially if you go as far as building the base images from scratch.

mirrors can be configured in dockerd or buildkit. if you can update the config (might need a self-hosted runner?) it’s a quick fix - see https://cloud.google.com/artifact-registry/docs/pull-cached-... for an example. aws and azure are similar.

I came here to mention that any non-trivial company depending on Docker images should look into a local proxy cache. It’s too much infra for a solo developer / tiny organization, but is a good hedge against DockerHub, GitHub repo, etc downtime and can run faster (less ingress transfer) if located in the same region as the rest of your infra.

I’m not sure that the impact will be that big. Most organizations have their own mirrors for artifacts.

Yes I noticed Signal being down too

You don’t want a Rube Goldberg contraption doing everything?

So not agile!

Pull-through caches are still useful even when the upstream is down... assuming the image(s) were pulled recently. The HEAD to upstream will obviously fail [when checking currency], but the software is happy to serve what it has already pulled.

Depends on the implementation, of course: I'm speaking to 'distribution/distribution', the reference. Harbor or whatever else may behave differently, I have no idea.

How hard is it to go to the GitHub repository and open the LICENSE file that is in almost every repository? Would have taken you less time than writing that comment, and showed you it's under MIT.

After some digging - https://github.com/spegel-org/spegel/blob/main/LICENSE says MIT

It's been a while since I looked at kuik, but I would say the main difference is that Spegel doesn't do any of the pulling or storage of images. Instead it relies on Containerd to do it for you. This also means that Spegel does not have to manage garbage collection. The nice thing with this is that it doesn't change how images are initially pulled from upstream and is able to serve images that exist on the node before Spegel runs.

Also it looks kuik uses CRDs to store information about where images are cached, while Spegel uses its own p2p solution to do the routing of traffic between nodes.

If you are running k3s in your homelab you can enable Spegel with a flag as it is an embedded feature.

Spegel does not only mirror Docker Hub, and works a lot differently than the alternatives you suggested. Instead of being yet another failure point closer to your production environment, it runs a distributed stateless registry inside of your Kubernetes cluster. By piggy backing off of Containerds image store it will distribute already pulled images inside of the cluster.

I am having some discussions about getting things working on GKE but I can't give an ETA as it really depends on how things align with deployment schedules. I am positive however that this will soon be resolved.

Google Cloud has its own cache of Docker Hub that you can use for free, AWS does as well

That's informative, I wasn't aware of that way to view HN, thanks.

What does the "active" page sort by?

Is there a way to configure alternate mirrors in containerd?

"MOST of the time" we're nine 9s.

Wait, Google and Microsoft rely on AWS? That seems unlikely? (does it? I wouldn't really know to be honest)

The new left-pad