GitHub incident: ongoing issues with Actions, Issues and other Git operations (opens in new tab)

This is why I don't touch other people's keyboards.

Ideally both GitHub and HN would be down at the same time. With nowhere to discuss the former, people might actually get something done.

Easy to fix just adjust your system prompt add the following:

“Never ever type on keyboard with sandwich not even the most delicious tuna sandwich”

Next time try coffee, it's another worthy experience.

Many outages happen because something changed, and someone/something missed one of the effects of said change, bringing the platform down immediately, or after a while.

There was a period of time when GitHub basically didn't change, for years. And the platform was relatively stable (although "unicorns" (downtime)) still happened from time to time.

But nowhere near as often as now, but then again, there is a lot of more moving pieces now compared to before.

I wonder if it has anything to do with layoffs, if crucial Ops people left

About 10 years ago someone said we should move to self hosting because bitbucket who we used were unreliable. I looked at the status page and say 2 hours of downtime over 3-months, while we have 3-4 days of downtime on our self hosted jenkins during the same time. I always think of that when I see people complain about services being unreliable. Often we see one or two problems in short span and forget about the months were we didn't see any issues.

GitHub is probably as reliable now as it has been for the past 10 years. It's always had downtime.

They used to, some screenshots of it existing here:

https://interface.lt/github-uptime-and-microsoft

https://www.githubstatus.com/history seems to show incidents in the last 3 months:

* March - 20 incidents

* April - 12 incidents

* May - 4 incidents (so far)

We notice API outages that break core functionality every single day. It's gotten so bad over the last couple years.

Performance of the GitHub website in general feels increasingly bad.

It's a surprisingly unreliable service. It's been great for code management / reviews. But I can't imagine relying on it as the only option for deployments via CD. Imagine needing to deploy an important bug fix or a big project with a release date, but you can't because Github's having an outage.

You can checkout https://github.com/GitHub-Incident-History/GitHub-Incident-H... which provides all recorded GitHub incidents.

It's since M$FT took over, not so recently.

> You are better off self-hosting at this point

For medium/large companies, I fully agree. For smaller projects, specifically open source projects, I'd look at something like https://codeberg.org/

Maybe it's the case, but Github has always had issues with uptime and stability.

Where'd you get this from?

Where is their infra right now?

We've switched to selfhosted Gitea last month, no regrets. Only the CI story could be a bit better. We're currently using Woodpecker but need macOS runners, and Woodpecker's "local" agent implementation is still unstable. I'm watching Gitea Actions' progress with great interest.

Any reason to use Gitea over the Forgejo fork? Based on the drama I'd rather use Forgejo if they're functionally equal.[1]

[1]: https://forgejo.org/faq/#why-was-forgejo-created

I don't know about you, but the stuff I self-manage usually has worse uptime than SaaS products

GIT is actually a great protocol for keeping distributed copies of code. You can pretty easily with bash cycle through a list of backup urls for a git repo, looking for updates.

GitHub outages aren't nearly long or often enough to consider this. Git is distributed, just keep working locally until GitHub is back up. GitHub outages are nowhere near the threshold of pain I'd require to introduce a second Git hosting provider to the mix.

Really, GitHub outages barely hurt at all. It's not like an AWS or Cloudflare outage which is more likely to be a production disaster. Every outage a bunch of people on HN start screaming about owning their own on-prem destiny or wondering why we're still on GitHub. Nothing changes because it's not nearly as bad as those people are making it out to be. Life is all about tradeoffs.

Enterprise says I have to use this unreliable garbage.

https://status.gitlab.com/ lists 27 incidents this year, so far.

GitLab is quite a bit more expensive. If you have GitHub Enterprise with the security features, it's $70/month/user whereas you'll need to get GitLab Ultimate for the security features, which is $99/month/user.

Feature mismatch of anything outside Git. And no one wrote the tooling needed to synchronize issues, pull/merge requests, ... back and forth.

why particularly GitLab?

last I heard they've sharded some stuff, but some of the core tables/databases are still using a monolithic architecture.

And I guess you're only as strong as your weakest link, which can be not that bad, that is, if it isn't your core tables.

Though take this with a grain of salt, this is mostly hearsay =D

Indeed, I notice it when trying to push also:

  remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
  remote: fatal error in commit_refs
  To github.com:acme/foo.git
   ! [remote rejected] HEAD -> acme/foo (failure)
  error: failed to push some refs to 'github.com:acme/foo.git'

good catch

If this is your first time being affected, I guess you have a good reason to adjust your deployment structure so you can deploy manually if needed.

If this is not the first time, hopefully at least it will be the last one :)

Maybe a good time to investigate whether it's possible to hold those demos without relying on third party services.

Could you apply the patch from the URL? Pull requests aren't really needed.

Ignoring the fact that what people actually do with GitHub, git is such a small part. Issues, PRs, CI/CD and basically everything that isn't git, doesn't happen over git (besides the wiki, which somehow miraculously actually is via git).

Some people have their entire roadmap in GitHub, and every single bug report / feature request, without any offline backup. Don't ask me why, I don't get it. Especially since they have proven for the last few years that they cannot keep the platform up in a stable manner.

I mean, you joke but that's actually fairly true. P4 users always notice when the central server goes down because you can't reliably look at changelist history, draft CLs, and do a host of other operations that are possible on git locally. (using a central VCS confers other advantages of course).

You're kidding right? People / companies rely on git being available for deployments, builds, config changes, developer workflow etc

Wow huggingface is down completely

Goddamn and here I thought I was getting errors because the files i pushed were to large or something.

more like weekly when you look at the past incidents https://www.githubstatus.com/#past-incidents

I've started working on a Forgejo instance for myself (Gitea fork). It's honestly disappointing how bad GitHub has gotten, just in terms of uptime anymore. I hope they get their stuff together.

Yes. Once again. [0]

Due to GitHub's chronic unreliability, it is guaranteed to continue happening every month.

Looks like avoiding to 'centralize everything to GitHub' has aged very well [1] and at this point you would get better uptime with self-hosting instead of using GitHub.

Just ask many open source organizations like RedoxOS, ReactOS, wireguard, GNOME, KDE, etc.

[0] https://news.ycombinator.com/item?id=35817662

[1] https://news.ycombinator.com/item?id=22867803