> [Monitoring] GitLab.com is now recovering. We found 2 last DB nodes which had not reverted their change. Apologies for the disruption. [1]
The reason I favor GitLab over GitHub is that I can install it on a dedicated server and have blazing fast performance, which helps development performance on an every day basis.
I've been maintaining a bunch of deployments of all sorts (k8s, docker-compose, baremetal) for the last years for several customers of all sorts of size, the upgrade process has always been pretty smooth, sometimes I ran in edgecases yes but always found a solution.
My deployments are always up and kicking, unless I'm messing with the configuration and doing some mistakes.
I highly recommend hosting your own GitLab instance, even on a single server.
Sorry, but GitLab has never been blazing fast. The gitlab.com instance is notoriously slow — supposedly improved a lot over the past few years, but still feels pretty sluggish. My self-hosted instance isn’t much better.
In fact, a couple of open source maintainer friends looked into migrating to GitLab when GitHub was acquired by MS; they did not precisely because GitLab was too slow.
Another thing: CI starts the very second I push, and since I configure dedicated runners extremely well GitLab makes it easy keep every pipeline stage under 5 minutes, which is critical to me, as such I recon I have acquired countless tricks for that matter.
Also, the ChatOps integration (GitLab to Mattermost or Slack) seems instant for me.
Those factors matter for me because I'm an extremely fast iterator and often iterate on several repositories at the time.
The cherry on the top is that the mirroring between my gitlab instance and github is also blazing fast.
As for the web interface, well they all feel slow to me so I don't spend my time waiting for them so it's ruled out of the equation for me anyway, I'm just far more comfortable with command line anyway.
But, pushing to gitlab prints out the URL to create a pull request from that branch, or to view the existing one, also a time saver that's not available on github, not sure why though.
By contrast I've managed a self-hosted install a while back, upgrades regularly broke as much as they fixed. For example the time when LDAP logins all failed, the gitlab-pages stopped working, and more.
That's on top of the constant UI churn with the side-menu, and similar changing from release to release with no rhyme or reason.
I wanted to like it, and did like the runner/CI support, though even that had as much pain as love. (Hard to script installation of agents, etc.)
Anyway, for CI-runner setup, I have a one-liner that worked for me: bigsudo install lean_delivery.gitlab_runner @somehost gitlab_ci_token=yourcommand gitlab_host=yourlabs.io gitlab_runner_limit=4 gitlab_version=11.6
But also a docker run does the job... Well, there's still registration of runner that I did manually, but I just checked ansible documentation and it really looks like I got a fair chance to achieve that too, given all the new modules ansible got since 2.8 to deal with gitlab, including gitlab_runner and gitlab_runners modules: https://docs.ansible.com/ansible/latest/modules/gitlab_runne...
So yes, Gitlab strives to keep development and releases coming, that means sometimes foundations move, and that there's a learning curve too, I understand that some people may not like that, but as a dev myself I like it this way, and at the end of the day my team iterates great with GitLab-CI, we got automated review-deploys in less than 5 minutes with just docker-compose and that's really helpful to keep the master branch clean, this allows features to be merged only after they have received product team approval so that's a big win. With GitHub you can also do it with Drone-CI which you might like better because it's easier to setup even though it remains behind GitLab-CI in terms of features that matter to me.
I don't mind the general sluggishness of the system that much (as I love the platform in general) but when you can't get your work (nor hobbies) done because of tools breaking, it gets really annoying really fast.
I always assumed that the publicly hosted version of gitlab is basically a giant demo version of the enterprise edition you buy to host it yourself. Hell, you can even host the community edition for free.
If your work relies on it, why rely on a free online product?
EDIT: TIL gitlab.com also has a paid options. I stand corrected.
We have a hosted Gitlab at 2 of my clients. Both are up.
We're not using the free product, we're paying customers (using the hosted saas version of the product).
Some orgs are paying 99$ per user-month for the SaaS version of gitlab.com, they probably rely on it pretty heavily :)
https://web.archive.org/web/20191119174637/https://about.git...
Edit: others made the same point, but I'll leave this here for the link to the pricing page.
Yes, the issues, MR, reviews are not available, but honestly I wouldn't want to be in a job where I could not get by without it for several hours.
For me the real test here is how they respond to this. As a paying customer I want to understand the issue, the efforts to prevent this in future and how they communicate this.
I like gitlab as a product but they don’t have a service mindset, and I think not hiring operations-centric people is a symptom of that which causes these kinds of issues.
He was asked to clone a repo during the interview, he told me it was almost a GB.
When he commented it was slow compared to Github, the engineer interviewing him told him it was his fault for living in Australia and that they have bad internet and they're too far away from everything.
Go figure.
It would be great if you understand that just saying something is not enough on the internet.
I’ve been looking at job postings and watching the way they work too for a little time since it’s all open.
DBAs are “ruby devs who have used Postgres”
https://about.gitlab.com/jobs/apply/backend-engineer-databas...
SREs are “ruby devs who have used docker/kubernetes”
(No job listing currently)
The only open job labelled “ops” is telling.
https://about.gitlab.com/jobs/apply/frontend-engineer---conf...
"Until very recently I was the only database specialist which meant I had a lot of work on my plate."
https://about.gitlab.com/blog/2017/10/02/scaling-the-gitlab-...
How come all of them are down all at once.
(NOTE: I am speculating here, if they do have a staging system and this wasn't reproduced there then the last sentence doesn't apply.)
Some guesses would be:
Automation/orchestration - They've migrated to k8s (I don't believe they've actually done this yet), but it could be their orchestration / automation tool automated a broken thing everywhere.
Database/Auth - Pretty much everything in gitlab will touch the database as far as I'm aware. Otherwise, how do you check whether users are auth'd to take action something. You wouldn't expect this to break the static website, i.e. the sales landing pages, but these could be based off an internal CMS, or could be checking for "guest" role session.
DNS/Service Discovery - As a sibling posted, "it's always DNS". It's good practice to use names for services instead of IP addresses, but this means your DNS needs to generally work, or everything will go down. Service Discovery could rely on DNS, but it could also be an API call that finds out DNS addresses or IP addresses directly.
CDN - You wouldn't typically put this in front of auth'd usage, and typically a CDN might not be helpful in front of something like SSH, but a quick look at fastly suggests they might support this. The main downside is sharing all the user data / auth tokens.
Security Product / CA - All you need is a requirement to encrypt internal traffic and rotate secrets, and you end up with a secret store that sits in the middle of everything.
Storage Layer - I believe they were big on Ceph for a while. If everything is backed by Ceph, everything will go down if you fail with Ceph.
Obviously, whatever it is, you'd expect them to split up their fail over plan a bit more in the future if it is something like that, but usually there's a single point of failure somewhere.
This points to there being:
- a lack of process and testing on key networking changes. Aren't they doing CI/CD, automated testing and peer review for this?
- A SPOF in the database; why couldn't things connect to a secondary for a read-only mode?
Quite a lot of the time, things break for stupid reasons. The main difference is when a normal company does something stupid, they can hide it, lie about it, or make it sound more complex.
The fact Gitlab publishes their fuck ups, is supposed to force them to do a better job and actually look at root causes and apply proper fixes that we can all judge. I wouldn't hold any particular fuck-up against them.
> CDN in front of everything.
Check their status page, it's just a simple reroute since the 13th [1].
I would have done anything to stay on the platform, for at home and the office and setup gitlab instances in two different offices.
Since somewhere last year, I moved back to Github primarily, and I'm sad to hear that my company is likely to make the same choice soon.The only reason is stability, and that's a little sad to me. I really want to love the product, but I need something that just works; not bells and whistles
The thing that strikes me each time is how fragile everyone's setup is if GitHub/lab is a single point of failure for them...
At least gitlab let's you self host, which would let you run backups on offsite hosting meaning zero downtime.
GitLab may well be focused on providing full-stack dev-services (VCS, build-server, CI and stuff), but in the end they are a hosting company - and for hosting, uptime is one of the most important metrics.
EDIT: 99% was exaggerating, but I got so many 50Xs the last days that it was from time to time unusable.
My experience with github is much better though on average. Haven't had a single bad experience personally the past year for my personal stuff. Not once. Github does only one thing, the CI is usually somewhere else so there have less reasons to fail as well... It might be comparing apples and oranges.
We use Gitlab at the office and it is constantly slow. Might be our setup I agree, but in the end it also tarnishes the brand to me.
All in all, my general experience is that speed and stability has degraded over the past years, in favor of the 'whole in one devops pipeline'. Github stayed focus on its core market.
Zero downtime but not zero maintainence cost. I think the administration of a full CI/CD environment (not only the gitlab machine, also the runners, the network etc.) are the main factor for this.
GitLab is a rare software that enables you to be in control.
"[Identified] We have identified firewall misconfiguration was applied that is preventing applications from connecting to the database."
https://gitlab.com/gitlab-com/gl-infra/production/issues/142...
The rollout of new iptables rules blocked database server connections
> was applied that is preventing applications from connecting
> to the database. We've rolling back that change and expect
> to be operational again shortly.
Heh, happens to the best of us! Seems to be coming back online now.
git center: http://127.0.0.1:43110/1GitLiXB6t5r8vuU2zC6a8GYj9ME6HMQ4t/
client: https://zeronet.io/
proxy: https://zero.acelewis.com/
I think it's about time they dedicated some resources or addressed the public about the efforts they're making to address these issues. It's becoming excruciating.
I've used Github every day for years and years and I feel like every page load has been pretty much instant forever.
Self-hosting has multiple different options as well.
Or does exist a mirror on github.com? ;)
gem install gitlab-development-kit
gdk init
cd gitlab-development-kit
gdk run
For Docker, check out https://hub.docker.com/r/gitlab/gitlab-ce I haven't tried it out for myself but it seems popular :)
Otherwise I would try the docker container OR just install it with omnibus in a VM: https://about.gitlab.com/install/
> with the latest Gitlab release
Just wait till later/tomorrow. Setting up your own instance is relatively easy. Still, you can get most of the way without their repos...
https://www.techrepublic.com/article/how-to-set-up-a-gitlab-...
> Or does exist a mirror on github.com? ;)
It seems so: https://github.com/gitlabhq/gitlabhq
i root for gitlab when i can, but theres a reason i mirror my repos to github.
AFAIK, nothing material happened after the announcement. There are lots of companies which do not hire in certain countries.
Note that this was always only about excluding certain job roles who have administrative access to production servers, not all jobs in general.
As another one said, their policy in not hiring people "in" rather than "from", as tax rules change from country to country.
In all seriousness, hopefully these things are not related.