Does the choice of RoR play a part in performance problems?
Can performance be improved by deploying more servers?
PS: I really enjoy using Gitlab, and would be ready to replace Github with Gitlab in my workflow if the performance improves.
> what are the major areas where you face performance issues? > Does the choice of RoR play a part in performance problems?
RoR does not play a part in the performance problems as much as any other language choice. Our performance problems come from at least 3 different fronts: lack of caching in some specific points making us call the same complex/slow operations many times, NFS (filesystem) performance as a whole, and algorithms that worked really well at small scale, but not anymore, both at app level and at DB level.
I think that RoR is a really good option for building a product fact, and eventually it is necessary to start specializing specific parts that do not perform anymore, and just replacing what cannot be specialized. The key element here is that we need to measure first to see where the problem is.
> Can performance be improved by deploying more servers?
Not really, more front end servers means more load on the NFS backend, so there is no easy solution here. The first step into fixing this issue right now is this one: https://gitlab.com/gitlab-com/infrastructure/issues/139 and then we will start playing with a distributed FS as a longer term solution: https://gitlab.com/gitlab-com/operations/issues/1
As a final note, we are using some specific issues to measure performance as a blackbox, and those issues are seeing some really good progress lately. So stay tuned :)
Could you also comment on this https://news.ycombinator.com/item?id=12056991
We're working on switching to CephFS which looks very promising. See the relevant issue for more info: https://gitlab.com/gitlab-com/operations/issues/1
I can get someone from the backend performance team to comment too if you'd like :)
Sure!
> the biggest bottleneck at this point is with the file system.
Would something like AWS EFS [1] solve the problem?
It might not be a practical solution from a cost perspective though. Eg: for 50TB of data at $0.3/GB/month
0.3*1024*50 = $15360/month
We build a product that you can host yourself. If we only solve the scaling issue by pushing it down to a specific vendor, then there is no actual solution.
The way we are facing the problem is first by enabling a really easy and simple form of sharding (that would solve most of the issues that a lot of big customers may face), and then by using an open source underlaying filesystem that can scale reasonably well.