[META] Improve GitLab Performance
[META]
Let's Make GitLab Faster
NB: Work in Progress
Priority items marked with a
Monitoring
- Add metrics instrumentation to GitLab Internals. #28465
-
⭐ ️ Create a publically visible dashboard of key metrics. #948- Easy to understand! -- everyone in the team should immediately be able to comprehend how fast things are running
Caching
Consider RoR Caching Techniques
- Fragment and Russian doll Caching
- This may not be useful, since most models will need to be cached per-user.
- Per-user, per-model caching
➡ ️ Low hit-rates
- SQL Caching
- Investigate RoR/Redis cache tuning and performance
- How well is the cache performing?
- How big is it? Are keys being evicted early
- Investigate RoR/Redis cache tuning and performance
Improve Conditional HTTP Support in the Application
Rails has fantastic support for generating Etag
headers, let's take advantage of it. Unfortunately at the moment, the Etag is being generated from the response
-
Take Control of Your HTTP Caching in Rails - Jan 2015
-
How key-based cache expiration works - DHH, Feb 2012
Polling and Realtime
Two separate issues here, but they are closely related:
- Realtime Issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/26396
- From the client side, we want to get realtime updates
- CI runners (potentially tens of thousands concurrently) listening to GitLab for build jobs to run.
Application Performance
-
⭐ ️ Find badly performing routes- n + 1 queries
- Count number of postgres and redis transactions per request and monitor w/ prometheus
- n + 1 queries
-
Slowlogs
- postgres slowlog
➡ ️ kibana - redis slowlog
➡ ️ kibana - slowlog
- postgres slowlog
-
⭐ ️ Disableseqscan
in postgres query planner in tests, so that any queries causing a scan will cause test failure
Git Access
File System Access
- Git Storage
-
⭐ ️ NFS optimisation - Migration to
gitaly
-
- Attachments
- Need to find out more information about where attachments are stored
- Ideally in an object store
- minio already in use?
Frontend Performance
- Frontend Performance Issue #23213