### Problem to solve Making lots of small IO calls to Gitaly is a bad access pattern. This is magnified for customers using NFS where each Gitaly call could cost 200ms regardless of the Git command. We need to address the worst IO access patterns so that NFS performance is tolerable without using Rugged while we wait for HA Gitaly https://gitlab.com/groups/gitlab-org/-/epics/842 ### Further details After a major customer ran into serious performance problems, five hotspots were addressed by bringing back the more performant Rugged implementation. This made performance acceptable in the short term. Feature flags: `rugged_find_commit`, `rugged_tree_entries`, `rugged_tree_entry`, `rugged_commit_is_ancestor`, `rugged_commit_tree_entry` Source: https://gitlab.com/gitlab-org/gitlab-ce/blob/6febe6634277a2aae0486a6d703b160ac4333438/lib/gitlab/git/rugged_impl/repository.rb#L15 ### Proposal - [x] **Communication** - [x] Blog post: explanation of the problems we've observed, the approaches we are trying, and the vision for the future https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/21851 - [x] **Mitigation** - use Rugged while we fix Gitaly - [X] Feature flagged Rugged patches https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25477, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25896, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25674, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25896, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25702 %11.10 - [X] Back port feature flagged Rugged patches https://gitlab.com/gitlab-org/release/tasks/issues/749 %11.5, %11.6, %11.7, %11.8, %11.9 - [x] Enable patches by default https://gitlab.com/gitlab-org/gitlab-ce/issues/60931 ~~%"12.0"~~ %"12.1" - [x] Add NFS telemetry https://gitlab.com/gitlab-org/gitlab-ce/issues/60602 ~~%"12.0"~~ %"12.1" - [x] **Measurement** - identify worst hot spots and implement measurement to prevent future regressions - [x] Add feature flag to enforce gitaly request limits https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25931 %"11.10" - this will allow us to test in production - [x] ~~Investigate already whitelisted N+1 hot spots https://gitlab.com/gitlab-org/gitlab-ce/issues/60593~~ - [ ] **Reduce Gitaly disk access** - reduce or prevent disk access for some Git operations - [x] Pack loose refs more frequently https://gitlab.com/gitlab-org/gitlab-ce/issues/59715 %"11.11" - [x] Caching/process pooling `cat-file` https://gitlab.com/gitlab-org/gitaly/issues/1505 %"11.11" - [ ] Cache `info/refs` https://gitlab.com/gitlab-org/gitaly/issues/1631 ~~%"12.0"~~ ~~%"12.1"~~ ... %"12.5" - [x] **Improve Git performance** - for operations where caching isn't possible, improve Git performance - [x] → Refer to Git performance tuning using delta islands, commit graph etc https://gitlab.com/groups/gitlab-org/-/epics/1200 - [ ] **Reduce frequency that data is requested from Gitaly** - see table below - [ ] **Remove Rugged... again** - [ ] https://gitlab.com/gitlab-org/gitaly/issues/1708 | Controller/API endpoint | Score | Issue | Fixed | |-------------------------|-------|-------|-------| | Gitaly InfoRefsUploadPack | 1326902503 | https://gitlab.com/gitlab-org/gitaly/issues/1631 | ~~%12.0~~ ~~%12.1~~ %12.2 | | Projects::MergeRequestsController#show | 354168635 | https://gitlab.com/gitlab-org/gitlab-ce/issues/37432 | %11.10 :white\_check\_mark: | | Projects::GitHttpController#info_refs | 151747382 | https://gitlab.com/gitlab-org/gitaly/issues/1631 | ~~%12.0~~ ~~%12.1~~ %12.2 | | ProjectsController#show | 74905523 | | | | Projects::RefsController#logs_tree | 67926420 | https://gitlab.com/gitlab-org/gitlab-ce/issues/37433 | %11.4 :white\_check\_mark:| | Projects::JobsController#show | 64902385 | https://gitlab.com/gitlab-org/gitlab-ce/issues/60708 | %12.0 :white\_check\_mark: | | Projects::MergeRequestsController#ci_environments_status | 61300455 | https://gitlab.com/gitlab-org/gitlab-ce/issues/62589 | %12.1 :white_check_mark: | | /api/:version/internal/allowed | 52409369 | https://gitlab.com/gitlab-org/gitlab-ce/issues/60588 | %11.11 :white\_check\_mark: | | /api/:version/jobs/:id/trace | 48969154 | https://gitlab.com/gitlab-org/gitlab/issues/28168 | | | Projects::CommitController#pipelines | 45944101 | https://gitlab.com/gitlab-org/gitlab-ce/issues/60449 | %12.2 :white_check_mark: | | RootController#index | 38979604 | | | | Projects::PipelinesController#index | 36796196 | https://gitlab.com/gitlab-org/gitlab/issues/19697 | ~~%12.1~~ ~~%12.2~~ %12.5 | | Projects::CommitsController#signatures | 32990004 | https://gitlab.com/gitlab-org/gitlab-ce/issues/60449 | ~~%12.1~~ %12.2 :white_check_mark: | | Projects::BlobController#show | 30221775 | https://gitlab.com/gitlab-org/gitlab/issues/27201 | %12.4 :white_check_mark: | | Projects::TreeController#show | 18423618 | https://gitlab.com/gitlab-org/gitlab-ce/issues/60449 | ~~%12.1~~ %12.2 :white_check_mark: | | /api/:version/projects/:id/jobs/:job_id | 16950686 | https://gitlab.com/gitlab-org/gitlab/issues/30255 | %"12.5" | | ~~Projects::NotesController#index~~ no I/O involved; DB queries | 16871682 | | | | Projects::MergeRequestsController#discussions | 15157733 | https://gitlab.com/gitlab-org/gitlab-ce/issues/58297 | %12.0 :white_check_mark: | | ~~UploadsController#show~~ (likely not related to Gitaly but filesystem) | 14009856 | | | | /api/:version/jobs/request | 13657325 | https://gitlab.com/gitlab-org/gitlab/issues/30256 | | | /api/:version/projects/:id/pipelines/:pipeline_id/jobs | 12312438 | https://gitlab.com/gitlab-org/gitlab/issues/28169 | |