[ENGINEERING] Switch (Object Storage back-end) (#21711) · Epics · GitLab.org

[ENGINEERING] Switch (Object Storage back-end)

Project switch aims to scale git access 100x for the agentic era ## Hours Worked | Epic | Category | DRI | Total Hours | |------|----------|-----|-------------| | [Git Backend Stage 3 - Pluggable object databases](https://gitlab.com/groups/gitlab-org/-/epics/15061) | ~"infra-category::Internal Innovation" | @pks-gitlab | 0.0 | | [Git Backend Stage 1 - MVCC-Reftable infrastructure](https://gitlab.com/groups/gitlab-org/-/epics/22613) | | @eric.p.ju | 0.0 | | [Benchmark object storage backed Gitaly](https://gitlab.com/groups/gitlab-org/-/epics/21928) | ~"infra-category::Internal Innovation" | @echui-gitlab | 0.0 | | [Git Backend Stage 0: Design](https://gitlab.com/groups/gitlab-org/-/epics/22612) | | | 0.0 | ## Project Work ### :hourglass: Work In Progress These epics are currently in progress | **Topic** | **Start Date** | **Target End Date** | **Summary** | |-----------|----------------|---------------------|-------------| | [Git Backend Stage 3 - Pluggable object databases](https://gitlab.com/groups/gitlab-org/-/epics/15061) @pks-gitlab (+0 participants) ~"group::git" | 2024-10-08 | 2026-09-28 | **2026-07-02**: :tada: **achievements**: - A bunch of patch series have been merged: - Generic promisor remote connectivity checks. - Removal of global state in "setup.c", which is a prerequisite for the object storage extension. - A bunch of patch series are close to being merged: - The refactoring that drops the `whence` field and replaces it with a source. - Generic "prepare" checks. :issue-blocked: **blockers**: - A couple of patch series are stuck in "next" due to the Git 2.55 release. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/15061#note_3515181364_  **Nested Epics: 8** • https://gitlab.com/groups/gitlab-org/-/epics/22300+ • https://gitlab.com/groups/gitlab-org/-/epics/21206+ **2026-06-11**: The scaling Git POC has several ODB transaction related patches that need to be refactored and upstreamed. This includes patches to add transaction support for git-receive-pack(1) and git-fetch-pack(1). :clock1: **total hours spent this week by all contributors**: 8 :arrow_forward: **next**: - Finishing up refactoring for patch series to introduce ODB transactions to git-receive-pack(1) and should be sending it upstream soon. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/21206#note_3444045402_  • https://gitlab.com/groups/gitlab-org/-/epics/20440+ **2026-06-04**: :clock1: **total hours spent this week by all contributors**: 20 :tada: **achievements**: - The patch series that converts the "loose" backend into an ODB source has been merged upstream. :arrow_forward: **next**: - We only have one more backend that needs to be converted into an ODB source, which is the "packed" backend. Once that's done we have finished with the foundational refactorings, and the next step would then be to send out the patch series that convert more functionality to become pluggable. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/20440#note_3419391194_  • https://gitlab.com/groups/gitlab-org/-/epics/19719+ **2026-03-05**: :clock1: **total hours spent this week by all contributors**: 6 :tada: **achievements**: - The patch series that introduces the "files" backend has been merged now. This means that we have landed the initial pluggable infrastructure in upstream Git that allows us to swap out the object database format. While this initial infrastructure is still limited in scope, this is still a significant milestone. This epic can thus be closed. :arrow_forward: **next**: - While the initial infrastructure has landed, we'll need to make more functionality pluggable. This work is tracked as part of https://gitlab.com/groups/gitlab-org/-/work_items/20440+. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/19719#note_3132940536_  • https://gitlab.com/groups/gitlab-org/-/epics/18599+ • https://gitlab.com/groups/gitlab-org/-/epics/15062+ • https://gitlab.com/groups/gitlab-org/-/epics/17821+ **2026-01-06**: :clock1: **total hours spent this week by all contributors**: 4 :tada: **achievements**: - A [minor patch series](https://gitlab.com/gitlab-org/git/-/merge_requests/463) that fixes some bugs related to the object database has been merged. :issue-blocked: **blockers**: - Multiple patch series related to pluggable object databases are currently under review on the Git mailing list. We cannot progress further until those have been merged. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/17821#note_2985075263_  • https://gitlab.com/groups/gitlab-org/-/epics/15063+ | | [Git Backend Stage 1 - MVCC-Reftable infrastructure](https://gitlab.com/groups/gitlab-org/-/epics/22613) @eric.p.ju (+0 participants) ~"group::gitaly" | 2026-04-21 | 2026-09-30 | **Nested Epics: 1** • https://gitlab.com/groups/gitlab-org/-/epics/22476+ | | [Benchmark object storage backed Gitaly](https://gitlab.com/groups/gitlab-org/-/epics/21928) @echui-gitlab (+0 participants) ~"group::gitaly" | 2026-06-13 | 2026-07-10 | **2026-06-25**: Most of my time was spent on [Display benchmark results visually](https://gitlab.com/gitlab-org/gitaly/-/work_items/7266). The plan is to have the benchmark results show up as a posted note on the MR for now and in the MR's widget as a Metrics Report. It also adds a new job to the gitaly MR pipelines so it can be run and results shown visually to the user. Still needs a review. :clock1: **total hours spent this week by all contributors**: 15 :tada: **achievements**: * [Setup infrastructure required to target elastic cluster](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/8928) - https://gitlab.com/gitlab-org/gitaly/-/merge_requests/8928 * Sets up the infra required for a kubernetes job to run k6 benchmarks against the gitaly cluster deployed with the 'poc-scaling' branch. :issue-blocked: **blockers**: * n/a :arrow_forward: **next**: * Since the two issues i've been working on are infra focused, next week we can finally write some proper tests to find bottlenecks. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/21928#note_3491034707_  | | [Git Backend Stage 0: Design](https://gitlab.com/groups/gitlab-org/-/epics/22612) (+0 participants) ~"group::gitaly" | 2026-06-13 | 2026-07-10 | **Nested Epics: 1** • https://gitlab.com/groups/gitlab-org/-/epics/22205+ | | [Git Backend Stage 3 - Pluggable object databases](https://gitlab.com/groups/gitlab-org/-/epics/15061) @pks-gitlab (+0 participants) ~"group::git" | 2024-10-08 | 2026-09-28 | **2026-07-02**: :tada: **achievements**: - A bunch of patch series have been merged: - Generic promisor remote connectivity checks. - Removal of global state in "setup.c", which is a prerequisite for the object storage extension. - A bunch of patch series are close to being merged: - The refactoring that drops the `whence` field and replaces it with a source. - Generic "prepare" checks. :issue-blocked: **blockers**: - A couple of patch series are stuck in "next" due to the Git 2.55 release. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/15061#note_3515181364_  **Nested Epics: 8** • https://gitlab.com/groups/gitlab-org/-/epics/22300+ • https://gitlab.com/groups/gitlab-org/-/epics/21206+ **2026-06-11**: The scaling Git POC has several ODB transaction related patches that need to be refactored and upstreamed. This includes patches to add transaction support for git-receive-pack(1) and git-fetch-pack(1). :clock1: **total hours spent this week by all contributors**: 8 :arrow_forward: **next**: - Finishing up refactoring for patch series to introduce ODB transactions to git-receive-pack(1) and should be sending it upstream soon. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/21206#note_3444045402_  • https://gitlab.com/groups/gitlab-org/-/epics/20440+ **2026-06-04**: :clock1: **total hours spent this week by all contributors**: 20 :tada: **achievements**: - The patch series that converts the "loose" backend into an ODB source has been merged upstream. :arrow_forward: **next**: - We only have one more backend that needs to be converted into an ODB source, which is the "packed" backend. Once that's done we have finished with the foundational refactorings, and the next step would then be to send out the patch series that convert more functionality to become pluggable. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/20440#note_3419391194_  • https://gitlab.com/groups/gitlab-org/-/epics/19719+ **2026-03-05**: :clock1: **total hours spent this week by all contributors**: 6 :tada: **achievements**: - The patch series that introduces the "files" backend has been merged now. This means that we have landed the initial pluggable infrastructure in upstream Git that allows us to swap out the object database format. While this initial infrastructure is still limited in scope, this is still a significant milestone. This epic can thus be closed. :arrow_forward: **next**: - While the initial infrastructure has landed, we'll need to make more functionality pluggable. This work is tracked as part of https://gitlab.com/groups/gitlab-org/-/work_items/20440+. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/19719#note_3132940536_  • https://gitlab.com/groups/gitlab-org/-/epics/18599+ • https://gitlab.com/groups/gitlab-org/-/epics/15062+ • https://gitlab.com/groups/gitlab-org/-/epics/17821+ **2026-01-06**: :clock1: **total hours spent this week by all contributors**: 4 :tada: **achievements**: - A [minor patch series](https://gitlab.com/gitlab-org/git/-/merge_requests/463) that fixes some bugs related to the object database has been merged. :issue-blocked: **blockers**: - Multiple patch series related to pluggable object databases are currently under review on the Git mailing list. We cannot progress further until those have been merged. _Copied from https://gitlab.com/groups/gitlab-org/-/epics/17821#note_2985075263_  • https://gitlab.com/groups/gitlab-org/-/epics/15063+ | ### :soon: Ready Linked epics that are ready to start | **Topic** | |-----------| | [Git Backend Stage 2 — MVCC-Reftable migration](https://gitlab.com/groups/gitlab-org/-/epics/22226) ~"group::gitaly" | ### :rotating_light: Epics that need attention These linked epics are not in the correct state or missing a workflow label <details> | **Topic** | **Links** | **Reason** | |-----------|-----------|-------------| | [Scalable Git Backend POC](https://gitlab.com/groups/gitlab-org/-/epics/22230) (+0) | Labeling problem, epic has no workflow label | | [Private Beta Customer Engagements](https://gitlab.com/groups/gitlab-org/-/epics/22622) (+0) | Labeling problem, epic has no workflow label | | [Git Backend Stage 4 — MVCC-Packfile Infrastructure](https://gitlab.com/groups/gitlab-org/-/epics/22236) @pks-gitlab (+0) group::git | Labeling problem, epic has no workflow label | | [Git Backend Stage 5 — Migration to Object Storage](https://gitlab.com/groups/gitlab-org/-/epics/22227) (+0) group::gitaly | Labeling problem, epic has no workflow label | | [Git Backend Stage 6 — Clustering and Routing (PoC)](https://gitlab.com/groups/gitlab-org/-/epics/22229) @jamesliu-gitlab (+0) group::gitaly | Labeling problem, epic has no workflow label | | [Cloud storage connection pooling via unix socket proxy](https://gitlab.com/groups/gitlab-org/-/epics/21955) (+0) group::gitaly | Labeling problem, epic has no workflow label | </details>

epic