Reduce growth rate of ci_builds_metadata partitions (#13886) · Epics · GitLab.org

Reduce growth rate of ci_builds_metadata partitions

## The Problem Tables are huge, see https://gitlab.com/groups/gitlab-org/-/work_items/13886#note_2989935234 (internal) ## Proposal * https://gitlab.com/gitlab-org/gitlab/-/issues/521005+s - This archival period will ensure we greatly reduce the writes on old partitions, allowing cleanup tasks to run (pg_repack, data removal). * https://gitlab.com/gitlab-org/gitlab/-/issues/547859+s - we will deduplicate data from `p_ci_build_metadata` and later truncate processing data for archived pipelines. The SSoT for all the work related to engineering is https://gitlab.com/groups/gitlab-org/-/epics/13886+ ## Implementation plan * https://gitlab.com/groups/gitlab-org/-/epics/18267+ - Target completion: early FY26-Q3 * No direct benefits to the CI storage usage yet as this is prep work to simplify changes in Phase 2. * https://gitlab.com/groups/gitlab-org/-/epics/18268+ - Target completion: FY26-Q3 * At the end of Phase 2 we will write data in deduplicated way, greatly reducing the growth rate. * This phase alone should allow us to get 12 months headroom. * https://gitlab.com/groups/gitlab-org/-/epics/18271+ - Target completion: FY26-Q4 * At the end of Phase 3 we will reclaim a lot of storage by deleting `ci_builds_metadata` and running `pg_repack` on `ci_builds` * Risk: the data migration could take 2 months to complete since we have to update most of the records in `ci_builds` - Blocker for any operations to reclaim space * https://gitlab.com/groups/gitlab-org/-/epics/18273+ - Target completion: FY27-Q1 * This will put long-term optimizations in place to periodically remove unused data from archived partitions. ## Exit Criteria 1. The exit criteria is to drop `p_ci_builds_metadata` in favour of `p_ci_job_definitions`. ## Data ``` id | created_at | updated_at | status -----+-------------------------------+-------------------------------+-------- 102 | 2024-08-24 02:00:03.743692+00 | 2024-08-24 02:00:03.757141+00 | 3 103 | 2025-04-15 02:00:02.425296+00 | 2025-04-24 02:00:05.383722+00 | 2 100 | 2022-09-07 14:36:50.191528+00 | 2024-08-24 02:00:03.743692+00 | 3 101 | 2022-11-17 09:22:08.14387+00 | 2024-08-24 02:00:03.743692+00 | 3 104 | 2025-04-16 02:00:05.593491+00 | 2025-04-22 02:00:07.014264+00 | 1 (5 rows) ```  > [!important] > > This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

epic