Strategy to remove p_ci_builds_metadata partitions

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Background

We're migrating data from the p_ci_builds_metadata table to other tables (GitLab.com only) via MR !212935. The migration runs separately for each partition, allowing us to reclaim storage as soon as each partition's migration completes.

Initial Setup

  1. Stop creating new partitions - Remove p_ci_builds_metadata from the partitioning initializer and other partition management tools, since we no longer write to this table.

  2. Add partition constraints - Create a post-deployment migration that adds partitioning constraints as invalid check constraints on each partition.

  3. Validate constraints - Use the async validation workflow to validate these check constraints.

Removal Strategy

For each partition, wait until its background migration completes, then follow the appropriate removal process based on partition size:

Test partition (ci_builds_metadata_107 - 46 GB)

  1. Detach the partition using a post-deployment migration (possibly with a change management issue)
  2. Monitor the system for one day to ensure stability
  3. Truncate the partition using a post-deployment migration

Medium partitions (103-106, ranging from 1-1.2 TB each)

  • Create two post-deployment migrations per partition: one to detach, one to truncate
  • Important: Only truncate one partition per deployment pipeline to avoid system impact

Large partitions (ci_builds_metadata, 102, 101 - ranging from 2.9-8.3 TB each)

  1. Detach using post-deployment migrations
  2. Coordinate with DBREs to truncate at convenient times, or consider using background migrations to delete rows gradually and reduce replication lag risks

Current Partition Sizes

ci_builds_metadata     - 8,289 GB
ci_builds_metadata_102 - 6,311 GB  
ci_builds_metadata_101 - 2,947 GB
ci_builds_metadata_104 - 1,187 GB
ci_builds_metadata_106 - 1,079 GB
ci_builds_metadata_105 - 1,069 GB
ci_builds_metadata_103 - 1,058 GB
ci_builds_metadata_107 -    46 GB
ci_builds_metadata_108 -    48 kB
Edited by 🤖 GitLab Bot 🤖