Skip to content

Clean up after incident 8204 (2023-01-05)

During gitlab-com/gl-infra/production#8204 (closed) we created records with mixed partition ids that will need to be fixed for the partitioning work to resume:

gitlabhq_dblab=# select count(*) from p_ci_builds_metadata where partition_id = 101;
 count
--------
 188112
(1 row)

gitlabhq_dblab=# select count(*) from p_ci_builds_metadata inner join ci_builds on ci_builds.id = p_ci_builds_metadata.build_id where p_ci_builds_metadata.partition_id = 101 and ci_builds.partition_id <> 101;
 count
-------
 82282
(1 row)

Because of this mismatch we won't be able to update the foreign key between p_ci_builds_metadata and ci_builds to use partition_id. This is likely happening for all 17 tables, but we need to add indexes on partition_id to know more.

Proposal

Add temporary indexes to the following tables and write migrations to update partition_id to 100

table name temporary index created merge requests for indexes cleanup MR cleanup completed
ci_build_needs !108823 (merged) !111440 (closed)
ci_build_pending_states Not needed, will be updated by FK constraint on ci_builds
ci_build_report_results !108823 (merged) !111548 (merged)
ci_build_trace_chunks Not needed, will be updated by FK constraint on ci_builds Not needed
ci_build_trace_metadata !109084 (merged) !111549 (merged)
ci_builds !109085 (merged) !111287 (merged)
ci_job_artifacts !109084 (merged) !111436 (merged)
ci_job_variables !109084 (merged) !111173 (merged)
ci_pipeline_variables !108823 (merged) !111420 (merged)
ci_pipelines !109085 (merged) !109981 (merged)
ci_sources_pipelines !109094 (merged) !109364 (merged)
ci_stages !109085 (merged) !111439 (merged)
ci_unit_test_failures Not needed, will be updated by FK constraint on ci_builds Not needed
p_ci_builds_metadata Not needed, already covered by the partitioning constraint covered by the partitioning constraint !111576 (merged)
ci_running_builds !108788 (merged) Not needed
ci_pending_builds !108788 (merged) Not needed
ci_builds_runner_session !108788 (merged) Not needed, with !109316 (merged) it can be cascaded from ci_builds
Edited by Marius Bobin