Prevent duplicate pipeline IIDs -- Next step: BBM to migrate existing iids to ci_pipeline_iids
Background
We currently have the unique index (project_id, iid, partition_id) on the table p_ci_pipelines. This ensures uniqueness within the same partition but not across partitions. Due to a possible workflow bug/race condition, we occasionally save duplicate pipeline iids for the same project, which causes problems such as https://gitlab.com/gitlab-com/request-for-help/-/issues/2855#note_2518831537.
We've explored a couple different possible solutions (https://gitlab.com/gitlab-org/gitlab/-/issues/545167#note_2802709422, https://gitlab.com/gitlab-org/gitlab/-/issues/545167#note_2844491364), and we ultimately decided that the only way to guarantee Pipeline iid uniqueness on the database level is to create a new table to track the iids. (See proposed approach in !210789 (comment 2869213142).)
- In !213065 (merged), we introduced the static hash partitioned table
p_ci_pipeline_iids. It has 64 partitions with the hash onproject_idand primary key on(project_id, iid). - In !213457 (merged), we added database triggers on
p_ci_pipelinesso that it keep track of iids inp_ci_pipeline_iids. - In !213992 (merged), we are updating the internal ID
initial_valuelogic to readmaximum(:iid)fromci_pipeline_iidsinstead ofci_pipelines.
Implementation
This issue is intended to track the next task in our implementation steps.
Proposal
Create a BBM to migrate existing iids from p_ci_pipelines to p_ci_pipeline_iids.