Skip to content

Remove duplicate entries in Ci::RunnerProject

Ref: #350752 (closed)

What does this MR do and why?

This MR removes our duplicated records of Ci::RunnerProjects by doing the following:

  1. Add a new unique index with the highest ci_runner_projects_id which prevents further duplicates (very unlikely that we'll see duplicates).
  2. Iterate over ci_runner_projects and find duplicated entries (17 duplicate records)
  3. Eliminate the duplicated rows by "merging" them.
  4. Add global unique index on [:runner_id, :project_id] and remove the tmp index created by step 1.

This is the same approach we did for issue metrics in !55285 (merged).

Database review

Migration output

$ bin/rails db:migrate
== 20220124130028 DedupRunnerProjects: migrating ==============================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:ci_runner_projects, [:runner_id, :project_id], {:where=>"id > 2", :unique=>true, :name=>"tmp_unique_ci_runner_projects_by_runner_id_and_project_id", :algorithm=>:concurrently})
   -> 0.0026s
-- execute("SET statement_timeout TO 0")
   -> 0.0010s
-- add_index(:ci_runner_projects, [:runner_id, :project_id], {:where=>"id > 2", :unique=>true, :name=>"tmp_unique_ci_runner_projects_by_runner_id_and_project_id", :algorithm=>:concurrently})
   -> 0.0054s
-- execute("RESET statement_timeout")
   -> 0.0010s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:ci_runner_projects, [:runner_id, :project_id], {:unique=>true, :name=>"index_unique_ci_runner_projects_on_runner_id_and_project_id", :algorithm=>:concurrently})
   -> 0.0019s
-- add_index(:ci_runner_projects, [:runner_id, :project_id], {:unique=>true, :name=>"index_unique_ci_runner_projects_on_runner_id_and_project_id", :algorithm=>:concurrently})
   -> 0.0028s
-- transaction_open?()
   -> 0.0000s
-- indexes(:ci_runner_projects)
   -> 0.0021s
-- remove_index(:ci_runner_projects, {:algorithm=>:concurrently, :name=>"tmp_unique_ci_runner_projects_by_runner_id_and_project_id"})
   -> 0.0036s
-- transaction_open?()
   -> 0.0000s
-- indexes(:ci_runner_projects)
   -> 0.0019s
-- remove_index(:ci_runner_projects, {:algorithm=>:concurrently, :name=>"index_ci_runner_projects_on_runner_id_and_project_id"})
   -> 0.0040s
== 20220124130028 DedupRunnerProjects: migrated (0.0428s) =====================
$ bin/rails db:rollback
== 20220124130028 DedupRunnerProjects: reverting ==============================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:ci_runner_projects, [:runner_id, :project_id], {:name=>"index_ci_runner_projects_on_runner_id_and_project_id", :algorithm=>:concurrently})
   -> 0.0040s
-- Index not created because it already exists (this may be due to an aborted migration or similar): table_name: ci_runner_projects, column_name: [:runner_id, :project_id]
-- transaction_open?()
   -> 0.0000s
-- indexes(:ci_runner_projects)
   -> 0.0022s
-- execute("SET statement_timeout TO 0")
   -> 0.0008s
-- remove_index(:ci_runner_projects, {:algorithm=>:concurrently, :name=>"tmp_unique_ci_runner_projects_by_runner_id_and_project_id"})
   -> 0.0063s
-- execute("RESET statement_timeout")
   -> 0.0013s
-- transaction_open?()
   -> 0.0000s
-- indexes(:ci_runner_projects)
   -> 0.0016s
-- current_schema()
   -> 0.0002s
== 20220124130028 DedupRunnerProjects: reverted (0.0280s) =====================

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Max Orefice

Merge request reports