Skip to content

Prepare indexes for bigint column conversions

Patrick Bair requested to merge pb-prepare-indexes-for-bigint-conversions into master

What does this MR do?

Prepare indexes for asynchronous creation for tables that need to have columns converted from int to bigint. We need to do this for tables which have large indexes on GitLab.com, so deployments can complete in a reasonable time. The indexes should be created as part of the reindexing process that runs every weekend. Async index creation was added in !66478 (merged)

For example, with ci_job_artifacts we have 5 indexes that contain the id or job_id columns, both of which are being converted from int to bigint. The migration in this MR schedules those 5 indexes to be created asynchronously, before the migration to swap the columns is merged. Once the swap migration runs, the add_concurrent_index will be a no-op since the indexes will already exist. On self-managed installations, the indexes will be created as part of the swap migration as normal.

The related MRs for the swaps are: !65601 (merged), !66625 (merged), !66088 (merged)

How to setup and validate locally

Note the migration only inserts a record into postgres_async_indexes for each index, so runtime is not a concern.

rails db:migrate:up VERSION=20210804151444
== 20210804151444 PrepareIndexesForCiJobArtifactBigintConversion: migrating ===
-- index_exists?(:ci_job_artifacts, :id_convert_to_bigint, {:unique=>true, :name=>:index_ci_job_artifact_on_id_convert_to_bigint, :algorithm=>:concurrently})
   -> 0.0039s
-- add_index_options(:ci_job_artifacts, :id_convert_to_bigint, {:unique=>true, :name=>:index_ci_job_artifact_on_id_convert_to_bigint, :algorithm=>:concurrently})
   -> 0.0000s
-- index_exists?(:ci_job_artifacts, [:project_id, :id_convert_to_bigint], {:where=>"file_type = 18", :name=>:index_ci_job_artifacts_for_terraform_reports_bigint, :algorithm=>:concurrently})
   -> 0.0030s
-- add_index_options(:ci_job_artifacts, [:project_id, :id_convert_to_bigint], {:where=>"file_type = 18", :name=>:index_ci_job_artifacts_for_terraform_reports_bigint, :algorithm=>:concurrently})
   -> 0.0000s
-- index_exists?(:ci_job_artifacts, :id_convert_to_bigint, {:where=>"file_type = 18", :name=>:index_ci_job_artifacts_id_for_terraform_reports_bigint, :algorithm=>:concurrently})
   -> 0.0028s
-- add_index_options(:ci_job_artifacts, :id_convert_to_bigint, {:where=>"file_type = 18", :name=>:index_ci_job_artifacts_id_for_terraform_reports_bigint, :algorithm=>:concurrently})
   -> 0.0000s
-- index_exists?(:ci_job_artifacts, [:expire_at, :job_id_convert_to_bigint], {:name=>:index_ci_job_artifacts_on_expire_at_and_job_id_bigint, :algorithm=>:concurrently})
   -> 0.0026s
-- add_index_options(:ci_job_artifacts, [:expire_at, :job_id_convert_to_bigint], {:name=>:index_ci_job_artifacts_on_expire_at_and_job_id_bigint, :algorithm=>:concurrently})
   -> 0.0000s
-- index_exists?(:ci_job_artifacts, [:job_id_convert_to_bigint, :file_type], {:unique=>true, :name=>:index_ci_job_artifacts_on_job_id_and_file_type_bigint, :algorithm=>:concurrently})
   -> 0.0027s
-- add_index_options(:ci_job_artifacts, [:job_id_convert_to_bigint, :file_type], {:unique=>true, :name=>:index_ci_job_artifacts_on_job_id_and_file_type_bigint, :algorithm=>:concurrently})
   -> 0.0000s
== 20210804151444 PrepareIndexesForCiJobArtifactBigintConversion: migrated (0.0531s)
rails db:migrate:down VERSION=20210804151444
== 20210804151444 PrepareIndexesForCiJobArtifactBigintConversion: reverting ===
== 20210804151444 PrepareIndexesForCiJobArtifactBigintConversion: reverted (0.0183s)
rails db:migrate:up VERSION=20210804153307
== 20210804153307 PrepareIndexesForTaggingBigintConversion: migrating =========
-- index_exists?(:taggings, :id_convert_to_bigint, {:unique=>true, :name=>:index_taggings_on_id_convert_to_bigint, :algorithm=>:concurrently})
   -> 0.0032s
-- add_index_options(:taggings, :id_convert_to_bigint, {:unique=>true, :name=>:index_taggings_on_id_convert_to_bigint, :algorithm=>:concurrently})
   -> 0.0000s
-- index_exists?(:taggings, [:taggable_id_convert_to_bigint, :taggable_type], {:name=>:i_taggings_on_taggable_id_convert_to_bigint_and_taggable_type, :algorithm=>:concurrently})
   -> 0.0020s
-- add_index_options(:taggings, [:taggable_id_convert_to_bigint, :taggable_type], {:name=>:i_taggings_on_taggable_id_convert_to_bigint_and_taggable_type, :algorithm=>:concurrently})
   -> 0.0000s
-- index_exists?(:taggings, [:taggable_id_convert_to_bigint, :taggable_type, :context], {:name=>:i_taggings_on_taggable_bigint_and_taggable_type_and_context, :algorithm=>:concurrently})
   -> 0.0019s
-- add_index_options(:taggings, [:taggable_id_convert_to_bigint, :taggable_type, :context], {:name=>:i_taggings_on_taggable_bigint_and_taggable_type_and_context, :algorithm=>:concurrently})
   -> 0.0000s
-- index_exists?(:taggings, [:tag_id, :taggable_id_convert_to_bigint, :taggable_type, :context, :tagger_id, :tagger_type], {:unique=>true, :name=>:taggings_idx_tmp, :algorithm=>:concurrently})
   -> 0.0018s
-- add_index_options(:taggings, [:tag_id, :taggable_id_convert_to_bigint, :taggable_type, :context, :tagger_id, :tagger_type], {:unique=>true, :name=>:taggings_idx_tmp, :algorithm=>:concurrently})
   -> 0.0000s
== 20210804153307 PrepareIndexesForTaggingBigintConversion: migrated (0.0388s)
rails db:migrate:down VERSION=20210804153307
== 20210804153307 PrepareIndexesForTaggingBigintConversion: reverting =========
== 20210804153307 PrepareIndexesForTaggingBigintConversion: reverted (0.0175s)
rails db:migrate:up VERSION=20210804154407
== 20210804154407 PrepareIndexesForCiStageBigintConversion: migrating =========
-- index_exists?(:ci_stages, :id_convert_to_bigint, {:unique=>true, :name=>:index_ci_stages_on_id_convert_to_bigint, :algorithm=>:concurrently})
   -> 0.0032s
-- add_index_options(:ci_stages, :id_convert_to_bigint, {:unique=>true, :name=>:index_ci_stages_on_id_convert_to_bigint, :algorithm=>:concurrently})
   -> 0.0001s
-- index_exists?(:ci_stages, [:pipeline_id, :id_convert_to_bigint], {:where=>"status in (0, 1, 2, 8, 9, 10)", :name=>:index_ci_stages_on_pipeline_id_and_id_convert_to_bigint, :algorithm=>:concurrently})
   -> 0.0022s
-- add_index_options(:ci_stages, [:pipeline_id, :id_convert_to_bigint], {:where=>"status in (0, 1, 2, 8, 9, 10)", :name=>:index_ci_stages_on_pipeline_id_and_id_convert_to_bigint, :algorithm=>:concurrently})
   -> 0.0000s
== 20210804154407 PrepareIndexesForCiStageBigintConversion: migrated (0.0262s)
rails db:migrate:down VERSION=20210804154407
== 20210804154407 PrepareIndexesForCiStageBigintConversion: reverting =========
== 20210804154407 PrepareIndexesForCiStageBigintConversion: reverted (0.0140s)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Edited by Patrick Bair

Merge request reports