Skip to content

Migration for fixing missing traversal ids

Adam Hegyi requested to merge fix-missing-traversal-ids into master

What does this MR do?

The first background migration job to backfill namespaces.traversal_ids did not cover all namespaces. This is most likely due to a misunderstanding we had between the relationship of chatops and default feature flag enablement.

This MR schedules the same background migrations as from the first background migration. Namely Gitlab::BackgroundMigration::BackfillNamespaceTraversalIdsRoots and Gitlab::BackgroundMigration::BackfillNamespaceTraversalIdsChildren.

Database

  • Index creation took only a few minutes (both indexes). The indexes will contain about 100K namespaces.
  • Follow-up for removing the indexes: #333365 (closed)

UP:

== 20210611080951 FixMissingTraversalIds: migrating ===========================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:namespaces, :id, {:where=>"parent_id IS NULL AND traversal_ids = '{}'", :name=>"tmp_index_namespaces_empty_traversal_ids_with_root_namespaces", :algorithm=>:concurrently})
   -> 0.0091s
-- add_index(:namespaces, :id, {:where=>"parent_id IS NULL AND traversal_ids = '{}'", :name=>"tmp_index_namespaces_empty_traversal_ids_with_root_namespaces", :algorithm=>:concurrently})
   -> 0.0113s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:namespaces, :id, {:where=>"parent_id IS NOT NULL AND traversal_ids = '{}'", :name=>"tmp_index_namespaces_empty_traversal_ids_with_child_namespaces", :algorithm=>:concurrently})
   -> 0.0260s
-- add_index(:namespaces, :id, {:where=>"parent_id IS NOT NULL AND traversal_ids = '{}'", :name=>"tmp_index_namespaces_empty_traversal_ids_with_child_namespaces", :algorithm=>:concurrently})
   -> 0.0078s
-- Scheduled 1 BackfillNamespaceTraversalIdsRoots jobs with a maximum of 1000 records per batch and an interval of 120 seconds.

The migration is expected to take at least 120 seconds. Expect all jobs to have completed after 2021-06-11 08:21:25 UTC."
-- Scheduled 1 BackfillNamespaceTraversalIdsChildren jobs with a maximum of 1000 records per batch and an interval of 120 seconds.

The migration is expected to take at least 360 seconds. Expect all jobs to have completed after 2021-06-11 08:25:25 UTC."
== 20210611080951 FixMissingTraversalIds: migrated (0.1908s) ==================

DOWN:

== 20210611080951 FixMissingTraversalIds: reverting ===========================
-- transaction_open?()
   -> 0.0000s
-- indexes(:namespaces)
   -> 0.0117s
-- execute("SET statement_timeout TO 0")
   -> 0.0012s
-- remove_index(:namespaces, {:algorithm=>:concurrently, :name=>"tmp_index_namespaces_empty_traversal_ids_with_root_namespaces"})
   -> 0.0036s
-- execute("RESET ALL")
   -> 0.0006s
-- transaction_open?()
   -> 0.0000s
-- indexes(:namespaces)
   -> 0.0093s
-- remove_index(:namespaces, {:algorithm=>:concurrently, :name=>"tmp_index_namespaces_empty_traversal_ids_with_child_namespaces"})
   -> 0.0028s
== 20210611080951 FixMissingTraversalIds: reverted (0.0317s) ==================

Queries:

2 standard EachBatch queries are covered by index.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Adam Hegyi

Merge request reports