Skip to content

Scope epics internal_ids generation to `issues` usage

Alexandru Croitor requested to merge change_epic_internal_ids_usage_to_issues into master

What does this MR do and why?

We work on simlessly migrating Epics to WorkItems. Because we currently have no work items available at group level we will actually migrate Epics as the first group level WorkItem and as an added bonus we get to keep Epic's IIDs for the group level Epic WorkItem IID. Because links to epics use IID, this would allow us to maintain compatibility between old and new links with a simple redirect as the IID lookup would be the same.

In order to achieve that and unblock creation of other group level WorkItems, we want to make sure that IID allocation for Epics is now using the issues usage in inteternal_ids table and sync up internal_ids issues usage for groups the epics internal_ids usage.

In order for Epic model to allocate IID based on issues usage, we need to override the internal_id_scope_usage method to return issues. This however creates a challenge in rolling deployment environments where some app nodes can run on old code while new nodes would run on new code, thus creating a situation where Epic model with "old code" would try to allocate IID by looking up a usage: :epics record in internal_ids while "new code" would lookup usage: :issues record in internal_ids which can lead to duplicate IID allocation and application errors.

To solve that we are adding a couple triggers to run on insert/update to keep the last_value column for the records for the two usages issues and epics in sync. These triggers are only needed for the rolling deployment duration when we can have 2 versions of the code running. Once the new code is deployed Epic model will always allocate IID by using the issues usage.

Screenshots or screen recordings

  • AddTemporaryIndexInternalIdsOnIdAndUsage
[3] pry(main)> AddTemporaryIndexInternalIdsOnIdAndUsage.new.up
main: -- transaction_open?(nil)
main:    -> 0.0009s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.1855s
  Gitlab::Database::PostgresPartition Exists? (197.8ms)  SELECT 1 AS one FROM "postgres_partitions" WHERE (identifier = concat(current_schema(), '.', 'internal_ids')) LIMIT 1 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/lib/gitlab/database/postgres_partition.rb:33:in `partition_exists?'*/
main: -- index_exists?(:internal_ids, [:id, :usage], {:name=>"tmp_index_internal_ids_on_id_and_usage", :where=>"usage = 4", :algorithm=>:concurrently})
main:    -> 0.9449s
   (186.5ms)  SHOW statement_timeout /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/lib/gitlab/database/migrations/timeout_helpers.rb:56:in `statement_timeout_disabled?'*/
main: -- execute("SET statement_timeout TO 0")
   (188.3ms)  SET statement_timeout TO 0 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/lib/gitlab/database/migrations/timeout_helpers.rb:29:in `disable_statement_timeout'*/
main:    -> 0.1890s
main: -- add_index(:internal_ids, [:id, :usage], {:name=>"tmp_index_internal_ids_on_id_and_usage", :where=>"usage = 4", :algorithm=>:concurrently})
   (44189.6ms)  CREATE INDEX CONCURRENTLY "tmp_index_internal_ids_on_id_and_usage" ON "internal_ids" ("id", "usage") WHERE usage = 4 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/lib/gitlab/database/migration_helpers.rb:122:in `block in add_concurrent_index'*/
main:    -> 44.1908s
main: -- execute("RESET statement_timeout")
   (188.7ms)  RESET statement_timeout /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/lib/gitlab/database/migrations/timeout_helpers.rb:33:in `disable_statement_timeout'*/
main:    -> 0.1905s
  Gitlab::Database::AsyncIndexes::PostgresAsyncIndex Load (218.3ms)  SELECT "postgres_async_indexes".* FROM "postgres_async_indexes" WHERE "postgres_async_indexes"."name" = 'tmp_index_internal_ids_on_id_and_usage' LIMIT 1 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/lib/gitlab/database/async_indexes/migration_helpers.rb:24:in `unprepare_async_index_by_name'*/
=> nil
[4] pry(main)> load 'db/post_migrate/20231208145335_cleanup_group_level_work_items.rb'
=> true
  • BackfillInternalIdsWithIssuesUsageForEpics
[7] pry(main)> BackfillInternalIdsWithIssuesUsageForEpics.new.up
  BackfillInternalIdsWithIssuesUsageForEpics::MigrationInternalId Load (196.6ms)  SELECT "internal_ids"."id" FROM "internal_ids" WHERE "internal_ids"."usage" = 4 ORDER BY "internal_ids"."id" ASC LIMIT 1 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/app/models/concerns/each_batch.rb:62:in `each_batch'*/
  BackfillInternalIdsWithIssuesUsageForEpics::MigrationInternalId Load (370.8ms)  SELECT "internal_ids"."id" FROM "internal_ids" WHERE "internal_ids"."usage" = 4 AND "internal_ids"."id" >= 50688 ORDER BY "internal_ids"."id" ASC LIMIT 1 OFFSET 1000 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/app/models/concerns/each_batch.rb:81:in `block in each_batch'*/
   (7527.9ms)  INSERT INTO internal_ids (usage, last_value, namespace_id)
  SELECT 0, last_value, namespace_id
  FROM internal_ids
  WHERE internal_ids.id IN(SELECT "internal_ids"."id" FROM "internal_ids" WHERE "internal_ids"."usage" = 4 AND "internal_ids"."id" >= 50688 AND "internal_ids"."id" < 2596678)
ON CONFLICT (usage, namespace_id) WHERE namespace_id IS NOT NULL
DO UPDATE SET last_value = GREATEST(EXCLUDED.last_value, internal_ids.last_value)
RETURNING id;
 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:db/post_migrate/20231208185335_backfill_internal_ids_with_issues_usage_for_epics.rb:27:in `block in up'*/
  BackfillInternalIdsWithIssuesUsageForEpics::MigrationInternalId Delete All (246.1ms)  DELETE FROM "internal_ids" WHERE "internal_ids"."usage" = 4 AND "internal_ids"."id" >= 50688 AND "internal_ids"."id" < 2596678 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:db/post_migrate/20231208185335_backfill_internal_ids_with_issues_usage_for_epics.rb:39:in `block in up'*/
  BackfillInternalIdsWithIssuesUsageForEpics::MigrationInternalId Load (409.0ms)  SELECT "internal_ids"."id" FROM "internal_ids" WHERE "internal_ids"."usage" = 4 AND "internal_ids"."id" >= 2596678 ORDER BY "internal_ids"."id" ASC LIMIT 1 OFFSET 1000 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/app/models/concerns/each_batch.rb:81:in `block in each_batch'*/
   (6932.7ms)  INSERT INTO internal_ids (usage, last_value, namespace_id)
  SELECT 0, last_value, namespace_id
  FROM internal_ids
  WHERE internal_ids.id IN(SELECT "internal_ids"."id" FROM "internal_ids" WHERE "internal_ids"."usage" = 4 AND "internal_ids"."id" >= 2596678 AND "internal_ids"."id" < 3977376)
ON CONFLICT (usage, namespace_id) WHERE namespace_id IS NOT NULL
DO UPDATE SET last_value = GREATEST(EXCLUDED.last_value, internal_ids.last_value)
RETURNING id;
 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:db/post_migrate/20231208185335_backfill_internal_ids_with_issues_usage_for_epics.rb:27:in `block in up'*/
  BackfillInternalIdsWithIssuesUsageForEpics::MigrationInternalId Delete All (208.7ms)  DELETE FROM "internal_ids" WHERE "internal_ids"."usage" = 4 AND "internal_ids"."id" >= 2596678 AND "internal_ids"."id" < 3977376 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:db/post_migrate/20231208185335_backfill_internal_ids_with_issues_usage_for_epics.rb:39:in `block in up'*/
[9] pry(main)> RemoveInternalIdsTmpIndex.new.up
main: -- transaction_open?(nil)
main:    -> 0.0003s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.1852s
  Gitlab::Database::PostgresPartition Exists? (200.3ms)  SELECT 1 AS one FROM "postgres_partitions" WHERE (identifier = concat(current_schema(), '.', 'internal_ids')) LIMIT 1 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/lib/gitlab/database/postgres_partition.rb:33:in `partition_exists?'*/
main: -- indexes(:internal_ids)
main:    -> 1.1122s
   (183.1ms)  SHOW statement_timeout /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/lib/gitlab/database/migrations/timeout_helpers.rb:56:in `statement_timeout_disabled?'*/
main: -- remove_index(:internal_ids, {:algorithm=>:concurrently, :name=>"tmp_index_internal_ids_on_id_and_usage"})
   (212.0ms)  DROP INDEX CONCURRENTLY "tmp_index_internal_ids_on_id_and_usage" /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/lib/gitlab/database/migration_helpers.rb:213:in `block in remove_concurrent_index_by_name'*/
main:    -> 0.2133s
  Gitlab::Database::AsyncIndexes::PostgresAsyncIndex Load (185.0ms)  SELECT "postgres_async_indexes".* FROM "postgres_async_indexes" WHERE "postgres_async_indexes"."name" = 'tmp_index_internal_ids_on_id_and_usage' LIMIT 1 /*application:console,db_config_name:main,console_hostname:GTLB-Alexandru.local,console_username:acroitor,line:/lib/gitlab/database/async_indexes/migration_helpers.rb:24:in `unprepare_async_index_by_name'*/
=> nil

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Alexandru Croitor

Merge request reports