Add and backfill project_id for approval_merge_request_rules_approved_approvers
What does this MR do and why?
Add and backfill project_id for approval_merge_request_rules_approved_approvers.
This table has a desired sharding key configured (view configuration).
This merge request is the first step towards transforming the desired sharding key into a sharding key.
This involves three changes:
- Adding a new column that will serve as the sharding key (along with the relevant index and foreign key).
- Populating the sharding key when new records are created by adding a database function and trigger.
- Scheduling a batched background migration to set the sharding key for existing records.
Once the background migration has completed, a second merge request will be created to finalize the background migration and validate the not null constraint.
How to verify
We have assigned a random backend engineer from groupcode review to review these changes. Please review this merge request from a backend perspective. The main thing we are looking to verify is that the added column and association match the values specified by the desired sharding key configuration and that backfilling the column from this other table makes sense in the context of this feature.
When you are finished, please:
- Trigger the database testing pipeline as instructed by Danger.
- Request a review from the backend maintainer and database reviewer suggested by Danger.
If you have any questions or concerns, reach out to @tigerwnz
or @shubhamkrai.
This merge request was generated by a once off keep implemented in !143774
This change was generated by gitlab-housekeeper using the Keeps::BackfillDesiredShardingKeySmallTable keep.
To provide feedback on your experience with gitlab-housekeeper
please create an issue with the
label GitLab Housekeeper and consider pinging the author of this keep.
Merge request reports
Activity
added 1 commit
- 58d02d3a - Add and backfill project_id for approval_merge_request_rules_approved_ap
added devopscreate sectiondev labels
added pipelinetier-1 label
changed milestone to %17.10
assigned to @shubhamkrai
added backend databasereview pending labels
1 Warning ca4603b3: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines. 1 Message This merge request adds or changes files that require a review from the Database team. This merge request requires a database review. To make sure these changes are reviewed, take the following steps:
- Ensure the merge request has database and databasereview pending labels. If the merge request modifies database files, Danger will do this for you.
- Prepare your MR for database review according to the docs.
- Assign and mention the database reviewer suggested by Reviewer Roulette.
The following files require a review from the Database team:
db/docs/batched_background_migrations/backfill_approval_merge_request_rules_approved_approvers_project_id.yml
db/migrate/20250304131257_add_project_id_to_approval_merge_request_rules_approved_approvers.rb
db/post_migrate/20250304131258_index_approval_merge_request_rules_approved_approvers_on_project_id.rb
db/post_migrate/20250304131259_add_approval_merge_request_rules_approved_approvers_project_id_fk.rb
db/post_migrate/20250304131260_add_approval_merge_request_rules_approved_approvers_project_id_trigger.rb
db/post_migrate/20250304131261_queue_backfill_approval_merge_request_rules_approved_approvers_project_id.rb
db/schema_migrations/20250304131257
db/schema_migrations/20250304131258
db/schema_migrations/20250304131259
db/schema_migrations/20250304131260
db/schema_migrations/20250304131261
lib/gitlab/background_migration/backfill_approval_merge_request_rules_approved_approvers_project_id.rb
spec/migrations/20250304131261_queue_backfill_approval_merge_request_rules_approved_approvers_project_id_spec.rb
db/docs/approval_merge_request_rules_approved_approvers.yml
db/structure.sql
Reviewer roulette
Category Reviewer Maintainer backend @squadri
(UTC+0, 1 hour behind author)
@aakriti.gupta
(UTC+1, same timezone as author)
database @bmarjanovic
(UTC+1, same timezone as author)
@jon_jenkins
(UTC-4, 5 hours behind author)
Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
DangerEdited by ****added Data WarehouseImpact Check label
Database migrations (on the main database)
1 Warnings 20250304131259 - AddApprovalMergeRequestRulesApprovedApproversProjectIdFk had a query that exceeded
timing guidelines. Run time should not exceed 100ms, but it was 106.51ms. Please consider possible
options to improve the query performance.ALTER TABLE
approval_merge_request_rules_approved_approvers VALIDATE CONSTRAINT fk_8dfb93b836Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20250304131257 - AddProjectIdToApprovalMergeRequestRulesApprovedApprovers Regular 4.3 s +0.00 B 20250304131258 - IndexApprovalMergeRequestRulesApprovedApproversOnProjectId Post deploy 121.4 s +646.57 MiB 20250304131259 - AddApprovalMergeRequestRulesApprovedApproversProjectIdFk Post deploy 5.4 s +0.00 B 20250304131260 - AddApprovalMergeRequestRulesApprovedApproversProjectIdTrigger Post deploy 4.5 s +0.00 B 20250304131261 - QueueBackfillApprovalMergeRequestRulesApprovedApproversProjectId Post deploy 5.0 s +0.00 B Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 18 0.1 seconds - 1 second 5 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 1 5 minutes + 0 Migration: 20250304131257 - AddProjectIdToApprovalMergeRequestRulesApprovedApprovers
* Duration: 4.3 s
* Database size change: +0.00 BCalls Total Time Max Time Mean Time Rows Query 1 3.0 ms 3.0 ms 3.0 ms 0 ALTER TABLE "approval_merge_request_rules_approved_approvers" ADD "project_id" bigint
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddProjectIdToApprovalMergeRequestRulesApprovedApprovers
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 3 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20250304131258 - IndexApprovalMergeRequestRulesApprovedApproversOnProjectId
* Duration: 121.4 s
* Database size change: +646.57 MiBCalls Total Time Max Time Mean Time Rows Query 1 116250.3 ms 116250.3 ms 116250.3 ms 0 CREATE INDEX CONCURRENTLY "idx_approval_merge_request_rules_approved_approvers_project_id" ON "approval_merge_request_rules_approved_approvers" ("project_id")
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for IndexApprovalMergeRequestRulesApprovedApproversOnProjectId
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 2 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 1 5 minutes + 0 Migration: 20250304131259 - AddApprovalMergeRequestRulesApprovedApproversProjectIdFk
* Duration: 5.4 s
* Database size change: +0.00 BCalls Total Time Max Time Mean Time Rows Query 1 106.5 ms 106.5 ms 106.5 ms 0 ALTER TABLE approval_merge_request_rules_approved_approvers VALIDATE CONSTRAINT fk_8dfb93b836
1 13.2 ms 13.2 ms 13.2 ms 0 ALTER TABLE approval_merge_request_rules_approved_approvers ADD CONSTRAINT fk_8dfb93b836 FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE NOT VALID
1 0.6 ms 0.6 ms 0.6 ms 0 SELECT $1 AS one
FROM "postgres_foreign_keys" WHERE "postgres_foreign_keys"."constrained_table_name" = $2 AND "postgres_foreign_keys"."referenced_table_name" = $3 AND "postgres_foreign_keys"."name" = $4 AND "postgres_foreign_keys"."constrained_columns" = $5 AND "postgres_foreign_keys"."referenced_columns" = $6 AND "postgres_foreign_keys"."on_delete_action" = $7
LIMIT $82 0.5 ms 0.3 ms 0.2 ms 0 SELECT "postgres_partitioned_tables".*
FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), $1, $2))
LIMIT $32 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddApprovalMergeRequestRulesApprovedApproversProjectIdFk
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 4 0.1 seconds - 1 second 3 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20250304131260 - AddApprovalMergeRequestRulesApprovedApproversProjectIdTrigger
* Duration: 4.5 s
* Database size change: +0.00 BCalls Total Time Max Time Mean Time Rows Query 1 1.0 ms 1.0 ms 1.0 ms 0 CREATE OR REPLACE FUNCTION trigger_5ed68c226e97() RETURNS TRIGGER AS $1 LANGUAGE PLPGSQL
1 0.3 ms 0.3 ms 0.3 ms 0 CREATE TRIGGER trigger_5ed68c226e97 BEFORE INSERT OR UPDATE ON "approval_merge_request_rules_approved_approvers" FOR EACH ROW EXECUTE FUNCTION trigger_5ed68c226e97()
1 0.1 ms 0.1 ms 0.1 ms 0 DROP TRIGGER IF EXISTS trigger_5ed68c226e97 ON "approval_merge_request_rules_approved_approvers"
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddApprovalMergeRequestRulesApprovedApproversProjectIdTrigger
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 5 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20250304131261 - QueueBackfillApprovalMergeRequestRulesApprovedApproversProjectId
* Duration: 5.0 s
* Database size change: +0.00 BCalls Total Time Max Time Mean Time Rows Query 1 6.5 ms 6.5 ms 6.5 ms 1 SELECT MAX("id")
FROM "approval_merge_request_rules_approved_approvers"1 1.0 ms 1.0 ms 1.0 ms 1 INSERT INTO "batched_background_migrations" ("created_at", "updated_at", "max_value", "batch_size", "sub_batch_size", "interval", "status", "job_class_name", "table_name", "column_name", "job_arguments", "total_tuple_count", "started_at", "gitlab_schema", "queued_migration_version") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15) RETURNING "id"
1 0.7 ms 0.7 ms 0.7 ms 0 SELECT $1 AS one
FROM "batched_background_migrations" WHERE "batched_background_migrations"."job_class_name" = $2 AND "batched_background_migrations"."table_name" = $3 AND "batched_background_migrations"."column_name" = $4 AND (job_arguments = $5) AND (ARRAY[$6] <@ ARRAY[$7, $8, $9, $10, $11, $12, $13]) AND "batched_background_migrations"."gitlab_schema" IN ($14, $15, $16, $17, $18, $19, $20)
LIMIT $211 0.0 ms 0.0 ms 0.0 ms 0 SELECT $1 AS one
FROM "batched_background_migrations" WHERE "batched_background_migrations"."job_arguments" = $2 AND "batched_background_migrations"."job_class_name" = $3 AND "batched_background_migrations"."table_name" = $4 AND "batched_background_migrations"."column_name" = $5
LIMIT $62 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for QueueBackfillApprovalMergeRequestRulesApprovedApproversProjectId
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 4 0.1 seconds - 1 second 2 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Other information
Other migrations pending on GitLab.com
Migration Type Total runtime Result DB size change 20250227054952 - AddRedirectRoutesNamespaceId Regular 5.3 s +0.00 B 20250228142819 - AddNamespaceIdToIncidentManagementPendingIssueEscalations Regular 4.4 s +0.00 B 20250227091623 - IndexRedirectRoutesOnNamespaceId Post deploy 19.0 s +34.84 MiB 20250227092023 - AddRedirectRoutesNamespaceIdFk Post deploy 5.2 s +0.00 B 20250228142820 - IndexIncidentManagementPendingIssueEscalationsOnNamespaceId Post deploy 12.1 s +80.00 KiB 20250228142821 - AddIncidentManagementPendingIssueEscalationsNamespaceIdFk Post deploy 15.5 s +16.00 KiB 20250228142822 - AddIncidentManagementPendingIssueEscalationsNamespaceIdTrigger Post deploy 4.5 s +8.00 KiB [note] 20250228142823 - QueueBackfillIncidentManagementPendingIssueEscalationsNamespaceId Post deploy 4.8 s +0.00 B 20250228143727 - RequeueSyncUnlinkedSecurityPolicyProjectLinks Post deploy 5.7 s +0.00 B 20250228155146 - QueueBackfillOnboardingStatusSetupForCompany Post deploy 5.0 s +0.00 B 20250303122513 - RemoveBrokenFkOnCiBuildsAndCiPipelines Post deploy 4.9 s +0.00 B 20250303231437 - FinalizeHkBackfillProtectedEnvironmentApprovalRulesProtectedEnvir47878 Post deploy 5.1 s +0.00 B 20250304204818 - DeleteIndexTodosOnAuthorIdAndCreatedAtIndex Post deploy 6.0 s -3.35 GiB 20250304231245 - IncreaseTokenEncryptedConstraint Post deploy 5.5 s +0.00 B 20250305084646 - IndexScanResultPoliciesOnApprovalPolicyRuleId Post deploy 9.6 s +3.77 MiB Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-4261749-17517174-main
2025-03-06T15:32:28Z 2025-03-06T11:59:42Z 2025-03-07 03:51:23 +0000 database-testing-4261749-17517174-ci
2025-03-06T15:32:28Z 2025-03-06T12:45:05Z 2025-03-07 03:51:23 +0000 Database migrations (on the ci database)
Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20250304131257 - AddProjectIdToApprovalMergeRequestRulesApprovedApprovers Regular 7.2 s +0.00 B 20250304131258 - IndexApprovalMergeRequestRulesApprovedApproversOnProjectId Post deploy 8.1 s +8.00 KiB [note] 20250304131259 - AddApprovalMergeRequestRulesApprovedApproversProjectIdFk Post deploy 8.2 s +0.00 B 20250304131260 - AddApprovalMergeRequestRulesApprovedApproversProjectIdTrigger Post deploy 7.4 s +0.00 B 20250304131261 - QueueBackfillApprovalMergeRequestRulesApprovedApproversProjectId Post deploy 7.1 s +0.00 B Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 0 0.1 seconds - 1 second 20 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20250304131257 - AddProjectIdToApprovalMergeRequestRulesApprovedApprovers
* Duration: 7.2 s
* Database size change: +0.00 BCalls Total Time Max Time Mean Time Rows Query 1 2.8 ms 2.8 ms 2.8 ms 0 ALTER TABLE "approval_merge_request_rules_approved_approvers" ADD "project_id" bigint
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddProjectIdToApprovalMergeRequestRulesApprovedApprovers
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 0 0.1 seconds - 1 second 3 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20250304131258 - IndexApprovalMergeRequestRulesApprovedApproversOnProjectId
* Duration: 8.1 s
* Database size change: +8.00 KiB [note]Calls Total Time Max Time Mean Time Rows Query 1 8.4 ms 8.4 ms 8.4 ms 0 CREATE INDEX CONCURRENTLY "idx_approval_merge_request_rules_approved_approvers_project_id" ON "approval_merge_request_rules_approved_approvers" ("project_id")
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for IndexApprovalMergeRequestRulesApprovedApproversOnProjectId
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 0 0.1 seconds - 1 second 3 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20250304131259 - AddApprovalMergeRequestRulesApprovedApproversProjectIdFk
* Duration: 8.2 s
* Database size change: +0.00 BCalls Total Time Max Time Mean Time Rows Query 1 11.7 ms 11.7 ms 11.7 ms 0 ALTER TABLE approval_merge_request_rules_approved_approvers VALIDATE CONSTRAINT fk_8dfb93b836
1 6.3 ms 6.3 ms 6.3 ms 0 ALTER TABLE approval_merge_request_rules_approved_approvers ADD CONSTRAINT fk_8dfb93b836 FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE NOT VALID
1 0.7 ms 0.7 ms 0.7 ms 0 SELECT $1 AS one
FROM "postgres_foreign_keys" WHERE "postgres_foreign_keys"."constrained_table_name" = $2 AND "postgres_foreign_keys"."referenced_table_name" = $3 AND "postgres_foreign_keys"."name" = $4 AND "postgres_foreign_keys"."constrained_columns" = $5 AND "postgres_foreign_keys"."referenced_columns" = $6 AND "postgres_foreign_keys"."on_delete_action" = $7
LIMIT $82 0.4 ms 0.2 ms 0.2 ms 0 SELECT "postgres_partitioned_tables".*
FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), $1, $2))
LIMIT $32 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddApprovalMergeRequestRulesApprovedApproversProjectIdFk
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 0 0.1 seconds - 1 second 7 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20250304131260 - AddApprovalMergeRequestRulesApprovedApproversProjectIdTrigger
* Duration: 7.4 s
* Database size change: +0.00 BCalls Total Time Max Time Mean Time Rows Query 1 1.0 ms 1.0 ms 1.0 ms 0 CREATE OR REPLACE FUNCTION trigger_5ed68c226e97() RETURNS TRIGGER AS $1 LANGUAGE PLPGSQL
1 0.2 ms 0.2 ms 0.2 ms 0 CREATE TRIGGER trigger_5ed68c226e97 BEFORE INSERT OR UPDATE ON "approval_merge_request_rules_approved_approvers" FOR EACH ROW EXECUTE FUNCTION trigger_5ed68c226e97()
1 0.1 ms 0.1 ms 0.1 ms 0 DROP TRIGGER IF EXISTS trigger_5ed68c226e97 ON "approval_merge_request_rules_approved_approvers"
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddApprovalMergeRequestRulesApprovedApproversProjectIdTrigger
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 0 0.1 seconds - 1 second 5 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20250304131261 - QueueBackfillApprovalMergeRequestRulesApprovedApproversProjectId
* Duration: 7.1 s
* Database size change: +0.00 BCalls Total Time Max Time Mean Time Rows Query 2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for QueueBackfillApprovalMergeRequestRulesApprovedApproversProjectId
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 0 0.1 seconds - 1 second 2 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Other information
Other migrations pending on GitLab.com
Migration Type Total runtime Result DB size change 20250227054952 - AddRedirectRoutesNamespaceId Regular 8.3 s +0.00 B 20250228142819 - AddNamespaceIdToIncidentManagementPendingIssueEscalations Regular 7.2 s +0.00 B 20250227091623 - IndexRedirectRoutesOnNamespaceId Post deploy 8.5 s +8.00 KiB [note] 20250227092023 - AddRedirectRoutesNamespaceIdFk Post deploy 8.3 s +0.00 B 20250228142820 - IndexIncidentManagementPendingIssueEscalationsOnNamespaceId Post deploy 16.4 s +72.00 KiB 20250228142821 - AddIncidentManagementPendingIssueEscalationsNamespaceIdFk Post deploy 20.8 s +32.00 KiB 20250228142822 - AddIncidentManagementPendingIssueEscalationsNamespaceIdTrigger Post deploy 7.5 s +0.00 B 20250228142823 - QueueBackfillIncidentManagementPendingIssueEscalationsNamespaceId Post deploy 7.1 s +0.00 B 20250228143727 - RequeueSyncUnlinkedSecurityPolicyProjectLinks Post deploy 7.1 s +0.00 B 20250228155146 - QueueBackfillOnboardingStatusSetupForCompany Post deploy 7.1 s +0.00 B 20250303122513 - RemoveBrokenFkOnCiBuildsAndCiPipelines Post deploy 8.0 s +0.00 B 20250303231437 - FinalizeHkBackfillProtectedEnvironmentApprovalRulesProtectedEnvir47878 Post deploy 6.8 s +0.00 B 20250304204818 - DeleteIndexTodosOnAuthorIdAndCreatedAtIndex Post deploy 9.1 s -8.00 KiB 20250304231245 - IncreaseTokenEncryptedConstraint Post deploy 12.5 s +0.00 B 20250305084646 - IndexScanResultPoliciesOnApprovalPolicyRuleId Post deploy 8.3 s +8.00 KiB [note] Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-4261749-17517174-main
2025-03-06T15:32:28Z 2025-03-06T11:59:42Z 2025-03-07 03:51:23 +0000 database-testing-4261749-17517174-ci
2025-03-06T15:32:28Z 2025-03-06T12:45:05Z 2025-03-07 03:51:23 +0000
Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic
Edited by ****- Resolved by Shubham Kumar
- Resolved by Shubham Kumar
mentioned in issue #493768
added database-testing-automation label
Migration Failure Notice
The following migrations have failed. Please see the latest Database Testing pipeline message for further details. This message must be resolved before the MR can be merged.
Failed Database Migrations
Database Migration Name main IndexApprovalMergeRequestRulesApprovedApproversOnProjectId ci IndexApprovalMergeRequestRulesApprovedApproversOnProjectId