Finalize conversion for FKs targeting p_ci_builds - 3 self-managed
What does this MR do and why?
Completes the FK swap for p_ci_builds_metadata
and p_ci_runner_machine_builds
for self-managed. These migrations were executed on .com with:
-
!124310 (merged) skipped on CI database because of wraparound autovacuum on
ci_builds
-
!126122 (merged) -
ReplaceOldFkPCiBuildsMetadataToBuildsV2
failed to get the exclusive lock onci_builds
-
!126701 (merged) - successfully executed
ReplaceOldFkPCiBuildsMetadataToBuildsV2
instructions during the weekend.
They will be no-op on .com.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #414396 (closed)
Merge request reports
Activity
changed milestone to %16.2
assigned to @mbobin
- A deleted user
added database databasereview pending labels
2 Warnings New migrations added but db/structure.sql wasn't updated Usually, when adding new migrations, db/structure.sql should be
updated too (unless the migration isn't changing the DB schema
and isn't the most recent one).You've made some app changes, but didn't add any tests.
That's OK as long as you're refactoring existing code,
but please consider adding any of the maintenancepipelines, maintenancerefactor, maintenanceworkflow, documentation, QA labels.1 Message This merge request adds or changes files that require a review from the Database team. This merge request requires a database review. To make sure these changes are reviewed, take the following steps:
-
Ensure the merge request has database and databasereview pending labels. If the merge request modifies database files, Danger will do this for you.
-
Prepare your MR for database review according to the docs.
-
Assign and mention the database reviewer suggested by Reviewer Roulette.
-
Kick off the
db:gitlabcom-database-testing
manual job. This job can also be used before requesting review to test your migrations against production data.
The following files require a review from the Database team:
db/post_migrate/20230724085146_replace_old_fk_p_ci_builds_metadata_to_builds_v3.rb
db/post_migrate/20230724085149_replace_old_fk_p_ci_runner_machine_builds_to_builds_v3.rb
db/schema_migrations/20230724085146
db/schema_migrations/20230724085149
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer database Carla Drago (
@carlad-gl
) (UTC+2, 1 hour behind@mbobin
)Jon Jenkins (
@jon_jenkins
) (UTC-5, 8 hours behind@mbobin
)~"migration" No reviewer available No maintainer available To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger-
mentioned in issue #414396 (closed)
Allure report
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for fb5353c2expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Plan | 47 | 0 | 0 | 40 | 47 | ❗ | | Manage | 12 | 0 | 1 | 12 | 13 | ❗ | | Data Stores | 20 | 0 | 0 | 15 | 20 | ❗ | | Create | 19 | 0 | 0 | 18 | 19 | ❗ | | Govern | 19 | 0 | 0 | 15 | 19 | ❗ | | Verify | 8 | 0 | 0 | 8 | 8 | ❗ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 125 | 0 | 1 | 108 | 126 | ❗ | +-------------+--------+--------+---------+-------+-------+--------+
mentioned in epic &7522
- A deleted user
added Data WarehouseImpact Check label
added 2422 commits
-
a00fe643...180c25b3 - 2421 commits from branch
master
- 18a3edef - Finalize conversion for FKs targeting p_ci_builds - 3 self-managed
-
a00fe643...180c25b3 - 2421 commits from branch
changed milestone to %16.3
added missed:16.2 label
added 2603 commits
-
18a3edef...a794b256 - 2602 commits from branch
master
- fb5353c2 - Finalize conversion for FKs targeting p_ci_builds - 3 self-managed
-
18a3edef...a794b256 - 2602 commits from branch
marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed
added pipeline:skip-undercoverage label
requested review from @tianwenchen and @mattkasa
Started database testing pipeline (limited access). This comment will be updated once the pipeline has finished running.
- A deleted user
added database-testing-automation label
- Resolved by Jon Jenkins
Error: Database testing pipeline failureFailed pipeline ID: 2144169
The database testing pipeline has failed for pipeline ID 2144169. Please reach out in
#database
or to a database maintainer to help troubleshoot.Why aren't details of the failure posted here?
Because migration testing pipelines use production data, and this merge request is public. Production data may contain sensitive information, so we only expose this data in job artifacts and output from the pipeline itself.
Why don't I have access to this pipeline?
Because migration testing pipelines use production data, sensitive information could be leaked in job logs. As a result, the reports are limited to:
- Members of the database group
- Database maintainers
- GitLab team members with an approved access request for Reporter access to https://ops.gitlab.net/gitlab-com/database-team/gitlab-com-database-testing/.
Can I just re-run the pipeline?
Possibly yes, if the errors were caused by an incident (or services being slow) and re-running could fix it. If you're doing something slow, remember:
- Migration testing jobs time out after 10 hours. Synchronous work on a big table can reach this limit, and re-running will mean you wait another 10 hours for it.
- Migration sampling runs 30 minutes per background migration. If you add 20 background migrations, you will hit the timeout. Consider breaking multiple background migrations apart into separate merge requests.
If you think re-running might fix it, re-trigger the pipeline by running the manual job
db:gitlabcom-database-testing
.
mentioned in merge request mbobin/headway!1 (closed)
mentioned in merge request gitlab-org/ci-cd/ops-eng-managers/headway!1 (merged)
@mbobin thanks for the changes! LGTM!
@tianwenchen
, thanks for approving this merge request.This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.
For more info, please refer to the following links:
added pipeline:mr-approved label
removed review request for @tianwenchen
Database migrations (on the main database)
1 Warnings 20230724085146 - ReplaceOldFkPCiBuildsMetadataToBuildsV3 had a query that exceeded timing
guidelines. Run time should not exceed 100ms, but it was 418.28ms. Please consider possible options
to improve the query performance.SELECT $1 AS one
FROM "postgres_foreign_keys" WHERE "postgres_foreign_keys"."constrained_table_name" = $2 AND
"postgres_foreign_keys"."referenced_table_name" = $3 AND "postgres_foreign_keys"."name" = $4
LIMIT $5Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20230724085146 - ReplaceOldFkPCiBuildsMetadataToBuildsV3 Post deploy 3.7 s +0.00 B 20230724085149 - ReplaceOldFkPCiRunnerMachineBuildsToBuildsV3 Post deploy 1.9 s +0.00 B Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 5 0.1 seconds - 1 second 1 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230724085146 - ReplaceOldFkPCiBuildsMetadataToBuildsV3- Type: Post deploy
- Duration: 3.7 s
- Database size change: +0.00 B
Calls Total Time Max Time Mean Time Rows Query 1 418.3 ms 418.3 ms 418.3 ms 1 SELECT $1 AS one
FROM "postgres_foreign_keys" WHERE "postgres_foreign_keys"."constrained_table_name" = $2 AND "postgres_foreign_keys"."referenced_table_name" = $3 AND "postgres_foreign_keys"."name" = $4
LIMIT $52 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for ReplaceOldFkPCiBuildsMetadataToBuildsV3
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 2 0.1 seconds - 1 second 1 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230724085149 - ReplaceOldFkPCiRunnerMachineBuildsToBuildsV3
- Type: Post deploy
- Duration: 1.9 s
- Database size change: +0.00 B
Calls Total Time Max Time Mean Time Rows Query 1 11.8 ms 11.8 ms 11.8 ms 1 SELECT $1 AS one
FROM "postgres_foreign_keys" WHERE "postgres_foreign_keys"."constrained_table_name" = $2 AND "postgres_foreign_keys"."referenced_table_name" = $3 AND "postgres_foreign_keys"."name" = $4
LIMIT $52 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for ReplaceOldFkPCiRunnerMachineBuildsToBuildsV3
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 3 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Other information
No other migrations pending on GitLab.com
Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-2148504-10693712-main
2023-07-25T12:04:53Z 2023-07-25T08:12:04Z 2023-07-26 00:09:19 +0000 database-testing-2148504-10693712-ci
2023-07-25T12:04:53Z 2023-07-25T08:45:10Z 2023-07-26 00:09:19 +0000 Database migrations (on the ci database)
Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20230724085146 - ReplaceOldFkPCiBuildsMetadataToBuildsV3 Post deploy 3.2 s +0.00 B 20230724085149 - ReplaceOldFkPCiRunnerMachineBuildsToBuildsV3 Post deploy 2.8 s +0.00 B Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 5 0.1 seconds - 1 second 1 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230724085146 - ReplaceOldFkPCiBuildsMetadataToBuildsV3
- Type: Post deploy
- Duration: 3.2 s
- Database size change: +0.00 B
Calls Total Time Max Time Mean Time Rows Query 1 36.1 ms 36.1 ms 36.1 ms 1 SELECT $1 AS one
FROM "postgres_foreign_keys" WHERE "postgres_foreign_keys"."constrained_table_name" = $2 AND "postgres_foreign_keys"."referenced_table_name" = $3 AND "postgres_foreign_keys"."name" = $4
LIMIT $52 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for ReplaceOldFkPCiBuildsMetadataToBuildsV3
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 2 0.1 seconds - 1 second 1 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230724085149 - ReplaceOldFkPCiRunnerMachineBuildsToBuildsV3
- Type: Post deploy
- Duration: 2.8 s
- Database size change: +0.00 B
Calls Total Time Max Time Mean Time Rows Query 1 3.9 ms 3.9 ms 3.9 ms 1 SELECT $1 AS one
FROM "postgres_foreign_keys" WHERE "postgres_foreign_keys"."constrained_table_name" = $2 AND "postgres_foreign_keys"."referenced_table_name" = $3 AND "postgres_foreign_keys"."name" = $4
LIMIT $52 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for ReplaceOldFkPCiRunnerMachineBuildsToBuildsV3
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 3 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Other information
No other migrations pending on GitLab.com
Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-2148504-10693712-main
2023-07-25T12:04:53Z 2023-07-25T08:12:04Z 2023-07-26 00:09:19 +0000 database-testing-2148504-10693712-ci
2023-07-25T12:04:53Z 2023-07-25T08:45:10Z 2023-07-26 00:09:19 +0000
Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic
added 1 deleted label and removed workflowin dev label
- Resolved by Jon Jenkins
Makes sense to me, thanks @mbobin
added databasereviewed label and removed databasereview pending label
requested review from @jon_jenkins and removed review request for @mattkasa
added Data WarehouseNot Impacted label and removed Data WarehouseImpact Check label
added databaseapproved label and removed databasereviewed label
removed review request for @jon_jenkins
- Resolved by Jon Jenkins
@jon_jenkins, did you forget to run a pipeline before you merged this work? Based on our code review process, if the latest pipeline was created more than 6 hours ago OR finished more than 2 hours ago, you should:
- Ensure the merge request is not in Draft status.
- Start a pipeline (especially important for Community contribution merge requests).
- Set the merge request to auto-merge.
This is a guideline, not a rule. Please consider replying to this comment for transparency.
This message was generated automatically. You're welcome to improve it.
Hello @mbobin
The database team is looking for ways to improve the database review process and we would love your help!
If you'd be open to someone on the database team reaching out to you for a chat, or if you'd like to leave some feedback asynchronously, just post a reply to this comment mentioning:
@gitlab-org/database-team
And someone will be by shortly!
Thanks for your help!
This message was generated automatically. You're welcome to improve it.
mentioned in commit 065b7379
added workflowstaging-canary label and removed 1 deleted label
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
added workflowpost-deploy-db-staging label and removed workflowproduction label
added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label
added releasedcandidate label
added releasedpublished label and removed releasedcandidate label
mentioned in merge request kubitus-project/kubitus-installer!2364 (merged)