Create iid sequence for ci_pipelines with new projects
What does this MR do and why?
Create iid sequence for ci_pipelines with new projects
The initialization of the sequence is not atomic and due to race conditions some pipelines could reinitialize multiple times, resulting in attempts to insert rows with duplicate values. This fix moves the initialization from the first pipeline creation into the project creation, so the sequence will be already initialized when the first pipelines are created.
Changelog: fixed
Screenshots or screen recordings
Before
When the first pipeline is created, we have an update followed by an insert to generate the sequence. On edge cases, the first pipeline will not be one pipeline, but many created concurrently from Sidekiq. This can initialize the sequence more than once.
[11] pry(main)> FactoryBot.create(:ci_pipeline, project: project)
Update InternalId (0.7ms) UPDATE "internal_ids" SET "last_value" = ("internal_ids"."last_value" + 1) WHERE "internal_ids"."project_id" = 231 AND "internal_ids"."usage" = 5 RETURNING "last_value" /*application:console,db_config_name:main,console_hostname:rocket-sled.local,console_username:marius,line:/app/models/internal_id.rb:164:in `update_record!'*/
Ci::Pipeline Maximum (2.5ms) SELECT MAX("ci_pipelines"."iid") FROM "ci_pipelines" WHERE "ci_pipelines"."project_id" = 231 /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:/app/models/ci/pipeline.rb:66:in `block in <class:Pipeline>'*/
Ci::Pipeline Count (0.2ms) SELECT COUNT(*) FROM "ci_pipelines" WHERE "ci_pipelines"."project_id" = 231 /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:/app/models/ci/pipeline.rb:66:in `block in <class:Pipeline>'*/
InternalId Insert (2.0ms) INSERT INTO "internal_ids" ("project_id","namespace_id","usage","last_value") VALUES (231, NULL, 5, 1) ON CONFLICT DO NOTHING RETURNING "id" /*application:console,db_config_name:main,console_hostname:rocket-sled.local,console_username:marius,line:/app/models/internal_id.rb:178:in `create_record!'*/
Ci::Ref Load (0.5ms) SELECT "ci_refs".* FROM "ci_refs" WHERE "ci_refs"."project_id" = 231 AND "ci_refs"."ref_path" = 'refs/heads/master' LIMIT 1 /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:/app/models/application_record.rb:79:in `safe_find_or_create_by'*/
TRANSACTION (0.0ms) BEGIN /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:/lib/gitlab/database/schema_cache_with_renamed_table.rb:21:in `primary_keys'*/
Ci::Ref Create (0.4ms) INSERT INTO "ci_refs" ("project_id", "ref_path", "lock_version") VALUES (231, 'refs/heads/master', 0) RETURNING "id" /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:/app/models/application_record.rb:87:in `block in safe_find_or_create_by'*/
TRANSACTION (0.1ms) COMMIT /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:/lib/gitlab/database.rb:423:in `commit'*/
TRANSACTION (0.1ms) BEGIN /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:<internal:kernel>:90:in `tap'*/
Ci::Pipeline Create (1.7ms) INSERT INTO "ci_pipelines" ("ref", "sha", "created_at", "updated_at", "project_id", "status", "source", "protected", "iid", "ci_ref_id", "partition_id", "lock_version") VALUES ('master', 'b83d6e391c22777fca1ed3012fce84f633d7fed0', '2023-08-30 13:58:30.060043', '2023-08-30 13:58:30.060043', 231, 'pending', 1, FALSE, 1, 244, 100, 0) RETURNING "id" /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:<internal:kernel>:90:in `tap'*/
TRANSACTION (0.1ms) COMMIT
After
The internal_ids
record is created right after the project gets created, so the first pipelines will get the iid
value from the increment
call, without needing to initialize the sequence:
[18] pry(main)> FactoryBot.create(:ci_pipeline, project: project)
Update InternalId (0.7ms) UPDATE "internal_ids" SET "last_value" = ("internal_ids"."last_value" + 1) WHERE "internal_ids"."project_id" = 230 AND "internal_ids"."usage" = 5 RETURNING "last_value" /*application:console,db_config_name:main,console_hostname:rocket-sled.local,console_username:marius,line:/app/models/internal_id.rb:166:in `update_record!'*/
Ci::Ref Load (0.4ms) SELECT "ci_refs".* FROM "ci_refs" WHERE "ci_refs"."project_id" = 230 AND "ci_refs"."ref_path" = 'refs/heads/master' LIMIT 1 /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:/app/models/application_record.rb:79:in `safe_find_or_create_by'*/
TRANSACTION (0.2ms) BEGIN /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:/app/models/application_record.rb:87:in `block in safe_find_or_create_by'*/
Ci::Ref Create (0.5ms) INSERT INTO "ci_refs" ("project_id", "ref_path", "lock_version") VALUES (230, 'refs/heads/master', 0) RETURNING "id" /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:/app/models/application_record.rb:87:in `block in safe_find_or_create_by'*/
TRANSACTION (0.2ms) COMMIT /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:/lib/gitlab/database.rb:423:in `commit'*/
TRANSACTION (0.1ms) BEGIN /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:<internal:kernel>:90:in `tap'*/
Ci::Pipeline Create (1.3ms) INSERT INTO "ci_pipelines" ("ref", "sha", "created_at", "updated_at", "project_id", "status", "source", "protected", "iid", "ci_ref_id", "partition_id", "lock_version") VALUES ('master', 'b83d6e391c22777fca1ed3012fce84f633d7fed0', '2023-08-30 13:52:14.803045', '2023-08-30 13:52:14.803045', 230, 'pending', 1, FALSE, 1, 243, 100, 0) RETURNING "id" /*application:console,db_config_name:ci,console_hostname:rocket-sled.local,console_username:marius,line:<internal:kernel>:90:in `tap'*/
TRANSACTION (0.2ms) COMMIT
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #423421 (closed)
Merge request reports
Activity
changed milestone to %16.4
assigned to @mbobin
- A deleted user
added backend label
1 Warning There were no new or modified feature flag YAML files detected in this MR. If the changes here are already controlled under an existing feature flag, please add
the feature flagexists. Otherwise, if you think the changes here don't need
to be under a feature flag, please add the label feature flagskipped, and
add a short comment about why we skipped the feature flag.For guidance on when to use a feature flag, please see the documentation.
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend Rad Batnag (
@radbatnag
) (UTC+8, 5 hours ahead of@mbobin
)Ethan Urie (
@eurie
) (UTC-4, 7 hours behind@mbobin
)~"Verify" Reviewer review is optional for ~"Verify" Stan Hu (
@stanhu
) (UTC-7, 10 hours behind@mbobin
)Please check reviewer's status!
Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
Sidekiq queue changes
This merge request contains changes to Sidekiq queues. Please follow the documentation on changing a queue's urgency.
These queues were added:
ci_initialize_pipelines_iid_sequence
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger@mbobin Some end-to-end (E2E) tests should run based on the stage label.
Please start the
trigger-omnibus-and-follow-up-e2e
job in theqa
stage and ensure tests in thefollow-up-e2e:package-and-test-ee
pipeline pass before this MR is merged. (E2E tests are computationally intensive and don't run automatically for every push/rebase, so we ask you to run this job manually at least once.)To run all E2E tests, apply the pipeline:run-all-e2e label and run a new pipeline.
E2E test jobs are allowed to fail due to flakiness. See current failures at the latest pipeline triage issue.
Once done, apply the
emoji on this comment.Team members only: for any questions or help, reach out on the internal
#quality
Slack channel.requested review from @reprazent
- Resolved by Laura Montemayor
Thanks @mbobin, this seems like a simple fix to a complicated edge case. This looks good to me! @lauraX would you have a moment for the maintainer review here?
requested review from @lauraX and removed review request for @reprazent
@reprazent
, thanks for approving this merge request.This is the first time the merge request has been approved. To ensure full test coverage, a new pipeline will be started shortly.
For more info, please refer to the following links:
added pipeline:mr-approved label
added 604 commits
-
87ac9099...91643bf7 - 601 commits from branch
master
- 5e8eff8a - Create iid sequence for ci_pipelines with new projects
- f559a86e - Use the project creation service
- 6bf6fb18 - Update query limits
Toggle commit list-
87ac9099...91643bf7 - 601 commits from branch
added 1 commit
- 542c2330 - Create iid sequence for ci_pipelines with new projects
- Resolved by Bob Van Landuyt
added 1 commit
- 07fdcf4b - Create iid sequence for ci_pipelines with new projects
requested review from @reprazent
- Resolved by Bob Van Landuyt
- Resolved by Bob Van Landuyt
Thanks @mbobin, just 2 questions. I think using the published event for this is a great idea!
removed review request for @reprazent
Allure report
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for a9da1076expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Data Stores | 20 | 0 | 0 | 0 | 20 | ✅ | | Plan | 47 | 0 | 0 | 0 | 47 | ✅ | | Create | 38 | 0 | 0 | 2 | 38 | ❗ | | Govern | 34 | 0 | 0 | 0 | 34 | ✅ | | Manage | 12 | 0 | 1 | 0 | 13 | ✅ | | Verify | 8 | 0 | 0 | 0 | 8 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 159 | 0 | 1 | 2 | 160 | ❗ | +-------------+--------+--------+---------+-------+-------+--------+
added 2 commits
- Resolved by Laura Montemayor
Thanks @mbobin, nothing more from me!
@lauraX Once again, over to you
- Resolved by Marius Bobin
enabled an automatic merge when the pipeline for a0959300 succeeds
mentioned in commit 2c042ff8
added workflowstaging-canary label and removed workflowin review label
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
added workflowpost-deploy-db-staging label and removed workflowproduction label
added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label
mentioned in merge request !130835 (merged)
mentioned in merge request !130836 (merged)
added releasedcandidate label
added releasedpublished label and removed releasedcandidate label
mentioned in merge request kubitus-project/kubitus-installer!2437 (merged)
mentioned in issue #519457