Fix duplicate finished build sync events
What does this MR do and why?
This MRs removes 3 feature flags that were used to debug the duplicates being added to the ci_finished_builds_aggregated_queueing_delay_percentiles_mv
materialized view, and fixes the problem with sending duplicate build IDs to ClickHouse. We do this by loading the AR relationship before calling pluck(:build_id)
, so that the call to last
doesn't generate an extra query.
EE: true
Part of #428146 (closed)
This change was tested in the production console, where it was confirmed that @processed_record_ids
contained duplicate builds IDs at the end of execute
.
The builds in @processed_build_ids
looked like the following, if split by batch:
@processed_build_ids by batch
[[
# ... 498 unique IDs,
5_634_258_445,
5_634_259_020],
[
5_634_258_445,
5_634_259_020,
# ... 494 unique IDs,
5_634_474_423,
5_634_474_548,
5_634_474_559,
5_634_474_885],
[5_634_474_423,
5_634_474_548,
5_634_474_559,
5_634_474_885,
# ... 496 unique IDs
],
[ # ... 496 unique IDs,
5_634_583_521,
5_634_583_562,
5_634_583_577,
5_634_583_579,
5_634_583_582,
5_634_583_584],
[5_634_583_521,
5_634_583_562,
5_634_583_577,
5_634_583_579,
5_634_583_582,
5_634_583_584,
# ... 492 unique IDs,
5_634_614_703,
5_634_614_785],
[5_634_614_703,
5_634_614_785,
# ... 495 unique IDs,
5_634_636_196,
5_634_636_225,
5_634_636_274],
[5_634_636_196,
5_634_636_225,
5_634_636_274,
# ... 496 unique IDs,
5_634_651_184],
[5_634_651_184,
# ... 492 unique IDs,
5_634_667_919,
5_634_667_925,
5_634_667_929,
5_634_667_934,
5_634_667_943,
5_634_668_045,
5_634_668_092],
[5_634_667_919,
5_634_667_925,
5_634_667_929,
5_634_667_934,
5_634_667_943,
5_634_668_045,
5_634_668_092,
# ... 489 unique IDs,
5_634_680_068,
5_634_680_069,
5_634_680_072,
5_634_680_076],
[5_634_680_068,
5_634_680_069,
5_634_680_072,
5_634_680_076,
# ... 491 unique IDs,
5_634_688_597,
5_634_688_600,
5_634_688_603,
5_634_688_607,
5_634_688_608],
[5_634_688_597,
5_634_688_600,
5_634_688_603,
5_634_688_607,
5_634_688_608,
# ... 493 unique IDs,
5_634_693_846,
5_634_693_852],
[5_634_693_846,
5_634_693_852,
# ... 349 unique IDs
],
[]]
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
n/a
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
n/a
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Merge request reports
Activity
changed milestone to %16.7
assigned to @pedropombeiro
To speed up your revert merge request, please consider using the revert merge request template. Adding the appropriate labels for resolving master:broken before the merge request is created will skip several CI/CD jobs.
For this merge request, if this is for resolving master:broken you can add the appropriate labels present in the merge request template, and trigger a new pipeline. It will be faster
.- A deleted user
added backend label
2 Warnings 31209a48: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. There were no new or modified feature flag YAML files detected in this MR. If the changes here are already controlled under an existing feature flag, please add
the feature flagexists. Otherwise, if you think the changes here don't need
to be under a feature flag, please add the label feature flagskipped, and
add a short comment about why we skipped the feature flag.For guidance on when to use a feature flag, please see the documentation.
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend @rcobb
(UTC-8, 9 hours behind author)
@ahuntsman
(UTC-6, 7 hours behind author)
~"Clickhouse" Reviewer review is optional for ~"Clickhouse" @ahegyi
(UTC+1, same timezone as author)
Please check reviewer's status!
Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Dangermarked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed
removed backend label
added bugfunctional clickhouse labels
added typebug label
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@f7467bbf
- Resolved by Pedro Pombeiro
@vshushlin mind doing the initial backend/clickhouse maintainer reviews? This MR is composed by 3 revert commits + 1 commit to deal with duplicates.
Edited by Pedro Pombeiro
requested review from @vshushlin
mentioned in issue #428146 (closed)
mentioned in issue gitlab-runner#29486 (closed)
Allure report
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for 31209a48expand test summary
+-----------------------------------------------------------------------+ | suites summary | +------------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +------------------+--------+--------+---------+-------+-------+--------+ | Govern | 57 | 0 | 0 | 0 | 57 | ✅ | | Verify | 31 | 0 | 0 | 0 | 31 | ✅ | | Plan | 55 | 0 | 0 | 0 | 55 | ✅ | | Monitor | 8 | 0 | 0 | 0 | 8 | ✅ | | Create | 48 | 0 | 9 | 0 | 57 | ✅ | | Data Stores | 23 | 0 | 0 | 0 | 23 | ✅ | | ModelOps | 0 | 0 | 1 | 0 | 1 | ➖ | | Framework sanity | 0 | 0 | 1 | 0 | 1 | ➖ | | Package | 0 | 0 | 1 | 0 | 1 | ➖ | | Manage | 0 | 0 | 1 | 0 | 1 | ➖ | +------------------+--------+--------+---------+-------+-------+--------+ | Total | 222 | 0 | 13 | 0 | 235 | ✅ | +------------------+--------+--------+---------+-------+-------+--------+
e2e-package-and-test:
test report for 31209a48expand test summary
+-----------------------------------------------------------------------+ | suites summary | +------------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +------------------+--------+--------+---------+-------+-------+--------+ | Create | 147 | 0 | 16 | 0 | 163 | ✅ | | Verify | 184 | 0 | 36 | 0 | 220 | ✅ | | Govern | 6 | 0 | 0 | 0 | 6 | ✅ | | Plan | 8 | 0 | 0 | 0 | 8 | ✅ | | ModelOps | 0 | 0 | 2 | 0 | 2 | ➖ | | Framework sanity | 0 | 0 | 2 | 0 | 2 | ➖ | | Monitor | 8 | 0 | 0 | 0 | 8 | ✅ | | Data Stores | 4 | 0 | 0 | 0 | 4 | ✅ | | Package | 0 | 0 | 2 | 0 | 2 | ➖ | +------------------+--------+--------+---------+-------+-------+--------+ | Total | 357 | 0 | 58 | 0 | 415 | ✅ | +------------------+--------+--------+---------+-------+-------+--------+