[Test] spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb | Ci::PipelineProcessing::AtomicProcessingService Pipeline Processing Service when jobs change from stopped to alive status during pipeline processing runs ResetSkippedJobsSe...
Test metadata (don't modify)
Field | Value |
---|---|
File URL | spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb#L1013 |
Filename | spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb |
Description | Ci::PipelineProcessing::AtomicProcessingService Pipeline Processing Service when jobs change from stopped to alive status during pipeline processing runs ResetSkippedJobsService on the new alive jobs and logs event |
Test level | unit |
Hash | 2241e0515c433c708c53494129c6b4dab053f81d3 |
Reference duration | 3.23 seconds |
Expected duration | < 45.4 seconds |
Stack trace
Failure/Error: expect(all_builds_names_and_statuses).to eq(statuses_1)
expected: {:deploy1=>"skipped", :deploy2=>"skipped", :manual1=>"manual", :manual2=>"manual", :test1=>"skipped", :test2=>"skipped"}
got: {:deploy1=>"skipped", :deploy2=>"created", :manual1=>"manual", :manual2=>"manual", :test1=>"skipped", :test2=>"created"}
(compared using ==)
Diff:
@@ -1,7 +1,7 @@
:deploy1 => "skipped",
-:deploy2 => "skipped",
+:deploy2 => "created",
:manual1 => "manual",
:manual2 => "manual",
:test1 => "skipped",
-:test2 => "skipped",
+:test2 => "created",
Reports (1)
- 2024-03-14: https://gitlab.com/gitlab-org/gitlab/-/jobs/6394154237 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1213569840)
Resolution
We confirmed that the problem is the order in which jobs are processed by AtomicProcessingService
. The order is not consistent between each test run, which particularly affects the expected job statuses of DAG pipelines with same-stage dependent jobs. See !147875 (merged) for details.
We inferred that the query in AtomicProcessingService#update_jobs
only orders jobs by stage, but it should also account for DAG ordering.
The MR Fix handling of DAGs within a stage in AtomicPr... (!147875 - merged) was opened to resolve this issue.
Designs
- Show closed items
Relates to
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- A deleted user added Category:Continuous Integration failurenew found:main grouppipeline execution priority2 test labels
- Ghost User marked this issue as related to #446201 (closed)
marked this issue as related to #446201 (closed)
- Author Contributor
Flakiness reports (315)
Last 10 reports:
- 2024-03-27: https://gitlab.com/gitlab-org/gitlab/-/jobs/6485234816 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228937263) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6484831852 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228867428) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6484663588 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228833816) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6484653076 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228831871) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6484594186 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228819353) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6484407179 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228785922) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6484048269 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228721420) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6483909215 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228698652) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6483869847 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228691218) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6483488293 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228616929) found:master
See 305 more reports
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6483309717 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228585675) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6482941737 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228516608) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6482025336 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228377458) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6481897906 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228357493) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6481805295 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228344514) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6481538382 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228305234) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6481331830 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228274579) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6481321357 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228272994) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6481203728 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228256249) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6481110153 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228243770) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6480312106 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228122722) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6480101162 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228090471) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6479946940 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228066341) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6479564616 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1228004526) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6479496971 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227994708) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6479308709 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227966136) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6479290331 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227963306) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6479159334 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227944013) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6479134605 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227940930) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6478844456 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227895377) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6478668447 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227868354) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6478463179 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227834831) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6478182444 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227793786) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6477960770 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227758397) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6476947544 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227603616) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6476844109 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227584861) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6476400597 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227508122) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6476358707 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227500009) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6476279772 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227484236) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6475748337 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227389661) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6475506402 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227337380) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6475417814 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227321084) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6475417574 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227321084) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6475415965 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227321084) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6475415286 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227321084) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6475287105 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227297113) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6475121033 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227267906) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6475059883 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227257248) found:master
- 2024-03-26: https://gitlab.com/gitlab-org/gitlab/-/jobs/6475039859 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227254072) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6474883440 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227226971) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6474651460 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227193907) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6473680671 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227027509) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6473641172 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1227020132) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6473439294 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226984485) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6473377296 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226973985) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6473367043 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226972405) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6473262438 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226954997) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6473134578 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226935962) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6473071403 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226924681) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6472656508 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226862376) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6472648062 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226861187) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6472573796 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226848408) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6472001314 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226761124) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6471972981 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226756828) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6471842480 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226735854) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6470584539 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226540847) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6470428391 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226516305) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6469977360 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226448567) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6469560869 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226383142) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6469492271 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226372131) found:in MR
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6469160907 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226322919) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6469004822 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226300593) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6468914832 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226286903) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6468358465 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226201522) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6467103793 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1226003188) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6466872540 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225961648) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6466734890 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225936723) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6466501897 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225891155) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6466376731 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225869661) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6466205744 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225839191) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6465965073 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225788034) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6465899555 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225776828) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6465898552 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225776828) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6465898505 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225776828) found:master
- 2024-03-25: https://gitlab.com/gitlab-org/gitlab/-/jobs/6465709592 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225742320) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6465474055 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225692331) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6465374535 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225671198) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6465267997 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225647146) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6465077041 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225603229) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6464887478 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225558678) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6464829713 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225548786) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6464627383 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225500452) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6464036614 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225361807) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6463852102 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225321056) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6463677587 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225283102) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6463403105 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225216142) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6463402350 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225216142) found:master
- 2024-03-24: https://gitlab.com/gitlab-org/gitlab/-/jobs/6463402312 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225216142) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6463107115 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225146058) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6462767732 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1225067336) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6462129938 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224907203) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6461750041 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224822808) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6461381129 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224738696) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6461329387 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224726650) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6461071530 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224670296) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6461071293 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224670296) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6461070315 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224670296) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6461070270 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224670296) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6460903243 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224632471) found:master
- 2024-03-23: https://gitlab.com/gitlab-org/gitlab/-/jobs/6460824295 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224616598) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6460623338 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224580569) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6460567258 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224567513) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6460497552 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224555229) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6460143089 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224497064) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6459673407 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224423449) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6459614765 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224414777) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6459419461 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224383806) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6459318126 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224367502) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6459151057 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224344066) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6458725819 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224281745) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6458713225 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224279463) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6458530291 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224248563) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6458286910 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224212164) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6458175235 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224195971) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6458160286 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224193925) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6457851746 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224136759) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6457048828 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1224006965) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6456922500 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223982360) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6456452641 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223906253) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6456229535 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223870852) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6456131682 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223853300) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6455984204 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223830843) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6455559301 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223763840) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6455038671 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223682059) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6454983668 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223673650) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6454831020 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223649478) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6454652938 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223620696) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6454565929 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223605662) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6454005774 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223512973) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6453675799 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223454085) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452943522 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223306876) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452803710 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223278970) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452797935 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223278235) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452796974 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223278235) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452796898 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223278235) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452603896 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223238444) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452579255 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223234107) found:master
- 2024-03-22: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452419685 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223204027) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452285636 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223182191) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452173135 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223162928) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6452078562 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223148081) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6451403905 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223022526) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6451312034 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1223009336) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6450402676 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222858427) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6450347026 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222846959) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6450298917 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222838512) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6450292310 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222837616) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6450238341 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222829578) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6449792800 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222758915) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6449738055 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222751078) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6449405165 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222691610) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6449308693 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222674910) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6449078088 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222628105) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6448937846 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222598778) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6448748186 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222571540) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6448696517 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222564797) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6448686548 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222563432) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6448564200 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222546070) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6447667883 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222415300) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6447652585 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222412948) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6446602527 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222254971) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6446577121 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222251148) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6446148706 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222182948) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6445868382 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222141222) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6445760339 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222123158) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6445480893 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222082208) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6445308814 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222055782) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6445266684 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222048948) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6445161702 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1222029950) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6444414947 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221916729) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6444218346 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221887182) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6444212010 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221886581) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6443647311 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221786598) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6443213214 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221707648) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6442833797 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221632905) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6442760837 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221619664) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6442583062 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221602917) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6442582021 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221602917) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6442484682 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221583195) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6442350795 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221556783) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6442338229 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221555040) found:master
- 2024-03-21: https://gitlab.com/gitlab-org/gitlab/-/jobs/6442223927 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221531875) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6441777651 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221459102) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6441741862 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221451871) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6441640854 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221437568) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6440488013 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221257877) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6440463379 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221253700) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6439267663 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1221079494) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6438455922 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220956885) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6438026114 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220894070) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6437913747 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220876797) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6437789092 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220856996) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6437768984 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220854114) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6437522012 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220817603) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6437437734 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220804989) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6437195483 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220765503) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6437088731 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220751327) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6436722263 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220696919) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6436649509 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220683634) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6436446790 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220652044) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6436145150 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220605231) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6435797196 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220554502) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6435623206 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220527442) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6435428212 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220494186) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6435389936 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220488459) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6433900119 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220247181) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6433504564 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220173165) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6432950856 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220059008) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6432949476 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220059008) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6432949425 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220059008) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6432949304 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220059008) found:master
- 2024-03-20: https://gitlab.com/gitlab-org/gitlab/-/jobs/6432770339 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1220024903) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6432155206 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219921094) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6432149874 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219920322) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6431629146 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219842308) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6431578046 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219835055) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6430966375 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219741830) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6430781109 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219714537) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6430696135 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219701056) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6430689235 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219700118) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6430243722 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219630277) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6430115674 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219606450) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6429736820 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219553835) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6429587472 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219532855) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6429568685 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219530433) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6428563486 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219392170) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6428082401 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219320280) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6427908032 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219293627) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6426984851 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219152724) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6426955150 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219148595) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6426825852 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219127618) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6426815033 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219125743) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6425607509 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218939320) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6423963039 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218664480) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6422694126 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218488815) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6422447280 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218456002) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6422445570 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218456002) found:master
- 2024-03-19: https://gitlab.com/gitlab-org/gitlab/-/jobs/6422304807 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218431065) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6421881099 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218356913) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6421803646 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218345454) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6421724756 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218333349) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6421655181 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218320983) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6421417295 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218286347) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6421057165 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218230689) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6420925285 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218210229) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6420449872 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218138530) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6420337405 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218122925) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6420132537 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218091593) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6420090454 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218086049) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6419646920 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218021428) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6419529097 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1218004637) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6419111815 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1217945935) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6417913043 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1217770646) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6416743269 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1217597950) found:in MR
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6416712723 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1217592971) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6415552268 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1217403078) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6414526730 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1217236821) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6413584390 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1217059295) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6413320000 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1217003643) found:in MR
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6413251688 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216985954) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6413140812 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216962563) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6413140575 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216962563) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6413139483 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216962563) found:master
- 2024-03-18: https://gitlab.com/gitlab-org/gitlab/-/jobs/6412982595 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216928729) found:master
- 2024-03-17: https://gitlab.com/gitlab-org/gitlab/-/jobs/6412549480 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216834699) found:master
- 2024-03-17: https://gitlab.com/gitlab-org/gitlab/-/jobs/6412302684 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216781230) found:master
- 2024-03-17: https://gitlab.com/gitlab-org/gitlab/-/jobs/6412137370 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216743738) found:master
- 2024-03-17: https://gitlab.com/gitlab-org/gitlab/-/jobs/6411704500 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216621316) found:master
- 2024-03-17: https://gitlab.com/gitlab-org/gitlab/-/jobs/6411516883 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216577842) found:master
- 2024-03-17: https://gitlab.com/gitlab-org/gitlab/-/jobs/6411332087 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216536650) found:master
- 2024-03-17: https://gitlab.com/gitlab-org/gitlab/-/jobs/6410632602 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216384035) found:master
- 2024-03-17: https://gitlab.com/gitlab-org/gitlab/-/jobs/6410632537 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216384035) found:master
- 2024-03-17: https://gitlab.com/gitlab-org/gitlab/-/jobs/6410632492 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216384035) found:master
- 2024-03-17: https://gitlab.com/gitlab-org/gitlab/-/jobs/6410502617 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216355222) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6409771955 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216249828) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6409514584 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216204508) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6409340165 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216162396) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6409152374 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1216119980) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6408614675 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215993884) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6408219347 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215906326) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6407793180 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215825018) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6407793052 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215825018) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6407792904 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215825018) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6407792100 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215825018) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6407792066 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215825018) found:master
- 2024-03-16: https://gitlab.com/gitlab-org/gitlab/-/jobs/6407792034 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215825018) found:master
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6405697891 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215471977) found:master
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6405150011 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215389271) found:master
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6404124457 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215234111) found:master
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6403354838 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215111737) found:in MR
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6402978472 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1215050047) found:master
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6402645543 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1214992591) found:master
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6400550696 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1214616544) found:master
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6399985614 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1214513214) found:master
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6399606108 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1214440686) found:master
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6399246307 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1214366292) found:master
- 2024-03-15: https://gitlab.com/gitlab-org/gitlab/-/jobs/6398744207 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1214273159) found:master
- 2024-03-14: https://gitlab.com/gitlab-org/gitlab/-/jobs/6398726090 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1214270427) found:in MR
- 2024-03-14: https://gitlab.com/gitlab-org/gitlab/-/jobs/6397022261 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1214002795) found:master
- 2024-03-14: https://gitlab.com/gitlab-org/gitlab/-/jobs/6395999467 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1213849052) found:master
- 2024-03-14: https://gitlab.com/gitlab-org/gitlab/-/jobs/6395273710 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1213743226) found:in MR
- 2024-03-14: https://gitlab.com/gitlab-org/gitlab/-/jobs/6394966377 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1213695376) found:master
- 2024-03-14: https://gitlab.com/gitlab-org/gitlab/-/jobs/6394571419 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1213634635) found:master
- 2024-03-14: https://gitlab.com/gitlab-org/gitlab/-/jobs/6394507994 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1213624258) found:master
- 2024-03-14: https://gitlab.com/gitlab-org/gitlab/-/jobs/6394437379 (https://gitlab.com/gitlab-org/gitlab/-/pipelines/1213612797) found:master
Flaky tests were detected, please refer to the Flaky tests documentation to learn more about how to reproduce them.
- A deleted user added automation:bot-authored failureflaky-test labels and removed failurenew label
added automation:bot-authored failureflaky-test labels and removed failurenew label
- 🤖 GitLab Bot 🤖 added devopsverify sectionci labels
added devopsverify sectionci labels
- Caroline Simpson added typemaintenance label
added typemaintenance label
- Caroline Simpson mentioned in issue #450672 (closed)
mentioned in issue #450672 (closed)
This test is failing quite frequently. Latest example: https://gitlab.com/gitlab-org/gitlab/-/jobs/6469761722
In that job,
spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb
was the first tests to run, and that failed in 2 examples both related toResetSkippedJobsService
. Then when the spec retried, only the first example failed. This suggest this is not an order-dependent failure.In each failure, it appears that the jobs that are failing have a
created
state instead ofskipped
.This seems to suggest that the manual jobs were somehow played and
ResetSkippedJobsService
(or some equivalent) was called on those dependent jobs. Normally, the latter only happens if theplay
event.Collapse replies FYI @lma-git in case you have any ideas what might be happening here. I haven't reproduced this locally yet, but I'm attempting to replicate the Docker setup (outdated instructions: #390313 (comment 1283606446)) to see if repeated invocations might cause this to fail.
- Maintainer
Thanks for the ping @stanhu! Interesting. The code related to the test
when jobs change from stopped to alive status during pipeline processing
, was originally introduced as part of a fix for a race condition discovered in #388539 (closed). There are a couple analyses summarized on that issue if you'd like to see.My initial thought is that something must've changed in the pipeline execution process since then that affects the race condition, which is unstable and that's why it only happens sometimes.
🤔 Or it affects the conditions of the test itself. I'll give it a closer look.
Edited by Leaminn Ma Ok, I managed to induce the problem once more after resetting the DB (
RAILS_ENV=test bin/rake db:reset
) and restartingredis
andrediscluster
services. However, it doesn't seem to fail consistently.docker restart redis docker restart rediscluster bundle exec rspec --fail-fast spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb
I haven't made it fail if I just run
bundle exec rspec spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb:1013
alone.I wonder if we're just missing the
:redis
RSpec metadata. Maybe an exclusive lease is blocking the work inprocess_pipeline
.I added logging to
ResetSkippedJobsService
:diff --git a/app/services/ci/reset_skipped_jobs_service.rb b/app/services/ci/reset_skipped_jobs_service.rb index 9e5c887b3..f029b83ec 100644 --- a/app/services/ci/reset_skipped_jobs_service.rb +++ b/app/services/ci/reset_skipped_jobs_service.rb @@ -37,7 +37,10 @@ def dependent_jobs def process(job) Gitlab::OptimisticLocking.retry_lock(job, name: 'ci_requeue_job') do |job| + old_status = job.status job.process(current_user) + STDERR.puts "=== ResetSkippedJobService called with #{job.name} #{job.id}, old status was: #{old_status}, status was: #{job.status}" + STDERR.puts caller.join("\n") end end
With this logging, I didn't any invocations of
ResetSkippedJobService
here before the test on line 1013 failed.- Maintainer
I haven't yet seen any related application code that could cause the test to fail. So I'm assuming the problem is more likely related to the conditions of the test itself. Since Sidekiq jobs aren't processed in these tests, there shouldn't be a timing issue within them.
🤔 @stanhu The failures seem to suggest there is something else changing the statuses of the jobs during testing, which aligns with your thought:
Maybe an exclusive lease is blocking the work in
process_pipeline
.What if a previous test did enqueue
PipelineProcessWorker
asynchronously and it's still running by the time these tests are executed? - Maintainer
still running by the time these tests are executed
Actually never mind... the lease key includes
pipeline.id
, so the tests should all have their own exclusive leases provided they use a newpipeline
, which these tests do.Still, the failures do seem like
process_pipeline
isn't running and so the statuses aren't being skipped byCi::ProcessBuildService
.🤔 Not sure if this is a deterministic way of reproduce the error, but I got it again via:
docker restart redis docker restart rediscluster RAILS_ENV=test bin/rake db:reset && bundle exec rspec spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb:1013 && bundle exec rspec --format d --fail-fast spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb
I compared
test.log
between a non-working spec and working spec:In
success.log
, you can see 4 jobs marked asskipped
:% less -R success.log | grep UPDATE Ci::Build Update (1.1ms) UPDATE "p_ci_builds" SET "status" = 'manual', "updated_at" = '2024-03-25 19:23:14.785709', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 517 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:3cd9573c110a67328bbeedbed4a63343,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Build Update (0.7ms) UPDATE "p_ci_builds" SET "status" = 'manual', "updated_at" = '2024-03-25 19:23:14.863327', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 518 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:3cd9573c110a67328bbeedbed4a63343,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Build Update (0.7ms) UPDATE "p_ci_builds" SET "status" = 'skipped', "updated_at" = '2024-03-25 19:23:14.881339', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 519 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:3cd9573c110a67328bbeedbed4a63343,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Build Update (0.7ms) UPDATE "p_ci_builds" SET "status" = 'skipped', "updated_at" = '2024-03-25 19:23:14.898024', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 520 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:3cd9573c110a67328bbeedbed4a63343,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Stage Update (0.7ms) UPDATE "p_ci_stages" SET "updated_at" = '2024-03-25 19:23:14.918013', "status" = 6, "lock_version" = 1 WHERE "p_ci_stages"."id" = 383 AND "p_ci_stages"."lock_version" = 0 /*application:test,correlation_id:3cd9573c110a67328bbeedbed4a63343,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Build Update (0.8ms) UPDATE "p_ci_builds" SET "status" = 'skipped', "updated_at" = '2024-03-25 19:23:14.945092', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 521 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:3cd9573c110a67328bbeedbed4a63343,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Build Update (0.9ms) UPDATE "p_ci_builds" SET "status" = 'skipped', "updated_at" = '2024-03-25 19:23:14.965287', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 522 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:3cd9573c110a67328bbeedbed4a63343,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Stage Update (0.6ms) UPDATE "p_ci_stages" SET "updated_at" = '2024-03-25 19:23:14.986300', "status" = 6, "lock_version" = 1 WHERE "p_ci_stages"."id" = 384 AND "p_ci_stages"."lock_version" = 0 /*application:test,correlation_id:3cd9573c110a67328bbeedbed4a63343,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Pipeline Update (0.6ms) UPDATE "ci_pipelines" SET "updated_at" = '2024-03-25 19:23:14.991846', "status" = 'skipped', "lock_version" = 1 WHERE "ci_pipelines"."id" = 145 AND "ci_pipelines"."lock_version" = 0 /*application:test,correlation_id:3cd9573c110a67328bbeedbed4a63343,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ CommitStatus Update All (1.1ms) UPDATE "p_ci_builds" SET "processed" = TRUE WHERE "p_ci_builds"."commit_id" = 145 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 1 AND ("p_ci_builds"."id" = 517 OR "p_ci_builds"."id" = 518 OR "p_ci_builds"."id" = 519 OR "p_ci_builds"."id" = 520 OR "p_ci_builds"."id" = 521 OR "p_ci_builds"."id" = 522) /*application:test,correlation_id:3cd9573c110a67328bbeedbed4a63343,db_config_name:main,line:/app/models/commit_status.rb:239:in `update_as_processed!'*/
However, in the failure case, we only see 3
skipped
UPDATE calls:% less -R fail.log | grep UPDATE Ci::Build Update (1.1ms) UPDATE "p_ci_builds" SET "status" = 'manual', "updated_at" = '2024-03-25 19:30:50.307282', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 352 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:2033e49ad69ebba6e3eada1ef790af46,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Build Update (1.1ms) UPDATE "p_ci_builds" SET "status" = 'skipped', "updated_at" = '2024-03-25 19:30:50.339284', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 354 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:2033e49ad69ebba6e3eada1ef790af46,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Build Update (1.1ms) UPDATE "p_ci_builds" SET "status" = 'manual', "updated_at" = '2024-03-25 19:30:50.372033', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 351 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:2033e49ad69ebba6e3eada1ef790af46,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Stage Update (0.9ms) UPDATE "p_ci_stages" SET "updated_at" = '2024-03-25 19:30:50.454259', "status" = 1, "lock_version" = 1 WHERE "p_ci_stages"."id" = 268 AND "p_ci_stages"."lock_version" = 0 /*application:test,correlation_id:2033e49ad69ebba6e3eada1ef790af46,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Build Update (1.1ms) UPDATE "p_ci_builds" SET "status" = 'skipped', "updated_at" = '2024-03-25 19:30:50.499858', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 355 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:2033e49ad69ebba6e3eada1ef790af46,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Build Update (1.3ms) UPDATE "p_ci_builds" SET "status" = 'skipped', "updated_at" = '2024-03-25 19:30:50.533439', "processed" = FALSE, "lock_version" = 1 WHERE "p_ci_builds"."id" = 356 AND "p_ci_builds"."partition_id" = 101 AND "p_ci_builds"."lock_version" = 0 /*application:test,correlation_id:2033e49ad69ebba6e3eada1ef790af46,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Stage Update (0.8ms) UPDATE "p_ci_stages" SET "updated_at" = '2024-03-25 19:30:50.569878', "status" = 6, "lock_version" = 1 WHERE "p_ci_stages"."id" = 269 AND "p_ci_stages"."lock_version" = 0 /*application:test,correlation_id:2033e49ad69ebba6e3eada1ef790af46,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ Ci::Pipeline Update (0.9ms) UPDATE "ci_pipelines" SET "updated_at" = '2024-03-25 19:30:50.579484', "status" = 'pending', "lock_version" = 1 WHERE "ci_pipelines"."id" = 96 AND "ci_pipelines"."lock_version" = 0 /*application:test,correlation_id:2033e49ad69ebba6e3eada1ef790af46,db_config_name:main,line:/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'*/ CommitStatus Update All (1.7ms) UPDATE "p_ci_builds" SET "processed" = TRUE WHERE "p_ci_builds"."commit_id" = 96 AND "p_ci_builds"."partition_id" = 101 AND ("p_ci_builds"."id" = 352 AND "p_ci_builds"."lock_version" = 1 OR "p_ci_builds"."id" = 353 AND "p_ci_builds"."lock_version" = 0 OR "p_ci_builds"."id" = 354 AND "p_ci_builds"."lock_version" = 1 OR "p_ci_builds"."id" = 351 AND "p_ci_builds"."lock_version" = 1 OR "p_ci_builds"."id" = 355 AND "p_ci_builds"."lock_version" = 1 OR "p_ci_builds"."id" = 356 AND "p_ci_builds"."lock_version" = 1) /*application:test,correlation_id:2033e49ad69ebba6e3eada1ef790af46,db_config_name:main,line:/app/models/commit_status.rb:239:in `update_as_processed!'*/
That matches with this test failure:
1) Ci::PipelineProcessing::AtomicProcessingService Pipeline Processing Service when jobs change from stopped to alive status during pipeline processing runs ResetSkippedJobsService on the new alive jobs and logs eve nt Failure/Error: expect(all_builds_names_and_statuses).to eq(statuses_1) expected: {:deploy1=>"skipped", :deploy2=>"skipped", :manual1=>"manual", :manual2=>"manual", :test1=>"skipped", :test2=>"skipped"} got: {:deploy1=>"skipped", :deploy2=>"skipped", :manual1=>"manual", :manual2=>"manual", :test1=>"created", :test2=>"skipped"} (compared using ==) Diff: @@ -2,6 +2,6 @@ :deploy2 => "skipped", :manual1 => "manual", :manual2 => "manual", -:test1 => "skipped", +:test1 => "created", :test2 => "skipped", # ./spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb:1024:in `block (4 levels) in <top (required)>' # ./spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb:933:in `block (5 levels) in <top (required)>' # ./spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb:933:in `block (4 levels) in <top (required)>' # ./spec/spec_helper.rb:426:in `block (3 levels) in <top (required)>' # ./spec/support/sidekiq_middleware.rb:9:in `with_sidekiq_server_middleware' # ./spec/spec_helper.rb:417:in `block (2 levels) in <top (required)>' # ./spec/spec_helper.rb:413:in `block (3 levels) in <top (required)>' # ./lib/gitlab/application_context.rb:70:in `with_raw_context' # ./spec/spec_helper.rb:413:in `block (2 levels) in <top (required)>' # ./spec/spec_helper.rb:260:in `block (2 levels) in <top (required)>' # ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>' # ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <main>' # ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented' # ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <main>' Finished in 4 minutes 46.8 seconds (files took 20.28 seconds to load) 95 examples, 1 failure
1 Ok, I reproduced it without restarting Redis on my Docker setup. The following failed twice in a row for me, though it didn't fail in my GDK:
RAILS_ENV=test bin/rake db:reset && bundle exec rspec spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb:1013 && bundle exec rspec --format d --fail-fast spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb
I have to wonder if there is some hard-coded database ID somewhere.
I added more debugging. In the failing case, all 4 tests (
manual1
,manual2
,test1
, andtest2
) are being selected. However, for some reason, the status fortest1
iscreated
instead ofskipped
, so it's being skipped here: https://gitlab.com/gitlab-org/gitlab/-/blob/1741fa002b940086f63de0977bfddb1a86371315/app/services/ci/pipeline_processing/atomic_processing_service.rb#L94.One thing I noticed is that in the working case, the tests in
update_job!
in https://gitlab.com/gitlab-org/gitlab/-/blob/1741fa002b940086f63de0977bfddb1a86371315/app/services/ci/pipeline_processing/atomic_processing_service.rb#L67-77 are being called in this order:["manual1", "manual2", "test1", "test2"]
However, in the broken case, this is the order:
["manual2", "test1", "test2", "manual1"]
Note here that
manual1
doesn't get processed first, sotest1
might not get updated properly unless that happens? I wonder if this means we should sort by manual jobs first.Ok, I verified it is an database order issue. This hack makes the test work reliably:
diff --git a/app/services/ci/pipeline_processing/atomic_processing_service/status_collection.rb b/app/services/ci/pipeline_processing/atomic_processing_service/status_collection.rb index 9a53c6d8f..16ad8c3c0 100644 --- a/app/services/ci/pipeline_processing/atomic_processing_service/status_collection.rb +++ b/app/services/ci/pipeline_processing/atomic_processing_service/status_collection.rb @@ -116,6 +116,7 @@ def all_jobs raw_jobs = pipeline .current_jobs .ordered_by_stage + .order(:when) .pluck(*JOB_ATTRS) raw_jobs.map do |row|
We need to investigate whether this is a legitimate bug when manual jobs are on the same stage as dependent jobs.
Edited by Stan Hu1 - Maintainer
Thanks @stanhu ! This is a great find
🎉 This does look like a oversight.If we put the manual before test
stages: - test manual1: stage: test when: manual script: exit 0 test1: stage: test needs: [manual1] script: exit 0
The pipeline is skipped
✅ However if we do it the otherway
stages: - test test1: stage: test needs: [manual1] script: exit 0 manual1: stage: test when: manual script: exit 0
The pipeline is in created: https://gitlab.com/mfanGitLab/maxPublicProject/-/pipelines/1227275841 :question:It looks like there's a discrepancy here even though the jobs & stages are in the same order in the UI.
From the docs: There should be no problem having multiple jobs using needs in the same stage
@lma-git As this feels like a pipeline authoring kinda thing, do you want me to create an issue for this? If this edge case is something to be handled
cc: @drew @marknuzzo for the slack reference https://gitlab.slack.com/archives/C019R5JD44E/p1711040446124699
Edited by Max Fan1 - Maintainer
Note here that
manual1
doesn't get processed first, sotest1
might not get updated properly unless that happens? I wonder if this means we should sort by manual jobs first.@stanhu Thank you for finding this!
Yes, if
manual1
's status is still created by the timetest1
is processed, then it won't proceed to get skipped byCi::ProcessBuildService
.This is looking like it's not specific to manual jobs; any parent job likely should be processed before its dependents.
I think
update_job!
needs to use something likeordered_by_dag
which we run inResetSkippedJobsService
. Currently it only orders the jobs by stage.However if we do it the otherway
stages: - test test1: stage: test needs: [manual1] script: exit 0 manual1: stage: test when: manual script: exit 0
@mfanGitLab Great example!! ^ But now we know that this pipeline config may not always yield the same result. The flaky tests seem to indicate that the order of job creation is now less reliable/predictable based on their order of appearance in the Yaml.
💡
🤔 Interesting that this potential problem would've existed since the introduction of same-stage dependent jobs (#30632 (closed) in %14.2), but we're really just seeing the failures more now.I'm guessing it hasn't manifested as a noticeable bug because it's technically fine that the dependent jobs remain in
created
state in most scenarios. We can see in Max's example above that you can still play the manual job and the result is what you'd expect: it runs the dependenttest1
job after the manual job completes.Also, we often write the dependent jobs in the order we expect them to be run in the config Yaml, as most customers do as well.
I think these spec tests just so happened to reveal the problem because we don't test the processing order of dependent jobs elsewhere.
- Maintainer
As this feels like a pipeline authoring kinda thing, do you want me to create an issue for this? If this edge case is something to be handled
That would be great!
🙏 Yes PA should probably take it because it's related to the same-stage needs feature from #30632 (closed) and the subsequent work to fix job ordering in !81087 (merged).
I'm thinking that
AtomicProcessingService#update_jobs!
should have also accounted for DAG order. We implementedordered_by_dag
inResetSkippedJobsService
, but didn't do the same for processing the jobs inAtomicProcessingService
.cc @furkanayhan for visibility. I think you'll find the problem discussed in this thread interesting
😅 . Also, wdyt about my suggestion above?cc: @marknuzzo
As for the current flaky tests, I think we can just quarantine them for now. They're not directly related to the job ordering bug nor the original race condition issue that they're meant to test. Per #450395 (comment 1831103960), I think they just so happened to reveal an existing problem, one that we should solve in a new bug issue. Then we can un-quarantine them when it's fixed.
Thoughts?
cc: @drew
2 @mfanGitLab Great example! I had a feeling this would be a problem and was glad to see the bug illustrated so clearly.
flaky tests seem to indicate that the order of job creation is now less reliable/predictable based on their order of appearance in the Yaml.
@lma-git Right, the order may also depend on whatever the database returns.
I'm thinking that
AtomicProcessingService#update_jobs!
should have also accounted for DAG order. We implementedordered_by_dag
inResetSkippedJobsService
, but didn't do the same for processing the jobs inAtomicProcessingService
.Yeah, that makes sense. One thing to consider is that
StatusCollection
tries to retrieve all jobs in one go in https://gitlab.com/gitlab-org/gitlab/-/blob/3d7a5af7ce7b18c32c0a1e893f05f9c7bc6b4b92/app/services/ci/pipeline_processing/atomic_processing_service/status_collection.rb#L116-119, and then each job is checked for a DAG in https://gitlab.com/gitlab-org/gitlab/-/blob/1741fa002b940086f63de0977bfddb1a86371315/app/services/ci/pipeline_processing/atomic_processing_service.rb#L109.Edited by Stan Hu1 Something like this seems to work. We should probably use a feature flag for this refactor:
diff --git a/app/services/ci/pipeline_processing/atomic_processing_service.rb b/app/services/ci/pipeline_processing/atomic_processing_service.rb index 84e5089b0d53..c8e05fc83ba4 100644 --- a/app/services/ci/pipeline_processing/atomic_processing_service.rb +++ b/app/services/ci/pipeline_processing/atomic_processing_service.rb @@ -55,25 +55,44 @@ def update_stages! end def update_stage!(stage) - # Update jobs for a given stage in bulk/slices - @collection - .created_job_ids_in_stage(stage.position) - .in_groups_of(BATCH_SIZE, false) { |ids| update_jobs!(ids) } + ordered_jobs(stage).each { |job| update_job!(job) } status = @collection.status_of_stage(stage.position) stage.set_status(status) end - def update_jobs!(ids) - created_jobs = pipeline + def ordered_jobs(stage) + jobs = load_jobs_in_batches(stage) + sorted_job_names = sort_jobs(jobs).each_with_index.to_h + jobs.sort_by { |job| sorted_job_names.fetch(job.name) } + end + + def sort_jobs(jobs) + Gitlab::Ci::YamlProcessor::Dag.order( # rubocop: disable CodeReuse/ActiveRecord + jobs.to_h do |job| + [job.name, job.needs.map(&:name)] + end + ) + end + + def load_jobs_in_batches(stage) + @collection + .created_job_ids_in_stage(stage.position) + .in_groups_of(BATCH_SIZE, false) + .each_with_object([]) do |ids, jobs| + jobs.concat(load_jobs(ids)) + end + end + + def load_jobs(ids) + pipeline .current_processable_jobs .id_in(ids) .with_project_preload .created .ordered_by_stage .select_with_aggregated_needs(project) - - created_jobs.each { |job| update_job!(job) } end def update_pipeline! diff --git a/spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb b/spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb index 6e263e82432e..4c20183a14cf 100644 --- a/spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb +++ b/spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb @@ -937,16 +937,6 @@ def event_on_pipeline(event) <<-YAML stages: [test, deploy] - manual1: - stage: test - when: manual - script: exit 0 - - manual2: - stage: test - when: manual - script: exit 0 - test1: stage: test needs: [manual1] @@ -966,6 +956,16 @@ def event_on_pipeline(event) stage: deploy needs: [test2] script: exit 0 + + manual1: + stage: test + when: manual + script: exit 0 + + manual2: + stage: test + when: manual + script: exit 0 YAML end
1 Testing out !147875 (merged).
- Contributor
Thanks everyone for the ping here.
As this feels like a pipeline authoring kinda thing, do you want me to create an issue for this? If this edge case is something to be handled
Yes, @mfanGitLab if you could summarize the details in an issue and assign it to PA, ccing me, I can figure out with @dhershkovitch where we can prioritize this work soon. Thank you.
/cc @lma-git
- Maintainer
@stanhu From a brief look at !147875 (merged), that solution looks great! Using
Gitlab::Ci::YamlProcessor::Dag.order
looks like exactly what we needed.@mfanGitLab @marknuzzo So maybe we don't need a new issue since Stan's MR resolves the problem. We can just update the description with details on our findings so that it's easier to reference in the future.
Edited by Leaminn Ma2 Thanks! We should probably quarantine that test failure in
spec/services/ci/pipeline_processing/atomic_processing_service_spec.rb
for backports since we won't backport !147875 (merged).- Developer
Thanks Stan, Leaminn, and Max for your work figuring this out.
I set the group to pipeline authoring on this, since we won't have a separate issue. But if it makes more sense to put this back to pipeline execution, feel free to do so.
1 !147875 (merged) has been merged, so I'll close this issue.
I've submitted merge requests to quarantine the flaky tests for the stable branches:
-
16-10-stable-ee
: !147975 (merged) -
16-9-stable-ee
: !147985 (merged) -
16-8-stable-ee
: !147986 (merged)
1 -
- Maintainer
Thanks for the discussion here and solving the problem
🎉 As a side note, I created an issue to refactor the
app/services/ci/pipeline_processing/atomic_processing_service.rb
file. (#452167)
- Stan Hu mentioned in issue #451992 (closed)
mentioned in issue #451992 (closed)
- Stan Hu mentioned in merge request !147875 (merged)
mentioned in merge request !147875 (merged)
- Caroline Simpson added grouppipeline authoring label and removed grouppipeline execution label
added grouppipeline authoring label and removed grouppipeline execution label
- 🤖 GitLab Bot 🤖 added Category:Pipeline Composition label
added Category:Pipeline Composition label
- Leaminn Ma changed the description
Compare with previous version changed the description
- Leaminn Ma added typebug label and removed typemaintenance label
added typebug label and removed typemaintenance label
- Stan Hu mentioned in merge request !147975 (merged)
mentioned in merge request !147975 (merged)
- Stan Hu mentioned in merge request !147985 (merged)
mentioned in merge request !147985 (merged)
- Stan Hu mentioned in merge request !147986 (merged)
mentioned in merge request !147986 (merged)
- Stan Hu closed
closed
- Furkan Ayhan mentioned in issue #452167
mentioned in issue #452167
- Hordur Freyr Yngvason marked #449096 (closed) as a duplicate of this issue
marked #449096 (closed) as a duplicate of this issue
- Hordur Freyr Yngvason marked this issue as related to #449096 (closed)
marked this issue as related to #449096 (closed)
- Hordur Freyr Yngvason marked #450650 (closed) as a duplicate of this issue
marked #450650 (closed) as a duplicate of this issue
- Hordur Freyr Yngvason marked this issue as related to #450650 (closed)
marked this issue as related to #450650 (closed)
- Hordur Freyr Yngvason marked #446201 (closed) as a duplicate of this issue
marked #446201 (closed) as a duplicate of this issue
- Hordur Freyr Yngvason mentioned in issue #448654 (closed)
mentioned in issue #448654 (closed)
- Hordur Freyr Yngvason marked #448654 (closed) as a duplicate of this issue
marked #448654 (closed) as a duplicate of this issue
- Hordur Freyr Yngvason marked this issue as related to #448654 (closed)
marked this issue as related to #448654 (closed)
- Hordur Freyr Yngvason marked #448488 (closed) as a duplicate of this issue
marked #448488 (closed) as a duplicate of this issue
- Hordur Freyr Yngvason marked this issue as related to #448488 (closed)
marked this issue as related to #448488 (closed)
- Hordur Freyr Yngvason mentioned in issue #448488 (closed)
mentioned in issue #448488 (closed)
- Hordur Freyr Yngvason mentioned in issue #448540 (closed)
mentioned in issue #448540 (closed)