Pipeline Triage Report from 2023-03-27 to 2023-04-01
DRI
Please review the responsiblities and guidelines if you have not done so recently.
Above all else, please remember that the aim of pipeline triage is to identify problems and try to get them resolved (ideally) before they impact users.
You can view the Testcases project list of issues for non-quarantined test results to help identify failures and see if any were missed when reviewing pipelines.
Feel free to use the dri gem to help report and triage issues.
DRI Schedule
| Week of | APAC | EMEA | AMER |
|---|---|---|---|
| 2023-03-27 | Mark Lapierre (@mlapierre) | Nailia Iskhakova (@niskhakova) | Valerie Burton (@vburton) |
| 2023-04-03 | Jay McCure (@jay_mccure) | Andrejs Cunskis (@acunskis) | Zeff Morgan (@zeffmorgan) |
Carryovers
- Failure in browser_ui/11_fulfillment/license/cl... (gitlab-org/gitlab#388350 - closed)
- [Test] browser_ui/5_package/package_registry/ru... (gitlab-org/gitlab#366099 - closed)
- [Test] ee/browser_ui/3_create/repository/restri... (gitlab-org/gitlab#356816 - closed)
- Failure in browser_ui/3_create/merge_request/vi... (gitlab-org/gitlab#399477 - closed)
- Failure in api/1_manage/user_access_termination... (gitlab-org/gitlab#386670 - closed)
- Failure in api/1_manage/user_access_termination... (gitlab-org/gitlab#386671 - closed)
- Failure in ee/browser_ui/11_fulfillment/license... (gitlab-org/gitlab#396427 - closed)
- Failure in browser_ui/11_fulfillment/license/cl... (gitlab-org/gitlab#398682 - closed)
- Multiple failures seen on Production E2E pipeline (gitlab-org/gitlab#399147)
- Failure in ee/browser_ui/10_govern/security_rep... (gitlab-org/gitlab#382987 - closed)
- Failure in ee/browser_ui/11_fulfillment/utiliza... (gitlab-org/gitlab#398115 - closed)
- Failure in browser_ui/4_verify/ci_variable/pipe... (gitlab-org/gitlab#399149 - closed)
- Production E2E pipeline - QA user occasionally ... (gitlab-org/gitlab#399166 - closed)
- Failure in ee/browser_ui/4_verify/user_cancel_m... (gitlab-org/gitlab#398214 - closed)
- Staging failure ./qa/specs/features/browser_ui/... (gitlab-org/gitlab#361352 - closed)
- Failure in browser_ui/3_create/merge_request/me... (gitlab-org/gitlab#393854 - closed)
- Failure in browser_ui/3_create/web_ide_new/add_... (gitlab-org/gitlab#387609 - closed)
- Failure in browser_ui/8_monitor/alert_managemen... (gitlab-org/gitlab#395512 - closed)
- Failure in browser_ui/4_verify/runner/register_... (gitlab-org/gitlab#390184 - closed)
- Failure in ee/browser_ui/8_monitor/alert_manage... (gitlab-org/gitlab#397133 - closed)
- [Test] browser_ui/4_verify/pipeline/pipeline_wi... (gitlab-org/gitlab#396498)
- Failure in api/1_manage/group_access_token_spec... (gitlab-org/gitlab#396615 - closed)
- Failure in browser_ui/2_plan/pages/new_static_p... (gitlab-org/gitlab#383215)
- Failure in browser_ui/3_create/merge_request/me... (gitlab-org/gitlab#397651 - closed)
- Failure in browser_ui/3_create/merge_request/re... (gitlab-org/gitlab#397657 - closed)
- Failure in browser_ui/13_secure/merge_request_l... (gitlab-org/gitlab#397701 - closed)
- Failure in browser_ui/3_create/merge_request/me... (gitlab-org/gitlab#397495)
- Failure in api/3_create/repository/project_arch... (gitlab-org/gitlab#397727 - closed)
- Failure in ee/browser_ui/13_secure/license_comp... (gitlab-org/gitlab#397685 - closed)
- Failure in browser_ui/10_govern/vulnerability_m... (gitlab-org/gitlab#398409 - closed)
- Failure in browser_ui/3_create/repository/assig... (gitlab-org/gitlab#392984 - closed)
- Failure in ee/browser_ui/3_create/repository/co... (gitlab-org/gitlab#398116 - closed)
- Failure in api/3_create/merge_request/push_opti... (gitlab-org/gitlab#346425)
- Failure in ee/browser_ui/13_secure/license_comp... (gitlab-org/gitlab#397695 - closed)
- Failure in browser_ui/3_create/merge_request/me... (gitlab-org/gitlab#398113 - closed)
- Failure in browser_ui/3_create/snippet/share_sn... (gitlab-org/gitlab#393683 - closed)
- Failure in browser_ui/3_create/snippet/delete_f... (gitlab-org/gitlab#398194 - closed)
- Failure in browser_ui/3_create/repository/file_... (gitlab-org/gitlab#396927 - closed)
- Failure in ee/browser_ui/10_govern/group/group_... (gitlab-org/gitlab#396908 - closed)
- Failure in ee/browser_ui/10_govern/project_secu... (gitlab-org/gitlab#396502 - closed)
- Failure in ee/browser_ui/13_secure/enable_scann... (gitlab-org/gitlab#398193 - closed)
- Failure in api/1_manage/integrations/webhook_ev... (gitlab-org/gitlab#393274 - closed)
- Failure in ee/browser_ui/13_secure/license_comp... (gitlab-org/gitlab#365125 - closed)
Current known issues for CustomersDot project
Activity
- A deleted user
added Quality label
assigned to @niskhakova, @mlapierre, and @vburton
APAC, Monday - 2023-03-27
posted by: @mlapierre
Edited by Mark Lapierre-
Cloud activation tests are now passing: gitlab-org/gitlab#388350 (comment 1326489682)
Nightlies are failing because of a security release (this is expected - but maybe we could do better, e.g., automatically suspend the nightlies?
-
The CustomersDot failures involving prices revealed a bug: https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/6121
Edited by Mark Lapierre -
browser_ui/8_monitor/alert_management/alert_with_escalation_policy_spec.rbhas been failing for a while and keeps creating new issues because of the changing error message. Unfortunately I didn't get a chance to look into it today.Edited by Mark Lapierre 
@mlapierre thanks for the detailed notes
Nightlies are failing because of a security release (this is expected - but maybe we could do better, e.g., automatically suspend the nightlies?
It's being tracked at Explore Creating Release Tools or Chatops Capab... (gitlab-com/gl-infra/delivery#2659)
browser_ui/8_monitor/alert_management/alert_with_escalation_policy_spec.rbhas been failing for a while and keeps creating new issues because of the changing error message.Will keep an eye on it, so far failure didn't happen
Nightlies are failing because of a security release (this is expected - but maybe we could do better, e.g., automatically suspend the nightlies?
IMO it's better to simply move all tests in to
gitlabproject and just run via the standard nightly setup using common setup. Then the tests won't be affected by security releases and it will also be somewhat easier to track when issues got introduced since we will have actual commit diffs.I don't believe testing against nightly build specifically has much additional benefit.
Edited by Andrejs Cunskis-
Worth bringing this up in Audit of end-to-end test runs in various enviro... (gitlab-org/quality/engineering-productivity/team#174 - closed) perhaps? I'm not sure what specifically was needed from nightly pipelines when they were added, but overall I'm in favour of unification of pipelines if possible
-
Yeah, we are already discussing it here: gitlab-org/quality/engineering-productivity/team#174 (comment 1282858263)
-
EMEA, Monday - 2023-03-27
AMER, Monday - 2023-03-27
posted by: @vburton
-
CustomersDot Updates
- I've created this QA failure issue in customers-gitlab-com to keep track of the failures in
/spec/ui/self_managed/renew_license_spec.rbthat is caused by bug https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/6121- This also has the same cause as https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/6116+
- Based on recent discussions, this bug should be resolved by https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/5899, but that is currently scheduled for %15.11. It is the end of my day here, but I'll reach out to the team tomorrow to see if we can get eyes on that sooner or if we will need to quarantine or adjust these tests in the meantime
- I've also created an issue for a separate failure in
/spec/ui/self_managed/renew_license_spec.rb: https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/6141- This should be resolved by https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/6938
- I've created this QA failure issue in customers-gitlab-com to keep track of the failures in
-
This should be resolved by https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/6938
The failure didn't appear in the two pipelines after the fix hit staging
Edited by Mark Lapierre
-
APAC, Tuesday - 2023-03-28
posted by: @mlapierre
Edited by Mark Lapierre -
no-opversion of job for gdk as in package-and-test when MR is only quarantining specs? Noticed that in gitlab-org/gitlab!116002 (merged) - https://gitlab.com/gitlab-org/gitlab/-/pipelines/820503977 -> GDK smoke was triggered. Wasn't able to find it in team tasks
-
EMEA, Tuesday - 2023-03-28
-
Handover notes
- Failure in api/1_manage/migration/gitlab_migrat... (gitlab-org/gitlab#402400 - closed) - wasn't able to find the root cause of this flakiness. Please keep an eye on it? So far Import team hasn't commented on this
- gitlab-org/gitlab!116002 (merged) - please also keep an eye on MR - hopefully maintainers will merge this soon
-
AMER, Tuesday - 2023-03-28
posted by: @vburton
-
Updates
-
https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/6140
- The team has notified me that the bug causing this should be fixed within the week. In the meantime, I've opened an MR for a potential workaround while the bug is being fixed: https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/6957
-
https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/6146
- Stale tests, caused by a UI update, just waiting for the fix to be reviewed and merged: https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/6955
-
Failure in api/1_manage/migration/gitlab_migrat... (gitlab-org/gitlab#402400 - closed)
- Started to happen towards the end of my day again.
@rodrigo.tomonarihas been helping from the import team looking into this. Tried to assist as much as I could while still online, but if someone is able to help follow-up while I'm offline I'd greatly appreciate it
- Started to happen towards the end of my day again.
-
gitlab-org/gitlab#402889 (closed)
- Started happening towards the end of my day as well, caused by a handful of user cap feature flags being enabled in Staging. I notified
@dstull, who is looking into a fix for the tests
- Started happening towards the end of my day as well, caused by a handful of user cap feature flags being enabled in Staging. I notified
-
https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/6140
-
I added some notes on the bulk import failure: gitlab-org/gitlab#402400 (comment 1332134900)
Looks like the importer thinks there's a version conflict, even though it's importing from Staging to Staging.
-
APAC, Wednesday - 2023-03-29 - in-progress
posted by: @mlapierre
Title Issue Pipelines Stack Trace Actions Failure in browser_ui/3_create/merge_request/merge_when_pipeline_succeeds_spec.rb| Create Merge requests when merging several times behaves like merge when pipeline succeeds merges after pipeline succeeds
canary, gitlab.com, main, nightly, staging 
1) Create Merge requests when merging several times behaves like merge when pipeline succeeds merges after pipeline succeeds Got 3 failures from failure aggregation block. Shared Example Group: "merge when pipeline succeeds" called from ./qa/specs/f...Status: - investigating
- Assigned
Edited by Mark Lapierre -
A few jobs have failed in the last 20 mins or so because or runner errors. An alert was raised in
#productionso infra is aware. -
Nothing to add that hasn't been reported above, so I had the chance to dig a little into the failure in
gitlab_migration_project_specgitlab-org/gitlab#402400 (comment 1332134900) -
I have to go to the dentist so @jay_mccure has kindly agreed to keep an eye on things for the last hour of my shift.
Signing off now!
Nothing else from me, except for this new failure issue: gitlab-org/gitlab#402981 (closed)
I closed it and gitlab-org/gitlab#402457 (closed) as a duplicate of gitlab-org/gitlab#397133 (closed)
Since the assertion is checking a randomized string, I suspect this is creating additional failure issues rather than linking to an existing issue.
-
EMEA, Wednesday - 2023-03-29
-
Handover notes
Quite a lot of fires today
- Failure in browser_ui/11_fulfillment/saas_user_... (gitlab-org/gitlab#402889 - closed) - do we need to explore quarantining the spec while fix is being worked on to reduce the noise?
- 2023-03-29: Degraded performance on Gitlab.com (gitlab-com/gl-infra/production#8619 - closed) - please keep an eye on discussions in the incident. It would be helpful to learn if the root cause could have been caught earlier
-
Thanks @niskhakova
-
AMER, Wednesday - 2023-03-29
posted by: @vburton
-
Updates
-
https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/6957 has been merged so we should be seeing green CDot pipelines again
- Waiting for pipeline to finish to be merged for gitlab-org/gitlab!116166 (merged), which will remove
user_access_termination_specfrom reliable bucket (root cause is also appears to be bug gitlab-org/gitlab#403001 (closed)) - The fix for Failure in browser_ui/11_fulfillment/saas_user_... (gitlab-org/gitlab#402889 - closed) is currently in review, so I decided to hold off quarantining for now: gitlab-org/gitlab!116053 (merged)
-
https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/6957 has been merged so we should be seeing green CDot pipelines again
-
APAC, Thursday - 2023-03-30 -- in-progress
posted by: @mlapierre
Edited by Mark Lapierre -
Several tests have failed with
Resource at https://gitlab.com/api/v4/runners/:id?private_token=[****] could not be found (403): `{"message":"403 Forbidden - No access granted"}`.I opened an issue to gather them together: gitlab-org/gitlab#403174 (closed)
I started doing some investigation (gitlab-org/gitlab#403174 (comment 1334144010)) but kept getting interrupted by all those other failures
It could be another race condition caused by slow sidekiq processing.
Edited by Mark Lapierre -
I wasted a little time frantically trying to figure out why gitlab-org/gitlab#403180 (closed) was failing
-
There were three other new failures that I haven't investigated further because they only failed once and look like transient flakiness:
mentioned in issue gitlab-org/gitlab#403309 (closed)
-
EMEA, Thursday - 2023-03-30
-
Handover notes
Updates:
- gitlab-org/gitlab!116269 (merged) - should help to reduce the noise
- https://gitlab.com/gitlab-org/gitlab/-/issues/402675#note_1335406717 - we did a huge dive into Staging Ref 500 issues, turned out to be a bug in Omnibus
@vburton @mlapierre could you please assist with the following:
- Failures in multiple E2E tests | api/v4/runners... (gitlab-org/gitlab#403174 - closed) - please continue to investigate this issue. Unfortunately Staging Ref debugging took too much time and I hasn't been able to dive into this, but looks like it's now on Pre as well - which might indicate that it's some runner issue? maybe a new version was released recently?
- gitlab-org/gitlab!116053 (merged) - could it be helpful to share the fix with maintainers for a quicker merge?
-
Failures in multiple E2E tests | api/v4/runners... (gitlab-org/gitlab#403174 - closed) - please continue to investigate this issue. Unfortunately Staging Ref debugging took too much time and I hasn't been able to dive into this, but looks like it's now on Pre as well - which might indicate that it's some runner issue? maybe a new version was released recently?
I left a comment with some additional findings that could possibly be related to project auth update delays again. I also reached out to
@manojmjto see if he may be able to assist in investigating as well.gitlab-org/gitlab!116053 (merged) - could it be helpful to share the fix with maintainers for a quicker merge?
I reached out to
#quality_maintainers, and the fix had been merged
-
AMER, Thursday - 2023-03-30
posted by: @vburton
-
APAC, Friday - 2023-03-31 -- in-progress
posted by: @mlapierre
Edited by Mark Lapierre -
A few tests failed after a few feature flags were enabled on staging:
- gitlab-org/gitlab#403589 (closed)
- gitlab-org/gitlab#403588 (closed)
- gitlab-org/gitlab#403587 (closed)
- gitlab-org/gitlab#403586 (closed)
- gitlab-org/gitlab#403585 (closed)
- gitlab-org/gitlab#403584 (closed)
- gitlab-org/gitlab#403592 (closed)
The feature flags were:
- identity_verification (already enabled on earlier on staging and 2 days ago on prod )
- identity_verification_phone_number
- soft_email_confirmation
- arkose_labs_signup_challenge (already enabled on earlier on staging and 2 days ago on prod )
merge_with_code_owner_in_root_group_specfails with a verification error, which is clearly related to the feature flags. But it seems the other tests fail when trying to create a second group. The tests create the first group and then expect the browser to be at the dashboard, but for some reason it's still at the page showing the first group's details, so when it tries to create another group, it fails because that menu item is found on the dashboard.Discussion with Hinam in Slack
merge_with_code_owner_in_root_group_specpassed afteridentity_verification_phone_numberandsoft_email_confirmationwere disabled, but the other tests still fail. It's strange that they would still fail even thougharkose_labs_signup_challengeandidentity_verificationwere already enabled on staging.Edited by Mark Lapierre -
Oh, almost forgot! I think I know how to fix
unlocking_job_artifacts_across_parent_child_pipelines_specbut didn't have time to try it: gitlab-org/gitlab#403391 (comment 1336075752)
mentioned in issue gitlab-org/gitlab#403592 (closed)
mentioned in issue gitlab-org/gitlab#403589 (closed)
mentioned in issue gitlab-org/gitlab#403588 (closed)
mentioned in issue gitlab-org/gitlab#403587 (closed)
mentioned in issue gitlab-org/gitlab#403586 (closed)
mentioned in issue gitlab-org/gitlab#403584 (closed)
-
EMEA, Friday - 2023-03-31
-
Handover notes
Notes:
- Multiple QA failures due to WaitExceededError (gitlab-org/gitlab#403696 - closed) - declared a production incident due to multiple errors (thanks @vburton for assistance!), so far it looks like the issue was mitigated by itself.
- Quarantine new branch rule E2E spec (gitlab-org/gitlab!116396 - merged) - quarantined flaky spec
- investigated few
- gitlab-org/gitlab#403573 (closed) - fix is in review gitlab-org/gitlab!116343 (merged) so not sure about quarantining this spec
Follow-ups:
- Fix for Fulfillment specs that use go_to_groups (gitlab-org/gitlab!116381 - merged) - please keep an eye on request in maintainers, thanks to Will this should fix fullfilment issues on Staging from #191 (comment 1336084438)
- I wasn't able to follow up on #191 (comment 1336135785) and Failures in multiple E2E tests | api/v4/runners... (gitlab-org/gitlab#403174 - closed)
It was very busy today! Hope there won't be any new incidents
Stepping away
-
AMER, Friday - 2023-03-31
posted by: @vburton
-
Updates
-
gitlab-com/gl-infra/production#8638 (closed) began to occur again today at around
18:30 UTC, and caused similar failures as gitlab-org/gitlab#403696 (closed) showingTimed out waiting for eventfor specs that were pushing to projects. Based on the latest comment in the incident issue, it is looking like this is also caused by gitlab-org/gitlab#403001 (closed)- Infradev issue was created to address this: gitlab-org/gitlab#403862 (closed)
- Staging Ref also started failing again today with 500 errors /
Failed to determine the write location of the primary database: https://gitlab.com/gitlab-org/gitlab/-/issues/402675#note_1337491390- I unfortunately did not get enough time to look into this as it was towards the end of my day and I needed to fix some CDot failures at that time
-
gitlab-com/gl-infra/production#8638 (closed) began to occur again today at around
- A deleted user
closed
