Pipeline Triage Report from 2023-04-24 to 2023-04-29

DRI

Please review the responsiblities and guidelines if you have not done so recently.

Above all else, please remember that the aim of pipeline triage is to identify problems and try to get them resolved (ideally) before they impact users.

You can view the Testcases project list of issues for non-quarantined test results to help identify failures and see if any were missed when reviewing pipelines.

Feel free to use the dri gem to help report and triage issues.

DRI Schedule

Week of	APAC	EMEA	AMER
2023-04-24	Mark Lapierre (@mlapierre)	Will Meek (@willmeek)	Tiffany Rea (@treagitlab)
2023-05-01	Sanad Liaquat (@sliaquat)	Sofia Vistas (@svistas)	Désirée Chevalier (@dchevalier2)

Carryovers

• Failure in browser_ui/3_create/repository/push_... (gitlab-org/gitlab#408334 - closed)
• Failure in ee/browser_ui/3_create/repository/pu... (gitlab-org/gitlab#408335 - closed)
• Failure in ee/browser_ui/1_manage/group/group_l... (gitlab-org/gitlab#408310 - closed)
• undefined method `namespaces_user_namespace_url... (gitlab-org/gitlab#408156 - closed)
• Failure in browser_ui/8_monitor/incident_manage... (gitlab-org/gitlab#408321 - closed)
• Failure in browser_ui/1_manage/login/login_via_... (gitlab-org/gitlab#408317 - closed)
• Failure in ee/browser_ui/3_create/repository/as... (gitlab-org/gitlab#403189 - closed)
• Failure in ee/browser_ui/10_govern/security_rep... (gitlab-org/gitlab#408331 - closed)
• Failure in ee/browser_ui/10_govern/vulnerabilit... (gitlab-org/gitlab#407467 - closed)
• Failure in api/3_create/merge_request/push_opti... (gitlab-org/gitlab#346425)
• Failure in ee/browser_ui/10_govern/fake_tests_s... (gitlab-org/gitlab#408324 - closed)
• Failure in api/4_verify/file_variable_spec.rb |... (gitlab-org/gitlab#407836 - closed)
• Failure in browser_ui/4_verify/runner/register_... (gitlab-org/gitlab#390184 - closed)
• Failure in browser_ui/4_verify/ci_job_artifacts... (gitlab-org/gitlab#403391 - closed)
• Failure in browser_ui/1_manage/login/login_via_... (gitlab-org/gitlab#408132 - closed)
• Failure in browser_ui/1_manage/login/login_via_... (gitlab-org/gitlab#408163 - closed)
• Failure in browser_ui/1_manage/login/oauth_logi... (gitlab-org/gitlab#407850 - closed)
• Failure in api/12_systems/gitaly/gitaly_mtls_sp... (gitlab-org/gitlab#401132 - closed)
• Failure in browser_ui/5_package/package_registr... (gitlab-org/gitlab#378221 - closed)
• Failure in browser_ui/5_package/package_registr... (gitlab-org/gitlab#408144 - closed)
• Failure in browser_ui/8_monitor/incident_manage... (gitlab-org/gitlab#404353 - closed)
• Failure in ee/browser_ui/4_verify/new_discussio... (gitlab-org/gitlab#402768 - closed)
• Failure in api/1_manage/group_access_token_spec... (gitlab-org/gitlab#404520 - closed)
• Failure in api/1_manage/group_access_token_spec... (gitlab-org/gitlab#404521 - closed)
• Failure in browser_ui/8_monitor/incident_manage... (gitlab-org/gitlab#407810 - closed)
• Failure in ee/browser_ui/3_create/repository/fi... (gitlab-org/gitlab#407771 - closed)
• Failure in ee/browser_ui/4_verify/pipelines_for... (gitlab-org/gitlab#388662 - closed)
• Failure in ee/browser_ui/13_secure/enable_scann... (gitlab-org/gitlab#406971 - closed)
• Failure in browser_ui/4_verify/ci_variable/pipe... (gitlab-org/gitlab#399149 - closed)
• Failure in browser_ui/13_secure/merge_request_l... (gitlab-org/gitlab#397701 - closed)
• Failure in ee/browser_ui/10_govern/vulnerabilit... (gitlab-org/gitlab#407828 - closed)
• Failure in browser_ui/3_create/merge_request/su... (gitlab-org/gitlab#407843 - closed)
• Failure in browser_ui/3_create/merge_request/me... (gitlab-org/gitlab#393854 - closed)
• Failure in browser_ui/2_plan/design_management/... (gitlab-org/gitlab#407600 - closed)
• Failure in browser_ui/2_plan/design_management/... (gitlab-org/gitlab#407479 - closed)
• Failure in browser_ui/1_manage/project/add_proj... (gitlab-org/gitlab#393626 - closed)
• Failure in browser_ui/1_manage/project/invite_g... (gitlab-org/gitlab#393628 - closed)
• Failure in ee/browser_ui/10_govern/vulnerabilit... (gitlab-org/gitlab#407639 - closed)
• Failure in ee/browser_ui/3_create/repository/as... (gitlab-org/gitlab#387081 - closed)
• Failure in api/1_manage/group_access_token_spec... (gitlab-org/gitlab#407827 - closed)
• Failure in api/1_manage/group_access_token_spec... (gitlab-org/gitlab#399212 - closed)
• Failure in browser_ui/2_plan/design_management/... (gitlab-org/gitlab#407478 - closed)
• Failure in ee/api/1_manage/import/import_github... (gitlab-org/gitlab#407297 - closed)
• Failure in api/1_manage/import/import_github_re... (gitlab-org/gitlab#407296 - closed)
• Failure in browser_ui/2_plan/issue/create_issue... (gitlab-org/gitlab#408113 - closed)
• Failure in browser_ui/8_monitor/incident_manage... (gitlab-org/gitlab#408118 - closed)
• Failure in browser_ui/3_create/web_ide/add_new_... (gitlab-org/gitlab#408076 - closed)
• [Test] browser_ui/5_package/package_registry/ru... (gitlab-org/gitlab#366099 - closed)
• Failure in browser_ui/2_plan/issue/real_time_as... (gitlab-org/gitlab#408064 - closed)
• Failure in browser_ui/3_create/repository/push_... (gitlab-org/gitlab#408077 - closed)
• Failure in ee/browser_ui/13_secure/enable_scann... (gitlab-org/gitlab#398193 - closed)
• Failure in browser_ui/5_package/infrastructure_... (gitlab-org/gitlab#378534 - closed)
• Failure in browser_ui/3_create/repository/prote... (gitlab-org/gitlab#407812 - closed)
• Failure in ee/browser_ui/2_plan/iterations/crea... (gitlab-org/gitlab#392919 - closed)
• Failure in ee/browser_ui/2_plan/iterations/assi... (gitlab-org/gitlab#392920 - closed)
• Failure in browser_ui/4_verify/ci_job_artifacts... (gitlab-org/gitlab#403180 - closed)
• Failure in ee/browser_ui/2_plan/contribution_an... (gitlab-org/gitlab#407728 - closed)
• Failure in ee/browser_ui/10_govern/fix_vulnerab... (gitlab-org/gitlab#407737 - closed)
• Failure in browser_ui/3_create/repository/push_... (gitlab-org/gitlab#407741 - closed)
• Failure in ee/browser_ui/2_plan/epic/epics_mana... (gitlab-org/gitlab#407742 - closed)

Current known issues for CustomersDot project

added Quality label

assigned to @treagitlab, @svistas, and @mlapierre

So far only known failures, Review App misbehaving (but fine now), and a failure due to preprod instability.

APAC, Monday - 2023-04-24

posted by: @mlapierre

Title	Issue	Pipelines	Stack Trace	Actions
Failure in browser_ui/1_manage/login/login_via_oidc_with_gitlab_as_idp_spec.rb | Manage OIDC creates GitLab OIDC application and uses it to login	gitlab-org/gitlab#408317 (closed)	nightly	Failure/Error: gitlab.pullQA::Service::Shellout::CommandError: Command: docker pull CE failed ✘First happened in https://gitlab.com/gitlab-org/quality/nightly/-/jobs/4152874422.Related test case: https://gitlab.com/gitlab-org/gitlab/-/quality/test_ca...	Status: new Assigned Potential fixes: [QUARANTINE] [nightly] Manage OIDC cr...
Failure in browser_ui/4_verify/runner/register_group_runner_spec.rb | Verify Group runner registration user registers a new group runner	gitlab-org/gitlab#390184 (closed)	master, nightly	Failure/Error: expect(page).to have_content(executor) expected to find text "qa-runner-1675312525" in "Skip to content GitLab / 1 Help G group-with-ci-cd-58a0b7b0d1bd26ec Subgroup information Epics 0 Issues 0 Merge requests 0 Security & Compliance CI...	Status: new Assigned Potential fixes: [QUARANTINE] [nightly] Verify Group r... Draft: Add wait for EE job timeouts gitaly: Do not start gitaly-ruby Update gitlab_spec to reduce brittleness Move node-exporter settings to unders... Symlink the public ssh key to /etc/ss... Update curl to 8.0.1 to resolve Febru...
Failure in browser_ui/4_verify/ci_job_artifacts/unlocking_job_artifacts_across_parent_child_pipelines_spec.rb | Verify Unlocking job artifacts across parent-child pipelines without strategy:depend when latest child pipeline failed unlocks job artifacts...	gitlab-org/gitlab#403391 (closed)	canary, main	Failure/Error: job.id = project.job_by_name(job_name):idNoMethodError: undefined method ' for nil:NilClass### Screenshotverify_unlocking_job_artifacts_across_parent-child_pipelines_without_strategy_de_2023-03-30-17-52-23.460(/uploads/acd3925161d4d18d...	Status: new Assigned Potential fixes: QA tests on unlocking of job artifact...
Failure in ee/browser_ui/3_create/repository/assign_code_owners_spec.rb | Create Codeowners merge request assigns code owners as approvers	gitlab-org/gitlab#403189 (closed)	canary, gitlab.com, main, staging	Failure/Error: Resource::MergeRequest.fabricate do |merge_request| merge_request.project = project merge_request.target_new_branch = false merge_request.source_branch = branch_name + '-patch' merge_request.target_branch = branch_name merge_request.no...	Status: test-environment Assigned Potential fixes: [15.10] Fix automatically-retried job...
Failure in browser_ui/3_create/web_ide_new/add_new_directory_in_web_ide_spec.rb | Create Add a directory in Web IDE when a directory with the same name already exists throws an error	gitlab-org/gitlab#387609 (closed)	nightly	Failure/Error: ide.wait_for_ide_to_loadQA::Support::Repeater::WaitExceededError: Wait failed after 60 secondsFirst happened in https://gitlab.com/gitlab-org/quality/nightly/-/jobs/3577340352.Related test case: https://gitlab.com/gitlab-org/gitlab/-/q...	Status: flaky-test Assigned Potential fixes: E2E - add new upload file VSCode spec... [QUARANTINE] [nightly] Create Add a d... Fixing web_ide_spec by adding more ti...

Hello

There is a fix for the following 3 failures:

• gitlab-org/gitlab#408333 (closed)
• gitlab-org/gitlab#408334 (closed)
• gitlab-org/gitlab#408335 (closed)

These failures started with the mirror_only_branches_match_regex feature flag being enabled in staging. The fix should be backwards compatible and should work without the FF turned off also.

Just to clarify that I swapped with @willmeek, I will be on call next week.

Changed description too

FYI I also made the change accordingly in the schedules !148 (merged)

assigned to @willmeek and unassigned @svistas

changed the description

EMEA, Monday - 2023-04-24

posted by: @willmeek

Title	Issue	Pipelines	Stack Trace	Actions
Production/Staging E2E pipelines - QA user gets blocked and API requests return 403, can block deployments	gitlab-org/gitlab#375166 (closed)	gitlab.com, staging	Around 3:30 PM UTC on 09/22/2022, multiple errors were reported to #qa-staging similar to the following:An unexpected error occurred while reporting test results in issues.The error occurred in job: https://ops.gitlab.net/gitlab-org/quality/staging/-...	Status: investigating Assigned Potential fixes: Skip failure issue creation on blocke...
Failure in browser_ui/4_verify/pipeline/include_multiple_files_from_a_project_spec.rb | Verify Include multiple files from a project runs the pipeline with composed config	gitlab-org/gitlab#408498 (closed)	gitlab.com	QA::Resource::Errors::ResourceNotFoundError: Resource at https://gitlab.com/api/v4/runners/22896762?private_token=**** could not be found (403): {"message":"403 Forbidden - No access granted"}.Correlation Id: a1ce5806cb4047fd63e74898cf539140Sentry Ur...	Status: new Assigned Potential fixes:
Failure in browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can dismiss a vulnerability with a reason from mr security widget	gitlab-org/gitlab#406889 (closed)	staging	QA::Resource::Errors::ResourceFabricationFailedError: Fabrication of QA::EE::Resource::VulnerabilityItem using the API failed (200) with {"data":{"vulnerabilityCreate":null},"errors":{"message":"The resource that you are attempting to access does not...	Status: new Assigned Potential fixes:
Failure in browser_ui/8_monitor/incident_management/recovery_alert_closes_correct_incident_spec.rb | Monitor Recovery alert when using Prometheus integration behaves like triggers recovery alert only closes the correct incident	gitlab-org/gitlab#408321 (closed)	gitlab.com, main	Failure/Error: expect(index).to have_no_incident(title: resolve_title), 'Did not expect to see the resolved incident in the Open tab' Did not expect to see the resolved incident in the Open tabShared Example Group: "triggers recovery alert" called fr...	Status: new Assigned Potential fixes: Adds custom error message for expects
Failure in ee/browser_ui/3_create/repository/pull_mirroring_over_http_spec.rb | Create Pull mirror a repository over HTTP configures and syncs a (pull) mirrored repository with password auth	gitlab-org/gitlab#408335 (closed)	staging	Failure/Error: mirror_settings.update(source_project_uri) # rubocop:disable Rails/SaveBangURI::InvalidURIError: bad URI(is not URI?): "https://*****:*****@staging.gitlab.com/gitlab-qa-sandbox-group-6/qa-test-2023-04-21-09-12-27-65ab47609c9ab85f/pull-...	Status: new Assigned Potential fixes: E2E test: remove badge from mirror ur...
Failure in browser_ui/3_create/repository/push_mirroring_over_http_spec.rb | Create Push mirror a repository over HTTP configures and syncs a (push) mirrored repository	gitlab-org/gitlab#408333 (closed)	staging	Failure/Error: mirror_settings.update(target_project_uri) # rubocop:disable Rails/SaveBangURI::InvalidURIError: bad URI(is not URI?): "https://*****:*****@staging.gitlab.com/gitlab-qa-sandbox-group-6/qa-test-2023-04-21-09-12-18-b90afe033feb96a7/push-...	Status: stale-test Assigned Potential fixes: E2E test: remove badge from mirror ur...
Failure in browser_ui/13_secure/merge_request_license_widget_spec.rb:231 | Secure License merge request widget when a CycloneDX SBOM file exists manage licenses from the merge requestfai	gitlab-org/gitlab#397701 (closed)	staging	Retry failed after 5 attempts/home/gitlab/qa/qa/support/repeater.rb:68:in repeat_until'/home/gitlab/qa/qa/support/retrier.rb:44:in retry_until'/home/gitlab/qa/qa/specs/features/ee/browser_ui/13_secure/merge_request_license_widget_spec.rb:239:in block...	Status: test-environment Assigned Potential fixes: Update SBOM test on merge_request_lic...
Failure in browser_ui/3_create/web_ide/add_new_directory_in_web_ide_spec.rb | Create Add a directory in Web IDE when user adds a new empty directory shows successfully but not able to be committed	gitlab-org/gitlab#407141 (closed)	nightly	Failure/Error: ide.wait_for_ide_to_loadQA::Support::Repeater::WaitExceededError: Wait failed after 60 secondsFirst happened in https://gitlab.com/gitlab-org/quality/nightly/-/jobs/4104857530.Related test case: https://gitlab.com/gitlab-org/gitlab/-/q...	Status: new Assigned Potential fixes: Fixing web_ide_spec by adding more ti... [QUARANTINE] [nightly] Create Add a d...
Failure in browser_ui/1_manage/login/login_via_oidc_with_gitlab_as_idp_spec.rb | Manage OIDC creates GitLab OIDC application and uses it to login	gitlab-org/gitlab#408317 (closed)	nightly	Failure/Error: gitlab.pullQA::Service::Shellout::CommandError: Command: docker pull CE failed ✘First happened in https://gitlab.com/gitlab-org/quality/nightly/-/jobs/4152874422.Related test case: https://gitlab.com/gitlab-org/gitlab/-/quality/test_ca...	Status: new Assigned Potential fixes: [QUARANTINE] [nightly] Manage OIDC cr...
Failure in browser_ui/4_verify/runner/register_group_runner_spec.rb | Verify Group runner registration user registers a new group runner	gitlab-org/gitlab#390184 (closed)	master, nightly	Failure/Error: expect(page).to have_content(executor) expected to find text "qa-runner-1675312525" in "Skip to content GitLab / 1 Help G group-with-ci-cd-58a0b7b0d1bd26ec Subgroup information Epics 0 Issues 0 Merge requests 0 Security & Compliance CI...	Status: new Assigned Potential fixes: [QUARANTINE] [nightly] Verify Group r... Draft: Add wait for EE job timeouts gitaly: Do not start gitaly-ruby Update gitlab_spec to reduce brittleness Move node-exporter settings to unders... Symlink the public ssh key to /etc/ss... Update curl to 8.0.1 to resolve Febru...
Failure in browser_ui/5_package/container_registry/online_garbage_collection_spec.rb | Package Container Registry Online Garbage Collection runs the online garbage collector tool	gitlab-org/gitlab#371463 (closed)	pre	Failure/Error: expect(job).to be_successful(timeout: 3900) expected QA::Page::Project::Job::Show.successful?({:timeout=>3900}) to be truthy, got falseFirst happened in https://ops.gitlab.net/gitlab-org/quality/preprod/-/jobs/7817906.Related test case...	Status: new Assigned Potential fixes:
Multiple E2E failures	gitlab-org/customers-gitlab-com#6293 (closed)		rubyFailures: 1) User changes plan namespace when from group to group with add-on purchase only does not allow user to change namespace Failure/Error: portal_sign_in_as(customer.email, customer.password) Watir::Wait::TimeoutError: timed out after 30 ...	Status: new Assigned Potential fixes:

@mlapierre , @willmeek , @treagitlab - The DBRE team is looking for a DRI in quality to help with tests for the PG14 upgrade - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16443#note_1365114889. Shall I assign you three based on the timezone as my understanding is we would need to monitor our e2e test pipeline in staging.

Issue template - https://gitlab.com/gitlab-com/gl-infra/db-migration/-/blob/master/.gitlab/issue_templates/pg14_upgrade.md

The Staging upgrade is scheduled from 10:30 am - 2:30 pm ET hence am adding @treagitlab and @willmeek to the meeting invite. Please let me know if you have any concerns.

Thanks Vincy. It kicks off 3.30pm my time, so I'll be there to kick off pipelines etc but I'll likely be handing over before they complete.

AMER, Monday - 2023-04-24

posted by: @treagitlab

Title	Issue	Pipelines	Stack Trace	Actions
Failure in ee/browser_ui/3_create/repository/pull_mirroring_over_http_spec.rb | Create Pull mirror a repository over HTTP configures and syncs a (pull) mirrored repository with password auth	gitlab-org/gitlab#408335 (closed)	staging	Failure/Error: mirror_settings.update(source_project_uri) # rubocop:disable Rails/SaveBangURI::InvalidURIError: bad URI(is not URI?): "https://*****:*****@staging.gitlab.com/gitlab-qa-sandbox-group-6/qa-test-2023-04-21-09-12-27-65ab47609c9ab85f/pull-...	Status: new Assigned Potential fixes: E2E test: remove badge from mirror ur...
Failure in ee/browser_ui/3_create/repository/assign_code_owners_spec.rb | Create Codeowners merge request assigns code owners as approvers	gitlab-org/gitlab#387081 (closed)	gitlab.com, main, master, nightly, staging	Failure/Error: approvers = show.approversCapybara::ElementNotFound: Unable to find css "data-qa-selector=\"approver_list_content\",.qa-approver-list-content"First happened in https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/8908281.Related te...	Status: new Assigned Potential fixes: Draft: Change FF yml flag to test qa ... [QUARANTINE] [nightly] Create Codeown... Update test and remove from quarantine

Pretty quiet day today on my end.

Staging-ref blew up for just one deployment. Almost all the tests failed, but then passed again after the next deployment. Looks like a network change initiated by kubernetes after deployment caused services to lose track of each other.

More in https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/staging-ref/-/issues/84

APAC, Tuesday - 2023-04-25

posted by: @mlapierre

Title	Issue	Pipelines	Stack Trace	Actions
Failure in ee/browser_ui/3_create/repository/project_templates_spec.rb | Create Project templates when built-in successfully imports the project using template	gitlab-org/gitlab#407699 (closed)	main	Failure/Error: new_page.choose_namespace(namespace)Capybara::Ambiguous: Ambiguous match, found 3 elements matching visible button "qa-test-2023-04-18-12-23-18-b15943e642697bbb" that is not disabledFirst happened in https://gitlab.com/gitlab-org/gitla...	Status: new Assigned Potential fixes:
New license key not generated after a legacy subscription is upgraded	gitlab-org/customers-gitlab-com#5059		<---Please read thisBefore opening a new issue, make sure to search for keywords in the issuesfiltered by the "bug" label in relevant trackers:- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=bug- https://gitlab.com/gitlab-org/customers...	Status: new Assigned Potential fixes:

Quiet other than the staging-ref mentioned , and a few other failures already reported earlier.

FYI PG14 upgrade tomorrow (26th April) https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16443#note_1365114889

EMEA, Tuesday - 2023-04-25

posted by: @willmeek

Title	Issue	Pipelines	Stack Trace	Actions
Production/Staging E2E pipelines - QA user gets blocked and API requests return 403, can block deployments	gitlab-org/gitlab#375166 (closed)	gitlab.com, staging	Around 3:30 PM UTC on 09/22/2022, multiple errors were reported to #qa-staging similar to the following:An unexpected error occurred while reporting test results in issues.The error occurred in job: https://ops.gitlab.net/gitlab-org/quality/staging/-...	Status: investigating Assigned Potential fixes: Skip failure issue creation on blocke...
Failure in browser_ui/3_create/repository/push_mirroring_over_http_spec.rb | Create Push mirror a repository over HTTP configures and syncs a (push) mirrored repository	gitlab-org/gitlab#408333 (closed)	staging	Failure/Error: mirror_settings.update(target_project_uri) # rubocop:disable Rails/SaveBangURI::InvalidURIError: bad URI(is not URI?): "https://*****:*****@staging.gitlab.com/gitlab-qa-sandbox-group-6/qa-test-2023-04-21-09-12-18-b90afe033feb96a7/push-...	Status: stale-test Assigned Potential fixes: E2E test: remove badge from mirror ur...
Failure in ee/browser_ui/3_create/repository/pull_mirroring_over_http_spec.rb | Create Pull mirror a repository over HTTP configures and syncs a (pull) mirrored repository with password auth	gitlab-org/gitlab#408335 (closed)	staging	Failure/Error: mirror_settings.update(source_project_uri) # rubocop:disable Rails/SaveBangURI::InvalidURIError: bad URI(is not URI?): "https://*****:*****@staging.gitlab.com/gitlab-qa-sandbox-group-6/qa-test-2023-04-21-09-12-27-65ab47609c9ab85f/pull-...	Status: new Assigned Potential fixes: E2E test: remove badge from mirror ur...
If MR diff is not ready, Overview page has never ending spinner ("Checking if mr can be merged")	gitlab-org/gitlab#398559 (closed)		### SummaryAs surfaced by this automated test: https://gitlab.com/gitlab-org/gitlab/-/issues/392984 (Also occurred in staging and prod)As we now load diffs asynchronously, if a user navigates to the Overview tab of an MR before the diffs are availabl...	Status: new Assigned Potential fixes:
Failure in qa/specs/features/api/3_create/repository/files_spec.rb | Status code 400	gitlab-org/gitlab#241762 (closed)	canary, gitlab.com, main, staging	Failures: 1) API basics user creates a project with a file and deletes them afterwards Failure/Error: expect_status(200) expected: 200 got: 400 (compared using ==) # /usr/local/bundle/gems/airborne-0.3.4/lib/airborne/request_expectations.rb:36:in exp...	Status: investigating Assigned Potential fixes:
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an auto-remediation MR from mr security widget	gitlab-org/gitlab#407467 (closed)	canary, gitlab.com	Failure/Error: merge_request.resolve_vulnerability_with_mr(remediable_vuln_name)Capybara::Ambiguous: Ambiguous match, found 2 elements matching visible css "data-qa-selector=\"vulnerability_report_grouped\",.qa-vulnerability-report-grouped"First happ...	Status: flaky-test Assigned Potential fixes:
Failure in browser_ui/8_monitor/incident_management/recovery_alert_closes_correct_incident_spec.rb | Monitor Recovery alert when using Prometheus integration behaves like triggers recovery alert only closes the correct incident	gitlab-org/gitlab#408321 (closed)	canary, gitlab.com, main	Failure/Error: expect(index).to have_no_incident(title: resolve_title), 'Did not expect to see the resolved incident in the Open tab' Did not expect to see the resolved incident in the Open tabShared Example Group: "triggers recovery alert" called fr...	Status: new Assigned Potential fixes: Adds custom error message for expects

FYI if staging-ref has PG problems again there are troubleshooting tips in these issues:

• https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/staging-ref/-/issues/84
• https://gitlab.com/gitlab-org/gitlab/-/issues/402675

AMER, Tuesday - 2023-04-25

posted by: @treagitlab

Title	Issue	Pipelines	Stack Trace	Actions
Failure in ee/browser_ui/4_verify/pipeline_for_project_mirror_github_spec.rb | Verify Pipeline for project mirrors Github user commits to GitHub triggers CI pipeline	gitlab-org/gitlab#408738 (closed)	staging	Capybara::ElementNotFound: Unable to find css "data-qa-selector=\"project_import_row\"data-qa-source-project=\"gitlab-qa-github/test-project\",.qa-project-import-row"First happened in https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/9873109.R...	Status: new Assigned Potential fixes:
After creating an MR, "MR has no changes" displayed to user for too long	https://gitlab.com/gitlab-org/gitlab/-/issues/407115		<code data-sourcepos="7:151-7:403">### SummaryAfter creating a very simple MR (1 small file added) the This merge request contains no changes message is displayed for ~10 seconds, which is concerning for users as they think their MR has no changes.This is a recent regression as we are...</code>	Status: new Assigned Potential fixes:
Production/Staging E2E pipelines - QA user gets blocked and API requests return 403, can block deployments	gitlab-org/gitlab#375166 (closed)	gitlab.com, staging	Around 3:30 PM UTC on 09/22/2022, multiple errors were reported to #qa-staging similar to the following:An unexpected error occurred while reporting test results in issues.The error occurred in job: https://ops.gitlab.net/gitlab-org/quality/staging/-...	Status: investigating Assigned Potential fixes: Skip failure issue creation on blocke...
Failure in ee/browser_ui/3_create/repository/assign_code_owners_spec.rb | Create Codeowners merge request assigns code owners as approvers	gitlab-org/gitlab#387081 (closed)	gitlab.com, main, master, nightly, staging	Failure/Error: approvers = show.approversCapybara::ElementNotFound: Unable to find css "data-qa-selector=\"approver_list_content\",.qa-approver-list-content"First happened in https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/8908281.Related te...	Status: investigating Assigned Potential fixes: Draft: Change FF yml flag to test qa ... [QUARANTINE] [nightly] Create Codeown... Update test and remove from quarantine
Failure in browser_ui/3_create/repository/protected_tags_spec.rb | Create Repository tags when not protected behaves like successful tag creation can be created by developer_user	gitlab-org/gitlab#407812 (closed)	canary	Failure/Error: Page::Project::Tag::Index.perform(&:click_new_tag_button)Capybara::ElementNotFound: Unable to find css "data-qa-selector=\"new_tag_button\",.qa-new-tag-button"Shared Example Group: "successful tag creation" called from ./qa/specs/featu...	Status: new Assigned Potential fixes:

We did tokens rotation today

• https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1266
• https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23511

I followed https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/blob/main/runbooks/rotating-credentials.md but that really only got me as far as 4 tokens (QA_GITHUB_ACCESS_TOKEN, QA_GITLAB_CI_TOKEN, QA_TEST_SESSION_TOKEN, QA_THIRD_PARTY_DOCKER_PASSWORD). The rest either don't have anything written in doc or rely on Andrey and Sofia to regenerate them If you know how to, please help.

APAC, Wednesday - 2023-04-26

posted by: @mlapierre

Title	Issue	Pipelines	Stack Trace	Actions
Failure in browser_ui/3_create/snippet/create_project_snippet_with_multiple_files_spec.rb | Create Multiple file snippet creates a project snippet with multiple files	gitlab-org/gitlab#408767 (closed)	pre	Failure/Error: expect(snippet).to have_file_content(file_content(i), i) expected QA::Page::Dashboard::Snippet::Show to have file content "9 file content" and 9First happened in https://ops.gitlab.net/gitlab-org/quality/preprod/-/jobs/9876267.Related ...	Status: new Assigned Potential fixes:

Pretty quiet apart from issues already reported, the token rotation, and lots and lots of 403s.

AMER, Wednesday - 2023-04-26

posted by: @treagitlab

Title	Issue	Pipelines	Stack Trace	Actions
Failure in ee/browser_ui/4_verify/pipeline_for_project_mirror_github_spec.rb | Verify Pipeline for project mirrors Github user commits to GitHub triggers CI pipeline	gitlab-org/gitlab#408738 (closed)	staging	Capybara::ElementNotFound: Unable to find css "data-qa-selector=\"project_import_row\"data-qa-source-project=\"gitlab-qa-github/test-project\",.qa-project-import-row"First happened in https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/9873109.R...	Status: new Assigned Potential fixes:
Failure in ee/browser_ui/3_create/repository/assign_code_owners_spec.rb | Create Codeowners merge request assigns code owners as approvers	gitlab-org/gitlab#387081 (closed)	gitlab.com, main, master, nightly, staging	Failure/Error: approvers = show.approversCapybara::ElementNotFound: Unable to find css "data-qa-selector=\"approver_list_content\",.qa-approver-list-content"First happened in https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/8908281.Related te...	Status: new Assigned Potential fixes: Draft: Change FF yml flag to test qa ... [QUARANTINE] [nightly] Create Codeown... Update test and remove from quarantine
Failure in browser_ui/9_data_stores/project/create_project_badge_spec.rb | Data Stores Create project badge creates project badge successfully	gitlab-org/gitlab#408877 (closed)	pre	Failure/Error: expect(project_settings).to have_notice('New badge added.') expected QA::Page::Project::Settings::Main.has_notice?("New badge added.") to be truthy, got falseFirst happened in https://ops.gitlab.net/gitlab-org/quality/preprod/-/jobs/98...	Status: new Assigned Potential fixes:
Failure in ee/browser_ui/2_plan/epic/epics_management_spec.rb | Plan Epics Management Resources created via API Visit epic first closes and reopens an epic	gitlab-org/gitlab#404523 (closed)	canary, main	Failure/Error: expect(show).to have_system_note('closed') expected QA::EE::Page::Group::Epic::Show to have system note "closed"First happened in https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/9676159.Related test case: https://gitlab.com/git...	Status: new Assigned Potential fixes:
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can dismiss a security finding with reason from pipeline security tab	gitlab-org/gitlab#406607 (closed)	master	Failure/Error: Resource::Repository::ProjectPush.fabricate do |push| push.project = project push.directory = Pathname.new(File.join(EE::Runtime::Path.fixtures_path, 'secure_premade_reports')) push.commit_message = 'Check in premade security reports'...	Status: new Assigned Potential fixes:
Failure in spec/ui/purchase/cancel_plan_spec.rb	gitlab-org/customers-gitlab-com#6347 (closed)		rubyUser cancels auto renew of SaaS subscription can cancel and resume auto renew from billing page Failure/Error: to_create { |instance| instance.create } QA::Helper::ApiRequestHelper::ResourceCreationFailedError: Failed to create subscription # ./h...	Status: new Assigned Potential fixes:

• Customer-dot continues to be unstable because of Zuora.
• Staging seems to be back to normal now with gitlab-com/gl-infra/production#9134 (closed) resolved.
• Staging-ref still not happy. Lots of 403s and the latest pipeline failed because tests couldn't create any pipeline https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/staging-ref/-/issues/86
• PG14 upgrade dry run is finished. Issue https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23527 was opened to determine when QA should step in.

Staging has been misbehaving a bit. Could be related to gitlab-com/gl-infra/production#8655 (closed)

APAC, Thursday - 2023-04-27

posted by: @mlapierre

Title	Issue	Pipelines	Stack Trace	Actions
Failure in api/5_package/container_registry_spec.rb | Package Container Registry pushes, pulls image to the registry and deletes tag	gitlab-org/gitlab#408930 (closed)	canary.staging, staging	QA::Support::Repeater::WaitExceededError: Wait failed after 60 secondsFirst happened in https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/jobs/9885419.Related test case: https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/348001....	Status: new Assigned Potential fixes:
Failures on staging-ref | issues_shortcut_button did not appear on QA::Page::Main::Menu as expected	gitlab-org/gitlab#408927 (closed)	staging-ref	1) Plan Custom email customizes email with additional text Failure/Error: Flow::Login.sign_in_as_admin QA::Page::Validatable::PageValidationError: issues_shortcut_button did not appear on QA::Page::Main::Menu as expected # ./qa/page/validatable.rb:15...	Status: new Assigned Potential fixes:
Failure in purchase_ci_minutes_spec | TimeoutError waiting for /You have successfully purchased CI minutes/	gitlab-org/customers-gitlab-com#6349 (closed)		ruby 1) User purchases CI minutes pack with group namespace with an active subscription behaves like successful purchase shows up on manage plan page and usage quota page Failure/Error: expect(group_quota_page).to have_ci_purchase_successful_alert Wa...	Status: new Assigned Potential fixes:

EMEA, Thursday - 2023-04-27

posted by: @willmeek

Title	Issue	Pipelines	Stack Trace	Actions
Failure in ee/api/2_plan/epics_milestone_dates_spec.rb | Plan Epics milestone dates API updates epic dates when removing issue	gitlab-org/gitlab#408979 (closed)	gitlab.com	Failure/Error: expect_status(200) expected: 200 got: 404 (compared using ==)First happened in https://ops.gitlab.net/gitlab-org/quality/production/-/jobs/9892676.Related test case: https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347957....	Status: new Assigned Potential fixes:
Failure in ee/browser_ui/10_govern/security_reports_spec.rb | Govern Security Reports displays the Dependency List	gitlab-org/gitlab#408965 (closed)	staging	Failure/Error: expect(dependency_list).to have_dependency_count_of number_of_dependencies_in_fixture expected QA::EE::Page::Project::Secure::DependencyList.has_dependency_count_of?(13) to be truthy, got falseFirst happened in https://ops.gitlab.net/g...	Status: new Assigned Potential fixes:
Failure in ee/browser_ui/11_fulfillment/utilization/saas_user_caps_spec.rb | Fulfillment Utilization when admin sets user cap limit for group shows members over limit as pending for approvals	gitlab-org/gitlab#408958 (closed)	staging	Failure/Error: Page::Group::Menu.perform(&:click_group_general_settings_item)Capybara::ElementNotFound: Unable to find css "data-qa-selector=\"sidebar_menu_link\"data-qa-menu-item=\"Settings\",.qa-sidebar-menu-link" within #<Capybara::Node::Element t...	Status: new Assigned Potential fixes:
Failure in ee/browser_ui/3_create/repository/assign_code_owners_spec.rb | Create Codeowners merge request assigns code owners as approvers	gitlab-org/gitlab#403189 (closed)	canary, gitlab.com, main, staging	Failure/Error: Resource::MergeRequest.fabricate do &#124;merge_request&#124; merge_request.project = project merge_request.target_new_branch = false merge_request.source_branch = branch_name + '-patch' merge_request.target_branch = branch_name merge_request.no...	Status: test-environment Assigned Potential fixes: [15.10] Fix automatically-retried job...
Production/Staging E2E pipelines - QA user gets blocked and API requests return 403, can block deployments	gitlab-org/gitlab#375166 (closed)	gitlab.com, staging	Around 3:30 PM UTC on 09/22/2022, multiple errors were reported to #qa-staging similar to the following:An unexpected error occurred while reporting test results in issues.The error occurred in job: https://ops.gitlab.net/gitlab-org/quality/staging/-...	Status: investigating Assigned Potential fixes: Skip failure issue creation on blocke...

A few 403s when trying to get runners via API, worth keeping an eye on.

AMER, Thursday - 2023-04-27

posted by: @treagitlab

Title	Issue	Pipelines	Stack Trace	Actions
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an issue from a vulnerability from mr security widget	gitlab-org/gitlab#409002 (closed)	canary	Failure/Error: Resource::MergeRequest.fabricate_via_api do |mr| mr.project = project mr.source_branch = 'secure-mr' mr.target_branch = project.default_branch mr.source = source_repository mr.target = project.default_branch mr.target_new_branch = fals...	Status: new Assigned Potential fixes:
Failures on staging-ref | issues_shortcut_button did not appear on QA::Page::Main::Menu as expected	gitlab-org/gitlab#408927 (closed)	staging-ref	1) Plan Custom email customizes email with additional text Failure/Error: Flow::Login.sign_in_as_admin QA::Page::Validatable::PageValidationError: issues_shortcut_button did not appear on QA::Page::Main::Menu as expected # ./qa/page/validatable.rb:15...	Status: new Assigned Potential fixes:
Failure in ee/browser_ui/11_fulfillment/utilization/saas_user_caps_spec.rb | Fulfillment Utilization when admin sets user cap limit for group shows members over limit as pending for approvals	gitlab-org/gitlab#408958 (closed)	staging	Failure/Error: Page::Group::Menu.perform(&:click_group_general_settings_item)Capybara::ElementNotFound: Unable to find css "data-qa-selector=\"sidebar_menu_link\"data-qa-menu-item=\"Settings\",.qa-sidebar-menu-link" within #<Capybara::Node::Element t...	Status: new Assigned Potential fixes: Quarantine saas_user_caps_spec
Failure in ee/browser_ui/3_create/repository/assign_code_owners_spec.rb | Create Codeowners merge request assigns code owners as approvers	gitlab-org/gitlab#387081 (closed)	gitlab.com, main, master, nightly, staging	Failure/Error: approvers = show.approversCapybara::ElementNotFound: Unable to find css "data-qa-selector=\"approver_list_content\",.qa-approver-list-content"First happened in https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/8908281.Related te...	Status: new Assigned Potential fixes: Draft: Change FF yml flag to test qa ... [QUARANTINE] [nightly] Create Codeown... Update test and remove from quarantine

Pretty quiet day overall

APAC, Friday - 2023-04-28

posted by: @mlapierre

Title	Issue	Pipelines	Stack Trace	Actions
Failure in browser_ui/3_create/merge_request/revert/reverting_merge_request_spec.rb | Create Merged merge request can be reverted	gitlab-org/gitlab#409021 (closed)	canary	Failure/Error: merge_request.mergeCapybara::ElementNotFound: Unable to find css "data-qa-selector=\"merge_button\",.qa-merge-button"First happened in https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/9900833.Related test case: https://gitlab.co...	Status: new Assigned Potential fixes:

FYI - I'll be covering for Mark until Will is online.

FYI - https://gitlab.slack.com/archives/C0259241C/p1682639791514439?thread_ts=1682037383.442409&cid=C0259241C

APAC, Friday - 2023-04-28

posted by: @sliaquat

Title	Issue	Pipelines	Stack Trace	Actions
cannot load such file -- /home/gitlab/qa/qa/specs/features/browser_ui/3_create/pages/*_spec.rb	gitlab-org/gitlab#409035 (closed)	canary	An error occurred while loading ./qa/specs/features/browser_ui/3_create/pages/*_spec.rb.Failure/Error: __send__(method, file)LoadError: cannot load such file -- /home/gitlab/qa/qa/specs/features/browser_ui/3_create/pages/*_spec.rb# ./qa/specs/runner....	Status: stale-test Assigned Potential fixes: Changing product group for wiki snipp... Update pages spec job with correct path
Failure in ee/browser_ui/11_fulfillment/utilization/saas_user_caps_spec.rb | Fulfillment Utilization when admin sets user cap limit for group shows members over limit as pending for approvals	gitlab-org/gitlab#408958 (closed)	staging	Failure/Error: Page::Group::Menu.perform(&:click_group_general_settings_item)Capybara::ElementNotFound: Unable to find css "data-qa-selector=\"sidebar_menu_link\"data-qa-menu-item=\"Settings\",.qa-sidebar-menu-link" within #<Capybara::Node::Element t...	Status: new Assigned Potential fixes: Quarantine saas_user_caps_spec

EMEA, Friday - 2023-04-28

posted by: @willmeek

Title	Issue	Pipelines	Stack Trace	Actions
Staging - sanity job timed out but log showed executing for 8 minutes, blocked deployment	gitlab-org/gitlab#409119	staging	On Staging a job failed that was part of a sanity pipeline, a deployment blocker.https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/9907523The failure was a timeout, executing for more than one hour.However looking at the job it was noted:- The...	Status: Assigned Potential fixes:
Production/Staging E2E pipelines - QA user gets blocked and API requests return 403, can block deployments	gitlab-org/gitlab#375166 (closed)	gitlab.com, staging	Around 3:30 PM UTC on 09/22/2022, multiple errors were reported to #qa-staging similar to the following:An unexpected error occurred while reporting test results in issues.The error occurred in job: https://ops.gitlab.net/gitlab-org/quality/staging/-...	Status: investigating Assigned Potential fixes: Skip failure issue creation on blocke...
E2E pipelines - 403 Forbidden - No access granted when fabricating project runners	gitlab-org/gitlab#409089 (closed)	gitlab.com	Across a number of E2E tests in production we are intermittently seeing the following when a test fabricates a project runner:Resource at https://gitlab.com/api/v4/runners/23017652?private_token=MASKED could not be found (403): {"message":"403 Forbid...	Status: investigating Assigned Potential fixes:
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an auto-remediation MR from mr security widget	gitlab-org/gitlab#409046 (closed)	staging	Failure/Error: merge_request.visitQA::Support::Repeater::WaitExceededError: Page did not fully load. This could be due to an unending async request or loading icon.First happened in https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/9904999.Rel...	Status: new Assigned Potential fixes:
Failure in browser_ui/8_monitor/alert_management/automatically_creates_incident_for_alert_spec.rb | Monitor Alert when using Prometheus integration behaves like new alert automatically creates new incident	gitlab-org/gitlab#409054 (closed)	staging	Failure/Error: expect(index).to have_incidentQA::Support::Repeater::WaitExceededError: Wait failed after 60 secondsShared Example Group: "new alert" called from ./qa/specs/features/browser_ui/8_monitor/alert_management/automatically_creates_incident_...	Status: new Assigned Potential fixes:
Failure in ee/browser_ui/4_verify/pipeline_subscription_with_group_owned_project_spec.rb | Verify Pipeline subscription with a group owned project when upstream project new tag pipeline finishes triggers pipeline in downstream project	gitlab-org/gitlab#409055 (closed)	staging	Failure/Error: expect { downstream_project.pipelines.size }.to eventually_eq(2), "There are currently #{downstream_project.pipelines.size} pipelines in downstream project." There are currently 1 pipelines in downstream project.First happened in https...	Status: new Assigned Potential fixes:
cannot load such file -- /home/gitlab/qa/qa/specs/features/browser_ui/3_create/pages/*_spec.rb	gitlab-org/gitlab#409035 (closed)	canary	An error occurred while loading ./qa/specs/features/browser_ui/3_create/pages/*_spec.rb.Failure/Error: __send__(method, file)LoadError: cannot load such file -- /home/gitlab/qa/qa/specs/features/browser_ui/3_create/pages/*_spec.rb# ./qa/specs/runner....	Status: stale-test Assigned Potential fixes: Changing product group for wiki snipp... Update pages spec job with correct path
Failure in ee/browser_ui/4_verify/pipelines_for_merged_results_and_merge_trains_spec.rb | Verify Pipelines for merged results and merge trains merges via a merge train	gitlab-org/gitlab#388729 (closed)	canary.staging, staging	Failure/Error: expect(show).to be_merged, "Expected content 'The changes were merged' but it did not appear." Expected content 'The changes were merged' but it did not appear.First happened in https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/...	Status: new Assigned Potential fixes:
e2e:saas Failed to Create Customer	gitlab-org/customers-gitlab-com#6358 (closed)		ruby 1) User purchases extra storage with group namespace with an active subscription behaves like successful purchase shows up on manage plan page Failure/Error: to_create { |instance| instance.create } QA::Runtime::Retrier::ReachedMaxAttemptsError:...	Status: new Assigned Potential fixes:

AMER, Friday - 2023-04-28

posted by: @treagitlab

Title	Issue	Pipelines	Stack Trace	Actions
Failure in ee/browser_ui/10_govern/vulnerability_management_spec.rb | Govern Vulnerability management in a merge request can create an issue from a vulnerability from mr security widget	gitlab-org/gitlab#409002 (closed)	canary, staging	Failure/Error: Resource::MergeRequest.fabricate_via_api do |mr| mr.project = project mr.source_branch = 'secure-mr' mr.target_branch = project.default_branch mr.source = source_repository mr.target = project.default_branch mr.target_new_branch = fals...	Status: new Assigned Potential fixes:
Failure in ee/browser_ui/3_create/repository/assign_code_owners_spec.rb | Create Codeowners merge request assigns code owners as approvers	gitlab-org/gitlab#403189 (closed)	canary, gitlab.com, main, staging	Failure/Error: Resource::MergeRequest.fabricate do |merge_request| merge_request.project = project merge_request.target_new_branch = false merge_request.source_branch = branch_name + '-patch' merge_request.target_branch = branch_name merge_request.no...	Status: test-environment Assigned Potential fixes: [15.10] Fix automatically-retried job...
cannot load such file -- /home/gitlab/qa/qa/specs/features/browser_ui/3_create/pages/*_spec.rb	gitlab-org/gitlab#409035 (closed)	canary	An error occurred while loading ./qa/specs/features/browser_ui/3_create/pages/*_spec.rb.Failure/Error: __send__(method, file)LoadError: cannot load such file -- /home/gitlab/qa/qa/specs/features/browser_ui/3_create/pages/*_spec.rb# ./qa/specs/runner....	Status: stale-test Assigned Potential fixes: Changing product group for wiki snipp... Update pages spec job with correct path

closed