Pipeline Triage Report 2021-09-13 to 2021-09-17
DRI
Please review the responsiblities and guidelines if you have not done so recently.
Above all else, please remember that the aim of pipeline triage is to identify problems and try to get them resolved (ideally) before they impact users.
You can view the Testcases project list of issues for non-quarantined test results to help identify failures and see if any were missed when reviewing pipelines.
| EMEA | AMER | APAC | |
|---|---|---|---|
| Primary | @zeffmorgan | @sliaquat | |
| Secondary | @richard.chong | @a_mcdonald |
Issues carried over from last week
| Test | Issue | Notes |
|---|---|---|
add_batch_comments_in_merge_request_spec |
 stale element reference after show.submit_pending_reviews
|
Quarantine MR gitlab-org/gitlab!70057 (closed) / Potential fix MR gitlab-org/gitlab!70065 (merged) |
maven_gradle_repository_spec |
expected QA::Page::Project::Job::Show.successful?({:timeout=>800}) to be truthy, got false
|
SET having a look at this, informed infrastructure of maven issue |
Current known issues for CustomersDot project
Summary
Highlights
Staging 
Staging Orchestrated
Pre-Production
Production
Nightly
Master
Remaining Issues
Handoff template
This is a template you can copy into the comments each day to help facilitate the hand-off between DRIs in different time zones.
| Test | Issue | Pipeline | Failure / Notes | Action Taken |
|---|---|---|---|---|
Merged
- link to MR with note about the effects
In Progress
- link to MR
Designs
Relates to
Activity
added Quality label
assigned to @sliaquat, @richard.chong, @zeffmorgan, and @a_mcdonald
Monday 2021-09-13, APAC
Test Issue Pipeline Failure / Notes Action Taken container_registry_spec
Staging Fabrication of QA::Resource::Repository::Commit using the API failed (400)Investigating add_batch_comments_in_merge_request_specstaging, production, master stale element referenceaftershow.submit_pending_reviewsPotential fix MR group_saml_non_enforced_sso_spec.rbMaster expected to find text "Already signed in with SAML for saml_sso_group_bc43fe36e1ff6e3c"...Tagged SET push_options_mwps_specStaging Wait condition not met after 60 secondsSET is CC'd epics_milestone_dates_specStaging expected: 201 got: 400 on create_projectPinged on issue code_owners_with_protected_branch_and_squashed_commits_specmaster Unable to find css "[data-qa-selector=\"squash_checkbox\"],.qa-squash-checkbox"Mentioned on issue change_namespace_specCustomerDot e2e:saas timed out after 60 seconds, Namespace change failedPinged SET purchase_license_specCustomerDot e2e:self-managed QA::Runtime::Retrier::ExceededMaxAttemptsErroroncopy_activation_codePinged SET Edited by Sanad Liaquat
Thanks @sliaquat for setting up the issue for us.
-
Monday 2021-09-13, AMER
Test Issue Pipeline Failure / Notes Action Taken project_security_dashboard_specproduction Wait condition not metformerge_request.has_pipeline_status?('passed')Under investigation add_batch_comments_in_merge_request_specproduction, canary, staging stale element referenceaftershow.submit_pending_reviewsPotential fix MR pipelines_for_merged_results_and_merge_trains_specproduction "Expected content 'The changes were merged' but it did not appear." Page did not fully load. This could be due to an unending async request or loading icon.Pinged SET. Seems a common issue in this widget and related to ongoing work user_views_commit_diff_patch_specproduction API 400 creating file resource - server 503 error Passed on subsequent runs cancelling_merge_request_in_merge_train_specstaging expect(todos).not_to have_todo_list, 'This user should not have any to-do item but found at least one!'Transient issue under investigation merge_when_pipeline_succeeds_specstaging Unable to find visible css "[data-qa-selector=\"merge_button\"],.qa-merge-button" with text "Merge when pipeline succeeds"Under investigation push_options_mwps_specmaster Wait condition not met after 60 secondsSET is CC'd Edited by Zeff Morgan -
Tuesday 2021-09-14, APAC
Test Issue Pipeline Failure / Notes Action Taken add_file_template_specmaster Error 500 getting groups via the API reverting_merge_request_spec.rb:Master submit_commit_button did not appear on QA::Page::Component::CommitModal as expectedNoted on issue cancelling_merge_request_in_merge_train_specStaging Unable to find css "[data-qa-selector=\"project_sidebar\"],.qa-project-sidebar"Pinged SET custom_commit_suggestion_specstaging expected to find text "Applying suggested change for testing purposes."Pinged SET push_options_remove_source_branch_specMaster There was a problem creating the merge requestchange_namespace_specCustomerDot e2e:saas QA::Runtime::Retrier::ExceededMaxAttemptsErroroncreate_subscriptionPinged SET Edited by Sanad Liaquat
mentioned in issue gitlab-org/gitlab#340763 (closed)
-
Tuesday 2021-09-14, AMER
Test Issue Pipeline Failure / Notes Action Taken pipelines_for_merged_results_and_merge_trains_specproduction - multi expect(show).to have_pipeline_status('passed'), 'Expected the merge request pipeline to pass.' QA::Support::Repeater::WaitExceededError:Wait condition not met after 60 secondsMonitored by SET. Seems a common issue in this widget and related to ongoing work Multiple specs with MRs failing intermittentlycanary, canary Failure/Error: merge_request.has_pipeline_status?('passed') QA::Support::Repeater::WaitExceededError:Wait condition not met after 60 secondsMonitored by SET. Suspect environment load issue that exacerbates sync problem. cancelling_merge_request_in_merge_train_specstaging @project.add_member RetriesExceededErrorUnder investigation merge_when_pipeline_succeeds_specstaging Unable to find visible css "[data-qa-selector=\"merge_button\"],.qa-merge-button" with text "Merge when pipeline succeeds"Under investigation. Error is intermittent. Edited by Zeff Morgan -
Wednesday 2021-09-15, APAC
Test Issue Pipeline Failure / Notes Action Taken rubygems_registry_specMaster QA::Page::Project::Job::Show.successful?({:timeout=>800})to return true, got falsePinged SET enable_sast_from_configuration_specmaster Got 2 failures from failure aggregation block "test SAST status is Enabled"Pinged SET epics_milestone_dates_specStaging expected: 201 got: 400 on create_projectPinged on issue push_options_labels_specmaster There was a problem creating the merge requestPined on issue group_audit_logs_1_specmaster Failure/Error: expect(page).to have_text("Group '#{group.path}' was successfully created")Pinged on issue add_list_delete_branches_specnightly Unable to find css "[data-qa-selector=\"delete_branch_button\"],.qa-delete-branch-button"Pinged on issue bulk_import_group_specnightly Failure/Error: expect(import_page).to have_imported_group(imported_group.path, wait: 300)Pinged SET free_trial_specstaging timed out after 30 seconds, waiting for true condition on ...Pinged SET merge_when_pipeline_succeeds_specstaging Unable to find visible css "[data-qa-selector=\"merge_button\"],.qa-merge-button" with text "Merge when pipeline succeeds"Under investigation Multiple tests failing on pipeline status confirmation Canary, Canary, Canary Wait condition not met after 60 secondsUnder investigation squash_merge_request_specStaging Unable to find css "[data-qa-selector=\"squash_checkbox\"],.qa-squash-checkboxInvestigating remove_license_specCustomerDot e2e:SaaS, CustomerDot e2e:self-managed QA::Runtime::Retrier::ExceededMaxAttemptsError on create_subscriptionPinged Set real_time_assignee_specmaster expected QA::Page::Project::Issue::Show not to have assigneePinged SET Edited by Sanad Liaquat -
@zeffmorgan Apologies for not adding handover notes this week so far. There wasn't anything to point out other than what was reported that day.
Today, there were some failures due to pipelines not starting on canary but when I went in to see the project, it had passed. A potential solution could be to increase the wait time. Other than this, no consistent failures seen.
-
@sliaquat no apologies necessary! You keep excellent records which is generally good enough. Always glad to work with you on pipeline triage.
-
Wednesday 2021-09-15, AMER
Test Issue Pipeline Failure / Notes Action Taken pipelines_for_merged_results_and_merge_trains_specproduction, canary expect(show).to have_pipeline_status('passed'), 'Expected the merge request pipeline to pass.' QA::Support::Repeater::WaitExceededError:Wait condition not met after 60 secondsRemoving from reliable while investigating to reduce noise and unnecessary blocker cancelling_merge_request_in_merge_train_specstaging expect(todos).not_to have_todo_list, 'This user should not have any to-do item but found at least one!'Transient issue under investigation Multiple specs fail on MR creation staging, staging Assignees is invalidin API and UI creationUnable to repro manually on staging. Stale? Edited by Zeff Morgan -
Thursday 2021-09-16, APAC
Test Issue Pipeline Failure / Notes Action Taken free_trial_specstaging timed out after 30 seconds, waiting for true condition on ...Bug. Asked for help on s_fulfillment slack Multiple specs fail on MR creation staging Assignees is invalidin API and UI creationPotential fix epics_milestone_dates_specStaging expected: 201 got: 400 on create_projectUnder investigation bulk_import_group_specnightly Failure/Error: expect(import_page).to have_imported_group(imported_group.path, wait: 300)Under investigation maven_gradle_repository_specstaging-orchestrated expected QA::Page::Project::Job::Show.successful?({:timeout=>800})to be truthy, got falseadd_list_delete_branches_specmaster Unable to find css "[data-qa-selector=\"delete_branch_button\"]Edited by Sanad Liaquat -
Thursday 2021-09-16, AMER
Test Issue Pipeline Failure / Notes Action Taken pipelines_for_merged_results_and_merge_trains_speccanary expect(show).to have_pipeline_status('passed'), 'Expected the merge request pipeline to pass.' QA::Support::Repeater::WaitExceededError:Wait condition not met after 60 secondsRemoved from reliable while investigating to reduce noise and unnecessary blocker conan_repository_specproduction expected QA::Page::Project::Packages::Index not to have package "conantest"Issue not reproducible. Notified counterpart. custom_commit_suggestion_specstaging expected to find text "Applying suggested change for testing purposes."Pinged SET ssh_key_support_specproduction Unable to find link "E2E test key:...Cleaned up excess keys with rake task container_registry_specStaging Fabrication of QA::Resource::Repository::Commit using the API failed ...with 400 and 500 errors on different runsInvestigating. Intermittent failure. Passed on subsequent runs. epics_milestone_dates_specStaging expected: 201 got: 500 on create_projectPinged on issue. Passed on retry. restrict_push_protected_brnach_specmaster QA::Support::Run::CommandError:...remote: GitLab: You are not allowed...Pinged in issue push_options_mwps_specmaster Wait condition not met after 60 secondsSET is CC'd license_compliance_specmaster Resource::Project.fabricate_via_api! ... Failed to GET ... 500Succeeded on retry. CC'd SET Edited by Zeff Morgan -
Friday 2021-09-17, APAC
Test Issue Pipeline Failure / Notes Action Taken push_options_mwps_specmaster Wait condition not met after 60 secondsPinged on issue Mutliple tests Staging Fabrication of QA::Resource::Sandbox using the API failed (429) withRate limit exceeded`MR for fix bulk_import_group_specnightly Failure/Error: expect(import_page).to have_imported_group(imported_group.path, wait: 300)Under investigation add_list_delete_branches_specnightly Unable to find css "[data-qa-selector=\"delete_branch_button\"]purchase_license_specCustomerDot e2e:self-managed timed out after 30 seconds, waiting for true condition on #<Watir::Anchor: located: false; {:text=>/license file/, :tag_name=>"a"}>Pinged Set Edited by Sanad Liaquat -
@zeffmorgan The rate limit on staging should be fixed with: gitlab-com/gl-infra/k8s-workloads/gitlab-com!1245 (closed)
Nothing else notable today.
Friday 2021-09-17, AMER
Test Issue Pipeline Failure / Notes Action Taken epics_milestone_dates_specStaging create_project–expected: 201 got: 400Passed on retry. Left a comment in the issue. cancelling_merge_request_in_merge_train_specStaging @project.add_member–QA::Support::Repeater::RetriesExceededError:– Retry condition not met after 3 attemptsPassed on retry. Left a comment in the issue for me to circle back later. license_compliance_specStaging Orchestrated, Master Resource::Project.fabricate_via_api!–QA::Resource::ApiFabricator::InternalServerError– (500):{"message":"500 Internal Server Error"}Passed on retry. Left a comment in the issue. push_options_title_description_specMaster There was a problem creating the merge requestPinged in the issue. enable_sast_from_configuration_specMaster merge_request.merge_immediately!–Capybara::ElementNotFound:– Unable to find css"[data-qa-selector=\"merge_button\"],.qa-merge-button"Edited by Richard Chongmarked this issue as related to #100 (closed)
Please register or sign in to reply