Rapid Action - Zero Quarantined QA Tests and Test Framework stabilization

Flakiness in the tests is causing instability in the testsuite. Also, it makes the Quality Triage Rotations overwhelming. We are advocating on helping the developers to write E2E tests as part of feature completion, which requires a permanent fix for this. Also, this is essential to enable package-and-qa job to be run by default on MRs.

Stakeholders

DRI	Task
@mlapierre	gitlab-qa#282 (closed)
@sliaquat	#43663 (closed)
@sliaquat	#55286 (closed)
@at.ramya	Process Improvement - Improve un-quarantining process

Exit Criteria

1. Fixed all tests under Zero Quarantined tests
2. A process for reliable tests and an official on-call schedule
3. Framework Improvements to reduce flakiness

1. Zero Quarantined tests

NOTE: The single source of truth is the board. We are not keeping the below section up-to-date. https://gitlab.com/groups/gitlab-org/-/boards/1385578?label_name[]=QA&label_name[]=Quality&label_name[]=bug

List of Quarantined Tests that need immediate attention:

• features/browser_ui/non_devops/performance_bar_spec.rb- # #38315 (closed)

• features/browser_ui/7_configure/auto_devops/create_project_with_auto_devops_spec.rb- # Failure issue: #118481 (closed)

• features/ee/browser_ui/secure/create_merge_request_with_secure_spec.rb- # #36559 (closed)

• features/ee/browser_ui/secure/security_reports_spec.rb- # Failure issue: #34342 (closed)

• features/ee/browser_ui/3_create/repository/push_rules_spec.rb- # Failure issue: #34978 (closed)

• features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb- # Failure issue: #35690 (closed)

• features/ee/browser_ui/3_create/merge_request/add_batch_comments_in_merge_request_spec.rb- # Failure issue: #118473 (closed)

• features/ee/browser_ui/enablement/elasticsearch/advanced_global_advanced_syntax_search_spec.rb- # Failure issue: #43732 (closed)

• features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb- # Issue: #35370 (closed)

• ee/browser_ui/secure/create_merge_request_with_secure_spec.rb- # #36559 (closed)

• ee/browser_ui/2_plan/issue_boards/delete_group_board_spec.rb- # Failure issue: #34936 (closed)

• ee/browser_ui/1_manage/group/restrict_by_ip_address_spec.rb- # #34351 (closed)

• ee/browser_ui/secure/merge_request_license_widget_spec.rb- # #36696 (closed)

• ee/browser_ui/secure/security_reports_spec.rb- # #38199 (closed)

2. New process for reliable test and updated on-call schedule

• https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/338
• Process to have reliable test bucket Have tests with the reliable tag guard deployments (not allowed to fail) https://gitlab.com/gitlab-org/quality/team-tasks/issues/337#note_285594666

3. Immediate Framework Improvements

The following are the issues that will be addressed as part of this Rapid Action: &705 (closed)

• Improve wait/retry helper modules
• Wait for client-side AJAX requests in E2E tests and Consider waiting for the loading spinner to disappear before all actions - Both are related. (Or redundant?)
• Dynamic Element Validation Iteration: Improve waiting
• Remove Animations
• Remove all_elements method

Moved out

• Improve E2E test performance by using waits more efficiently
• Create Quality Triage package for this list
• MR redefining De-quarantining process.
• Run QA tests in a throttled network #36015 (closed)
• Refactor existing tests to use dynamic element validation. https://gitlab.com/gitlab-org/quality/team-tasks/issues/427
  • Dynamic element validation, not only adds stability to the tests due to it's waiting, it also implicitly tests GitLab.
  • Many Pages are utilizing click_element but not specifying a page / partial that we expect to be on. We should start adding these in to ensure our test suite is waiting a sufficient amount of time, and to test these pages/partials.