Multiple E2E failures in main/master due to DB performance issues with new instances

Summary

Repeated failures in master pipelines however cannot be reliably reproduced on local (intermittent), similar failures against GDK appear to resolve after gdk restart. Trying to find offending commit via git bissect also proved to be unsuccessful.

First failing pipeline: https://gitlab.com/gitlab-org/gitlab/-/pipelines/806371641

Latest: https://gitlab.com/gitlab-org/gitlab/-/pipelines/806640401

Failing specs:

• ./qa/specs/features/browser_ui/1_manage/user/follow_user_activity_spec.rb
• ./qa/specs/features/browser_ui/3_create/merge_request/suggestions/batch_suggestion_spec.rb
• ./qa/specs/features/browser_ui/3_create/merge_request/suggestions/custom_commit_suggestion_spec.rb
• ./qa/specs/features/browser_ui/4_verify/ci_variable/pipeline_with_protected_variable_spec.rb
• ./qa/specs/features/browser_ui/3_create/web_ide/review_merge_request_spec.rb
• ./qa/specs/features/browser_ui/3_create/web_ide/open_web_ide_from_diff_tab_spec.rb
• ./qa/specs/features/api/1_manage/group_access_token_spec.rb
• ./qa/specs/features/browser_ui/3_create/merge_request/view_merge_request_diff_patch_spec.rb
• ./qa/specs/features/browser_ui/3_create/merge_request/revert/reverting_merge_request_spec.rb
• ./qa/specs/features/browser_ui/3_create/merge_request/squash_merge_request_spec.rb
• ./qa/specs/features/browser_ui/3_create/merge_request/cherry_pick/cherry_pick_commit_spec.rb
• ./qa/specs/features/browser_ui/3_create/merge_request/merge_when_pipeline_succeeds_spec.rb
• ./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb
• ./qa/specs/features/ee/browser_ui/2_plan/contribution_analytics_spec.rb
• ./qa/specs/features/browser_ui/3_create/merge_request/revert/revert_commit_spec.rb
• ./qa/specs/features/api/3_create/merge_request/push_options_mwps_spec.rb
• ./qa/specs/features/ee/browser_ui/13_secure/enable_scanning_from_configuration_spec.rb
• ./qa/specs/features/ee/browser_ui/3_create/repository/code_owners_with_protected_branch_and_squashed_commits_spec.rb
• ./qa/specs/features/ee/browser_ui/3_create/merge_request/approval_rules_spec.rb

Stack trace (example as it varies from test to test)

Failure/Error: expect(merge_request).to have_file(revertible_merge_request.file_name)
     
     QA::Support::Repeater::WaitExceededError:
       Page did not fully load. This could be due to an unending async request or loading icon.

Screenshot / HTML page

Expand for screenshot

d

Possible fixes