[E2E] Harden E2E specs that rely on searching filtered search result set

What does this MR do and why?

[E2E] Harden E2E specs that rely on searching filtered search result set

It was noted in #426248 (closed) that under certain circumstances we can see some tests fail if sidekiq hasn't run some background processes that are required for a project to appear in the search result page. Under these conditions we should instead add a retry to the spec, as it isn't expected that the search should be instantaneous given the dependance on sidekiq here.

This may not resolve every issue, but should help with addressing some of the observed flaky failures

How to set up and validate locally

1. I found the best way to simulate a failure here was to add a delay to sidekiq (apply sample diff from below)
2. gdk restart rails
3. bundle exec rspec \ qa/specs/features/browser_ui/9_data_stores/project/invite_group_to_project_spec.rb \ qa/specs/features/browser_ui/9_data_stores/project/project_owner_permissions_spec.rb \ qa/specs/features/ee/browser_ui/9_data_stores/group/share_group_with_group_spec.rb
   1. During the test execution we should see the screenshot as noted in #426248 (closed) on the first try
   2. On subsequent retries (make take 2/3 attempts) the search should work
   3. It may be useful to monitor /api/v4/sidekiq/queue_metrics during the test execution for additional info about sidekiq's status

Sample Diff to add delay in sidekiq

diff --git a/config/initializers/sidekiq.rb b/config/initializers/sidekiq.rb
index 8df12671f265..ce34a1b6e6a5 100644
--- a/config/initializers/sidekiq.rb
+++ b/config/initializers/sidekiq.rb
@@ -32,7 +32,17 @@ def enable_semi_reliable_fetch_mode?
 
 enable_json_logs = Gitlab.config.sidekiq.log_format != 'text'
 
+class DelayedProcessingMiddleware
+  def call(worker, job, queue)
+    sleep(5)
+    yield
+  end
+end
+
 Sidekiq.configure_server do |config|
+  config.server_middleware do |chain|
+    chain.add DelayedProcessingMiddleware
+  end
   config[:strict] = false
   config[:queues] = Gitlab::SidekiqConfig.expand_queues(config[:queues])

Screen recording of test

Screen_Recording_2023-12-14_at_17.44.15

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.