Find and fix root cause of flakiness in local E2E test suite

MR: Workaround workspaces e2e suite local flakiness (!188656 - merged) • Chad Woolley • 18.0

Description

Overview

There is some problem with the Workspaces E2E tests, where the "Allow agent" button cannot be clicked, and the test fails.

This only happens:

  1. On local development environment, never on CI
  2. When multiple RSpec test files are run as part of a suite.

Investigation

Here are debugging notes from internal slack thread. The related code from the debugging is on branch caw-ws-debug-capybara

I spent some time debugging this. Here’s what I found:

  1. Good news, there’s a workaround. Patch attached, I can turn it into an MR
  2. I don’t think it’s directly related to the popver MR. I think we (I) started thinking that based on Paul’s comment above about them both being absolute positioned.
  3. It is definitely related to the visibility of the button element. For some reason, the element is not visible in the failure case, which has “something” to do with some other capybara tests having been run previously in the same suite.
  4. Calling screenshot_and_open_image makes it visible for some reason (and also moves it down 50 pixels).
  5. I pushed a branch, caw-ws-debug-capybara, which has a lot of attempts to debug why it’s not visible. I could find no reason: It’s positioned in the window’s bounds, and doesn’t seem to have any other element hiding it, the HTML seems identical, and waiting does not make it visible. Only screenshot_and_open_image seems to make it visible. We could debug further into all the exact styling, but I timeboxed my debugging and gave up, since I already had a workaround patch to just click it even though it’s invisible.

Here’s the fix, just the code and in patch form. If you think this is OK, I’ll make an MR out of it (and make a linkable issue to continue debugging “when we get around to it” which will likely be never unless it crops up again).

    begin
      allow_agent_button = find('button', text: 'Allow agent', visible: true, wait: 2)
    rescue Capybara::ElementNotFound
      # Work around bug when sometimes the button is not visible when this test is run in local
      # development as part of a suite. See branch `caw-ws-debug-capybara` for more info and debugging code.
      allow_agent_button = find('button', text: 'Allow agent', visible: false)
    end

    allow_agent_button.click
Subject: [PATCH] gitlab.iml update
---
Index: ee/spec/features/remote_development/workspaces_spec.rb
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/ee/spec/features/remote_development/workspaces_spec.rb b/ee/spec/features/remote_development/workspaces_spec.rb
--- a/ee/spec/features/remote_development/workspaces_spec.rb	(revision c15cfceda2c6519c8439a2695042fed0bef3abb1)
+++ b/ee/spec/features/remote_development/workspaces_spec.rb	(date 1744832982202)
@@ -62,7 +62,17 @@
       click_button 'Allow'
       wait_for_requests
     end
-    click_button 'Allow agent'
+
+    begin
+      allow_agent_button = find('button', text: 'Allow agent', visible: true, wait: 2)
+    rescue Capybara::ElementNotFound
+      # Work around bug when sometimes the button is not visible when this test is run in local
+      # development as part of a suite. See branch `caw-ws-debug-capybara` for more info and debugging code.
+      allow_agent_button = find('button', text: 'Allow agent', visible: false)
+    end
+
+    allow_agent_button.click
+
     expect(page).to have_selector(gitlab_badge_selector, text: 'Allowed')
 
     nil

References

Acceptance criteria

  • Find root cause of problem and revert/remove workaround
Edited by 🤖 GitLab Bot 🤖