E2E: Investigate RSpec Predicate Matcher Not Waiting When Asserting on Modals to Be Present
Problem
As seen in https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/6904, remove_license_spec
would occasionally fail when asserting expect(license).to have_activate_subscription_modal
with stack trace
1) User removes activated self-managed subscription with legacy license successfully switches to use a cloud license
Failure/Error: expect(license).to have_activate_subscription_modal
expected `#<QA::Vendor::Gitlab::Page::SubscriptionPage:0x00007fa6f3b26f28 @site=#<QA::Runtime::Site:0x00007fa70...tp://10.0.0.43:3000/admin/subscription" title="Subscription · Admin Area · GitLab">, @url_param=nil>.has_activate_subscription_modal?` to be truthy, got false
# ./spec/ui/self_managed/remove_license_spec.rb:45:in `block (4 levels) in <module:QA>'
# ./page/base.rb:20:in `perform'
# ./spec/ui/self_managed/remove_license_spec.rb:42:in `block (3 levels) in <module:QA>'
At first glance, it appears that when using an RSpec assertion, find_element
in our predicate methods in base.rb
is not always being called, which waits by default.
Interestingly, if we change this to just be the predicate method called directly from the page, license.has_activate_subscription_modal?
, the test does wait as expected.
Detailed findings copied from MR:
Click to expand
I ran this spec 10 times with expect(license).to have_activate_subscription_modal
and 10 times with license.has_activate_subscription_modal?
against Staging, here were the results:
expect(license).to have_activate_subscription_modal
- Failed 5 times out of 10
- While the test is running, when the modal is in a transition state (still somewhat transparent and in the process of opening), the test fails during this time
Example stacktrace:
1) User removes activated self-managed subscription with legacy license successfully switches to use a cloud license
Failure/Error: expect(license).to have_activate_subscription_modal
expected `#<QA::Vendor::Gitlab::Page::SubscriptionPage:0x00007fa6f3b26f28 @site=#<QA::Runtime::Site:0x00007fa70...tp://10.0.0.43:3000/admin/subscription" title="Subscription · Admin Area · GitLab">, @url_param=nil>.has_activate_subscription_modal?` to be truthy, got false
# ./spec/ui/self_managed/remove_license_spec.rb:45:in `block (4 levels) in <module:QA>'
# ./page/base.rb:20:in `perform'
# ./spec/ui/self_managed/remove_license_spec.rb:42:in `block (3 levels) in <module:QA>'
license.has_activate_subscription_modal?
- Failed 0 times out of 10
- I noticed at times the test will actually pause momentarily to wait for the modal to appear.
It's as if the RSpec predicate matcher is not hitting the method we define in base.rb
, and thus not hitting the wait that is part of find_element
. I tried to do a breakpoint debugging session during this time but I was able to trace through and see that it does get to that point, so I'm not quite sure what could be causing this.
A couple other interesting things I noticed:
If I comment out license.visit.activate_cloud_licensing
, so that it tries to look for the modal on a page where it doesn't exist, the stacktrace becomes:
1) User removes activated self-managed subscription with legacy license successfully switches to use a cloud license
Failure/Error: expect(license).to have_activate_subscription_modal
Watir::Wait::TimeoutError:
timed out after 30 seconds, waiting for true condition on #<Watir::Heading: located: false; {:text=>/Activate subscription/, :tag_name=>"h4"}>
# ./page/base.rb:215:in `find_element'
# ./page/base.rb:42:in `block (3 levels) in singleton class'
# ./page/base.rb:53:in `public_send'
# ./page/base.rb:53:in `block (3 levels) in singleton class'
# ./spec/ui/self_managed/remove_license_spec.rb:45:in `block (4 levels) in <module:QA>'
# ./page/base.rb:20:in `perform'
# ./spec/ui/self_managed/remove_license_spec.rb:42:in `block (3 levels) in <module:QA>'
And we can see we are actually reaching that wait here
Proposal
Investigate the root cause for this issue and determine a solution.
Some things to try:
- Look into recent Watir 7.x upgrade as a possible culprit
- Revert Watir back to previous version 6.19 locally, run the test and see if the issue still persists