Flaky failures in spec/system/existing_customers/view_company_details_spec.rb
Problem
Some specs in spec/system/existing_customers/view_company_details_spec.rb
are causing frequent test failures. As an example, see the last 5 pipelines of this MR and this failure.
- This test is often the cause.
- This test has also been cropping up a few times over the last few days. Example Job #5486727370
This spec file has had a couple of failed flaky spec in pipelines recently, so they have been quarantined as well.:
https://gitlab.com/gitlab-org/customers-gitlab-com/-/jobs/6336964537)
Failures:
1) Viewing company details when on the billing account settings page when a customer has a billing account when changing bill to contact details when changing custom contact with invalid email does not update the billing account contacts
Failure/Error:
within "[data-testid='#{contact_type.dasherize}-details']", wait: zuora_max_wait_time, match: :first do
find('button.dropdown-toggle').click
find('p', text: 'Enter a custom contact').click
fill_in 'First name', with: contact_details[:first_name]
fill_in 'Last name', with: contact_details[:last_name]
fill_in 'Email', with: contact_details[:work_email]
end
Capybara::ElementNotFound:
Unable to find css "[data-testid='bill-to-details']"
[Screenshot Image]: /builds/gitlab-org/customers-gitlab-com/tmp/capybara/failures_r_spec_example_groups_viewing_company_details_when_on_the_billing_account_settings_page_when_a_customer_has_a_billing_account_when_changing_bill_to_contact_details_when_changing_custom_contact_with_in_368.png
# ./spec/fixtures/vcr_cassettes/viewing_company_details/when_on_the_billing_account_settings_page/when_a_customer_has_a_billing_account/when_changing_bill_to_contact_details/when_changing_custom_contact/with_invalid_email/does_not_update_the_billing_account_contacts.yml <- from VCR cassette
# ./spec/system/existing_customers/view_company_details_spec.rb:339:in `change_custom_contact_details'
# ./spec/system/existing_customers/view_company_details_spec.rb:282:in `block (6 levels) in <top (required)>'
# ./spec/support/vcr/zuora_time_traveller.rb:20:in `block (2 levels) in apply'
# ./spec/support/vcr/zuora_time_traveller.rb:39:in `block in freeze_example'
# ./spec/support/vcr/zuora_time_traveller.rb:38:in `freeze_example'
# ./spec/support/vcr/zuora_time_traveller.rb:19:in `block in apply'
# ./spec/support/vcr/zuora_time_traveller.rb:14:in `apply'
# ./spec/support/vcr.rb:99:in `block (2 levels) in <top (required)>'
Finished in 8 minutes 57 seconds (files took 22.88 seconds to load)
773 examples, 1 failure, 8 pending
Failed examples:
rspec ./spec/system/existing_customers/view_company_details_spec.rb:295 # Viewing company details when on the billing account settings page when a customer has a billing account when changing bill to contact details when changing custom contact with invalid email does not update the billing account contacts
Randomized with seed 37615
https://gitlab.com/gitlab-org/customers-gitlab-com/-/jobs/6337092604
Failures:
1) Viewing company details when on the billing account settings page when a customer has a billing account when changing sold to contact details behaves like changing contact details with valid details updates the billing account contacts
Failure/Error:
within "[data-testid='#{contact_type.dasherize}-details']", wait: zuora_max_wait_time, match: :first do
fill_in 'Address 1', with: contact_details[:address1]
fill_in 'Address 2', with: contact_details[:address2]
fill_in 'City', with: contact_details[:city]
fill_in 'Postal Code', with: contact_details[:postal_code]
select(contact_details[:country], from: "billing_account[#{contact_type}_attributes][country]")
select(contact_details[:state], from: "billing_account[#{contact_type}_attributes][state]")
click_button 'Save changes'
end
Capybara::ElementNotFound:
Unable to find css "[data-testid='sold-to-details']"
[Screenshot Image]: /builds/gitlab-org/customers-gitlab-com/tmp/capybara/failures_r_spec_example_groups_viewing_company_details_when_on_the_billing_account_settings_page_when_a_customer_has_a_billing_account_when_changing_sold_to_contact_details_behaves_like_changing_contact_detail_716.png
Shared Example Group: "changing contact details" called from ./spec/system/existing_customers/view_company_details_spec.rb:214
# ./spec/fixtures/vcr_cassettes/viewing_company_details/when_on_the_billing_account_settings_page/when_a_customer_has_a_billing_account/when_changing_sold_to_contact_details/behaves_like_changing_contact_details/with_valid_details/updates_the_billing_account_contacts.yml <- from VCR cassette
# ./spec/system/existing_customers/view_company_details_spec.rb:318:in `change_contacts_details'
# ./spec/system/existing_customers/view_company_details_spec.rb:131:in `block (5 levels) in <top (required)>'
# ./spec/support/vcr/zuora_time_traveller.rb:20:in `block (2 levels) in apply'
# ./spec/support/vcr/zuora_time_traveller.rb:39:in `block in freeze_example'
# ./spec/support/vcr/zuora_time_traveller.rb:38:in `freeze_example'
# ./spec/support/vcr/zuora_time_traveller.rb:19:in `block in apply'
# ./spec/support/vcr/zuora_time_traveller.rb:14:in `apply'
# ./spec/support/vcr.rb:99:in `block (2 levels) in <top (required)>'
Finished in 9 minutes 16 seconds (files took 25.48 seconds to load)
773 examples, 1 failure, 9 pending, 1 error occurred outside of examples
Failed examples:
rspec './spec/system/existing_customers/view_company_details_spec.rb[1:1:2:4:2:1:1]' # Viewing company details when on the billing account settings page when a customer has a billing account when changing sold to contact details behaves like changing contact details with valid details updates the billing account contacts
Randomized with seed 33181
Proposal
Investigate the spec and see if any improvements can be made to reduce flakeyness, or otherwise quarantine the test.
Determine if the test assertions in renders all available billing account customers in the dropdown
are needed since we're verifying the same behavior in frontend unit tests.
Result
Less failures during merged pipelines which will increase developer productivity
Edited by Aman Luthra