Evaluate Effort to Migrate Chemlab E2E Tests to GitLab QA Framework
Summary
This issue will be used to evaluate the effort to migrate the current Chemlab E2E tests in GitLab (under qa/qa/specs/*
) back to the GitLab E2E test framework, as well as determine challenges we may foresee with the migration
Files that use Chemlab
Tests
ee/browser_ui/11_fulfillment/utilization/namespace_storage_limit_spec.rb
ee/browser_ui/11_fulfillment/saas_user_limit_experience_spec.rb
ee/browser_ui/11_fulfillment/purchase/user_registration_billing_spec.rb
ee/browser_ui/11_fulfillment/license/cloud_activation_spec.rb
ee/browser_ui/11_fulfillment/license/license_spec.rb
ee/browser_ui/11_fulfillment/purchase/overage_modal_spec.rb
ee/browser_ui/11_fulfillment/utilization/billing_seats_usage_data_spec.rb
ee/browser_ui/11_fulfillment/purchase/purchase_ci_spec.rb
ee/browser_ui/11_fulfillment/purchase/purchase_storage_spec.rb
ee/browser_ui/11_fulfillment/purchase/upgrade_group_spec.rb
ee/browser_ui/11_fulfillment/utilization/usage_quotas_seats_spec.rb
ee/browser_ui/11_fulfillment/utilization/saas_user_caps_spec.rb
ee/browser_ui/11_fulfillment/utilization/free_namespace_storage_spec.rb
Percentage of Tests
Percentage of test files that use Chemlab out of the total number of test files in qa/specs/features
- Chemlab tests: 15
- Total tests: 374
- Percentage: 4%
Pages
Slack::Page::Oauth
Slack::Page::Chat
Slack::Page::Login
QA::Page::Project::Settings::Services::Slack
QA::Page::Profile::ChatNames::New
QA::Page::Alert::StorageLimit
QA::Page::Alert::FreeTrial
Gitlab::Page::Admin::Dashboard
Gitlab::Page::Admin::Subscription
Gitlab::Page::Group::Settings::Billing
Gitlab::Page::Group::Settings::UsageQuotas
Gitlab::Page::Main::SignUp
Gitlab::Page::Subscriptions::New
Gitlab::Page::Trials::New
Gitlab::Page::Trials::Select
-
Chemlab::Vendor::GitlabHandbook::Page::About
(this originates from https://gitlab.com/gitlab-org/quality/chemlab-library-www-gitlab-com and is in use byfree_trial_spec
, so we would also need to create a new page for this in the GitLab framework)
Percentage of Pages
Percentage of page files that use Chemlab out of the total number of page files in qa/ee/page
and qa/page
-
Chemlab pages: 16 (including the new
GitlabHandbook
page that will need to be created, mentioned above) - Total pages: 342
- Percentage: 5%
Flows
Other
chemlab-library-gitlab.gemspec
qa/lib/gitlab.rb
qa/lib/slack.rb
qa/lib/slack/mixins/browser.rb
qa.rb
qa/qa/runtime/browser.rb
- Stub files (ex:
qa/lib/gitlab/page/subscriptions/new.stub.rb
)
POC
As a POC, we can attempt to migrate one of the tests as part of this spike to help get a better estimate for the full scope as well.
Implementation Plan
- Update CustomersDot E2E test framework to not use
data-testid
by default- Once complete, update corresponding Danger rule to reflect these changes
- For each Chemlab page class:
- Create a new corresponding QA page class
- Update the
data-testid
names in the frontend to follow the GitLab E2E test element naming convention- Ex:
foo
will now need to be renamed tofoo-button
- This may require updating some frontend specs as well
- Ex:
- Migrate all tests and flows that use the old Chemlab page class to use the new QA page class
- Update any corresponding elements in the CustomersDot framework (as some
data-testid
s are shared between the two)- Ex:
button :foo, data_testid: 'foo'
now becomesbutton :foo, data_testid: 'foo-button'
, to keep aligned with the CustomersDot naming convention
- Ex:
- Remove Chemlab page class
- Update
reliable_report.rb
to userainbow
instead ofcolorize
- Remove other Chemlab code and stub files in the framework
- Remove Chemlab gem
- Update / deprecate documenation:
- Capybara to Chemlab Migration Guide
- (others TBD)
Risks / Challenges
- Some
data-testid
s are shared between tests in both GitLab and CustomersDot. CustomersDot uses the same naming convention as Chemlab, so renaming thedata-testid
s to match the GitLab naming convention will pose a bit of a challenge / less clean code for CustomersDot (ex:button :foo
would becomebutton :foo, data_testid: 'foo-button'
)- When updating the
data-testid
s in GitLab, this will cause temporary test failures in CustomersDot. We can lessen this time by making sure we have the MR to update CustomersDot reviewed and ready to merge before the GitLab MR is merged, and ensuring we are updating these pages /data-testid
s in smaller iterations. Then, once the GitLab MR has propagated to Staging, the CustomersDot MR can be merged right away. Self-managed CustomersDot tests will still fail, however, until the next nightly package is released. - We could also consider updating the CustomersDot framework to be compatible for both conventions temporarily (ex: attempt to find element by
data_testid: 'foo-button'
, if not found, fall back to finding by element namefoo
)
- When updating the
- QA page classes will be larger / have more methods than their Chemlab counterparts, due to differences mentioned here
- Some
data-testid
s are dynamically generated in the frontend. Chemlab offered a clean way to still define these elements, making it easier to see what elements a page is using. We can't add a dynamic element to aview
block in a QA page class, so we lose some of that visibility. - The majority of the Fulfillment tests cannot be run locally or verified in MRs, which makes verifying
data-testid
changes longer, since we would need to alter the spec to run locally, have a local CustomersDot instance running, etc. -
Performance: As a POC, I migrated
ee/browser_ui/11_fulfillment/purchase/upgrade_group_spec.rb
and ran the test against Staging from my local machine. The test was a bit slower when using the GitLab page objects rather than Chemlab page objects. This mostly seemed to be due to the test filling out the customer info and credit card form slower than Chemlab on the GitLab checkout page (usingfill_element
/select_element
methods)- With Chemlab: 1 minute 38.52 seconds
- Without Chemlab: 1 minute 56.30 seconds
Business Justification
A summary on why we should migrate from Chemlab to the GitLab E2E framework can be found below. Please see Migrate Chemlab E2E Tests in the GitLab Project... (&65) for more specific details and examples.
Benefits:
- Simplifies E2E testing in GitLab by having one single framework instead of two, reducing confusion, cognitive load and maintenance costs.
- Reduces barriers to developer contribution by using the same underlying tools and guidelines as feature specs which teams are already familiar with (ex: Capybara vs. Watir,
data-testid
naming conventions, etc.) - Leverages robust and more mature existing capabilities in the GitLab E2E framework like built-in logging, detecting changed selectors, dynamic selectors and better scoping that are not yet supported in Chemlab (just to name a few examples). This, in turn, allows for quicker troubleshooting, catching stale tests earlier, and consistency in development.
While there is room for improvement in the GitLab E2E framework, only about 4% of total E2E test files in the GitLab project use Chemlab today. It does not make sense to continue to support and maintain two separate frameworks for such a small portion of tests. The effort to migrate all tests to Chemlab would also cost significantly more effort, versus optimizing the current GitLab E2E framework where necessary.
We should address migrating away from Chemlab sooner rather than later, before more tests are added in the future, where migration costs will be increased even more.
In summary, migrating away from Chemlab will allow us to reduce costs, encourage contribution, leverage existing capabilities and simplify the testing framework.