Automation: fix MV3 E2E ABP tests
Background
We will be switching to MV3 version of ABP in Chrome in March 2024, so we need to adjust our E2E tests to run with the MV3 version of the extension.
Running the tests with MV3 extension in the CI will be done here: #1664 (closed)
As a part of this issue, we will to go through all the existing tests and determine if any code changes are needed, or if some tests need to be skipped (not all features might be supported on MV3)
What to change
- Design: n/a
- Research: n/a
- Spec: n/a
- Legal: n/a
- Development: Here is a list of testcases that need to be updated:
The following tests need to be updated:
-
test abp premium license check retries
- because the background page does not exist anymore, we have to get the responses from the service worker
- AFAIK we only run these in Chrome anyway, so they can be skipped for Chrome MV2 builds and only run on Chrome MV3 builds from now on (ABP MV3 release is planned in less than 2 weeks so by then, Chrome MV2 tests will become redundant)
- updated test case in cucumber: License server responses
-
test abp premium license server responses
- because the background page does not exist anymore, we have to get the errors from the service worker
- AFAIK we only run these in Chrome anyway, so they can be skipped for Chrome MV2 builds and only run on Chrome MV3 builds from now on (ABP MV3 release is planned in less than 2 weeks so by then, Chrome MV2 tests will become redundant)
- updated test case in cucumber: License check retries
-
test advanced tab - filter lists: should go to filter list source page
- here we check that the URL of the filterlist is correct, but MV3 uses different filterlists than MV2
- updated test case in cucumber: Filter list webpages
-
test subscriptions as part of the integration tests: should disable/enable subscriptions
- It think this is a timing issue, MV3 is a bit slower with handling subscriptions and applying filters. Adding a pause before the last 4
expects
(so the last step, when we check that the subscription is enabled) was enough to make the test pass. There is probably a better solution thanbrowser.pause
, but at least this means that the test steps are OK and the test should be passing.
- It think this is a timing issue, MV3 is a bit slower with handling subscriptions and applying filters. Adding a pause before the last 4
-
test subscriptions as part of the integration tests: should add/remove subscriptions
- same issue as above, and same "fix" (in the same place even!) as above
-
Snippet filter list is named ABP filters (ABP Anti-Circumvention Filter List) on MV2 and ABP filters (compliance) ABP Anti-Circumvention Filter List) on MV3, the testcase was updated: https://studio.cucumber.io/projects/283030/test-plan/folders/2012388/scenarios/6606323
-
test advanced tab filter lists: should display default state: ABP filters is named ABP filters (complicance) on MV3, the testcase was update: https://studio.cucumber.io/projects/283030/test-plan/folders/2012388/scenarios/6606319
-
test advanced tab filter lists: should display disabled filters error: we can't subscribe to custom filterlists in MV3, so I've rewritten the testcase a bit so that subscription is no longer necesarry. See the updated testcase here: https://studio.cucumber.io/projects/283030/test-plan/folders/2012388/scenarios/6606373
The following tests should be skipped for MV3, as the features they test are not supported in MV3 (but they should continue to run on MV2!)
- test advanced tab - filter lists: should add a filter list via URL
- test advanced tab - filter lists: should display an error for invalid filter list via URL
- test advanced tab - filter lists: should display disabled filters error
- test subscriptions as part of the integration tests: should add new subscription via link
- test adblocking as part of the smoke tests: should test sitekey --> sitekey doesn't yet work with MV3. However, the test should also be updated for MV2- trucking.com and cook.com domains should be changed church.com and mckowen.com, see updated test case in cucumber: Sitekey
NOTES:
- test options page general tab language: should remove a language --> will fail because of bug, see: https://eyeo.atlassian.net/browse/EE-407
- test ABP IPM in page dialog CTA sometimes fails as the OPD randomly decides not to show up, but afaik, we're seeing the same with MV2 extensions already, so this can be looked at later
- all tests in
test options page general tab language
should be passing, features are supported and work as expected - tests in
test advanced tab - filter lists
, except for the 3 that should be skipped (that are mentioned above) should be passing - We have a bug with updating the filterlist (status is not displayed correctly). I know there is an issue opened for this but I can't find it for the life of me, however these two are likely to fail because of it:
- test advanced tab filter lists: should update a filter list
- test advanced tab filter lists: should update all filter lists
Other considerations
- update localisation tests so they run on MV3 builds, not MV2 builds (we currently don't run them on Firefox, which will remain in MV2 anyway)
- unskip should support multiline paste testcase (fix was merged, see: #1672 (closed))
- set globalRetriesNumber to 2, after most of the pipeline is green most of the time
🙂 (we set it to 0 here: #1673 (closed))
NOTE: this list is currently in progress
Hints for testers
None.
Hints for translators
n/a