Enhancement: Replace behave/gherkin with Playwright automated browser testing
Problem/Opportunity Statement
Gherkin/BDD is useful conceptually, however writing and maintaining more behavioral test suites is difficult. Breakages are hard to diagnose, and there is limited visibility into failures.
As an alternative, Playwright does a lot to reduce the effort required to build and maintain these tests.
Benefits:
-
playwright codegen
launches a browser, records your actions, and saves them to tests -
playwright test
is a full-featured test runner that generates detailed error reports- Running tests with
--debug
launches a full inspector interface, allowing you to step through your tests visually - tests can generate traces, which are effectively a timeline of DOM events allowing you to retroactively step through a failed test
-
expect().toHaveScreenshot()
andexpect().toMatchSnapshot()
allows for full visual comparisons, snapshots (either images or text) are taken, then subsequent runs compare against this. When large changes are made,--update-snapshots
can be used to regenerate all snapshots.
- Running tests with
Considerations:
- E2E testing still needs Openstack credentials, and care would be needed to ensure the CI's integration credentials are not leaked. Playwright again has a partial solution for this, although we would need to experiment with how to keep things secure.
What would success / a fix look like?
In stages,
- A single test should be implemented with Playwright, and added to the CI.
- This may be a larger lift than it sounds, requiring figuring out how to handle secrets properly. Local testing could require something like an
openrc.sh
in a known location (added to .gitignore).
- This may be a larger lift than it sounds, requiring figuring out how to handle secrets properly. Local testing could require something like an
- Reimplement all current behave tests in Playwright
- After a number of separate merged MRs, proving Playwright effective and resilient, remove behave
Edited by Frank Tackitt