Setup Playwright testing

• Setup Playwright testing into the project (so that one can run it with npm run ...). For now just on headless Chromium. At one desktop-sized viewport size (maybe default 1280x720 is good?).
• Implement one test for one flow (for example, authentication flow with password into Charon Dashboard).
• Integrate it into our CI configuration.
  • Configure it so that any results are stored into artifacts. Make those results be ignored in .gitignore. Some info about that.
  • Configure it so that HTML reports are shown as dynamic environments. See how this is done for test_go job in CI.
  • Enable also JUNIT reports and add get GitLab to parse them.
• Add snapshot assertions at various places in the flow (e.g., after every view change).
  • Make them full page. I hope this is enough and that we do not need to resize viewport which might introduce artifacts.
  • Also configure all snapshots to go under some top-level directory so that it is easy to store them separately (under a different name) into artifacts. Configure storing in artifacts. Make this directory be ignored in .gitignore.
    • I am not sure if output of this directory is needed for HTML report for visual comparisons - I am guessing the HTML report needs both images to be able to show differences? Investigate this.
  • Configure comparison threshold to be 0 (we want pixel perfect comparisons).
  • Add these flags when running Chromium.
• Enable coverage coverage reporting both on frontend and backend.
  • We should get HTML reports where we can easily see which code has coverage and which does not.
  • We should get CI artifacts for coverage, so that GitLab can then color that in MR reviews.
  • For backend, see notes below.
  • For frontend, I think we need vite-plugin-instanbul package with nyc to generate HTML report.
    • I found some instructions and code template.
• Add e2e test for password authentication.
• Add e2e test for code to e-mail authentication. For that testing instance of Charon should be configured with e-mail sending and mailhog testing SMTP server can be used, like here to receive those e-mails and then you can check the e-mail using its simple API.
• Add e2e test which when user exists with e-mail and password and invalid password is used, flow automatically continues to code to e-mail authentication.
• Add e2e test for SIPASS. We should use their testing instance for this.
• Add e2e test for Passkey (both sign-in and sign-up).
• Add e2e test which a) creates an OIDC application template for OIDC Debugger b) creates a test organization c) installs OIDC Debugger in the organization d) get a user to sign-in into this app e) verifies in Charon Dashboard admin view for the organization, that user is visible f) checks activity for the organization g) checks activity for the user who signed-in into the app
• Setup testing so that at every view it is checked that there are not duplicate IDs present in DOM. I think Chromium might print a warning to dev console in such case? Maybe we could just make sure that at every end of every test run, nothing should be in dev console?
  • Is there a way to configure a hook to be run "at every view" - which we could define as "always after waiting for idle network"?
• Setup accessibility testing at every view. It is fine if these tests for now fail.
  • Maybe you can use snapshots for now to allow all existing issues which will then just prevent future issues. We will fix all issues as part of #23.
  • Export scan results as artifacts and make a link to accessibility report (if it is separate from Playwright report) from MR.
  • Does this default accessibility tool for Playwright check focus order?
• With this issue finally been fixed in Chromium I hope that the Chromium version we are using for testing includes it. If it does, then please add another test where in code input box you type in more letters than is the width of the input box, the vertical lines should properly move left and stay aligned with the the underlying characters. You can just use visual testing for this in the test itself just make sure you make a screenshot before and then after typing a long code. Then another screenshot for a so long code that vertical lines disappear.

After you finish this, we will configure it so that during tests it fetches previous snapshots from artifacts of the target branch (it looks like there is API endpoint for this and CI_MERGE_REQUEST_TARGET_BRANCH_NAME predefined variable. But let's first get those artifacts be generated at all. And reports.

The goal is to use Playwright testing for end-to-end testing of various user stories and user flows we want to make sure they keep working. The goal is to prevent regressions. Another goal is also to prevent visual regressions as we change our UI implementations, etc.