Consider enhancements the test cases/sessions feature (best case scenario)

Preface

There are a couple of contexts to consider the test case/sessions feature in.

  1. Plan - a manual test planning tool to organize and keep track of test cases with imperative steps.
  2. Verify - a tool for automated e2e tests, when we want store metadata against a test, keep track of history, and visualize reports.

This issue is for considering enhancements to the test case/session feature in the latter, which is an automated context.

Best case scenario first (Pie in the sky)

Ideally...

GitLab scans the test cases out from the actual project repository

Imagine the test cases are not linked with ad-hoc api calls and manual tagging, but populated from the project repository itself.

Since GitLab is a full DevOps platform, and e2e tests ideally live with the code that they are testing, this could present a significant win for GitLab as a whole solution.

Actually...

Currently within the gitlab-org/gitlab ecosystem, E2E tests are developed in isolation from the test cases feature itself, and linked later with API calls.

Controlling the run (quarantining, excluding certain environments) is done with the use of environment variables and hardcoded metadata on the test itself.

Reports are uploaded to external storage from artifacts and run metrics are sent to another third party dashboard.

Consequentially...

Currently:

  • There is a lot of complexity and networking at play for managing e2e tests.
  • The solution is based around team conventions, and can't easily be packaged for customers.

Possibly:

  • The actual test code informs the test cases feature, a single source of truth.
  • Tests and test cases could be associated with arbitrary metadata, including project data, like:
  • Less ad-hoc networking and a more integrated source of truth for test reporting

What could be even cooler (and perhaps less practical):

  • Associating the test metadata with other files that live in the repository. When they change in an MR, run only those e2e tests against the Review app.

Risks & Challenges

  • Parsing different e2e tests in different languages into a common format.
  • Too tight of a coupling to the app code, ex: The test code lives in a different project.

In conclusion

This issue describes one possibility for thinking of enhancing test case/sessions with a best case scenario. Other issues will be used to discuss other strategies.