Improve E2E test setup - planning

Summary

In general, we set up E2E tests by using Resource classes to create the setup from scratch, and leave any changes in place once the test is complete. For example, if a test needs a project we use Resource::Project to create a new project. Once the test is complete it typically leaves the project behind (we now try to remember to delete the project after the test, but we don't do so consistently). If the next test needs a project with the same content, it creates a new one rather than reusing the previous one.

That works, and it allows us to run the tests on any environment from production environments to fresh local instances. However, it's inefficient in terms of time (creating everything every time is slow) and in terms of space (we leave changes behind that need to be removed later).

We should improve the E2E test setup so that it's faster, cleaner, less brittle, more readable, more efficient, portable, and easy to maintain.

Tasks

  • Identify problems with how we set up E2E tests, as well as opportunities for improvements.
  • Propose solutions that we can select from to develop a plan.
  • Come up with a solution for efficiently (re-)using test setup without unduly increasing maintenance costs or decreasing usability.
  • Come up with a plan to implement the improvements.

Problems and opportunities for improvement

Current problems:

  • Creating everything in every test is slow and inefficient.
  • Tests leave their changes behind by default, creating clutter in test environments.
  • Tests that do clean up after themselves do so even if the test fails, making it harder to troubleshoot the failure.
  • It not clear exactly what changes are made when a resource is fabricated.
    • It's also possible for fabricate! to result in a resource with different attributes when used to create a new resource vs. when used to get an existing resource.

Potential problems to avoid:

  • Prefilling an environment outside of tests could increase maintenance costs and decrease usability.

Proposal (The plan)

  1. Allow Resources to be reused in a way that avoids conflicts

    • If a test doesn't make changes to resources that could interfere with other tests, mark the resources as reusable.
    • When a resource is created, if it's marked as reusable check first to see if another test has already created it.
    • When tests or reusable resource classes are added/edited, validate that no reusable content is modified

  2. When resources are created, record their creation in a JSON file to be cleaned up later.

    • In CI, clean up the resources in another job after the test jobs.
    • When not in CI, clean up the resources after all tests are complete, with the option to leave resources behind.
    • In both cases, leave resources of failed tests, with the option to delete them too when the rest are deleted.

  3. Measure the time taken during setup, execution, and cleanup

    • Include logging of fetching, creation, and deletion.
    • Store the data so that it can be analysed.

  4. Additional changes if time permits

    • Make it possible to run tests using bundle exec rspec ${spec_name}.
      • The GitLab address could be provided via an environment variable.
      • Use localhost by default to be able to quickly run against local GDK (most likely usecase for bundle exec rspec)
    • Require gems automatically using Bundler.require. Implemented
    • Provide a consistent way to change the way resources are created if they're behind a feature flag.

TBC

Related links

Edited by Mark Lapierre