Skip to content

Testing Framework POC Part 2

Andrew Newdigate requested to merge test-improvements into main

This extends the jsonnet-tool test testing framework substantially, based on the feedback from attempting to retrofit this tool to GitLab Dedicated's Instrumentor project, which has a very large test suite.

  1. Adds two new matchers, one for plain text, one for arbitrary JSON not backed by a fixture file.
  2. Moves test bodies to a function() closure. This allows control of which tests are run, as well as allowing timing of manifestation to be measures.
  3. Treats manifestation as a precursor to matching. This is how it was happening under the hood previously, but not it allows the speed of each test to be understood.
  4. Replaces the diff library with something easier to read
  5. Consumes traces and only emits the traces for failed tests (by default, there's a switch to fix this), The trace is displayed after the test failure message.
  6. Adds some consistency in the naming of Visitor event function names
  7. Adds a base class for visitors, so that each visitor only needs to implement the methods it will override.

See https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/instrumentor/-/merge_requests/2636 for an example of this testing tool in action.

Demo

screenshot-andrewn-2024-01-17T21h11Z_2x

Edited by Andrew Newdigate

Merge request reports