Detect (and time) new tests introduced by a feature

What

System to find new tests introduced by an MR.

We can then run these tests in their own job and time how long they take.

Probably just for the gitlab-ce/ee projects rather than as a generic feature.

Why

  • Provide data so we realize when an MR is introducing a large amount of additional test time
  • We might be able to use this to spot tests written at the wrong level, e.g. as long running integration tests instead of unit tests
  • We can use this to run new tests in isolation so we can spot failures quicker instead of waiting for all tests to complete

Ideas how

  • rspec --format json --color --dry-run | jq '.examples[] | .full_description'
  • Could get list of names of tests before/after an MR and have special job to run only new tests
  • Could store metadata for which tests exist for a given commit, or could checkout previous commit and run
Edited Sep 28, 2017 by James Edwards-Jones
Assignee Loading
Time tracking Loading