Can't run single tests quickly in Auto DevOps qa-integration tests
Problem to solve
Running the qa integrations tests for Auto DevOps currently takes awhile. Furthermore, whenever a new feature needs integration testing it's hard to execute just one specific test since there's a bin/qa file needed to preset up a lot of things so the tests can be run by rpec.
Target audience
Backend Engineer
Further details
- New Auto DevOps feature needs integrations tests
- Debugging a broken spec
Proposal
1 - Setup a different script than bin/qa
or modify it, to accept running single spec tests, indicating spec by line number. Even though it would be nice to specify the test one wants bin/qa
to run, still the every time creation of clusters is the most consuming part.
2 - Setup scripts to spinup and spindown qa clusters, so they can be up all times and we can run specs by the traditional way. So it would be possible to run commands like:
~> ./bin/qa/create-qa-cluster
~> bundle exec rspec ./qa/specs/my_integration_test_spec.rb:34
~> ./bin/qa/drop-qa-cluster
What does success look like, and how can we measure that?
If it's possible to spinup a qa-cluster only once, then only run specs manually, then it's easily measurable by checking the time taken to run the whole suite against the time taken to run a specific test after the cluster is up.