Improve test suite performance

Before raising this MR, consider whether the following are required, and complete if so:

• Unit tests
• Metrics
• Documentation update(s)

No need for metrics or documentation updates, this is improvements to test execution speed and the CI configuration.

Description

This MR updates various tests to allow the test suite to run more quickly. There were some high-value improvements to be made here, notably the pool disposal tests which previously were taking multiples of 75 seconds, and the Bots service integration tests where previously all requests sent were using a 5 second connection duration, which was needlessly long for many tests.

This MR also updates the CI configuration, to further speed up the test execution in CI. The main change here is splitting the unit-tests job into integration tests and unit tests, and splitting each group further to optimize for execution time. The downside of this approach is the chance to miss new directories of tests added to the test suite if we forget to update the CI config when adding them. Speeding up the unit-tests part of our CI to take around 15 minutes seems like its worth this trade-off.

The new CI configuration takes advantage of a couple of GitLab features we were not previously utilizing. Firstly, relationships between jobs are defined using needs. This allows jobs to execute outside of the full stage order and instead execute strictly in dependency order. This means all the jobs start executing as quickly as possible, maximizing the parallelism available.

The other new feature being used is test reports. The unit and integration test job groups now produce JUnit test reports and publish them to GitLab, so that information about test duration and failures is available in the GitLab UI, rather than requiring digging into the job logs.

With the changes in this MR, the CI duration bottleneck is the end-to-end test. For this repo, that spends about 20 minutes building all of buildbox world from the current master branch of each repo. An MR to make that more efficient belongs in BuildGrid/buildbox-e2e.

Validation

Witness the CI speed on this MR.

The tests can be run locally with

tox -e rabbitmq-venv -- pytest