Add tests for monitoring/metrics
Context
Currently the monitoring and metrics code is not well tested. We should rectify that by expanding the test suite in a way that makes orchestrating multiple full (but not necessarily real) builds and status changes easily possible.
We can run some builds like that, and add then verify that the produced metrics and monitoring output is as we expected. We should test multiple monitoring methods and output formats.
Task Description
-
Add test code to spin up a BuildGrid instance with metrics enabled (ideally in multiple ways) -
Add tests to orchestrate running some real or mock jobs on this BuildGrid instance and verifying the produced metrics
Acceptance Criteria
Monitoring/metrics functionality is tested to a reasonable extent (as compared with the current situation of only somewhat testing the bots-count metric).