Add integration testing for Monitor features
Problem to solve
The functionality around deploying a Prometheus instance and around setting alerts is not adequately tested and requires engineers to manually test these features when they make code changes in the same area to ensure they haven't introduced regressions. The problem is that engineers on other teams have to dig in and understand these features well enough to go through the proper steps to make sure they still function.
One example is where an engineer on the configure team makes a change to kubernetes deployments and can't be sure whether their change unintentionally broke the way our Prometheus deployment works on top of kubernetes.
Target audience
-
Sasha, Software Developer, https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas#sasha-software-developer
-
Devon, DevOps Engineer, https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas#devon-devops-engineer
Further details
Ultimately, we want our team and other teams to be confident that they are not introducing regressions to the Monitor features.
Proposal
One likely approach to addressing this problem is to add integration tests to the gitlab-qa suites. Before diving in, we should consult with @DylanGriffith and @ddavison who have a lot of knowledge about how the Configure team manages their automated test suite.
The biggest areas that could use integration tests are:
- Installing a prometheus application
- Setting an alert
What does success look like, and how can we measure that?
We will know we have been successful when engineers can feel confident enough that running the full test suite will catch any potential regressions in the Monitor features and won't feel that they need to manually test the features.
Links / references
cc/ @joshlambert @meks