Blackbox testing of Environment URL's
Once we start provisioning Prometheus automatically for Kubernetes environments (#28916), we then have direct control over the configuration and can begin to build on top of this control.
A key area we can now try to accomplish is to provide OOTB blackbox testing to the environment. This can be used to gauge response times and any error rates, which are presently difficult to obtain with most common exporters.
Start by spinning up a blackbox exporter pod, and add the blackbox exporter to the scrape list. As environments are spun up, add their Environment URL to the list for blackbox testing.
We could also consider allowing additional URL's to be specified (perhaps deeper links to test API's, etc.) in
gitlab-ci.yml or another configuration area.
Further by shipping the Blackbox exporter, it could also probe any properly annotated services: https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L122
Links / references
(Write the start of the documentation of this feature here, include:
- Why should someone use it; what's the underlying problem.
- What is the solution.
- How does someone use this
During implementation, this can then be copied and used as a starter for the documentation.)
If we could do this without requiring k8s that would be even better. Maybe building an optional blackbox tester into the GitLab package.