Support monitoring projects that do not use Auto Deploy
Description
As part of #26910 (closed), we will be building support for monitoring the performance of auto-deployed apps on k8s, which laid the foundation for our monitoring functionality. However since the supported methods of deployment is limited to Auto Deploy for now, a large number of potential projects are not able to make use of this feature of GitLab.
If we could support deployment types other than Auto Deploy, we could significant increase the number of end-users who can take advantage of this feature.
Proposal
We need to be able to retrieve and correlate metrics from Kubernetes deployed apps, which do not leverage Auto Deploy. This means that we will need an alternate way to be able to map a particular time series to a particular deployment. One way we could achieve this is by having a Kubernetes tag, much like the Auto Deploy $CI_BUILD_REF_NAME
, be included or defined for a particular deployment job. This could then be annotated inside of Kubernetes, providing a way for Prometheus to determine one deployment from another.
We need to do some exploration to see how we can expose this in the lowest effort way for customers to mark the proper nodes. Maybe there is something we can do ourselves with k8s annotation to assist and take the burden off the customer? Or is an environment variable during the CI runner phase sufficient, and customers can take it from there?