Allow prometheus in K8s cluster to be installed manually

@splattael @rpereira2

#217859 (closed) inspires me to ask about this:

Is it feasible to allow user to install Prometheus themselves then tell GitLab that where it is installed ?

• For initial iteration, they can install it in a conventional location in gitlab-managed-apps
• They send a API call to /api/v4/clusters/:cluster_id/prometheus/installed to let GitLab know prometheus is installed.

Questions

• I see a lot of configuration in https://gitlab.com/gitlab-org/gitlab/-/blob/master/vendor/prometheus/values.yaml. Which config is mandatory and which are not ?
• Does GitLab update the prometheus Deployment in any way ?
• How does GitLab send a request to Prometheus ? Is this configurable ?

Proposal

I would suggest that :

1. User can install Prometheus with whatever config in the gitlab-managed-apps namespace, but the service name has to match https://gitlab.com/gitlab-org/gitlab/-/blob/e731618eaf8d7425ade89984550d60d823719f79/app/models/clusters/applications/prometheus.rb#L55
2. There is a way for user to hint to GitLab (webhook?) that Prometheus is now installed. GitLab may open a connection to the Prometheus service to verify Prometheus is indeed installed correctly
3. Optional: user can install with suggested scrape_configs from https://gitlab.com/gitlab-org/gitlab/-/blob/master/vendor/prometheus/values.yaml. If the scrape_configs are not present, then GitLab should gracefully fail
4. Alerts: User will have setup alerting manually as described in https://docs.gitlab.com/ee/operations/metrics/alerts.html#external-prometheus-instances

/cc @nicholasklick @nagyv-gitlab @dhershkovitch