For Prometheus, we should strive to use the official Helm chart and container. This will reduce the burden on us for maintaining these, and will align with their general path forward.
Support for this should simply be included in the top level
gitlab chart, with options to disable as needed. Configuration can be managed via ConfigMap, along with support for creation of its own PVC claim. There are existing configuration settings for all of these.
Some additional settings for the chart:
- Disable all extraneous services like alert manager, kube-state-metrics, push gateway, and node_exporter for now.
- Node Exporter could be particularly surprising as it is installed as a DaemonSet, and you would probably not expect GitLab to install this on all your nodes. We can always enable later as needed.
- Restrict the Kubernetes SD to just the namespace that GitLab is installed into. This way we don't go scraping the entire cluster, when the intent of this service is to provide metrics about GitLab. For large clusters, this could increase load on other pods, as well as significantly increase resources used by the Prometheus server.
- For storage, I'd recommend a PVC so we can persist metrics across version changes, pod restarts, etc. The default (8gb) should be fine for most installs.
- I'd suggest a default retention period of 2 weeks.