Limit Prometheus k8s SD to Gitlab namespace
We should limit the k8s SD config to just the namespace GitLab is installed into: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Ckubernetes_sd_config%3E
For very large clusters, we could crush the Prometheus server by discovering everything.
The upstream Prometheus chart populates server configuration via config map and values: https://github.com/kubernetes/charts/blob/master/stable/prometheus/values.yaml#L678 https://github.com/kubernetes/charts/blob/master/stable/prometheus/templates/server-configmap.yaml
The upstream chart allows for overriding the server configmap with an external one. This is probably what we'd need to do to populate with dynamic values like namespace name.
The upstream chart also creates a ClusterRole and ClusterRoleBinding which is overkill for a namespace-scoped deployment. We should disable the service account creation in the chart (https://github.com/kubernetes/charts/blob/master/stable/prometheus/values.yaml#L6) and create our own rbac config that is limited to the namespace.