Project specific Prometheus servers
While GitLab can support sharing a Prometheus server across projects, we should consider restricting a single server to at most a single Project:
- General best practice for operating Prometheus at scale, is to horizontally shard by deploying a Prometheus service for each task. This can start with a single server per Project (supported), then scale to a single server per environment (supported with #28916 (closed)), and if necessary we can consider building features to scale beyond that.
- As we gather more data within GitLab, we should also consider warehousing this data in a more scalable way. In particular, consider whether we should have dedicated storage per project (PG, ElasticSearch, etc.). (#41450)