Spike: Add the ability to capture and present any metric using external prometheus

Problem to solve

We currently have automatically generated dashboards, based on well-known Prometheus exporters like NGINX Ingress. This is nice and convenient, but we need to go much further and expand this model to any metrics at an organization, or beyond. We would like to allow our users and customers to add any exporters they'd like in order to monitor their application

Further details

We would like to investigate, improve and document the flow a user needs to take in order to monitor their application using External Prometheus

Proposal

Based on #33544 (comment 239355474)

With #7983 (closed), I envision that applications will follow this flow:

• Installation / upgrade via CI job
• GitLab polls / queries cluster state; cache as needed, handling the following states:
  • Application is not installed
  • Application is not installed as required
  • Application is installed and configured as required (e.g. Deploy boards require a specific annotation)

This means GitLab can now support:

• Applications installed entirely by template GitLab CI job (will be developed in #7983 (closed))
• Application installed using template GitLab CI job with different configuration
• Application installed by same other method

/cc @tkuah

What does success look like, and how can we measure that?

• Detailed documentation on how to add new exporter and create new dashboards with custom metrics
• "How to" blog

What is the type of buyer?

Links / references