Allow multiple Prometheus configurations per project, scoped per environment
Problem to solve
For quite a while we can configure multiple Kubernetes clusters per project. We aren't using the application installation feature, since we have tons of projects and just two k8s clusters - staging and production, and the installation detection is just not there. We do have Prometheus in each cluster to enable auto-scrapping of metrics. However, we can only configure one Prometheus using the Manual configuration, which limits Metrics to one cluster only.
Intended users
Customers that use multiple k8s clusters and want to expose metrics for each cluster from different Prometheus instances, using Manual configuration on the Metrics page.
Proposal
Instead of being able to configure one Prometheus URL and set of queries, we need to be able to configure multiple Prometheus URLs. We can use one set of metrics though since they should be identical.
- We will display a data source dropdown in Metrics UI which will contain both auto (managed on cluster) and manual Prometheus configurations. Each option will have an indicator of what type of Prometheus instance this is.
- The dropdown will have an option to "Manage data sources" which links to the Prometheus configuration page.
- With multiple auto and manual configurations, we'll have to default to one of the manual configurations when multiple manual configurations are available. The easiest way is to default to the first one alphabetically and the user will have the option to switch to a different data source.
- Data sources will be ordered alphabetically in the dropdown, so users can define names for the data sources in a way that certain data sources appear at the top of the dropdown.
- When filling out the form, saved configuration cards collapse automatically, but only if there are no validation errors. If a configuration contains any validation messages or haven't been saved, we should not collapse the configuration card.
- Each configuration can be individually enabled or disabled with a toggle.
- In further iterations users will be able to choose a default configuration.
- In further iterations we'll also show connectivity indicators for each Prometheus configuration.
Activation of Premium users
Multiple Prometheus configurations will be available to Premium users, so we need to have a way to surface and enable the feature for Core users.
This is what it could look like in Prometheus configuration page.
- The accordion expands along with the first manual configuration form after they click "Add manual configuration".
- This feature discovery element will only be available to admin of a project/group with the ability to start a trial.
- Once the feature is enabled, we show a confirmation and show the button to "Add another configuration".
This is what it will look like in Prometheus configuration page |
---|
For more context, check out this flow promoting the Gold feature of required approvers in MRs.
Solution Validation Results
Open questions
- Can we have a Pajamas styled confirmation dialog for when user wants to delete a Prometheus configuration?
Documentation
We need to make sure to clearly document the precedence rules for which Prometheus instance is used and when.
Additional input
This is likely to become a premium feature