<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.* <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> ### Problem to solve Currently we display cluster metrics to users, but we don't support alerts on them. This can be a problem if a cluster starts to exceed its resources or requests, as no one knows unless they look at the page. ### Intended users <!-- Who will use this feature? If known, include any of the following: types of users (e.g. Developer), personas, or specific company roles (e.g. Release Manager). It's okay to write "Unknown" and fill this field in later. Personas can be found at https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/ --> ### Further details <!-- Include use cases, benefits, and/or goals (contributes to our vision?) --> ### Proposal To achieve this, we should do two things: 1. Support setting an alert which analyzes multiple queries. For example: `(Requests) / (Capacity) > 0.95`. 1. Collect usage ping data when a new alert is setup 1. Allow alerts to be configured on Cluster Monitor screen 1. Enable this alert rule by default (if possible) #### Roadmap 1. Move Cluster Metrics to Prometheus Endpoint https://gitlab.com/gitlab-org/gitlab-ce/issues/63044 - Move queries in cluster_metrics.yml into the DB, using an importer similar to `common_metrics_importer`. - Create a dashboard for cluster metrics and add it to the dashboards folder. - Change the FE to start using the dashboard API instead of the existing API which reads from the cluster_metrics.yml file. - Remove existing API. 1. Add alerting support for the new version of cluster metrics https://gitlab.com/gitlab-org/gitlab-ee/issues/12256 1. Add support for alerting on multiple queries https://gitlab.com/gitlab-org/gitlab-ee/issues/12258 ### Design Implementing this would mean introducing the alert buttons from the current metrics page onto the two charts on the cluster health page, as well as updating the alerts form. Proposal is to replace the first dropdown in the 'add alerts' dropdown with a field so users can input multiple queries if/as needed:  * Recommend adding in a tooltip to give users a little more help formatting the query since we are losing the dropdown:  * After a user has added an alert, alerts can be listed within a badge that would serve as a link to the alert edit/delete functionality:  ### Permissions and Security <!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)? --> ### Documentation <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html Add all known Documentation Requirements here, per https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements --> ### What does success look like, and how can we measure that? <!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. --> ### What is the type of buyer? <!-- Which leads to: in which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers --> ### Links / references