Custom metrics
Description
As part of https://gitlab.com/gitlab-org/gitlab-ce/issues/28717, we are expanding our support of metrics to include all well known Exporters. This is great for many customers, but for users who have instrumented their own code for Prometheus, like we are doing, these metrics will not be known.
This is particularly important when a customer has instrumented their application to export key business metrics: conversions, users, sales, etc.
While it would be possible to manually edit the Metric Library YAML file, we should provide a first class method for adding these via the UI.
Proposal
We should offer the ability to add these metrics so they can be monitored within GitLab. Since these metrics are not common across all customers, we will have to expose a way for project specific metrics to be added via the Prometheus service configuration screen. This would augment the existing UI, for listing what was found from the metric library.
This would then get added into the metric library for this project and autodetected following the same process. As part of this, we would also have to implement the priority
field.
To keep focused on the MVC, each project specific metric will fall into one of the following groups based on the “Type” field: Business, Response, or System. These metric groups should appear in alphabetical order, before any of the common metrics. In the future based on demand and resources, we can allow more flexibility here.
- Group Name:
Business|Response|System
- Group Priority: 1
- Weight: 10
- Required Metric: Ignore for now, always attempt to scan these. In the future can extract the metric names out of the query.
We prompt for these fields when creating/editing a metric:
- Title: Used as title for the chart
- Type (Business, Response or System): For grouping similar metrics
- Query: Must be a valid PromQL query. Prometheus Querying documentation
- Y-axis label: Label of the chart’s vertical axis. Usually this is the type of unit being charted. The horizontal axis (X-axis) always represents time.
- Unit label: Unit type returned by the query. For example "MB" or "Requests/sec".
- Legend label (optional): Used if the query returns a single series. If it returns multiple series, their legend labels will be picked up from the response.
The form to create/edit metrics should be in a new, separate screen. For the breadcrumb, they should be:
- Creating a metric:
GROUP > PROJECT > Settings > Integrations > Prometheus > New metric
- Editing an existing metric:
GROUP > PROJECT > Settings > Integrations > Prometheus > Edit metric
Once we have the query, we will not attempt to do any validation. That can be kept as a stretch goal: https://gitlab.com/gitlab-org/gitlab-ee/issues/3426
Links / references
Documentation blurb
(Write the start of the documentation of this feature here, include:
- Why should someone use it; what's the underlying problem.
- What is the solution.
- How does someone use this
During implementation, this can then be copied and used as a starter for the documentation.)