In a couple of places we use hardcoded values to specify the maximum value in saturation points, in other places we specify hardcoded values in one-off recording rules for GitLab.com or a dedicated environment directly. Some examples: - [`pvs_cloud_run_container_instances`](https://gitlab.com/gitlab-com/runbooks/blob/6a92550e3dbe50fadaf001f00f724256ee9960b8/libsonnet/saturation-monitoring/pvs_cloudrun_container_instances.libsonnet#L22) hardcoded max instances - [`aws_rds_used_connections`](https://gitlab.com/gitlab-com/runbooks/blob/6a92550e3dbe50fadaf001f00f724256ee9960b8/libsonnet/saturation-monitoring/aws_rds_db.libsonnet#L29), hardcoded, to change in to a recording rule calculation in https://gitlab.com/gitlab-com/runbooks/-/merge_requests/6777 - [`aws_rds_disk_space`](https://gitlab.com/gitlab-com/runbooks/blob/6a92550e3dbe50fadaf001f00f724256ee9960b8/libsonnet/saturation-monitoring/aws_rds_disk.libsonnet#L32) hardcoded disk space - [`aws_rds_freeable_memory`](https://gitlab.com/gitlab-com/runbooks/blob/6a92550e3dbe50fadaf001f00f724256ee9960b8/libsonnet/saturation-monitoring/aws_rds_memory.libsonnet#L10): hardcoded, to change to a recording rule in https://gitlab.com/gitlab-com/runbooks/-/merge_requests/6777 - [`nat_gateway_port_allocation`](https://gitlab.com/gitlab-com/runbooks/blob/6a92550e3dbe50fadaf001f00f724256ee9960b8/libsonnet/saturation-monitoring/nat_gateway_port_allocation.libsonnet#L33) hardcoded for GitLab.com - [`pgbouncer_client_connections`](https://gitlab.com/gitlab-com/runbooks/blob/6a92550e3dbe50fadaf001f00f724256ee9960b8/libsonnet/saturation-monitoring/pgbouncer_client_connections.libsonnet#L45-46), hardcoded for GitLab.com - [`kube_node_ips`](https://gitlab.com/gitlab-com/runbooks/blob/6a92550e3dbe50fadaf001f00f724256ee9960b8/libsonnet/saturation-monitoring/kube_node_ips.libsonnet#L29), [recording rules](https://gitlab.com/gitlab-com/runbooks/blob/6a92550e3dbe50fadaf001f00f724256ee9960b8/thanos-rules-jsonnet/subnet-sizes.jsonnet#L51), for GitLab.com We should avoid hardcoding values in saturation points, but have them generate recording rules. So we can have more historical information about the saturation point in metrics: Did we resolve an incident by increasing capacity or did we reduce utilisation? Instead of hardcoding them directly from a recording rule or in the saturation point, we should allow configuring this through `gitlab-metrics-config`, so we can easily adjust them for different tenants. Ideally, we could define the recording rules from static values passed in through config directly in the Saturation point: ```jsonnet staticRules: { rds_max_connections: { labels: { some: from_config, labels: from_config, }, expr: from_config, }, }, ``` ____ The following discussion from gitlab-com/runbooks!6777 should be addressed: - [ ] @cmiskell started a [discussion](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/6777#note_1746902328): (+5 comments) > It is possible to express this as a function of `rds_instance_ram_bytes` based on the AWS provided formula (what is being done in https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/instrumentor/-/merge_requests/2658 by the caller). Requiring a distinct recording rule raises the requirements on the consumer slightly, and it seems unfortunate we can't be more helpful. > > *question*: Is there a particular reason you chose this approach vs using `clamp_min(rds_instance_ram_bytes/9531392, 5000)` in this expression?