Use expanded labels recording rule for alerting dashboards
Coming from gitlab-com/runbooks!7048 (comment 1862060557).
Once we've expanded the labels for all saturation points in #3390, the alerting dashboards can utilize the recorded saturation point instead of querying from source metrics.
For example:
-
Now:
clamp_min( clamp_max( max by(environment, tier, type, stage, shard, model_engine) ( clamp_min( clamp_max( sum by (environment, tier, type, stage, shard, model_engine)(max_over_time(model_inferences_in_flight{env="gprd",stage="main",type="ai-gateway"}[10m])) / max by (environment, tier, type, stage, shard, model_engine)(min_over_time(model_inferences_max_concurrent{env="gprd",stage="main",type="ai-gateway"}[10m])) , 1) , 0) ) ,1) ,0)
-
Proposed:
clamp_min( clamp_max( gitlab_component_saturation:ratio{env="gprd", stage="main", component="max_concurrent_inferences"} ,1) ,0)
Edited by Gregorius Marco