Rule Evaluation Failure RedisMemoryUsageTooHigh

level=warn ts=2019-07-18T14:49:35.084Z caller=manager.go:512 component="rule manager" group="Redis Alerts" msg="Evaluating rule failed" rule="alert: RedisMemoryUsageTooHigh
expr: (redis_memory_used_rss_bytes / on(fqdn, tier, type, environment) node_memory_MemTotal_bytes)
  > 0.9
for: 5m
labels:
  pager: pagerduty
  severity: s1
annotations:
  description: Redis node {{$labels.fqdn}} is using over 80% of the host memory for
    the last 5 minutes.
  runbook: howto/clear_anonymous_sessions.md
  title: Redis memory usage is too high
" err="found duplicate series for the match group {} on the right hand-side of the operation: [{__name__=\"node_memory_MemTotal_bytes\", endpoint=\"metrics\", instance=\"10.232.20.217:9100\", job=\"node-exporter\", namespace=\"monitoring\", pod=\"gitlab-monitoring-prometheus-node-exporter-vthc7\", service=\"gitlab-monitoring-prometheus-node-exporter\"}, {__name__=\"node_memory_MemTotal_bytes\", endpoint=\"metrics\", instance=\"10.232.20.194:9100\", job=\"node-exporter\", namespace=\"monitoring\", pod=\"gitlab-monitoring-prometheus-node-exporter-jlkl7\", service=\"gitlab-monitoring-prometheus-node-exporter\"}];many-to-many matching not allowed: matching labels must be unique on one side"

For rule: https://gitlab.com/gitlab-com/runbooks/blob/d989dc1a94f6525675dda6f9245532a6fb8d8e33/rules/redis.yml#L68

/cc @andrewn /cc @bjk-gitlab

Edited Jul 18, 2019 by Ben Kochie
Assignee Loading
Time tracking Loading