Rule Evaluation Failure RedisMemoryUsageTooHigh
level=warn ts=2019-07-18T14:49:35.084Z caller=manager.go:512 component="rule manager" group="Redis Alerts" msg="Evaluating rule failed" rule="alert: RedisMemoryUsageTooHigh
expr: (redis_memory_used_rss_bytes / on(fqdn, tier, type, environment) node_memory_MemTotal_bytes)
> 0.9
for: 5m
labels:
pager: pagerduty
severity: s1
annotations:
description: Redis node {{$labels.fqdn}} is using over 80% of the host memory for
the last 5 minutes.
runbook: howto/clear_anonymous_sessions.md
title: Redis memory usage is too high
" err="found duplicate series for the match group {} on the right hand-side of the operation: [{__name__=\"node_memory_MemTotal_bytes\", endpoint=\"metrics\", instance=\"10.232.20.217:9100\", job=\"node-exporter\", namespace=\"monitoring\", pod=\"gitlab-monitoring-prometheus-node-exporter-vthc7\", service=\"gitlab-monitoring-prometheus-node-exporter\"}, {__name__=\"node_memory_MemTotal_bytes\", endpoint=\"metrics\", instance=\"10.232.20.194:9100\", job=\"node-exporter\", namespace=\"monitoring\", pod=\"gitlab-monitoring-prometheus-node-exporter-jlkl7\", service=\"gitlab-monitoring-prometheus-node-exporter\"}];many-to-many matching not allowed: matching labels must be unique on one side"
/cc @andrewn /cc @bjk-gitlab
Edited by Ben Kochie