Redis recording rules can't distinguish between redis cache and shared state servers
We received alerts that redis-cache was broken (https://gitlab.slack.com/archives/C9YAJ0YFP/p1529329103000182) on GPRD, but in fact, the shared state Redis cluster was not working.
It looks like the recording rules assume the master that any connected slave is the right one:
- record: gitlab:redis_disconnected_slaves
expr: count(redis_connected_slaves) WITHOUT (fqdn, instance, job) - sum(redis_connected_slaves)
WITHOUT (fqdn, instance, job) - 1
- record: gitlab:redis_master
expr: redis_connected_slaves > 1
That breaks down if none of the slaves are connected. :)
/cc: @jarv