Template not expanded in the prometheus alert about low disk space

  1. Alert - https://gitlab.com/gitlab-com/runbooks/blob/master/alerts/available-disk.rules
  2. Today we had alerts about low disk space on docker-omnibus.gitlap.com node about low disk space, but template not expanded for this alert. Screenshot_from_2017-03-22_13_19_09
  3. After investigating and looking for result from query, I found that node_exporter lists all mountpoints, including dummy ones. So prometheus tries to alert all mountpoints which results in error in template generation. Here is results from df -a run:
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda1       30202916 27171820   1765064  94% /
/dev/sda1       30202916 27171820   1765064  94% /var/lib/docker/devicemapper

I see following ways to solve this:

  1. node_exporter is not listing dummy mountpoints or there is option to disable listing of dummy mountpoints.
  2. We add new group_by value here. I think it should be mountpoint. But I actually don't like this option because our alertmanager config will contain info about specific labels. But it is the most easiest and right I think.
  3. Rewrite alert in order to contain alerts for many nodes (mountpoints) in one alert. But I don't how they will change it's state to resolved. When only one of several node disk space is resolved.

/cc @bjk-gitlab @juliusv @northrup @pcarranza