Daemonset monitoring insufficient for alerting us during issues
Recently it was discovered that our logging daemonset was not spinning up Pods on some nodes. This remediated here: delivery#1033 (closed)
We had no alerts fire to let us know something is wrong. In looking at our current queries that fire off alerts, it would appear they are either misconfigured, or not looking at the correct data. Utilize this issue to investigate and fix alerting such that we are notified if a daemonset is unable to schedule pods appropriately.
Reference current configuration file: https://gitlab.com/gitlab-com/runbooks/-/blob/f650af9e9584ac3f0e1be5371ee8112dfa9e4fba/rules/kubernetes.yml#L145-182