Merge branch 'jarv/add-omnibus-alerts' into 'master'

Add alert for omnibus package

See merge request !1080

rules/omnibus.yml

+groups:
+  - name: omnibus.rules
+    rules:
+      - alert: OmnibusPackageUpdatesDisabled
+        expr: count(omnibus_package_install{enable="true"}) / count(omnibus_package_install) < .25
+        for: 6h
+        labels:
+          severity: warn
+        annotations:
+          description: |
+            At least 75% of the fleet have omnibus package updates disabled.
+            This prevents automatic package installations for newly provisioned
+            VMs
+          runbook: troubleshooting/omnibus-package-updates.md
+          title: Omnibus package updates have been disabled for over 6 hours

troubleshooting/omnibus-package-updates.md

+# Omnibus package troubleshooting
+
+## First and foremost
+
+*Don't Panic*
+
+## Symptoms
+
+* Omnibus packages are set to disabled preventing automatic updates
+
+## Troubleshooting
+
+Check the omnibus role for the environment
+
+```
+# Example for the gprd environment
+knife role show gprd-omnibus-version
+```
+
+See whether the `enable` attribute is set to `true`.
+
+If it is not, ensure there are not any recent failed deployments by checking the
+`#announcements` slack channel or recent [deployer pipelines](https://ops.gitlab.net/gitlab-com/gl-infra/deployer/pipelines)
+
+If unsure, contact someone in the delivery team in `#g_delivery` about the error so they can
+investigate.