Zoekt: Add OOM graph to Zoekt dashboard

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Background

Right now we don't have a lot of visibility into OOMs.

Proposal

I believe that we should add

(sum by (container, pod, environment) (kube_pod_container_status_last_terminated_reason{pod=~"gitlab-gitlab-zoekt-[0-9]+", reason="OOMKilled", env="gprd"})
          * on (container, pod, environment) group_left
          sum by (container, pod, environment) (changes(kube_pod_container_status_restarts_total{pod=~"gitlab-gitlab-zoekt-[0-9]+", env="gprd"}[1m]) > 0))

to https://dashboards.gitlab.net/d/search-zoekt/search3a-zoekt-info?from=now-3h&orgId=1&timezone=utc&to=now&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd

source