Skip to content

chore: upgrade prom chart 19.1.0 pre

Steve Xuereb requested to merge steveazz/update-prom-stack into master

Background

This is the second attempt to upgrade the Prometheus chart and Operator. The first attempt had to be reverted.

We need to upgrade to newer versions of the CRDs and helm chart so that we can specify startupProbe for a corrective action.

Solution

Upgrade the operator and chart version to all environments except gprd.

Specify a version of the Prometheus operator that supports our alertmanager configuration, specifically this commit we can that that that image is using the correct version by running the commands below.

$ docker run --rm -it ghcr.io/prometheus-operator/prometheus-operator:master@sha256:bb79240165868c7d73d3db2b45bd065bf2b3050729aa4809f6de79cace232feb --version
prometheus-operator, version 0.51.2 (branch: master, revision: 0a38647379a5e93f639bf8e634deabcc32e01fb6)
  build user:       Action-Run-ID-1345540349
  build date:       20211015-10:23:54
  go version:       go1.17.2
  platform:         linux/amd64⏎

Use the image from GitHub Container registry since quay.io has a small retention policy. We can this specific digest was pushed looking at GitHub Actions for that commit.

Upgrade

In gitlab-com/gl-infra/production#5711 (closed) we updated the CRDs for v0.50.0 however, now we are using a newer version of the operator so we need to use the CRDs available in 0a38647379a5e93f639bf8e634deabcc32e01fb6

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/0a38647379a5e93f639bf8e634deabcc32e01fb6/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/0a38647379a5e93f639bf8e634deabcc32e01fb6/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/0a38647379a5e93f639bf8e634deabcc32e01fb6/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/0a38647379a5e93f639bf8e634deabcc32e01fb6/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/0a38647379a5e93f639bf8e634deabcc32e01fb6/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/0a38647379a5e93f639bf8e634deabcc32e01fb6/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/0a38647379a5e93f639bf8e634deabcc32e01fb6/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/0a38647379a5e93f639bf8e634deabcc32e01fb6/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml

reference https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13973

Edited by Steve Xuereb

Merge request reports