Upgrade Thanos to v0.24.0
Production Change
Change Summary
Upgrade Thanos from version v0.21.0
to v0.24.0
in production part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14624
Change Details
- Services Impacted - ServiceThanos
- Change Technician - @swainaina
- Change Reviewer - @steveazz
- Time tracking - 1 hour
- Downtime Component - None
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 5
-
Set label changein-progress on this issue -
Revert gstg upgrade by merging this MR <code data-sourcepos="26:48-26:56">chef-repo</code>
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 45
-
Merge the Prometheus cookbook 👉 gitlab-cookbooks/gitlab-prometheus!577 (merged)-
Update berks file 👉 gitlab-cookbooks/gitlab-prometheus!578 (merged)
-
-
Wait for cookbook publisher to version bump cookbook versions in https://gitlab.com/gitlab-com/gl-infra/chef-repo -
Merge the merge requests opened by cookbook publisher -
Upgrade Thanos Store and Thanos Query -
Merge gitlab-com/gl-infra/k8s-workloads/tanka-deployments!327 (merged) -
Wait for pipeline on ops.GitLab.net to pass and fully roll out -
Check ops
rollout is successful.# In 1 terminal window glsh kube use-cluster ops # In another terminal window kubectl -n monitoring get deploy thanos-query kubectl -n monitoring get deploy thanos-query-frontend kubectl -n monitoring get sts thanos-store-0
-
Check if https://thanos.gitlab.net/ is still accessible. -
Check if https://thanos-query.ops.gitlab.net/ is still accessible. -
Check if gprd
rollout is successful.# In 1 terminal window glsh kube use-cluster gprd # In another terminal window kubectl -n monitoring get sts -l app.kubernetes.io/name=thanos-store
-
-
Finally merge the Thanos Side car MR 👉 gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!582 (merged)-
Wait for the pipeline on ops.gitlab.net
to pass and fully roll out -
Check if gprd
rollout is successful.# In 1 terminal window glsh kube use-cluster gprd # In another terminal window kubectl -n monitoring get sts -l app=prometheus-operator-prometheus
-
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 15
-
Confirm that Thanos Store has been upgraded and version is v0.24.0 in
kubectl -n monitoring get po -l app.kubernetes.io/name=thanos-store -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'
-
Confirm that Thanos sidecar version is v0.24.0
kubectl -n monitoring get po -l app=prometheus -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'
-
Confirm the compactor has been successfully migrated
ssh thanos-compact-01-inf-gstg.c.gitlab-staging-1.internal -- sudo chef-client
ssh thanos-compact-01-inf-gstg.c.gitlab-staging-1.internal -- /opt/prometheus/thanos/thanos --version
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 45
-
Revert the merge requests that we merged.
Monitoring
Key metrics to observe
- Metric: Apdex
- Location: https://dashboards.gitlab.net/d/monitoring-main/monitoring-overview?orgId=1&from=1645687260000&to=1645708919999&viewPanel=712482646
- What changes to this metric should prompt a rollback: Dip in the apdex
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Change Reviewer checklist
-
The scheduled day and time of execution of the change is appropriate. -
The change plan is technically accurate. -
The change plan includes estimated timing values based on previous testing. -
The change plan includes a viable rollback plan. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). -
The change plan includes success measures for all steps/milestones during the execution. -
The change adequately minimizes risk within the environment/service. -
The performance implications of executing the change are well-understood and documented. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? -
The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. 👉 https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14624#gstg -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.
Edited by Steve Xuereb