Upgrade Thanos to v0.24.0
Production Change
Change Summary
Upgrade Thanos from version v0.21.0
to v0.24.0
in production part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14624
Change Details
- Services Impacted - ServiceThanos
- Change Technician - @swainaina
- Change Reviewer - @steveazz
- Time tracking - 1 hour
- Downtime Component - None
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 5
-
Set label changein-progress on this issue
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 45
-
Upgrade Thanos Store and Thanos Query -
Merge gitlab-com/gl-infra/k8s-workloads/tanka-deployments!359 (merged) -
Wait for the pipeline on ops.GitLab.net to pass and fully roll out -
Check ops
rollout is successful.# In 1 terminal window glsh kube use-cluster ops # In another terminal window kubectl -n monitoring get deploy thanos-query kubectl -n monitoring get deploy thanos-query-frontend kubectl -n monitoring get sts thanos-store-0
-
Check if https://thanos.gitlab.net/ is still accessible. -
Check if https://thanos-query.ops.gitlab.net/ is still accessible. -
Check if gprd
rollout is successful.# In 1 terminal window glsh kube use-cluster gprd # In another terminal window kubectl -n monitoring get sts -l app.kubernetes.io/name=thanos-store
-
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 15
-
Confirm that Thanos Store has been upgraded and the version is v0.24.0 in
kubectl -n monitoring get po -l app.kubernetes.io/name=thanos-store -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 20
-
Revert the merge requests that we merged.
Monitoring
Key metrics to observe
- Metric: Apdex
- Location: https://dashboards.gitlab.net/d/monitoring-main/monitoring-overview?orgId=1&from=1645687260000&to=1645708919999&viewPanel=712482646
- What changes to this metric should prompt a rollback: Dip in the apdex
- Confirm metrics: https://thanos.gitlab.net/graph?g0.expr=gitlab_component_ops%3Arate%7Bcomponent%3D%22thanos_store%22%2C%20env%3D%22ops%22%7D&g0.tab=0&g0.stacked=0&g0.range_input=1w&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Change Reviewer checklist
-
The scheduled day and time of execution of the change is appropriate. -
The change plan is technically accurate. -
The change plan includes estimated timing values based on previous testing. -
The change plan includes a viable rollback plan. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). -
The change plan includes success measures for all steps/milestones during the execution. -
The change adequately minimizes risk within the environment/service. -
The performance implications of executing the change are well-understood and documented. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? -
The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. 👉 https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14624#gstg -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.
Edited by Silvester Wainaina