Upgrade to Thanos v0.24.0
Why do we need to upgrade
In https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14427 we are seeing error level logs because it fails to parse the host. This seems to be fixed in https://github.com/thanos-io/thanos/pull/4348 which was released in v0.22.0
. So we should upgrade to the latest version to make sure this problem goes away. As described in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14427#note_725134612 if we fix this problem we get better information inside of our logs such as query duration which will improve debuggability.
Previous updates
We've done updates in the past, most recently https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13360 you can take a look at the associated merge requests to see where we need to do the updates, in general gstg
should be done first to check if the upgrade works as expected and then roll it out to the other environments.
Changelog
Work
gstg
-
Thanos store 👉 gitlab-com/gl-infra/k8s-workloads/tanka-deployments!311 (merged) -
Thanos sidecar gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!568 (merged) -
Thanos Compactor 👉 https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/1335
global
(try 1)
-
Thanos on Prometheus cookbook 👉 gitlab-cookbooks/gitlab-prometheus!577 (merged)-
Fix berks.lock
👉 gitlab-cookbooks/gitlab-prometheus!578 (merged)
-
-
Thanos store / query 👉 gitlab-com/gl-infra/k8s-workloads/tanka-deployments!327 (merged) -
Thanos Side car 👉 gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!582 (merged) -
Revert Thanos in gstg 👉 https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/1388 -
Change management issue 👉 production#6396 (closed)
Set up Thanos-query staging environment
-
Setup a new Thanos Query staging environment 👉 https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/15341
Investgiate issues
-
Missing Thanos store metrics -
Add thanos.info.Info
metrics to alert monitoring rules👉 gitlab-com/runbooks!4471 (merged)
-
-
Thanos sidecar restarts on VMS
Thanos Query and Frontend
-
Upgrade Thanos Query in ops-stg
👉 gitlab-com/gl-infra/k8s-workloads/tanka-deployments!350 (merged) -
Upgrade Thanos Query in ops-stg
to0.24.0
👉 gitlab-com/gl-infra/k8s-workloads/tanka-deployments!360 (merged)
Kubernetes gprd/ops (Try 2)
-
Updgrade thanos query and store 👉 gitlab-com/gl-infra/k8s-workloads/tanka-deployments!359 (merged) -
Update the tools version in 👉 gitlab-com/runbooks!4516 (merged) -
Change management issue 👉 production#6806 (closed) -
Replace the --store
flag with--endpoint
flag👉 https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15584
GCE (Try 2)
-
Upgrade thanos compact gstg 👉 https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/1684 -
Upgrade thanos prod 👉 https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/1689 -
Change management issue 👉 production#6879 (closed)