2022-09-22: Remove corrupted series from Thanos

Production Change

Change Summary

For https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16424

After #7751 (closed) we had corrupted data in Thanos that is reflected in our availability number and error budgets for stage groups.

Using a thanos tool we're going to remove the corrupted data for the series affecting SLAs and error budgets for the timeframe of 2022-09-15 22:00:00 UTC to 2022-09-16 01:36:00 UTC

Config file for the deletion: deletion-config.yml

Change Details

  1. Services Impacted - ServiceThanos
  2. Change Technician - @igorwwwwwwwwwwwwwwwwwwww
  3. Change Reviewer - @reprazent
  4. Time tracking - 30m
  5. Downtime Component - None

Detailed steps for the change

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 30m

  • Set label changein-progress /label ~change::in-progress
  • Backup blocks 
    gsutil rsync -r gs://gitlab-ops-prometheus/01GD4Z87286SVT6V06FVDNCTV0 gs://gitlab-ops-prometheus/01GD4Z87286SVT6V06FVDNCTV0.bak
gsutil rsync -r gs://gitlab-ops-prometheus/01GD4ZEEGZ074E4FA84YT877KF gs://gitlab-ops-prometheus/01GD4ZEEGZ074E4FA84YT877KF.bak
gsutil rsync -r gs://gitlab-ops-prometheus/01GD4ZN8Q4KSJ75HTBJ0VNY7RB gs://gitlab-ops-prometheus/01GD4ZN8Q4KSJ75HTBJ0VNY7RB.bak
gsutil rsync -r gs://gitlab-ops-prometheus/01GD4ZXHHWMGXTGPE8CSKF1F2F gs://gitlab-ops-prometheus/01GD4ZXHHWMGXTGPE8CSKF1F2F.bak
  • Create objstore.yml: 
    type: GCS
config:
  bucket: gitlab-ops-prometheus
  • Dry-run 
    thanos tools bucket rewrite --rewrite.to-delete-config-file=deletion-config.yml --objstore.config-file=objstore.yml --id=01GD4Z87286SVT6V06FVDNCTV0 --id=01GD4ZEEGZ074E4FA84YT877KF --id=01GD4ZN8Q4KSJ75HTBJ0VNY7RB --id=01GD4ZXHHWMGXTGPE8CSKF1F2F --rewrite.add-change-log --dry-run
  • Proper run 
    thanos tools bucket rewrite --rewrite.to-delete-config-file=deletion-config.yml --objstore.config-file=objstore.yml --id=01GD4Z87286SVT6V06FVDNCTV0 --id=01GD4ZEEGZ074E4FA84YT877KF --id=01GD4ZN8Q4KSJ75HTBJ0VNY7RB --id=01GD4ZXHHWMGXTGPE8CSKF1F2F --rewrite.add-change-log --no-dry-run
  • Set label changecomplete /label ~change::complete

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Restore blocks from backup:

gsutil rsync -r gs://gitlab-ops-prometheus/01GD4ZN8Q4KSJ75HTBJ0VNY7RB.bak gs://gitlab-ops-prometheus/01GD4ZN8Q4KSJ75HTBJ0VNY7RB
gsutil rsync -r gs://gitlab-ops-prometheus/01GD4ZEEGZ074E4FA84YT877KF.bak gs://gitlab-ops-prometheus/01GD4ZEEGZ074E4FA84YT877KF
gsutil rsync -r gs://gitlab-ops-prometheus/01GD4Z87286SVT6V06FVDNCTV0.bak gs://gitlab-ops-prometheus/01GD4Z87286SVT6V06FVDNCTV0
gsutil rsync -r gs://gitlab-ops-prometheus/01GD4ZXHHWMGXTGPE8CSKF1F2F.bak gs://gitlab-ops-prometheus/01GD4ZXHHWMGXTGPE8CSKF1F2F

Monitoring

Key metrics to observe

Example series that should be removed: https://thanos-query.ops.gitlab.net/graph?g0.expr=gitlab_service_ops%3Arate_5m%7Btype%3D%22web%22%2C%20env%3D%22gprd%22%2C%20monitor%3D%22global%22%2C%20stage%3D%22main%22%7D&g0.tab=0&g0.stacked=0&g0.range_input=6h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.end_input=2022-09-16%2003%3A00%3A00&g0.moment_input=2022-09-16%2003%3A00%3A00

Change Reviewer checklist

C4 C3 C2 C1:

  • Check if the following applies:
    • The scheduled day and time of execution of the change is appropriate.
    • The change plan is technically accurate.
    • The change plan includes estimated timing values based on previous testing.
    • The change plan includes a viable rollback plan.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

  • Check if the following applies:
    • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
    • The change plan includes success measures for all steps/milestones during the execution.
    • The change adequately minimizes risk within the environment/service.
    • The performance implications of executing the change are well-understood and documented.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.
      • If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
    • The change has a primary and secondary SRE with knowledge of the details available during the change window.
    • The labels blocks deployments and/or blocks feature-flags are applied as necessary

Change Technician checklist

  • Check if all items below are complete:
    • The change plan is technically accurate.
    • This Change Issue is linked to the appropriate Issue and/or Epic
    • Change has been tested in staging and results noted in a comment on this issue.
    • A dry-run has been conducted and results noted in a comment on this issue.
    • The change execution window respects the Production Change Lock periods.
    • For C1 and C2 change issues, the change event is added to the GitLab Production calendar.
    • For C1 and C2 change issues, the SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
    • For C1 and C2 change issues, the SRE on-call provided approval with the eoc_approved label on the issue.
    • For C1 and C2 change issues, the Infrastructure Manager provided approval with the manager_approved label on the issue.
    • Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
    • There are currently no active incidents that are severity1 or severity2
    • If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.
Edited by Igor