Skip to content

2021-09-08: Update postgres backup buckets retention policies

Production Change

Change Summary

This will coordinate the rollout of the latest version of our database-backup-bucket terraform module (v5.0.1) and the corresponding attribute values in gitlab-com-infrastructure required to enable the addition of minimum retention policies, which will ensure that backups cannot be modified or deleted before the minimum retention period has past.

We will monitor these retention policies for a period of TESTING_PERIOD to ensure that there are no adverse operational impacts from the change, after which we will lock the bucket in production.

Note: Once set, this CANNOT be undone!

Change Details

  1. Services Impacted - ServicePostgres ServicePatroni ServicePraefect
  2. Change Technician - @craig
  3. Change Reviewer - @ahanselka
  4. Time tracking - 30 minutes
  5. Downtime Component - none

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 10

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 10

EXECUTE POST-CHANGE VALIDATION FOR EACH ENVIRONMENT BEFORE PROCEEDING TO THE NEXT

  • Update retention policy and lifecycle thresholds for gstg (MR)
  • Update retention policy and lifecycle thresholds for ops (MR)
  • Update retention policy and lifecycle thresholds for gprd (MR)

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - 2

  • gstg - gsutil retention get gs://gitlab-gstg-postgres-backup
  • ops - gsutil retention get gs://gitlab-ops-postgres-backup
  • gprd - gsutil retention get gs://gitlab-gprd-postgres-backup

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 10

  • Revert retention policy and lifecycle thresholds for gstg (MR to revert)
  • Revert retention policy and lifecycle thresholds for ops (MR to revert)
  • Revert retention policy and lifecycle thresholds for gprd (MR to revert)

Monitoring

  1. On patroni leader (run sudo gitlab-patronictl list from any patroni node to identify) Run sudo tail /var/log/wal-g/wal-g.log to list the latest WAL files
  2. From SRE workstation, verify WAL files are being streamed to GCS (gstg) Run gsutil ls -l gs://gitlab-gstg-postgres-backup/pitr-walg-pg12/wal_005/WAL_FILE_NAME [x] (ops) N/A - backup bucket is empty [ ] (gprd) Run gsutil ls -l gs://gitlab-gprd-postgres-backup/pitr-walg-pg12/wal_005/WAL_FILE_NAME

Summary of infrastructure changes

  • Does this change introduce new compute instances?
  • Does this change re-size any existing compute instances?
  • Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?

Summary of the above

Changes checklist

  • This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
  • This issue has the change technician as the assignee.
  • Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
  • This Change Issue is linked to the appropriate Issue and/or Epic
  • Necessary approvals have been completed based on the Change Management Workflow.
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
  • Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • There are currently no active incidents.
Edited by Craig Barrett