Enable Gitaly scheduled maintenance for a production storage

Production Change

Change Summary

This change introduces an experimental feature (gitlab-org/gitaly!2423 (merged)) that allows Gitaly to perform a background maintenance task to optimize repository access performance.

One important complication to understand: Gitaly is not able to discern storage locations outside the context of a request. Since all Gitaly servers are configured with all storage locations pointing to the same filesystem path, it makes it difficult to configure a Gitaly to perform a job on a single storage. To accomplish this on only one Gitaly, it will require only modifying the configuration of a single Gitaly server. In other words, we will need to pick a "pet" for this experiment.

Part of gitlab-org/gitaly#2820 (closed)

Change Details

1. Services Impacted - Gitaly
2. Change Technician - @pokstad1
3. Change Criticality - C2
4. Change Type - changeunscheduled
5. Change Reviewer - @nnelson
6. Due Date - 2020-10-15 22:30 UTC
7. Time tracking - Time, in minutes, needed to execute all change steps, including rollback
8. Downtime Component - N/A, there should be no need for downtime with graceful restart of Gitaly

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 15

• Pick a low traffic Gitaly storage (e.g. "default")
• Pick a low traffic time of day to run the maintenance task (e.g. Saturday at 10:30pm for 4 hours)

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 15

• Merge and apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/4396
• knife ssh roles:gprd-base-stor-gitaly-cny 'sudo chef-client'

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - 10

• Verify the affected Gitaly prints a log message with the correct maintenance window (e.g. level=info msg="maintenance: daily scheduled" scheduled="2020-07-29 23:04:00 -0700 PDT")

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 10

• Revert https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/4396
• knife ssh roles:gprd-base-stor-gitaly-cny 'sudo chef-client'
• Verify the log from the server indicates the change took effect

Monitoring

Key metrics to observe

• Metric: gitaly_daily_maintenance_repo_optimization_seconds

  • Location: https://dashboards.gitlab.net/d/9pLfuovGz/gitaly-background-maintenance?orgId=1
  • What changes to this metric should prompt a rollback: unknown

• Metric: All performance related metrics

  • Location: https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly-host-detail?orgId=1
  • What changes to this metric should prompt a rollback: any degradation of service during the specified maintenance window

Summary of infrastruture changes

• Does this change introduce new compute instances? No
• Does this change re-size any existing compute instances? No
• Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc? No

No infrastructure changes needed.

Changes checklist

• This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled).
• This issue has the change technician as the assignee.
• Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
• Necessary approvals have been completed based on the Change Management Workflow.
• Change has been tested in staging and resultes noted in a comment on this issue.
• A dry-run has been conducted and results noted in a comment on this issue.
• SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue.)
• There are currently no active incidents.