Enable Gitaly packfile cache on staging

Production Change

Change Summary

Part of scalability#931.

In &372 (closed) we are developing a caching mechanism in Gitaly meant to reduce CPU/RAM consumption due to massively parallel CI Git fetch workloads. A first iteration of this cache has been deployed behind a feature flag. The next step is for us to investigate the real world impact of the cache, and to see if we need to iterate more on its design.

What we are really interested in is the impact in gprd, but in this issue we will start with gstg. We don't expect this to answer many questions because gstg traffic is so different from gprd. But it will let us validate our experimentation process.

Change Details

1. Services Impacted - ServiceGit ServiceGitaly
2. Change Technician - @jacobvosmaer-gitlab
3. Change Criticality - C3,
4. Change Type - changeunscheduled, changescheduled
5. Change Reviewer - DRI for the review of this change
6. Due Date - Date and time (in UTC) for the execution of the change
7. Time tracking - Time, in minutes, needed to execute all change steps, including rollback
8. Downtime Component - If there is a need for downtime, include downtime estimate here

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 5

• Ensure the cache is enabled in gstg. Both queries should return the same number.
• Notify #quality and #g_delivery that we are toggling a feature flag that affects staging (although they should not notice anything)

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 30

• /chatops run feature set gitaly_upload_pack_gitaly_hooks 50 --project gitlab-org/gitlab-test --staging
• /chatops run feature set gitaly_upload_pack_gitaly_hooks true --project gitlab-org/gitlab-test --staging
• /chatops run feature set gitaly_upload_pack_gitaly_hooks true --project gitlab-com/www-gitlab-com --staging
• /chatops run feature set gitaly_upload_pack_gitaly_hooks true --project gitlab-org/gitlab-org --staging
• /chatops run feature set gitaly_upload_pack_gitaly_hooks true --staging

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - 20

• git clone --bare --depth=1 https://staging.gitlab.com/gitlab-org/gitlab-test.git
• git clone --bare --depth=1 https://staging.gitlab.com/gitlab-com/www-gitlab-com.git
• git clone --bare --depth=1 https://staging.gitlab.com/gitlab-org/gitlab.git

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 5

• /chatops run feature delete gitaly_upload_pack_gitaly_hooks --staging

Monitoring

Key metrics to observe

• Metric: Gitaly Apdex, Errors and Saturation

  • Location: https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gstg&var-stage=main&var-sigma=2
  • What changes to this metric should prompt a rollback: apdex decrease, error increase, saturation increase

• Metric: Disk space utilization

  • Location: https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview?viewPanel=2661375984&orgId=1&var-PROMETHEUS_DS=Global&var-environment=gstg&var-stage=main&var-sigma=2
  • What changes to this metric should prompt a rollback: notable increase

• Metric: Disk write throughput

  • Location: https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview?viewPanel=374696527&orgId=1
  • What changes to this metric should prompt a rollback: ??

Summary of infrastructure changes

• Does this change introduce new compute instances?
• Does this change re-size any existing compute instances?
• Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?

Summary of the above

Changes checklist

• This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
• This issue has the change technician as the assignee.
• Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
• Necessary approvals have been completed based on the Change Management Workflow.
• Change has been tested in staging and results noted in a comment on this issue.
• A dry-run has been conducted and results noted in a comment on this issue.
• SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
• There are currently no active incidents.