Skip to content

Enable `workhorse_use_sidechannel` for Gitlab's internal repositories

Production Change

Change Summary

For scalability#1193

In gitlab-org/gitlab!71047 (merged), we introduced a new RPC called PostUploadPackWithSidechannel to replace PostUploadPack. We added a flag named workhorse_use_sidechannel to control the roll out. At the moment, the change in the MR was deployed to production smoothly. It's time to turn on the flag to switch to the new RPC call.

The flag was turned on Staging in #5700 (closed). This change issue is to continue rolling out the flag to some of Gitlab's internal major repositories:

  • gitlab-org/gitaly
  • gitlab-org/gitlab
  • gitlab-com/www-gitlab-com

Change Details

  1. Services Impacted ServiceGitaly ServiceWorkhorse
  2. Change Technician @qmnguyen0711
  3. Change Reviewer @jacobvosmaer-gitlab
  4. Time tracking 1 hour
  5. Downtime Component - None

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 1 min

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 10 mins

  • /chatops run feature set workhorse_use_sidechannel true --project=gitlab-org/gitaly
  • /chatops run feature set workhorse_use_sidechannel true --project=gitlab-org/gitlab
  • /chatops run feature set workhorse_use_sidechannel true --project=gitlab-com/www-gitlab-com

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - 10 mins

  • Clone aforementioned repositories in the local environment

    • rm -rf /tmp/test.git; git clone --bare --depth=1 https://gitlab.com/gitlab-org/gitaly.git /tmp/test.git
    • rm -rf /tmp/test.git; git clone --bare --depth=1 https://gitlab.com/gitlab-org/gitlab.git /tmp/test.git
    • rm -rf /tmp/test.git; git clone --bare --depth=1 https://gitlab.com/gitlab-com/www-gitlab-com.git /tmp/test.git
  • All the calls should use PostUploadPackWithSidechannel. The flag propagation should take a while. This behavior could be observed via grpc client metrics below, or in the gitaly server logs.

  • Ensure that the metrics are fine

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 30 mins

  • /chatops run feature set workhorse_use_sidechannel false

Monitoring

Key metrics to observe

Summary of infrastructure changes

  • Does this change introduce new compute instances?
  • Does this change re-size any existing compute instances?
  • Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?

Summary of the above

Changes checklist

  • This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
  • This issue has the change technician as the assignee.
  • Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
  • This Change Issue is linked to the appropriate Issue and/or Epic
  • Necessary approvals have been completed based on the Change Management Workflow.
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
  • Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • There are currently no active incidents.
Edited by Quang-Minh Nguyen