Enable PackObjectsHookWithSidechannel in gprd
Production Change
Change Summary
Enable the gitaly_pack_objects_hook_with_sidechannel
feature flag in gprd. This will route existing PackObjectsHook gRPC traffic to use
PackObjectsHookWithSidechannel instead. The new RPC uses less CPU.
Already enabled in staging: #5422 (closed)
Change Details
- Services Impacted - ServiceGitaly
- Change Technician - @jacobvosmaer-gitlab
- Change Reviewer - DRI for the review of this change
- Time tracking - 60m
- Downtime Component - no downtime
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 10m
-
Set label changein-progress on this issue -
git clone --depth=1 --bare git@gitlab.com:gitlab-org/gitlab.git test.git
-
git clone --depth=1 --bare https://gitlab.com/gitlab-org/gitlab.git test.git
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 30m
-
/chatops run feature set gitaly_pack_objects_hook_with_sidechannel true --project=gitlab-org/gitlab
-
git clone --depth=1 --bare git@gitlab.com:gitlab-org/gitlab.git test.git
-
git clone --depth=1 --bare https://gitlab.com/gitlab-org/gitlab.git test.git
-
/chatops run feature delete gitaly_pack_objects_hook_with_sidechannel
-
/chatops run feature set gitaly_pack_objects_hook_with_sidechannel 1
-
/chatops run feature set gitaly_pack_objects_hook_with_sidechannel 10
-
/chatops run feature set gitaly_pack_objects_hook_with_sidechannel 100
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 20m
-
Observe RPC dashboard -
Observe logs
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 5m
-
/chatops run feature delete gitaly_pack_objects_hook_with_sidechannel
Monitoring
Key metrics to observe
- Metric: Gitaly error rate
- Location: https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main
- What changes to this metric should prompt a rollback: increase in errors
- Metric: Gitaly apdex
- Location: https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main
- What changes to this metric should prompt a rollback: decrease in apdex score
- Metric: PackObjectsHookWithSidechannel error rate and apdex
- Location: https://dashboards.gitlab.net/d/000000199/gitaly-feature-status?orgId=1&refresh=30s&var-environment=gprd&var-method=PackObjectsHookWithSidechannel&var-prometheus=prometheus-01-inf-gprd
- What changes to this metric should prompt a rollback: decrease in apdex score, increase in errors
Summary of infrastructure changes
-
Does this change introduce new compute instances? NO -
Does this change re-size any existing compute instances? NO -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc? NO
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents
Edited by Jacob Vosmaer