Enable gitaly cgroups in grpd cny
Production Change
Change Summary
Cgroups support in Gitaly provides protection from a git process consuming an inordinate amount of memory/cpu, potentially disabling an entire Gitaly node. See example.
Having Cgroups in place with memory and CPU limits protects the Gitaly node by monitoring and killing processes that end up breaching the thresholds that are configured. The risk is that a legitimate git process that takes a large amount of memory would be identified and killed by cgroups.
We have already enabled cgroups for Gitaly on staging with no apparent issues.
We also tested this on staging-ref by initiating a Git process that would have consumed memory beyond the threshold set, and verified that cgroups killed the process.
The next step is to enable cgroups on production cny.
Change Details
- Services Impacted - ServiceGitaly
- Change Technician - DRI for the execution of this change
- Change Reviewer - @alejandro
- Time tracking - 30
- Downtime Component - 0
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Set label changein-progress on this issue
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Merge and apply https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/1474 -
Change feature flag gitaly_run_cmds_in_cgroup
to true (ramp up traffic gradually)
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Add a new project with repository like https://gitlab.com/gitlab-org/gitlab/-/issues/331599 5 -
Manually migrate this project to the gitaly cny file server 10 -
Initiate download and see if the download stops prematurely and downloads an empty archive 10
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Change feature flag gitaly_run_cmds_in_cgroup
to false -
Revert https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/1474
Monitoring
Key metrics to observe
- Metric: Gitaly Cgroups Processes Killed
- Location: {+https://thanos-query.ops.gitlab.net/graph?g0.expr=sum(gitaly_cgroup_memory_failed_total%7Benvironment%3D%22gprd%22%7D)&g0.tab=0&g0.stacked=0&g0.range_input=1d&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D+}
- What changes to this metric should prompt a rollback: When there a high number of processes getting killed. I'd say more than a few dozen is sign of trouble
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Change Reviewer checklist
-
The scheduled day and time of execution of the change is appropriate. -
The change plan is technically accurate. -
The change plan includes estimated timing values based on previous testing. -
The change plan includes a viable rollback plan. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). -
The change plan includes success measures for all steps/milestones during the execution. -
The change adequately minimizes risk within the environment/service. -
The performance implications of executing the change are well-understood and documented. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? -
The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.