Enable gitlab-sshd on gprd CNY
Rolling phase
This CR is intended to roll out the change to CNY first, another CR will roll out to the other zonal clusters #6915 (closed).
Production Change
Change Summary
This is our 7th attempt at rolling out GitLab SSHD. Reference prior attempts and further details:
- Readiness Review: https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/git-ssh-gitlab-sshd
- First Attempt: #6269 (closed)
- Remediation effort after failed first attempt: &603 (comment 883599665)
- second attempt #6321 (closed)
- Fix for the HAProxy in order to release gitlab-sshd https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15627
- 3rd attempt we faced errors on the logs and broken metrics #6890 (comment 924657530)
- 4th attempt: #6925 (closed)
- 5th attempt was successful, however, we were above the error rate threshold, therefore we rolled back. MR to adjust the error rate threshold: gitlab-org/gitlab-shell!630 (merged)
- 6th attempt was successful, we had to rollback though due to memory spikes and apdex violations: https://gitlab.com/gitlab-org/gitlab/-/issues/361755#note_956263627
This CR targets gprd to enable gitlab-sshd
instead of openssh
. The rolling out plan is to enable it per cluster starting with
gprd-cny
After the application of the change on all clusters, there will be an extra MR to clean up the configuration files and to enable the change in one place targeting the whole production environment.
We are mitigating any potential performance issues by changing the weight of the clusters gradually and slowly. Which is important as we don't want to flood cny while shifting the traffic from any of the clusters.
The performance test for gitlab-sshd has been done here: readiness!88 (comment 826798137)
Change Details
- Services Impacted - ServiceGitlab Shell
- Change Technician - @ahyield
- Change Reviewer - @skarbek
- Time tracking - 20m
- Downtime Component - None
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 3 min
-
Validate all necessary requirements are deployed to .com -
Set label changein-progress on this issue -
Schedule time with teamDelivery and ServiceGitlab Shell team to watch the rollout together - via a synchronous Zoom call -
Approve gprd-cny
gitlab-com/gl-infra/k8s-workloads/gitlab-com!1501 (merged)
Change Steps - steps to take to execute the change
Estimated Time to Complete (20m)
gprd cny
-
Set label changein-progress /label ~change::in-progress
-
gprd cny
is already at 0 weight. No need to modify this -
port forward the Service object after merging the MR and test that gitlab-sshd works as expected. - glsh into the cluster
kubectl port-forward svc/gitlab-cny-gitlab-shell -n gitlab-cny 2222:2222
git remote add localhost ssh://git@localhost:2222/ahyield/test.git
git fetch localhost
-
set the weight ./bin/set-weights gprd cny 1 ssh
-
Wait for #⃣ 5
minutes -
monitor the dashboards listed here -
set the weight ./bin/set-weights gprd cny 5 ssh
-
Wait for #⃣ 5
minutes -
Remove changein-progress label -
monitor the dashboards listed here
-
-
Set label changecomplete /label ~change::complete
Post-Change Steps - steps to take to verify the change
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
gprd cny
-
set the weight back to 0 ./bin/set-weights gprd cny 0 ssh
-
Set label changeaborted /label ~change::aborted
Monitoring
Key metrics to observe
- Metric: Gitlab ssh-d RPS
- Location: https://dashboards.gitlab.net/d/git-main/git-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&from=now-15m&to=now
- There should be an increase in this panel.
- Metric: Gitlab ssh-d SLI detail
Change Reviewer checklist
-
Check if the following applies: - The scheduled day and time of execution of the change is appropriate.
- The change plan is technically accurate.
- The change plan includes estimated timing values based on previous testing.
- The change plan includes a viable rollback plan.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
- The change plan includes success measures for all steps/milestones during the execution.
- The change adequately minimizes risk within the environment/service.
- The performance implications of executing the change are well-understood and documented.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change.
- If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
- The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
Check if all items below are complete: - The change plan is technically accurate.
- This Change Issue is linked to the appropriate Issue and/or Epic
- Change has been tested in staging and results noted in a comment on this issue.
- A dry-run has been conducted and results noted in a comment on this issue.
- For C1 and C2 change issues, the SRE on-call has been informed prior to change being rolled out. (In #production channel, mention
@sre-oncall
and this issue and await their acknowledgement.) - Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention
@release-managers
and this issue and await their acknowledgment.) - There are currently no active incidents that are severity1 or severity2
- If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.