Enable gitlab-sshd on gprd
Production Change
Change Summary
This is our second attempt at rolling out GitLab SSHD. Reference prior attempts and further details:
- Readiness Review: https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/git-ssh-gitlab-sshd
- First Attempt: #6269 (closed)
- Remediation effort after failed first attempt: &603 (comment 883599665)
This CR targets gprd to enable gitlab-sshd
instead of openssh
. The rolling out plan is to enable it per cluster starting with
gprd-cny
gprd-b
gprd-c
gprd-d
After the application of the change on all clusters, there will be an extra MR to clean up the configuration files and to enable the change in one place targeting the whole production environment.
We are mitigating any potential performance issues by changing the weight of the clusters gradually and slowly. Which is important as we don't want to flood cny while shifting the traffic from any of the clusters.
The performance test for gitlab-sshd has been done here: readiness!88 (comment 826798137)
Change Details
- Services Impacted - ServiceGitlab Shell
- Change Technician - @ahyield
- Change Reviewer - @skarbek
- Time tracking - 8 Hours
- Downtime Component - none
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 3 min
-
Set label changein-progress on this issue
-
Approve gprd-cny
gitlab-com/gl-infra/k8s-workloads/gitlab-com!1501 (merged) -
Approve gprd-b
gitlab-com/gl-infra/k8s-workloads/gitlab-com!1502 (merged) -
Approve gprd-c
gitlab-com/gl-infra/k8s-workloads/gitlab-com!1503 (merged) -
Approve gprd-d
gitlab-com/gl-infra/k8s-workloads/gitlab-com!1504 (merged) -
Approve gprd
gitlab-com/gl-infra/k8s-workloads/gitlab-com!1505 (merged)
Change Steps - steps to take to execute the change
Estimated Time to Complete (240m) - 4hrs
→ ./bin/get-weights gprd ssh
3 ssh/gke-cny-ssh : 0 (initial 0)
3 ssh/shell-gke-us-east1-b : 100 (initial 100)
3 ssh/shell-gke-us-east1-c : 100 (initial 100)
3 ssh/shell-gke-us-east1-d : 100 (initial 100)
make sure we don't saturate cny when the traffic gets shifted
gprd cny
-
Make sure changein-progress is set -
gprd cny
is already at 0 weight. No need to modify this -
port forward the Service object after merging the MR and test that gitlab-sshd works as expected. -
glsh
into the cluster kubectl port-forward svc/gitlab-cny-gitlab-shell -n gitlab-cny 2222:2222
git remote add localhost ssh://git@localhost:2222/ahyield/test.git
git fetch localhost
-
-
set the weight ./bin/set-weights gprd cny 5 ssh
-
Wait for #⃣ 5
minutes -
set the weight ./bin/set-weights gprd cny 10 ssh
-
Wait for #⃣ 5
minutes -
Remove changein-progress label -
monitor the dashboards listed here
-
gprd b
-
Make sure we have a green light from RM -
Make sure changein-progress is set -
set the weight for gprd-b ./bin/set-weights gprd b 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight for gprd-b ./bin/set-weights gprd b 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight for gprd-b ./bin/set-weights gprd b 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight for gprd-b ./bin/set-weights gprd b 0 ssh
-
Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1502 (merged) -
port forward the Service object after merging the MR and test that gitlab-sshd works as expected. -
glsh
into the cluster kubectl port-forward svc/gitlab-gitlab-shell -n gitlab 2222:2222
git remote add localhost ssh://git@localhost:2222/ahyield/test.git
git fetch localhost
-
-
set the weight ./bin/set-weights gprd b 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight ./bin/set-weights gprd b 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight ./bin/set-weights gprd b 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight ./bin/set-weights gprd b 100 ssh
-
Remove changein-progress label -
monitor the dashboards listed here
-
gprd c
-
Make sure we have a green light from RM -
Make sure changein-progress is set -
set the weight for gprd-c ./bin/set-weights gprd c 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight for gprd-c ./bin/set-weights gprd c 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight for gprd-c ./bin/set-weights gprd c 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight for gprd-c ./bin/set-weights gprd c 0 ssh
-
Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1503 (merged) -
port forward the Service object after merging the MR and test that gitlab-sshd works as expected. -
glsh
into the cluster kubectl port-forward svc/gitlab-gitlab-shell -n gitlab 2222:2222
git remote add localhost ssh://git@localhost:2222/ahyield/test.git
git fetch localhost
-
-
set the weight ./bin/set-weights gprd c 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight ./bin/set-weights gprd c 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight ./bin/set-weights gprd c 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight ./bin/set-weights gprd c 100 ssh
-
Remove changein-progress label -
monitor the dashboards listed here
-
gprd d
-
Make sure we have a green light from RM -
Make sure changein-progress is set -
set the weight for gprd-d ./bin/set-weights gprd d 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight for gprd-d ./bin/set-weights gprd d 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight for gprd-d ./bin/set-weights gprd d 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight for gprd-d ./bin/set-weights gprd d 0 ssh
-
Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1504 (merged) -
port forward the pod after merging the MR and test that gitlab-sshd works as expected. -
glsh
into the cluster kubectl port-forward svc/gitlab-gitlab-shell -n gitlab 2222:2222
git remote add localhost ssh://git@localhost:2222/ahyield/test.git
git fetch localhost
-
-
set the weight ./bin/set-weights gprd d 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight ./bin/set-weights gprd d 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight ./bin/set-weights gprd d 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight ./bin/set-weights gprd d 100 ssh
-
Remove changein-progress label -
monitor the dashboards listed here
-
Post-Change Steps - steps to take to verify the change
The above steps contain verification. This section will be utilized to clean up our configurations
Estimated Time to Complete (40m)
-
Remove the configuration from the regional clusters, applied by these MRs -
Accept and apply gitlab-com/gl-infra/k8s-workloads/gitlab-com!1505 (merged)
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (180m) - 3hrs
gprd cny
-
revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1501 (merged) -
set the weight back to 0 ./bin/set-weights gprd cny 0 ssh
-
gprd b
-
set the weight to 80 ./bin/set-weights gprd b 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 50 ./bin/set-weights gprd b 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 20 ./bin/set-weights gprd b 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 0 ./bin/set-weights gprd b 0 ssh
-
revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1502 (merged) -
set the weight to 20 ./bin/set-weights gprd b 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 50 ./bin/set-weights gprd b 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 80 ./bin/set-weights gprd b 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 100 ./bin/set-weights gprd b 100 ssh
-
gprd c
-
set the weight to 80 ./bin/set-weights gprd c 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 50 ./bin/set-weights gprd c 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 20 ./bin/set-weights gprd c 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 0 ./bin/set-weights gprd c 0 ssh
-
revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1503 (merged) -
set the weight to 20 ./bin/set-weights gprd c 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 50 ./bin/set-weights gprd c 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 80 ./bin/set-weights gprd c 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 100 ./bin/set-weights gprd c 100 ssh
-
gprd d
-
set the weight to 80 ./bin/set-weights gprd d 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 50 ./bin/set-weights gprd d 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 20 ./bin/set-weights gprd d 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 0 ./bin/set-weights gprd d 0 ssh
-
revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1504 (merged) -
set the weight to 20 ./bin/set-weights gprd d 20 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 50 ./bin/set-weights gprd d 50 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 80 ./bin/set-weights gprd d 80 ssh
-
Wait for #⃣ 5
minutes -
set the weight to 100 ./bin/set-weights gprd d 10 ssh
-
Monitoring
Key metrics to observe
- Metric: Gitlab ssh-d RPS
- Location: https://dashboards.gitlab.net/d/git-main/git-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&from=now-15m&to=now
- There should be an increase in this panel.
- Metric: Gitlab ssh-d SLI detail
Summary of infrastructure changes
-
Does this change introduce new compute instances? - No
-
Does this change re-size any existing compute instances? - No
-
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc? - No
Change Reviewer checklist
-
The scheduled day and time of execution of the change is appropriate. -
The change plan is technically accurate. -
The change plan includes estimated timing values based on previous testing. -
The change plan includes a viable rollback plan. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). -
The change plan includes success measures for all steps/milestones during the execution. -
The change adequately minimizes risk within the environment/service. -
The performance implications of executing the change are well-understood and documented. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? -
The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.