Enable gitlab-sshd on gprd (b, c and d)
Production Change
Rolling phase
This CR is intended to roll out the change to the zonal clusters. Another CR rolled this out to CNY #6890 (closed)
Change Summary
This is our 4th attempt at rolling out GitLab SSHD. Reference prior attempts and further details:
- Readiness Review: https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/git-ssh-gitlab-sshd
- First Attempt: #6269 (closed)
- Remediation effort after failed first attempt: &603 (comment 883599665)
- second attempt #6321 (closed)
- Fix for the HAProxy in order to release gitlab-sshd https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15627
- 3rd attempt we faced errors on the logs and broken metrics #6890 (comment 924657530)
This CR targets gprd to enable gitlab-sshd instead of openssh. The rolling out plan is to enable it per cluster starting with
gprd-bgprd-cgprd-d
After the application of the change on all clusters, there will be an extra MR to clean up the configuration files and to enable the change in one place targeting the whole production environment.
We are mitigating any potential performance issues by changing the weight of the clusters gradually and slowly. Which is important as we don't want to flood cny while shifting the traffic from any of the clusters.
The performance test for gitlab-sshd has been done here: readiness!88 (comment 826798137)
Change Details
- Services Impacted - ServiceGitlab Shell
- Change Technician - @ahyield
- Change Reviewer - @skarbek
- Time tracking - 7 Hours
- Downtime Component - none
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 3 min
- Set label changein-progress on this issue
- Schedule time with ~"team::Delivery" and ServiceGitlab Shell team to watch the rollout together - via a synchronous Zoom call
- Approve
gprd-bgitlab-com/gl-infra/k8s-workloads/gitlab-com!1502 (merged) - Approve
gprd-cgitlab-com/gl-infra/k8s-workloads/gitlab-com!1503 (merged) - Approve
gprd-dgitlab-com/gl-infra/k8s-workloads/gitlab-com!1504 (merged) - Approve
gprdgitlab-com/gl-infra/k8s-workloads/gitlab-com!1505 (merged)
Change Steps - steps to take to execute the change
Estimated Time to Complete (240m) - 4hrs
→ ./bin/get-weights gprd ssh
3 ssh/gke-cny-ssh : 0 (initial 0)
3 ssh/shell-gke-us-east1-b : 100 (initial 100)
3 ssh/shell-gke-us-east1-c : 100 (initial 100)
3 ssh/shell-gke-us-east1-d : 100 (initial 100)
gprd b
- Make sure we have a green light from RM
- Make sure changein-progress is set
- set the weight for gprd-b
./bin/set-weights gprd b 80 ssh- Wait for
#️⃣ 5minutes - set the weight for gprd-b
./bin/set-weights gprd b 50 ssh - Wait for
#️⃣ 5minutes - set the weight for gprd-b
./bin/set-weights gprd b 20 ssh - Wait for
#️⃣ 5minutes - set the weight for gprd-b
./bin/set-weights gprd b 0 ssh - Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1502 (merged)
- port forward the Service object after merging the MR and test that gitlab-sshd works as expected.
glshinto the clusterkubectl port-forward svc/gitlab-gitlab-shell -n gitlab 2222:2222git remote add localhost ssh://git@localhost:2222/ahyield/test.gitgit fetch localhost
- set the weight
./bin/set-weights gprd b 20 ssh - Wait for
#️⃣ 5minutes - set the weight
./bin/set-weights gprd b 50 ssh - Wait for
#️⃣ 5minutes - set the weight
./bin/set-weights gprd b 80 ssh - Wait for
#️⃣ 5minutes - set the weight
./bin/set-weights gprd b 100 ssh - Remove changein-progress label
- monitor the dashboards listed here
- Wait for
gprd c
- Make sure we have a green light from RM
- Make sure changein-progress is set
- set the weight for gprd-c
./bin/set-weights gprd ssh/shell-gke-us-east1-c 80 ssh- Wait for
#️⃣ 5minutes - set the weight for gprd-c
./bin/set-weights gprd ssh/shell-gke-us-east1-c 50 ssh - Wait for
#️⃣ 5minutes - set the weight for gprd-c
./bin/set-weights gprd ssh/shell-gke-us-east1-c 20 ssh - Wait for
#️⃣ 5minutes - set the weight for gprd-c
./bin/set-weights gprd ssh/shell-gke-us-east1-c 0 ssh - Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1503 (merged)
- port forward the Service object after merging the MR and test that gitlab-sshd works as expected.
glshinto the clusterkubectl port-forward svc/gitlab-gitlab-shell -n gitlab 2222:2222git remote add localhost ssh://git@localhost:2222/ahyield/test.gitgit fetch localhost
- set the weight
./bin/set-weights gprd ssh/shell-gke-us-east1-c 20 ssh - Wait for
#️⃣ 5minutes - set the weight
./bin/set-weights gprd ssh/shell-gke-us-east1-c 50 ssh - Wait for
#️⃣ 5minutes - set the weight
./bin/set-weights gprd ssh/shell-gke-us-east1-c 80 ssh - Wait for
#️⃣ 5minutes - set the weight
./bin/set-weights gprd ssh/shell-gke-us-east1-c 100 ssh - Remove changein-progress label
- monitor the dashboards listed here
- Wait for
gprd d
- Make sure we have a green light from RM
- Make sure changein-progress is set
- set the weight for gprd-d
./bin/set-weights gprd d 80 ssh- Wait for
#️⃣ 5minutes - set the weight for gprd-d
./bin/set-weights gprd d 50 ssh - Wait for
#️⃣ 5minutes - set the weight for gprd-d
./bin/set-weights gprd d 20 ssh - Wait for
#️⃣ 5minutes - set the weight for gprd-d
./bin/set-weights gprd d 0 ssh - Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1504 (merged)
- port forward the pod after merging the MR and test that gitlab-sshd works as expected.
glshinto the clusterkubectl port-forward svc/gitlab-gitlab-shell -n gitlab 2222:2222git remote add localhost ssh://git@localhost:2222/ahyield/test.gitgit fetch localhost
- set the weight
./bin/set-weights gprd d 20 ssh - Wait for
#️⃣ 5minutes - set the weight
./bin/set-weights gprd d 50 ssh - Wait for
#️⃣ 5minutes - set the weight
./bin/set-weights gprd d 80 ssh - Wait for
#️⃣ 5minutes - set the weight
./bin/set-weights gprd d 100 ssh - Remove changein-progress label
- monitor the dashboards listed here
- Wait for
Post-Change Steps - steps to take to verify the change
The above steps contain verification. This section will be utilized to clean up our configurations
Estimated Time to Complete (40m)
- Revert HPA change that was introduce mid procedure
-
kubectl edit hpa -n gitlab-cny gitlab-webservice-git - Change the
minReplicasback to the default value of5
-
- Remove the configuration from the regional clusters, applied by these MRs
- Accept and apply gitlab-com/gl-infra/k8s-workloads/gitlab-com!1505 (merged)
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (180m) - 3hrs
gprd b
- set the weight to 80
./bin/set-weights gprd b 80 ssh- Wait for
#️⃣ 5minutes - set the weight to 50
./bin/set-weights gprd b 50 ssh - Wait for
#️⃣ 5minutes - set the weight to 20
./bin/set-weights gprd b 20 ssh - Wait for
#️⃣ 5minutes - set the weight to 0
./bin/set-weights gprd b 0 ssh - revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1502 (merged)
- set the weight to 20
./bin/set-weights gprd b 20 ssh - Wait for
#️⃣ 5minutes - set the weight to 50
./bin/set-weights gprd b 50 ssh - Wait for
#️⃣ 5minutes - set the weight to 80
./bin/set-weights gprd b 80 ssh - Wait for
#️⃣ 5minutes - set the weight to 100
./bin/set-weights gprd b 100 ssh
- Wait for
gprd c
- set the weight to 80
./bin/set-weights gprd shell/gprd-gke-us-east1-c 80 ssh- Wait for
#️⃣ 5minutes - set the weight to 50
./bin/set-weights gprd shell/gprd-gke-us-east1-c 50 ssh - Wait for
#️⃣ 5minutes - set the weight to 20
./bin/set-weights gprd shell/gprd-gke-us-east1-c 20 ssh - Wait for
#️⃣ 5minutes - set the weight to 0
./bin/set-weights gprd shell/gprd-gke-us-east1-c 0 ssh - revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1503 (merged)
- set the weight to 20
./bin/set-weights gprd shell/gprd-gke-us-east1-c 20 ssh - Wait for
#️⃣ 5minutes - set the weight to 50
./bin/set-weights gprd shell/gprd-gke-us-east1-c 50 ssh - Wait for
#️⃣ 5minutes - set the weight to 80
./bin/set-weights gprd shell/gprd-gke-us-east1-c 80 ssh - Wait for
#️⃣ 5minutes - set the weight to 100
./bin/set-weights gprd shell/gprd-gke-us-east1-c 100 ssh
- Wait for
gprd d
- set the weight to 80
./bin/set-weights gprd d 80 ssh- Wait for
#️⃣ 5minutes - set the weight to 50
./bin/set-weights gprd d 50 ssh - Wait for
#️⃣ 5minutes - set the weight to 20
./bin/set-weights gprd d 20 ssh - Wait for
#️⃣ 5minutes - set the weight to 0
./bin/set-weights gprd d 0 ssh - revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1504 (merged)
- set the weight to 20
./bin/set-weights gprd d 20 ssh - Wait for
#️⃣ 5minutes - set the weight to 50
./bin/set-weights gprd d 50 ssh - Wait for
#️⃣ 5minutes - set the weight to 80
./bin/set-weights gprd d 80 ssh - Wait for
#️⃣ 5minutes - set the weight to 100
./bin/set-weights gprd d 10 ssh
- Wait for
Monitoring
Key metrics to observe
- Metric: Gitlab ssh-d RPS
- Location: https://dashboards.gitlab.net/d/git-main/git-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&from=now-15m&to=now
- There should be an increase in this panel.
- Metric: Gitlab ssh-d SLI detail
Summary of infrastructure changes
- Does this change introduce new compute instances?
- No
- Does this change re-size any existing compute instances?
- No
- Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
- No
Change Reviewer checklist
- The scheduled day and time of execution of the change is appropriate.
- The change plan is technically accurate.
- The change plan includes estimated timing values based on previous testing.
- The change plan includes a viable rollback plan.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change.
- The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
- The change plan includes success measures for all steps/milestones during the execution.
- The change adequately minimizes risk within the environment/service.
- The performance implications of executing the change are well-understood and documented.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
- The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
- This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
- This issue has the change technician as the assignee.
- Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
- This Change Issue is linked to the appropriate Issue and/or Epic
- Necessary approvals have been completed based on the Change Management Workflow.
- Change has been tested in staging and results noted in a comment on this issue.
- A dry-run has been conducted and results noted in a comment on this issue.
- SRE on-call has been informed prior to change being rolled out. (In #production channel, mention
@sre-oncalland this issue and await their acknowledgement.) - Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention
@release-managersand this issue and await their acknowledgment.) - There are currently no active incidents.