Enable gitlab-sshd on gprd

Production Change

Change Summary

This is our second attempt at rolling out GitLab SSHD. Reference prior attempts and further details:

• Readiness Review: https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/git-ssh-gitlab-sshd
• First Attempt: #6269 (closed)
• Remediation effort after failed first attempt: &603 (comment 883599665)

This CR targets gprd to enable gitlab-sshd instead of openssh. The rolling out plan is to enable it per cluster starting with

1. gprd-cny
2. gprd-b
3. gprd-c
4. gprd-d

After the application of the change on all clusters, there will be an extra MR to clean up the configuration files and to enable the change in one place targeting the whole production environment.

We are mitigating any potential performance issues by changing the weight of the clusters gradually and slowly. Which is important as we don't want to flood cny while shifting the traffic from any of the clusters.

The performance test for gitlab-sshd has been done here: readiness!88 (comment 826798137)

Change Details

1. Services Impacted - ServiceGitlab Shell
2. Change Technician - @ahyield
3. Change Reviewer - @skarbek
4. Time tracking - 8 Hours
5. Downtime Component - none

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 3 min

• Set label changein-progress on this issue

• Approve gprd-cny gitlab-com/gl-infra/k8s-workloads/gitlab-com!1501 (merged)
• Approve gprd-b gitlab-com/gl-infra/k8s-workloads/gitlab-com!1502 (merged)
• Approve gprd-c gitlab-com/gl-infra/k8s-workloads/gitlab-com!1503 (merged)
• Approve gprd-d gitlab-com/gl-infra/k8s-workloads/gitlab-com!1504 (merged)
• Approve gprd gitlab-com/gl-infra/k8s-workloads/gitlab-com!1505 (merged)

Change Steps - steps to take to execute the change

Estimated Time to Complete (240m) - 4hrs

→ ./bin/get-weights gprd ssh
   3          ssh/gke-cny-ssh : 0 (initial 0)
   3          ssh/shell-gke-us-east1-b : 100 (initial 100)
   3          ssh/shell-gke-us-east1-c : 100 (initial 100)
   3          ssh/shell-gke-us-east1-d : 100 (initial 100)

make sure we don't saturate cny when the traffic gets shifted

gprd cny

• Make sure changein-progress is set
  • gprd cny is already at 0 weight. No need to modify this
  • port forward the Service object after merging the MR and test that gitlab-sshd works as expected.
    • glsh into the cluster
    • kubectl port-forward svc/gitlab-cny-gitlab-shell -n gitlab-cny 2222:2222
    • git remote add localhost ssh://git@localhost:2222/ahyield/test.git
    • git fetch localhost
  • set the weight ./bin/set-weights gprd cny 5 ssh
  • Wait for #⃣ 5 minutes
  • set the weight ./bin/set-weights gprd cny 10 ssh
  • Wait for #⃣ 5 minutes
  • Remove changein-progress label
  • monitor the dashboards listed here

gprd b

• Make sure we have a green light from RM
• Make sure changein-progress is set
• set the weight for gprd-b ./bin/set-weights gprd b 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight for gprd-b ./bin/set-weights gprd b 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight for gprd-b ./bin/set-weights gprd b 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight for gprd-b ./bin/set-weights gprd b 0 ssh
  • Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1502 (merged)
  • port forward the Service object after merging the MR and test that gitlab-sshd works as expected.
    • glsh into the cluster
    • kubectl port-forward svc/gitlab-gitlab-shell -n gitlab 2222:2222
    • git remote add localhost ssh://git@localhost:2222/ahyield/test.git
    • git fetch localhost
  • set the weight ./bin/set-weights gprd b 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight ./bin/set-weights gprd b 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight ./bin/set-weights gprd b 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight ./bin/set-weights gprd b 100 ssh
  • Remove changein-progress label
  • monitor the dashboards listed here

gprd c

• Make sure we have a green light from RM
• Make sure changein-progress is set
• set the weight for gprd-c ./bin/set-weights gprd c 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight for gprd-c ./bin/set-weights gprd c 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight for gprd-c ./bin/set-weights gprd c 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight for gprd-c ./bin/set-weights gprd c 0 ssh
  • Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1503 (merged)
  • port forward the Service object after merging the MR and test that gitlab-sshd works as expected.
    • glsh into the cluster
    • kubectl port-forward svc/gitlab-gitlab-shell -n gitlab 2222:2222
    • git remote add localhost ssh://git@localhost:2222/ahyield/test.git
    • git fetch localhost
  • set the weight ./bin/set-weights gprd c 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight ./bin/set-weights gprd c 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight ./bin/set-weights gprd c 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight ./bin/set-weights gprd c 100 ssh
  • Remove changein-progress label
  • monitor the dashboards listed here

gprd d

• Make sure we have a green light from RM
• Make sure changein-progress is set
• set the weight for gprd-d ./bin/set-weights gprd d 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight for gprd-d ./bin/set-weights gprd d 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight for gprd-d ./bin/set-weights gprd d 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight for gprd-d ./bin/set-weights gprd d 0 ssh
  • Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1504 (merged)
  • port forward the pod after merging the MR and test that gitlab-sshd works as expected.
    • glsh into the cluster
    • kubectl port-forward svc/gitlab-gitlab-shell -n gitlab 2222:2222
    • git remote add localhost ssh://git@localhost:2222/ahyield/test.git
    • git fetch localhost
  • set the weight ./bin/set-weights gprd d 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight ./bin/set-weights gprd d 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight ./bin/set-weights gprd d 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight ./bin/set-weights gprd d 100 ssh
  • Remove changein-progress label
  • monitor the dashboards listed here

Post-Change Steps - steps to take to verify the change

The above steps contain verification. This section will be utilized to clean up our configurations

Estimated Time to Complete (40m)

• Remove the configuration from the regional clusters, applied by these MRs
  • gprd-cny gitlab-com/gl-infra/k8s-workloads/gitlab-com!1501 (merged)
  • gprd-b gitlab-com/gl-infra/k8s-workloads/gitlab-com!1502 (merged)
  • gprd-cgitlab-com/gl-infra/k8s-workloads/gitlab-com!1503 (merged)
  • gprd-d gitlab-com/gl-infra/k8s-workloads/gitlab-com!1504 (merged)
• Accept and apply gitlab-com/gl-infra/k8s-workloads/gitlab-com!1505 (merged)

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (180m) - 3hrs

gprd cny

• revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1501 (merged)
  • set the weight back to 0 ./bin/set-weights gprd cny 0 ssh

gprd b

• set the weight to 80 ./bin/set-weights gprd b 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 50 ./bin/set-weights gprd b 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 20 ./bin/set-weights gprd b 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 0 ./bin/set-weights gprd b 0 ssh
  • revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1502 (merged)
  • set the weight to 20 ./bin/set-weights gprd b 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 50 ./bin/set-weights gprd b 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 80 ./bin/set-weights gprd b 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 100 ./bin/set-weights gprd b 100 ssh

gprd c

• set the weight to 80 ./bin/set-weights gprd c 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 50 ./bin/set-weights gprd c 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 20 ./bin/set-weights gprd c 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 0 ./bin/set-weights gprd c 0 ssh
  • revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1503 (merged)
  • set the weight to 20 ./bin/set-weights gprd c 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 50 ./bin/set-weights gprd c 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 80 ./bin/set-weights gprd c 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 100 ./bin/set-weights gprd c 100 ssh

gprd d

• set the weight to 80 ./bin/set-weights gprd d 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 50 ./bin/set-weights gprd d 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 20 ./bin/set-weights gprd d 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 0 ./bin/set-weights gprd d 0 ssh
  • revert gitlab-com/gl-infra/k8s-workloads/gitlab-com!1504 (merged)
  • set the weight to 20 ./bin/set-weights gprd d 20 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 50 ./bin/set-weights gprd d 50 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 80 ./bin/set-weights gprd d 80 ssh
  • Wait for #⃣ 5 minutes
  • set the weight to 100 ./bin/set-weights gprd d 10 ssh

Monitoring

Key metrics to observe

• Metric: Gitlab ssh-d RPS
  • Location: https://dashboards.gitlab.net/d/git-main/git-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&from=now-15m&to=now
  • There should be an increase in this panel.
• Metric: Gitlab ssh-d SLI detail
  • Location: https://dashboards.gitlab.net/d/git-main/git-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&from=now-15m&to=now

• git dashboard
• Git overview
• CNY dashboard

Summary of infrastructure changes

• Does this change introduce new compute instances?
  • No
• Does this change re-size any existing compute instances?
  • No
• Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
  • No

Change Reviewer checklist

C4 C3 C2 C1:

• The scheduled day and time of execution of the change is appropriate.
• The change plan is technically accurate.
• The change plan includes estimated timing values based on previous testing.
• The change plan includes a viable rollback plan.
• The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

• The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
• The change plan includes success measures for all steps/milestones during the execution.
• The change adequately minimizes risk within the environment/service.
• The performance implications of executing the change are well-understood and documented.
• The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
• The change has a primary and secondary SRE with knowledge of the details available during the change window.

Change Technician checklist

• This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
• This issue has the change technician as the assignee.
• Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
• This Change Issue is linked to the appropriate Issue and/or Epic
• Necessary approvals have been completed based on the Change Management Workflow.
• Change has been tested in staging and results noted in a comment on this issue.
• A dry-run has been conducted and results noted in a comment on this issue.
• SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
• Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
• There are currently no active incidents.