This Epic tracks all issues and Epics related to moving [gitlab-sshd into beta](https://gitlab.com/groups/gitlab-org/-/epics/5219) on GitLab SaaS. The ancestor Epic is https://gitlab.com/groups/gitlab-org/-/epics/6296+ :exclamation: Important Note: releasing this does not deliver instant value to customers. The value comes only with the release of [Enable GitLab SaaS Allowlisting](https://gitlab.com/groups/gitlab-org/-/epics/6296) which is blocked by this epic here. Anyone interested in the timeline to customer value should refer to [Enable GitLab SaaS Allowlisting](https://gitlab.com/groups/gitlab-org/-/epics/6296). :exclamation: - [Production Readiness Review calls](https://docs.google.com/document/d/1M8o0dUtOxzNqhafE0F1R3cjh-orb-dCxUSVbn0MyZlc/edit#heading=h.1x736oi4utzq) - [Make gitlab-sshd beta-quality](https://gitlab.com/groups/gitlab-org/-/epics/5219) Epic - [Production readiness review for running gitlab-sshd on GitLab SaaS](https://gitlab.com/gitlab-com/gl-infra/readiness/-/issues/24) issue and [MR](https://gitlab.com/gitlab-com/gl-infra/readiness/-/merge_requests/88). - Production rollout Slack channel [#proj_gitlab_sshd_rollout](https://gitlab.slack.com/archives/C03BWT8R4KH) ### DRIs - Source Code DRI @igor.drozdov - Infrastructure DRI @skarbek ### Action Items - [x] Identify Infrastructure DRI - @amyphillips - [x] Draft Production Change Request - https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14454 - [x] Involve the Quality team - @sean_carroll - [x] Document incremental rollout plan - @ahyield - [x] Add additional monitoring - @sean_carroll https://gitlab.com/gitlab-org/gitlab/-/issues/343274 ### Implementation Plan Based on the single phase approach outlined [here](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14454) 1. [x] Complete [Engineering work](https://gitlab.com/groups/gitlab-org/-/epics/5219) -> %"14.5" 2. [x] Enable on pre-prod - https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14453 3. [x] Enable on Staging - https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14454 4. [x] Assess observability, performance and security on Staging * [x] [Observability](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/603#note_827773393) is completed by @ahyield * [x] [Performance](https://gitlab.com/gitlab-com/gl-infra/readiness/-/merge_requests/88#note_826798137) is completed by @igor.drozdov * [x] [Security](https://gitlab.com/gitlab-com/gl-security/appsec/appsec-reviews/-/issues/55#note_825173934) is completed by @cmaxim 6. [x] Complete Readiness Review: ~~https://gitlab.com/gitlab-com/gl-infra/readiness/-/merge_requests/88~~ https://gitlab.com/gitlab-com/gl-infra/readiness/-/merge_requests/103 7. [x] Create Production Change Request https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/15011 -> [Change Request](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6269) (closed) 8. [x] Complete [Reuse Gitaly connections and sidechannel](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/575) MR into `gstg-ref`, but do not merge @igor.drozdov 9. [x] Simulate a heavy load using a custom pipeline @igor.drozdov 11. [x] Apply the MR in `gstg-ref` and re-assess performance @igor.drozdov 12. [x] Additional Security review based on updated code @cmaxim 10. [x] Update the [Production Readiness Review](https://gitlab.com/gitlab-com/gl-infra/readiness/-/merge_requests/103) @nick.thomas @igor.drozdov 13. [x] [Rollback gitlab sshd ](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6755)on gstg. _testing the rolling back procedure_ @ahyield 13. [x] [Rollout gitlab sshd on gstg](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6755#post-change-steps-steps-to-take-to-verify-the-change). _testing the rolling out procedure_ @ahyield 11. [x] Load test against Staging @igor.drozdov 11. [x] Fix Metrics: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/603#note_937529647 11. [x] Investigate Errors found on Canary enablement: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/559 14. [x] Enable on production * [x] ~https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6269~ - CR Failed, see - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6269#note_837343274 * [x] [~2022-02-09: OOM killed containers during cny rollout of gitlab-sshd~](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6327) * [x] ~2022-04-27 - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6890~ * [x] Canary Stage Rollout - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6925 * [x] Main Stage Rollout - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6915 ### Post Deployment 9. [x] Update documentation to move from Alpha to Beta @sean_carroll https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80708+ 10. [x] Retro issue https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15619+ @sean_carroll