### Description As expanded upon in the collapsed sections below, the existing SSH setup in Workspaces-proxy is not stable and not SSH-compliant in terms of commands. Workspaces-proxy is being decommissioned as part of https://gitlab.com/groups/gitlab-org/-/epics/16785+, where the workspaces architecture is being modified to proxy traffic via KAS. As a result of the architectural change, we have explored options involving leveraging KAS tunnels to get SSH traffic into the workspaces (see collapsible section and linked spike issues for more context). We have decided to adopt the approach of wrapping the SSH traffic into gRPC from the client machine and proxying the raw-byte stream to GitLab KAS and then to the workspace. So roughly: ``` Client ----(wrap SSH into gRPC) ---- > KAS ----(proxy byte payload to correct workspace)----> Workspace ----(forward byte payload to SSH daemon)----> ssh-daemon ``` ### Implementation plan - Configure [glab CLI](https://gitlab.com/gitlab-org/cli) to patch the client's SSH config to include relevant metadata about user workspaces we will leverage when proxying commands - Set up the KAS workspaces gRPC server to handle incoming connections for SSH - Set up the workspace(agentW embedded within the workspace) to receive incoming proxy connection from KAS and forward to local SSH Daemon <details><summary>Investigation description to conclude on an appropriate approach to SSH into workspaces via KAS</summary> ### Description Current SSH support in Workspaces has the following challenges - Standard SSH syntax is not followed. - Concurrency problem which results in unstable behaviour as described in https://gitlab.com/gitlab-org/workspaces/gitlab-workspaces-proxy/-/issues/1+ . - [Only supports user's PAT while authenticating](https://gitlab.com/groups/gitlab-org/-/epics/10912) This epic will focus on the first 2 items in the above list. The third can be optionally tackled or will be tackled in https://gitlab.com/groups/gitlab-org/-/epics/11437+ depending upon the complexity and nuances of the solution selected. ### Success criteria - [ ] Users can use standard SSH syntax for SSH'ing into a workspace. - [ ] Users can get a SSH shell into the workspace. - [ ] Users can run a command inside the Workspace through SSH without a dedicated SSH shell. - [ ] Users can port forward a local port into the workspace through SSH. - [ ] Users can port forward a remote port from the workspace through SSH. - [ ] Users can use other SSH features like jumping through servers etc. Basically, all SSH features should be supported. - [ ] Users should be able to connect their Desktop VS Code instance to the one running inside the workspace. - [ ] Users should be able to use JetBrains Gateway to connect to a workspace through SSH. This will additional validation that our SSH implementation is right. ### Possible approaches As we work on https://gitlab.com/groups/gitlab-org/-/epics/16650+ to remove GitLab Workspaces Proxy from our architecture to simplify setup, the work done here should be based on top of the architecture being worked on as part of https://gitlab.com/groups/gitlab-org/-/epics/16785+ i.e. the SSH traffic should be reverse tunnelled into the workspace through agentw. How should SSH traffic be routed to a workspace? The options so far are **Option 1 - Plain SSH from the developer's machine -\> SSH Server in GitLab Agent Server -\> Reverse tunnelled into workspace over gRPC** - Challenges - Since SSH works over TCP layer, it is not possible for us to know the host of the SSH connection i.e. the workspace name. To get over this, we could pass the workspace name in the SSH username field. But this is limiting and not SSH syntax compliant. We could explore passing this data through environment variables. Example - https://gitlab.com/gitlab-org/workspaces/gitlab-workspaces-proxy/-/merge_requests/22+ **Option 2 - SSH wrapped over WebSockets from the developer's machine -\> HTTP Server in GitLab Agent Server -\> Reverse tunnelled into workspace over gRPC** **Option 3 - SSH wrapped over gRPC from the developer's machine -\> HTTP/gRPC Server in GitLab Agent Server -\> Reverse tunnelled into workspace over gRPC** **Option 4 - Can we somehow use **[**GitLab SSHD**](https://docs.gitlab.com/administration/operations/gitlab_sshd/)** / **[**GitLab Shell**](https://gitlab.com/gitlab-org/gitlab-shell)** to SSH into a workspace?** **Option 5** - Anything else? **Do market analysis** - SSH proxy/jump-server - Option 1 - Similar to current logic we have in GitLab Workspaces Proxy - Option 2 - Something like https://gitlab.com/gitlab-org/workspaces/gitlab-workspaces-proxy/-/merge_requests/22+ In each of these options, we have to consider the following * Since GitLab Agent Server pods can be scaled up or down, how does connection retry work here? e.g. a user has a long running task in a workspace, the GitLab Agent Server pod which it was connected through has scaled down. What happens to the connection and the processes running through that connection in the workspace; specifically in terms of SSH retry? * Are we SSH syntax compliant? * Does it make it easy from a maintenance/operational perspective on .com especially? e.g. can we block some connections if we know the user is misusing it? </details> <details><summary>Previous description of this epic which was only focussed on WebSockets in GitLab Workspaces Proxy</summary> ### Description In https://gitlab.com/gitlab-org/workspaces/gitlab-workspaces-proxy/-/issues/1+ , we discussed of ways to rearchitect our SSH solution for workspaces. It was [decided](https://gitlab.com/gitlab-org/remote-development/gitlab-workspaces-proxy/-/issues/21#note_1898690123) `Using WebSockets to wrap/unwrap traffic` would be the preferred approach. A example workflow of how HTTP and SSH(or any other kind) traffic would flow through the system is shown below. ```plantuml node "Developer Machine" { [SSH Client] -right-> [WebSocket Server] note bottom of "WebSocket Server" Listens for TCP traffic/Unix Socket. Forwards it as WebSocket. end note [HTTP Client] -[hidden]down-> [SSH Client] } node "Kubernetes" { node "GitLab Workspaces Proxy" { [WebSocket Proxy] note bottom of "WebSocket Proxy" Authenticates and authorizes traffic based on "X-Secret-Key" header. Routes traffic based on request path. end note [HTTP Proxy] -[hidden]down-> [WebSocket Proxy] } node "Workspace1" { [Workspace1 WebSocket Server] -right-> [Workspace1 SSH Server] note bottom of "Workspace1 WebSocket Server" Listens for WebSocket traffic. Forwards it as TCP. end note [GitLab VS Code fork server] -[hidden]down-> [Workspace1 SSH Server] } } [WebSocket Server] -right-> [WebSocket Proxy] [WebSocket Proxy] --right--> [Workspace1 WebSocket Server] : If path is\n`workspaces/workspace1/tcp/60022` [HTTP Client] -right-> [HTTP Proxy] [HTTP Proxy] -right-> [GitLab VS Code fork server] : If path is\n`workspaces/workspace1/http/60001` ``` ### Workspace URL #### Metadata passed in request path - Workspace URL structure will be `domain.in.agent.configuration/workspaces/{{workspace_name}}/protocols/{{protocol}}/ports/{{port}}`. - GitLab VS Code fork server will be accessible at `domain.in.agent.configuration/workspaces/{{workspace_name}}/protocols/http/ports/60001`. - SSH server will be accessible at `domain.in.agent.configuration/workspaces/{{workspace_name}}/protocols/tcp/ports/60022`. #### Metadata passed in request headers - Workspace URL structure will be `domain.in.agent.configuration` for all ports of all workspaces created from that agent. All metadata will passed through the request headers which will be used for proxy decisions in gitlab-workspaces-proxy. - GitLab VS Code fork server will be accessible at `domain.in.agent.configuration` with headers `workspace-name=workspace1; protocol=http; port=60001` - SSH server will be accessible at `domain.in.agent.configuration` with headers `workspace-name=workspace1; protocol=tcp; port=60022` - Open question - Should we allow `domain.in.agent.configuration/with-optional-path-prefix` in the agent configuration? #### Metadata passed in request query parameters - Workspace URL structure will be `domain.in.agent.configuration` for all ports of all workspaces created from that agent. All metadata will passed through the request query parameters which will be used for proxy decisions in gitlab-workspaces-proxy. - GitLab VS Code fork server will be accessible at `domain.in.agent.configuration?workspace-name=workspace1&protocol=http&port=60001` - SSH server will be accessible at `domain.in.agent.configuration?workspace-name=workspace1&protocol=tcp&port=60022` - Open question - Should we allow `domain.in.agent.configuration/with-optional-path-prefix` in the agent configuration? ### What will we achieve at the end of this work - Stable SSH support - We can close https://gitlab.com/gitlab-org/remote-development/gitlab-workspaces-proxy/-/issues/21+ - Become SSH syntax compliant - We can now use any user to login during SSH. Earlier, it was always hardcoded to `gitlab-workspaces` due to our architecture choices/constraints. - Forward local port to become accessible inside workspace and vice versa using SSH local/remote port-forwarding - Maybe this is possible right now as well; we haven't tested it. But due to the buggy nature of the SSH architecture right now, wouldn't be surprised if something blocks it. - With the new architecture, we would be pretty confident that this would work and we should document with example use-cases. - Unlock https://gitlab.com/groups/gitlab-org/-/epics/10635+ by providing a stable SSH connection. - We would now be able to inject any IDE which supports SSH connection e.g. JetBrains ecosystem. - Users can now access HTTP traffic originating from within the workspace not just from the browser but also from tools like `curl`. - To enable this architecture, we would have to add support for authentication through the headers i.e. passing tokens in the headers. This will unlock developers to access any HTTP server that they have started in the workspace, to be accessible to them through tools like `curl` by passing the right headers. - Right now, it is not possible. You can only access it from within the browser since the authentication supported is OAuth only. - Users can now expose not just HTTP but any kind of traffic from within the workspace. - In contrast to existing architecture, where we can only allow HTTP traffic from within the browser and limited SSH traffic from the developer's terminal, this architecture would allow the users to start a server within the workspace(exposing any kind of traffic - HTTP, TCP, UDP, etc.) and then access it from within their terminals. - This is possible because all non-HTTP traffic would have to be wrapped as websockets and thus this architecture is extensible and unlocks new capabilities which are missing right now. - Codebase for gitlab-workspaces-proxy would become simpler, easier to maintain. It will now truly become a proxy and a simple authentication/authorization middleware doing just that and nothing more; as it should be. ### Open Questions Here are some of the open questions and my preliminary thoughts around them. - Should we wrap HTTP traffic into websockets as well? And then unwrap it inside the workspace? - I' don't think so. I'm inclined to say no for the initial iteration. But this could be considered for future iterations. - How should we make it easier for the developers to start the WebSocket server to wrap any kind of traffic? - Make `websocat` a dependency on developer machines. Add documentation with example use-cases. - Build convenience scripts or glab CLI subcommands. - How should we make it easier for the developers to start the WebSocket server inside workspace to unwrap any kind of traffic what they would like to expose from within the workspace? - Make `websocat` available in the path inside the workspace. This will be injected as part of tools-injector. Add documentation with an example use-case. - What is the best way to break down this work? </details>