Nautilus: a streamlined shell for GitLab
This idea (which I know is not original, and occasionally comes around again, this time by me) was originally spawned from this thread: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4738#note_106112000
Perhaps it's time to consider whether we should be using the actual linux login mechanisms to deal with potentially thousands of logins per minute on each host.
This got me thinking about how we can do this better. Right now, every git+ssh connection will traverse the entire Linux login process, spawn a shell and several other processes along the way. The main process runs for a second or two and then gets shutdown. This happens twenty times a second or so on each git host.
It reminds me a lot of cgi-bin
programs in the early days of the web, which were spawned to handle a single incoming request and then shutdown.
A much better approach would be to have a "GitLab Workhorse for Git+SSH". A single unprivileged service handling incoming SSH connections doing the following:
- Authentication (via internal API using authorized keys)
- Authorisation (via internal API)
- Forwarding the request on to Gitaly
In an attempt to see how difficult this would be, I put together this proof of concept: it uses Golang's stdlib SSH service, with the Gliderlabs wrapper over the top (see https://godoc.org/github.com/gliderlabs/ssh), which has a far more pleasant API than https://godoc.org/golang.org/x/crypto/ssh package, and makes building SSH servers as straight-forward as building net/http
servers.
Review the WIP here: https://gitlab.com/andrewn/nautilus/blob/master/cmd/nautilus/main.go
@jacobvosmaer-gitlab pointed out that:
- @nick.thomas is already working on a Golang (spawned) replacement for the existing shell and
- on single server installs, if you want to share port 22 between the the real sshd and GitLab-shell you need to stick to building a real shell
- There are some features that we probably couldn't support with our own server (although these are not used on GitLab.com)
One possible workaround would be to build Nautilus as part of the Golang gitlab-shell replacement and make much of the code common to both approaches. This would allow us to use a single process for high-performance instances like GitLab.com but still allow the same code to be used through a traditional ssh login model.
Comments?