Remove authorized_keys integration in favour of "fast SSH lookups" as the default git+ssh authorization mechanism
Currently, GitLab has three different mechanisms by which it can look up the user attached to an SSH key when logging in.
The first - and default - mechanism - is integration via authorized_keys. When an SSH key is added or removed in the API, an AuthorizedKeysWorker in sidekiq runs and modifies the authorized_keys file for the git user. This is not very scalable, and is vulnerable to race conditions, out-of-order execution issues, and so on. It also necessitates a shared filesystem for most deployments of more than one machine. We don't use it on GitLab.com
The second mechanism is "Fast lookup of SSH keys", which is documented here: https://docs.gitlab.com/ee/administration/operations/fast_ssh_key_lookup.html . In this mode, sshd is configured (via 2 lines in /etc/ssh/sshd_config) to call a binary which does an API lookup against GitLab.com to look up keys. This requires no integration with the authorized_keys file and is generally 1000x better. It's the mechanism we use on GitLab.com
A third mechanism is lookup via SSH certificate: https://docs.gitlab.com/ee/administration/operations/ssh_certificates.html . We don't use this either, but it was contributed by a large customer. I'm going to ignore it for now ^^.
The first mechanism has the advantage of working "out of the box" with omnibus - the user doesn't have to modify sshd_config for their machine when installing the omnibus package. However, the fact that we don't use it ourselves means that it's quite easy for a breakage in it to get through to production: #212178 (closed) . That it works out of the box also means that instances continue using it long past the time when they should have switched to the second mechanism.
At the cost of one manual step ("Add these two lines to your sshd_config"), we can remove the first mechanism altogether. It's probably too late to do it in time for %13.0, but I'd love to have a discussion about it and be ready, if we decide to, in time for %14.0!
WDYT @m_gill @danielgruesso ?