Consider a 'front-end' Chef role before implementing 'stage 1'
(After reading https://gitlab.com/gitlab-org/git-access-daemon/blob/0ed34d2f29ec55a225bee472d3ac47abe51a67c8/design/README.md)
The first step will be to just remove all the git processes from the workers into a specific fleet of git workers, it's good enough to have just a couple of those as the downside of being attacked will be that these hosts will be under heavy load, not so the workers.
(Side note: this sounds like we think of our users as 'attackers'. I don't think this is the right mindset for a discussion performance.)
I think this first step can be achieved (and put to the test) by creating a new 'frontend' Chef role that would look like:
- runs sshd to accept Git SSH sessions (gitlab-shell)
- runs gitlab-workhorse
- has file system access (identical layout to 'worker' machines)
- Redis access
- TCP access to 'worker' machines
- TCP access from HAproxy machines
I think this can be built with minor Omnibus changes (I am not sure if we have the flexibility to run workhorse on a different machine than Unicorn at the moment), and by making a variation on the gitlab.com 'worker' Chef role.