Use identities instead of tokens
Right now, we're using tokens to manage access control. While this mechanism works well enough, it has probably worked a lot better back when there was no split between Gitaly and Praefect. Nowadays, we need to exchange these tokens quite regularly and even exchange them over the wire in order to allow Gitaly to reach out to Praefect. As a result, they could be considered semi-public already where it's quite trivial to extract credentials via the network. While the move to TLS would solve parts of the issue, there is definitely better alternatives.
@stanhu mentioned the idea of using signatures, which led me to think about this issue a bit further. If we were to give each node an identity (public/private key pair), then we could establish trust in a much more fine-grained way. Instead of having to pass along tokens, we'd make e.g. the Praefect identity known to all Gitaly nodes and the other way round. Messages exchanged between them could then properly be signed and verified on the respective other side while allowing for easy rotation of compromised nodes without having to exchange secrets everywhere. And, most importantly, there would be no need to inject secrets into Gitaly nodes anymore.
Another benefit would be that we could assign roles to each of the identities, limiting the capabilities each one of them has. E.g. a Gitaly node has different permissions than the Rails client in that the Rails client may not access internal RPCs.
So the idea proposed by @stanhu seems very intriguing to me and would rev up our security to the 21st century. I'd be interested in your opinions on this, @gl-gitaly.