Read distribution causes failing access checks for snippets

When accepting pushes, Gitaly uses the pre-receive hook to verify whether a given push is allowed to happen or should be refused. As Gitaly doesn't have any information about the permission model, this check is not performed by Gitaly itself but instead by the /internal/allowed API, to which we're pushing various information like the user that's performing the push, the repo he's pushing to as well as the actual change he's doing (e.g. refs/heads/master $OLDREV -> $NEWREV).

The last part is the critical part here: while we're making the API aware of which revisions are getting pushed, it cannot figure out what these revisions contain without actually invoking additional Gitaly RPCs to see what they are. This is where the read-distribution part comes into play: if the mutating call was on Gitaly node A, but the subsequent accessor calls go to Gitaly node B, then their respective views may differ. As a result, we would fail to look up the newly pushed commit, fail the access check and ultimately fail the push.

The issue started popping up in the staging environment on August 20th (gitlab#239091 (closed), gitlab#244852 (closed)) -- feature flags for read distribution have been removed on August 18th. The issue currently isn't happening on production, most likely because snippet repositories aren't hosted by the Praefect cluster. gitlab#244852 (comment 414284002) also shows logs which quite clearly show that for a working request, both accessors and mutators go to the same Gitaly node, while for failing requests they go to different Gitaly nodes.