git clones are failing on git-08

While investigating https://gitlab.com/gitlab-com/support-forum/issues/2548 I was able to reproduce it consistently on git-08.

I can push via all servers except git-08:

$> for i in $(seq -w 1 12); do echo -n pushing via git-$i: && touch git-${i} && git add -A && git commit -m "push via git-$i" >/dev/null && GIT_SSH_COMMAND="ssh -o LogLevel=error" git push ssh://git@git-${i}.sv.prd.gitlab.com/ilyaf/test-repo.git >/dev/null 2>/dev/null ; echo $?; done                                                                                    
pushing via git-01:0                          
pushing via git-02:0                          
pushing via git-03:0                          
pushing via git-04:0                          
pushing via git-05:0                          
pushing via git-06:0                          
pushing via git-07:0                          
pushing via git-08:128                        
pushing via git-09:0                          
pushing via git-10:0                          
pushing via git-11:0                          
pushing via git-12:0                          

$> rm git-* && git add -A && git commit -m 'blah' > /dev/null && GIT_SSH_COMMAND="ssh -o LogLevel=error" git push ssh://git@git-08.sv.prd.gitlab.com/ilyaf/test-repo.git                
GitLab: API is not accessible                 
fatal: Could not read from remote repository. 

Please make sure you have the correct access rights                                         
and the repository exists.

gitlab-shell.log contains these:

E, [2017-10-27T01:06:43.628313 #47743] ERROR -- : API call <POST http://0.0.0.0:8080/api/v4/internal/allowed> failed: 500 => <{"message":"500 Internal Server Error"}>.                 
W, [2017-10-27T01:06:43.628376 #47743]  WARN -- : gitlab-shell: Access denied for git command <git-receive-pack '/ilyaf/test-repo.git'> by user with key key-1109191.

Unfortunately, I'm not sure where unicorn logs for that API calls are, as logs in the unicorn directory are mostly empty.

I tried gitlab-ctl restart unicorn but that didn't fix the issue. Stopping and starting gitlab services also didn't fix the issue.