While investigating https://gitlab.com/gitlab-com/support-forum/issues/2548 I was able to reproduce it consistently on git-08. I can push via all servers except git-08: ``` $> for i in $(seq -w 1 12); do echo -n pushing via git-$i: && touch git-${i} && git add -A && git commit -m "push via git-$i" >/dev/null && GIT_SSH_COMMAND="ssh -o LogLevel=error" git push ssh://git@git-${i}.sv.prd.gitlab.com/ilyaf/test-repo.git >/dev/null 2>/dev/null ; echo $?; done pushing via git-01:0 pushing via git-02:0 pushing via git-03:0 pushing via git-04:0 pushing via git-05:0 pushing via git-06:0 pushing via git-07:0 pushing via git-08:128 pushing via git-09:0 pushing via git-10:0 pushing via git-11:0 pushing via git-12:0 $> rm git-* && git add -A && git commit -m 'blah' > /dev/null && GIT_SSH_COMMAND="ssh -o LogLevel=error" git push ssh://git@git-08.sv.prd.gitlab.com/ilyaf/test-repo.git GitLab: API is not accessible fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ``` gitlab-shell.log contains these: ``` E, [2017-10-27T01:06:43.628313 #47743] ERROR -- : API call <POST http://0.0.0.0:8080/api/v4/internal/allowed> failed: 500 => <{"message":"500 Internal Server Error"}>. W, [2017-10-27T01:06:43.628376 #47743] WARN -- : gitlab-shell: Access denied for git command <git-receive-pack '/ilyaf/test-repo.git'> by user with key key-1109191. ``` Unfortunately, I'm not sure where unicorn logs for that API calls are, as logs in the unicorn directory are mostly empty. I tried `gitlab-ctl restart unicorn` but that didn't fix the issue. Stopping and starting gitlab services also didn't fix the issue.