On some routes GitLab returns 404 status codes when Gitaly is not responding
When Gitaly is down, some rails routes will timeout and fail with a 404 status code:
$ # in another terminal `lldb -p $(pgrep gitaly)`
$ time curl --head http://localhost:3000/root/gocv/commit/3f4d73bc6a6e20e4880c0ea208f3e80880e1c725
HTTP/1.1 404 Not Found
real 0m30.212s
user 0m0.010s
sys 0m0.015s
Returning a 404 error message is confusing to the user, because they're told that the object does not exist, when it does and the fault lies with the server.
Additionally, GitLab.com's monitoring depends on the application returning the correct status codes. A 4xx status code means that the fault lies with the client, and the server can ignore it. A 5xx status code means that the fault lies on the server, and is actionable (either through a bug fix or something else)
One of the side-effects of that since the application will see 4xx codes instead of 5xx, alerts will not be sent to operations teams.
To fix this, DeadlineExceeded
exceptions should result in 503 errors, not 404s.