Why are repository/archive requests so slow?

In gitlab-workhorse@1b274d0d we see benchmark results from @kirr where a simple request to Rails which only returns some JSON data for gitlab-workhorse took 500ms on average. For downloading archives this is not a major problem but it is probably a symptom of something slow that also has effects elsewhere.