Compress Git HTTP info/refs responses in workhorse
Problem to solve
We know that Git HTTP access to gitlab-org/gitlab-ce always incurs a download of the current refs on the server; this is the GET /info/refs part of Git HTTP. The other day we realized that for gitlab-org/gitlab-ce the response for this request is 6MB.
According to this query we pumped out 720GB of info/refs data in the last 24h. https://log.gitlab.net/goto/09d052971aa83732eb7bd3aa4f90ad2f
Further details
Many Git clients accepted a gzipped response body: Accept-Encoding: deflate, gzip. In the case of gitlab-ce, the info/refs response can be gzipped down to 26% of the uncompressed size (measured with command-line gzip, standard compression level).
It would be relatively easy to gzip-compress these responses in workhorse. This would be nice for us (less egress data) and for users.
Of course git procotol v2 is even better but this is a relatively cheap thing we can do for protocol v0.
Proposal
When supported, gzip the info/refs response.
This will improve performance for projects with a large number of refs.