Stream blob to compute X-Gitlab-Content-Sha256
What does this MR do and why?
GET …/repository/files/:file_path/raw and HEAD …/repository/files/:file_path loaded the whole blob into Ruby just to compute X-Gitlab-Content-Sha256. The Gitaly client joined every chunk into one string, so about 2x the blob size was live at once. On GitLab.com about 11.7k requests a week allocate over 1 GB this way.
This MR hashes the blob chunk by chunk as Gitaly streams it, so the full blob is never built in memory.
The header value does not change for any file. content_sha256 is the hash of the converted content, as documented. For valid UTF-8 and binary content, the converted content is the same as the raw bytes. To detect those cases while streaming:
- UTF-8 is validated chunk by chunk, carrying a character split between chunks over to the next one.
- Only the first 1 MB is kept for binary detection, which is all that CharlockHolmes looks at.
Only text that is not valid UTF-8, such as ISO-8859-1 or UTF-16, falls back to loading the full blob. Those blobs are read from Gitaly twice: once while streaming, then again by the fallback. This is deliberate. Stopping the stream early leaves the GetBlob call open until its deadline, and reusing the streamed bytes would mean buffering every large blob until its encoding is known, since an invalid byte can come at the very end. Blobs that are already fully loaded (GET …/files/:file_path, or blobs under 200 bytes) are still hashed in memory without another Gitaly call.
This MR fixes memory only. The stream still runs under Gitaly's fast_timeout, so the largest blobs can still time out. Adding the large-blob rate-limit guard to these endpoints is left for a follow-up.
References
- https://gitlab.com/gitlab-org/gitlab/-/work_items/637335
- https://gitlab.com/groups/gitlab-org/-/work_items/24151
Screenshots or screen recordings
Not applicable, no UI changes.
How to set up and validate locally
-
Pick a project with a file larger than 200 bytes and note its SHA256 from a clone, for example
sha256sum README.md. -
Request the raw file and check the header matches:
curl --head --header "PRIVATE-TOKEN: <token>" \ "http://gdk.test:3000/api/v4/projects/<id>/repository/files/README.md/raw?ref=master" | grep -i content-sha256 -
For a large file, compare
json.mem_total_bytesfor the request inlog/api_json.logbefore and after this change.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.