Geo: Switch blob download to use GitLab::HTTP to avoid llhttp-ffi issue
What does this MR do and why?
Adds an alternative HTTP backend for Geo blob downloads behind a feature flag (geo_blob_download_with_gitlab_http), to work around a crash/data corruption issue affecting Omnibus installs on Ubuntu 24.04 with kernel >= 6.8.
The existing BlobDownloader uses the http gem (httprb), which depends on llhttp-ffi for HTTP parsing via FFI callbacks. An interaction between rugged 1.9.0's native extension and Omnibus's bundled libffi 3.2.1 corrupts these FFI callbacks on newer kernels, causing blob downloads to either segfault or silently return incorrect HTTP status codes.
The new code path uses Gitlab::HTTP (HTTParty/Net::HTTP), which has no FFI dependency and is unaffected.
Root cause
- rugged was upgraded from 1.6.3 to 1.9.0 in GitLab 18.10
- Omnibus ships libffi 3.2.1, which uses legacy trampoline/closure mechanisms
- On kernel >= 6.8, stricter memory protections break libffi 3.2.1's executable memory allocation after rugged's native extension loads
- llhttp-ffi creates FFI closures via method(:callback).to_proc which rely on the now-corrupted trampoline memory
- The gem upgrade in !229530 (merged) (llhttp-ffi 0.5.1) does not fix this — the same callback pattern is used
- See #595139 for full investigation and customer reports.
References
How to set up and validate locally
- Setup Geo on Omnibus (Ubuntu 24.04.1 LTS, kernel 6.17.0-1010-gcp, running 18.10.1-ee)
- Create blob type (i.e. Upload) - observe sync failure/remains pending
- Enable FF in this MR via the rails console and patch the modified file then restart sidekiq
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #595139