### Summary Geo container registry replication fails on GitLab 18.10.x (Ubuntu 22.04). The same setup worked correctly on 18.9.5. The failure is specific to `EE::ContainerRegistry::Client` and manifests as a corrupted HTTP response status code (`:34` instead of `200`) when pulling blobs, despite the response body arriving intact. This is **not** the llhttp-ffi issue documented in #595139 — `http 5.3.1` and `llhttp-ffi 0.5.1` are confirmed loaded at runtime. The kernel is 5.15.x. ### Steps to reproduce 1. Set up a Geo secondary site with container registry replication enabled on GitLab 18.10.x (Omnibus, Ubuntu 22.04, single-node). 2. Trigger container registry replication from the primary to the secondary. 3. Observe `Geo::ContainerRepositorySyncService` failing with an `Invalid Unicode` error. ### Example Project N/A (customer environment — see RFH for details) ### What is the current *bug* behavior? `Geo::ContainerRepositorySyncService` fails with an `Invalid Unicode` error when pulling blobs via `EE::ContainerRegistry::Client`. The `http` gem returns a corrupted status code (`:34` instead of `200`), despite the response body being intact. ### What is the expected *correct* behavior? Container registry blob pulling should succeed with HTTP status `200 OK`, and Geo replication should complete without errors. ### Relevant logs and/or screenshots See the internal RFH for logs and command output (contains customer data): [gitlab-com/request-for-help#4663](https://gitlab.com/gitlab-com/request-for-help/-/work_items/4663) ### Output of checks **Ruled out:** - **#595139 (llhttp-ffi 0.4.0 on kernel 6.17):** ruled out — both primary and secondary are running `http 5.3.1` / `llhttp-ffi 0.5.1` as confirmed by `Gem.loaded_specs` in the Rails console, and the kernel is 5.15.x. - **`geo_blob_download_with_gitlab_http` feature flag:** enabled on the primary, confirmed visible on the secondary — no effect, as this flag covers `BlobDownloadService` not `EE::ContainerRegistry::Client`. - **Registry data integrity:** tag listing and manifest fetching succeed; only blob pulling fails. Manual pulls via `Net::HTTP` and `curl` return `200 OK` successfully. #### Results of GitLab environment info <details> <summary>Expand for output related to GitLab environment info</summary> <pre> - GitLab version: 18.10.x - OS: Ubuntu 22.04 - Deployment type: Omnibus - Single-node Geo setup - http gem: 5.3.1 - llhttp-ffi: 0.5.1 - Kernel: 5.15.x </pre> </details> ### Possible fixes !230361 switched `BlobDownloadService` to use `Gitlab::HTTP` (HTTParty/Net::HTTP) instead of the `http` gem to avoid the llhttp-ffi/FFI corruption issue. A similar approach may be needed for `EE::ContainerRegistry::Client#pull_blob`, which still uses the `http` gem and is not covered by the `geo_blob_download_with_gitlab_http` feature flag introduced in that MR. ### Workaround Disable container registry replication on the secondary: ```ruby # gitlab.rb #gitlab_rails['geo_registry_replication_enabled'] = true #gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://registry.example.com/' ``` ### Patch release information for backports If the bug fix needs to be backported in a [patch release](https://handbook.gitlab.com/handbook/engineering/releases/patch-releases) to a version under [the maintenance policy](https://docs.gitlab.com/policy/maintenance/), please follow the steps on the [patch release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/engineers.md). Refer to the [internal "Release Information" dashboard](https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1) for information about the next patch release, including the targeted versions, expected release date, and current status. #### High-severity bug remediation To remediate high-severity issues requiring an [internal release](https://handbook.gitlab.com/handbook/engineering/releases/internal-releases/) for single-tenant SaaS instances, refer to the [internal release process for engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/engineers.md?ref_type=heads).