## Summary On GitLab.com, a CI job on a hosted runner can clone a repository in a group that has **Restrict access by IP address** enabled, but it cannot fetch that repository's Git LFS objects. The job fails with: ``` LFS: Repository or object not found: https://gitlab.com/<group>/<project>.git/gitlab-lfs/objects/<oid> Check that it exists and that you have proper access to it ``` The cause is not the IP allowlist contents and not the runner fleet. GitLab Runner rewrites the repository URL onto an internal CI gateway, so the clone and the LFS `objects/batch` call both arrive internally and both succeed. Rails then builds the `actions.download.href` in the batch response from the **instance external URL** rather than from the host the request arrived on. `git-lfs` follows that absolute href to `https://gitlab.com`, so the object download leaves the runner over the public internet from its Cloud NAT address, and the group IP restriction denies it. Infrastructure has confirmed this is unrelated to the globally-allowed IP ranges and has asked for the href construction to be fixed here. See https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29616#note_4042396083 (internal). ## Steps to reproduce 1. On GitLab.com, enable **Settings > General > Permissions and group features > Restrict access by IP address** on a top-level group, listing only your own office or VPN ranges. 2. Add a repository in that group that tracks files with Git LFS. 3. Run a CI job on a GitLab.com hosted runner that checks out the repository with LFS smudge enabled (the default). ## Current behaviour The clone succeeds and the LFS lookup succeeds, then every object download is denied. GitLab Runner writes an `insteadOf` rule that routes Git traffic over the internal CI gateway: ``` git config --file /builds/<group>/<project>.tmp/.gitlab-runner.ext.conf --replace-all \ url.https://gitlab-ci-token:[MASKED]@us-east1.ci-gateway.int.gprd.gitlab.net:8989/<group>/<project>.git.insteadOf \ https://us-east1.ci-gateway.int.gprd.gitlab.net:8989/<group>/<project>.git ``` `git-lfs` honours that rule for the batch call, which reaches the gateway and returns `200`: ``` trace git-lfs: HTTP: POST https://gitlab-ci-token:[MASKED]@us-east1.ci-gateway.int.gprd.gitlab.net:8989/<group>/<project>.git/info/lfs/objects/batch > POST /<group>/<project>.git/info/lfs/objects/batch HTTP/1.1 > Host: us-east1.ci-gateway.int.gprd.gitlab.net:8989 > User-Agent: git-lfs/3.7.1 (GitHub; linux amd64; go 1.26.8) < HTTP/2.0 200 OK < Content-Type: application/vnd.git-lfs+json; charset=utf-8 < X-Request-Id: 01M52NCR21HVBSS59EB79S4B1J ``` The response body points the download back at the public host: ```json {"objects":[{"oid":"<oid>","size":13436619,"actions":{"download":{ "href":"https://gitlab.com/<group>/<project>.git/gitlab-lfs/objects/<oid>", "header":{"Authorization":"Basic [MASKED]"}}}}]} ``` `git-lfs` then requests that absolute URL. The request egresses from the runner VM's public Cloud NAT address, arrives through Cloudflare, and the group IP restriction denies it. `git-lfs` renders the denial as `Repository or object not found`. In the Rails logs for the affected job, the `objects/batch` request arrives with a `10.x` source address and returns `200`, while the `gitlab-lfs/objects/<oid>` requests arrive with public source addresses and are denied. ## Expected behaviour When the batch request arrives over the internal gateway, the download href should reference the same host, so the object transfer stays on the path that was already authorised. The documented contract is that group IP restrictions do not block hosted runners: > On GitLab.com, group IP address restrictions do not block hosted runners for GitLab.com. The hosted runner infrastructure is included in the GitLab.com globally-allowed IP address ranges, so hosted runners can run jobs for projects in your group without being added to your group's allowlist. [Group access and permissions: GitLab.com access restrictions](https://docs.gitlab.com/user/group/access_and_permissions/#gitlabcom-access-restrictions) Clones meet that contract. LFS object downloads do not. ## Root cause [`app/controllers/repositories/lfs_api_controller.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/repositories/lfs_api_controller.rb#L130) builds the proxied download href from `project.http_url_to_repo`, which is derived from the configured external URL and ignores the host the client actually used: ```ruby # Overridden in EE def proxy_download_actions_download_path(object) "#{project.http_url_to_repo}/gitlab-lfs/objects/#{object[:oid]}" end ``` The upload side has the same property through a different helper, so LFS pushes from the same context are expected to fail in the same way: ```ruby def upload_actions(object) { upload: { href: "#{upload_http_url_to_repo}/gitlab-lfs/objects/#{object[:oid]}/#{object[:size]}", header: upload_headers } } end # Overridden in EE def upload_http_url_to_repo Gitlab::RepositoryUrlBuilder.build(repository.full_path, protocol: :http) end ``` Which branch runs is decided in `download_actions`: ```ruby def download_actions(object, lfs_object) lfs_file = lfs_object.file if lfs_file.file_storage? || lfs_file.proxy_download_enabled? proxy_download_actions(object) else direct_download_actions(lfs_object) end end ``` GitLab.com keeps LFS objects in object storage with proxied download enabled, so the `proxy_download_actions` branch is the one that runs and the external-URL href is what clients receive. ## Why there is no workaround - The href comes from the server, so no client-side or project-side configuration changes it. `git-lfs` has no option to re-point an absolute href returned by the batch API. - Allowlisting the runner egress addresses is not viable. One affected group observed 19 distinct public egress addresses across two test pipelines, so covering the pool means allowlisting broad Google Cloud ranges, which defeats the control the group enabled. - `GIT_LFS_SKIP_SMUDGE` avoids the failure only for jobs that do not read LFS file contents. Affected groups are left choosing between the IP restriction and working CI. ## Proposed fix Either of these resolves it. The first is the narrower change: 1. **Build the href from the request host.** When the batch request arrives on a host that GitLab serves, derive the download and upload hrefs from that host instead of from the instance external URL, so the transfer continues on the same path. This mirrors how the clone already behaves and would fix every deployment that fronts Rails with an additional internal hostname, not only GitLab.com. ## Impact - **Affects:** GitLab.com groups with **Restrict access by IP address** enabled that use Git LFS and GitLab.com hosted runners. Self-managed and Dedicated deployments that place an internal hostname in front of Rails are affected by the same code path. - **Severity:** builds and deployments that need LFS file contents cannot run. Reported by an Ultimate customer at Severity 2 with all affected pipelines blocked. - **Not new:** this has been the behaviour since the CI gateway was introduced. It surfaces only when a group enables IP restrictions. - **Not fixed by the allowlist work:** `gitlab-com/gl-infra/production-engineering#30087` widened the globally-allowed ranges to cover internal traffic, which does not help here because the denied request arrives over the public path. The header-based exemption proposed in `gitlab-com/gl-infra/production-engineering#30056` would not cover it either, for the same reason. ## Related - `gitlab-com/gl-infra/production-engineering#29616` (internal) — infrastructure investigation, full trace, and the request to open this issue - [Git LFS development guidelines](https://docs.gitlab.com/development/lfs/) — documents that the client requests each object from the href returned by the batch call - [Group access and permissions: GitLab.com access restrictions](https://docs.gitlab.com/user/group/access_and_permissions/#gitlabcom-access-restrictions) - Zendesk ticket 760980 (internal), plus prior reports 674851, 610150 and 524859 covering the artifact equivalent of this failure