Git LFS batch builds the download href from the instance external URL, so LFS object fetches leave the internal CI gateway and fail under group IP restrictions

Summary

On GitLab.com, a CI job on a hosted runner can clone a repository in a group that has Restrict access by IP address enabled, but it cannot fetch that repository's Git LFS objects. The job fails with:

LFS: Repository or object not found: https://gitlab.com/<group>/<project>.git/gitlab-lfs/objects/<oid>
Check that it exists and that you have proper access to it

The cause is not the IP allowlist contents and not the runner fleet. GitLab Runner rewrites the repository URL onto an internal CI gateway, so the clone and the LFS objects/batch call both arrive internally and both succeed. Rails then builds the actions.download.href in the batch response from the instance external URL rather than from the host the request arrived on. git-lfs follows that absolute href to https://gitlab.com, so the object download leaves the runner over the public internet from its Cloud NAT address, and the group IP restriction denies it.

Infrastructure has confirmed this is unrelated to the globally-allowed IP ranges and has asked for the href construction to be fixed here. See gitlab-com/gl-infra/production-engineering#29616 (comment 4042396083) (internal).

Steps to reproduce

  1. On GitLab.com, enable Settings > General > Permissions and group features > Restrict access by IP address on a top-level group, listing only your own office or VPN ranges.
  2. Add a repository in that group that tracks files with Git LFS.
  3. Run a CI job on a GitLab.com hosted runner that checks out the repository with LFS smudge enabled (the default).

Current behaviour

The clone succeeds and the LFS lookup succeeds, then every object download is denied.

GitLab Runner writes an insteadOf rule that routes Git traffic over the internal CI gateway:

git config --file /builds/<group>/<project>.tmp/.gitlab-runner.ext.conf --replace-all \
  url.https://gitlab-ci-token:[MASKED]@us-east1.ci-gateway.int.gprd.gitlab.net:8989/<group>/<project>.git.insteadOf \
  https://us-east1.ci-gateway.int.gprd.gitlab.net:8989/<group>/<project>.git

git-lfs honours that rule for the batch call, which reaches the gateway and returns 200:

trace git-lfs: HTTP: POST https://gitlab-ci-token:[MASKED]@us-east1.ci-gateway.int.gprd.gitlab.net:8989/<group>/<project>.git/info/lfs/objects/batch
> POST /<group>/<project>.git/info/lfs/objects/batch HTTP/1.1
> Host: us-east1.ci-gateway.int.gprd.gitlab.net:8989
> User-Agent: git-lfs/3.7.1 (GitHub; linux amd64; go 1.26.8)

< HTTP/2.0 200 OK
< Content-Type: application/vnd.git-lfs+json; charset=utf-8
< X-Request-Id: 01M52NCR21HVBSS59EB79S4B1J

The response body points the download back at the public host:

{"objects":[{"oid":"<oid>","size":13436619,"actions":{"download":{
  "href":"https://gitlab.com/<group>/<project>.git/gitlab-lfs/objects/<oid>",
  "header":{"Authorization":"Basic [MASKED]"}}}}]}

git-lfs then requests that absolute URL. The request egresses from the runner VM's public Cloud NAT address, arrives through Cloudflare, and the group IP restriction denies it. git-lfs renders the denial as Repository or object not found.

In the Rails logs for the affected job, the objects/batch request arrives with a 10.x source address and returns 200, while the gitlab-lfs/objects/<oid> requests arrive with public source addresses and are denied.

Expected behaviour

When the batch request arrives over the internal gateway, the download href should reference the same host, so the object transfer stays on the path that was already authorised. The documented contract is that group IP restrictions do not block hosted runners:

On GitLab.com, group IP address restrictions do not block hosted runners for GitLab.com. The hosted runner infrastructure is included in the GitLab.com globally-allowed IP address ranges, so hosted runners can run jobs for projects in your group without being added to your group's allowlist.

Group access and permissions: GitLab.com access restrictions

Clones meet that contract. LFS object downloads do not.

Root cause

app/controllers/repositories/lfs_api_controller.rb builds the proxied download href from project.http_url_to_repo, which is derived from the configured external URL and ignores the host the client actually used:

# Overridden in EE
def proxy_download_actions_download_path(object)
  "#{project.http_url_to_repo}/gitlab-lfs/objects/#{object[:oid]}"
end

The upload side has the same property through a different helper, so LFS pushes from the same context are expected to fail in the same way:

def upload_actions(object)
  {
    upload: {
      href: "#{upload_http_url_to_repo}/gitlab-lfs/objects/#{object[:oid]}/#{object[:size]}",
      header: upload_headers
    }
  }
end

# Overridden in EE
def upload_http_url_to_repo
  Gitlab::RepositoryUrlBuilder.build(repository.full_path, protocol: :http)
end

Which branch runs is decided in download_actions:

def download_actions(object, lfs_object)
  lfs_file = lfs_object.file
  if lfs_file.file_storage? || lfs_file.proxy_download_enabled?
    proxy_download_actions(object)
  else
    direct_download_actions(lfs_object)
  end
end

GitLab.com keeps LFS objects in object storage with proxied download enabled, so the proxy_download_actions branch is the one that runs and the external-URL href is what clients receive.

Why there is no workaround

  • The href comes from the server, so no client-side or project-side configuration changes it. git-lfs has no option to re-point an absolute href returned by the batch API.
  • Allowlisting the runner egress addresses is not viable. One affected group observed 19 distinct public egress addresses across two test pipelines, so covering the pool means allowlisting broad Google Cloud ranges, which defeats the control the group enabled.
  • GIT_LFS_SKIP_SMUDGE avoids the failure only for jobs that do not read LFS file contents.

Affected groups are left choosing between the IP restriction and working CI.

Proposed fix

Either of these resolves it. The first is the narrower change:

  1. Build the href from the request host. When the batch request arrives on a host that GitLab serves, derive the download and upload hrefs from that host instead of from the instance external URL, so the transfer continues on the same path. This mirrors how the clone already behaves and would fix every deployment that fronts Rails with an additional internal hostname, not only GitLab.com.

Impact

  • Affects: GitLab.com groups with Restrict access by IP address enabled that use Git LFS and GitLab.com hosted runners. Self-managed and Dedicated deployments that place an internal hostname in front of Rails are affected by the same code path.
  • Severity: builds and deployments that need LFS file contents cannot run. Reported by an Ultimate customer at Severity 2 with all affected pipelines blocked.
  • Not new: this has been the behaviour since the CI gateway was introduced. It surfaces only when a group enables IP restrictions.
  • Not fixed by the allowlist work: gitlab-com/gl-infra/production-engineering#30087 widened the globally-allowed ranges to cover internal traffic, which does not help here because the denied request arrives over the public path. The header-based exemption proposed in gitlab-com/gl-infra/production-engineering#30056 would not cover it either, for the same reason.

Related

  • gitlab-com/gl-infra/production-engineering#29616 (internal) — infrastructure investigation, full trace, and the request to open this issue
  • Git LFS development guidelines — documents that the client requests each object from the href returned by the batch call
  • Group access and permissions: GitLab.com access restrictions
  • Zendesk ticket 760980 (internal), plus prior reports 674851, 610150 and 524859 covering the artifact equivalent of this failure
Edited by 🤖 GitLab Bot 🤖