CI job token LFS downloads return 404 when the job user lacks download_code

Summary

Scan execution policy jobs that run as the GitLab Security Policy Bot cannot fetch Git LFS objects. The LFS batch request succeeds, every object download after it 404s. Reported by a self-managed customer via support.

Steps to reproduce

1. Enable LFS on a private project and track a file with it.
2. Enforce a scan execution policy with a rule that runs a policy pipeline as the bot (for example a schedule rule) plus a SAST job.
3. Let the policy pipeline run.

Current behavior

git fetch works. The LFS batch request, authenticated with the job's CI job token, returns 200 with download actions. Every subsequent object GET returns 404 ("LFS: Repository or object not found" in git-lfs). git-lfs retries with backoff, so get_sources can run for around 2 hours before failing. Workhorse sees a high volume of LFS 404s during this.

Workaround: set GIT_LFS_SKIP_SMUDGE: "1" in the policy. Scanners then see LFS pointer files instead of file content.

Expected behavior

Object downloads succeed for the same batch that already returned 200 with download actions.

Root cause

Repositories::LfsApiController#lfs_auth_header (app/controllers/repositories/lfs_api_controller.rb) replaces the job token in each batch action's Authorization header with a user-scoped Gitlab::LfsToken, whenever the batch itself was authenticated with a CI job token.

The object request then authenticates as that user with read_write_project_authentication_abilities and is authorized via user_can_download_code? (can?(user, :download_code, project)) in app/controllers/concerns/lfs_request.rb, not build_can_download_code? (build_download_code), which is what authorized the batch.

The Security Policy Bot's permissions are restricted by an allowlist (config/authz/roles/security_policy_bot.yml, added around 19.0) that includes build_download_code but not download_code. The batch passes, the object download 404s.

Before that allowlist existed, the bot was a Guest member and got download_code on internal/public projects through the public_authenticated role, so LFS worked there. On private projects Guest never had download_code, so it likely never worked there.

Proposed fix

Stop swapping the job token for a user token on batch requests authenticated with a job token, so the object request is authorized by the same job token and the same build_* abilities as the batch. Proposed in !257736 (merged).