LFS push silently succeeds but objects not stored when stale DB references exist

When GitLab's database contains LFS object records that reference non-existent files on disk, git lfs push reports 100% success but transfers no data. The batch API returns "already present" based on stale DB records, causing uploads to be skipped entirely.

Steps to reproduce:

  1. Have LFS objects in GitLab database referencing missing files on disk (possible causes: storage unmounted during boot/upgrade or manual file deletion we're not sure yet)
  2. Run git lfs push origin --all from client
  3. Push reports success: Uploading LFS objects: 100% (54/54), 18 MB | 0 B/s, done.
  4. Fresh clone fails: "Repository or object not found"

Key indicator: 0 B/s - no actual data transferred despite "success"

Expected behavior

Push should either:

  • Detect missing files and overwrite stale DB records
  • Fail with an error indicating objects are referenced but missing

Actual behavior

Push silently "succeeds" but objects aren't stored. Client receives "already exists" response from batch API.

Evidence

Upload appears successful but doesn't transfer data:

$ git lfs push origin main --all
Uploading LFS objects: 100% (54/54), 18 MB | 0 B/s, done.

Server file count after "successful" upload:

$ find /var/opt/gitlab/gitlab-rails/shared/lfs-objects/ -type f | wc -l
13

Only 13 files exist, but client reported uploading 54.

Download fails:

$ git clone git@gitlab.example.com:namespace/project.git
Cloning into 'project'...
remote: Enumerating objects: 5181, done.
remote: Counting objects: 100% (5181/5181), done.
remote: Compressing objects: 100% (1885/1885), done.
remote: Total 5181 (delta 2726), reused 4919 (delta 2643), pack-reused 0 (from 0)
Receiving objects: 100% (5181/5181), 886.06 KiB | 19.69 MiB/s, done.
Resolving deltas: 100% (2726/2726), done.
Downloading path/to/large-file.wav (1.1 MB)
Error downloading object: path/to/large-file.wav (ee309c6): Smudge error: Error downloading path/to/large-file.wav (ee309c66fdad08ae970dc47dcf5d654e89389857ce04e3418d28ed8dc25169c4):
LFS: Repository or object not found: https://gitlab.example.com/namespace/project.git/gitlab-lfs/objects/ee309c66fdad08ae970dc47dcf5d654e89389857ce04e3418d28ed8dc25169c4
Check that it exists and that you have proper access to it

Errors logged to '/home/user/project/.git/lfs/logs/20251216T143010.232000443.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: path/to/large-file.wav: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

Server-wide integrity check shows massive failures:

$ gitlab-rake gitlab:lfs:check
Checking integrity of LFS objects
- 1..220: Failures: 200
- 221..420: Failures: 200
...
Done!
1827 out of 1860 LFS objects failed across all projects.

Additional issue: Wrong URL path when object storage disabled

With lfs_object_store_enabled: false, batch API still returns object storage URLs:

Wrong: /gitlab-lfs/objects/OID (object storage path) Correct: /info/lfs/objects/OID (local storage path)

GitLab logs show routing to wrong controller:

{"method":"GET","path":"/namespace/project.git/gitlab-lfs/objects/724fc970...","controller":"Repositories::LfsStorageController","action":"download","status":404}

Workaround

Single project

  1. Clear stale DB references (gitlab-rails console) Project.find(ID).lfs_objects.delete_all
  2. Re-push from client git lfs push origin --all

For server-wide cleanup: 

# gitlab-rails console
Project.find_each do |project|
  count = project.lfs_objects.count
  if count > 0
    puts "Deleting #{count} LFS objects from #{project.full_path}"
    project.lfs_objects.delete_all
  end
end

Environment

  • GitLab version: Affects at least 18.4.2-ee through 18.6.2-ee (current)
  • Installation type: Omnibus
  • LFS configuration: lfs_object_store_enabled: false (local storage)
Edited by Vasilii Iakliushin