Pushing of specific (previously lost) LFS file fails
Pushing a new file to lfs storage fails, although the web interface shows the file as "LFS" but GitLab web interface download link gives 404.
Steps to reproduce
$ cp model.caffemodel model2.caffemodel $ git add model2.caffemodel $ git commit -m "Adding model2" [add_models a483dd6] Adding model2 1 file changed, 3 insertions(+) create mode 100755 models/joint/caffe/model2.caffemodel ncgalopp@ncgalopp-mac05:~/tmp/cvdl_service/models/joint/caffe $ git push Locking support detected on remote "origin". Consider enabling it with: $ git config lfs.https://ncgalopp-nuc.jf.intel.com/TSD/cvdl_service.git/info/lfs.locksverify true Counting objects: 5, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (5/5), 387 bytes | 387.00 KiB/s, done. Total 5 (delta 2), reused 0 (delta 0) remote: remote: To create a merge request for add_models, visit: remote: http://ncgalopp-nuc.jf.intel.com:7070/tsd/cvdl_service/merge_requests/new?merge_request%5Bsource_branch%5D=add_models remote: To ncgalopp-nuc.jf.intel.com:TSD/cvdl_service beccedd..a483dd6 add_models -> add_models
It looks like the file was added from client side, as shown below.
$ git lfs ls-files 92d61b8634 - models/joint/caffe/deploy.prototxt 8a1b82c915 - models/joint/caffe/model.caffemodel 8a1b82c915 - models/joint/caffe/model2.caffemodel
However, on the server side,
/var/lfs-objects/8a is not present. On the other hand,
/var/lfs-objects/92/... is present from a previous push.
As a side note, the
model.caffemodel file had been added to the repository in the past, but was lost due to a server upgrade. Possibly there is a stale pointer somewhere that makes the server think that the object already exists in storage?
What is the current bug behavior?
Clean check out of the repository/branch fails due to server reporting missing lfs object:
ncgalopp@ncgalopp-mac05:~/tmp/cvdl_service $ git checkout add_models Downloading models/joint/caffe/model.caffemodel (10 MB) Error downloading object: models/joint/caffe/model.caffemodel (8a1b82c): Smudge error: Error downloading models/joint/caffe/model.caffemodel (8a1b82c9159d13070ece5ad3eabf51aa7fe66de59b9bec19dd6e631e76b10b19): LFS: Repository or object not found: http://ncgalopp-nuc.jf.intel.com:7070/tsd/cvdl_service.git/gitlab-lfs/objects/8a1b82c9159d13070ece5ad3eabf51aa7fe66de59b9bec19dd6e631e76b10b19 Check that it exists and that you have proper access to it Errors logged to /Users/ncgalopp/tmp/cvdl_service/.git/lfs/logs/20180713T151924.317920918.log Use `git lfs logs last` to view the log. error: external filter 'git-lfs filter-process' failed fatal: models/joint/caffe/model.caffemodel: smudge filter lfs failed
What is the expected correct behavior?
Clean checkout of the branch.
Output of checks
# gitlab-rake gitlab:lfs:check Checking integrity of LFS objects - 1..23: Failures: 7 Done!
Results of GitLab environment info
System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.4.4p296 Gem Version: 2.7.6 Bundler Version:1.16.2 Rake Version: 12.3.1 Redis Version: 3.2.11 Git Version: 2.17.1 Sidekiq Version:5.1.3 Go Version: unknown GitLab information Version: 11.0.3 Revision: aa62075 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://ncgalopp-nuc.jf.intel.com:7070 HTTP Clone URL: http://ncgalopp-nuc.jf.intel.com:7070/some-group/some-project.git SSH Clone URL: firstname.lastname@example.org:some-group/some-project.git Using LDAP: yes Using Omniauth: no GitLab Shell Version: 7.1.4 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git