Pushing of specific (previously lost) LFS file fails

Summary

Pushing a new file to lfs storage fails, although the web interface shows the file as "LFS" but GitLab web interface download link gives 404.

Steps to reproduce

$ cp model.caffemodel model2.caffemodel
$ git add model2.caffemodel
$ git commit -m "Adding model2"
[add_models a483dd6] Adding model2
 1 file changed, 3 insertions(+)
 create mode 100755 models/joint/caffe/model2.caffemodel
ncgalopp@ncgalopp-mac05:~/tmp/cvdl_service/models/joint/caffe $ git push
Locking support detected on remote "origin". Consider enabling it with:
  $ git config lfs.https://ncgalopp-nuc.jf.intel.com/TSD/cvdl_service.git/info/lfs.locksverify true
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (5/5), 387 bytes | 387.00 KiB/s, done.
Total 5 (delta 2), reused 0 (delta 0)
remote:
remote: To create a merge request for add_models, visit:
remote:   http://ncgalopp-nuc.jf.intel.com:7070/tsd/cvdl_service/merge_requests/new?merge_request%5Bsource_branch%5D=add_models
remote:
To ncgalopp-nuc.jf.intel.com:TSD/cvdl_service
   beccedd..a483dd6  add_models -> add_models

It looks like the file was added from client side, as shown below.

$ git lfs ls-files
92d61b8634 - models/joint/caffe/deploy.prototxt
8a1b82c915 - models/joint/caffe/model.caffemodel
8a1b82c915 - models/joint/caffe/model2.caffemodel

However, on the server side, /var/lfs-objects/8a is not present. On the other hand, /var/lfs-objects/92/... is present from a previous push.

As a side note, the model.caffemodel file had been added to the repository in the past, but was lost due to a server upgrade. Possibly there is a stale pointer somewhere that makes the server think that the object already exists in storage?

What is the current bug behavior?

Clean check out of the repository/branch fails due to server reporting missing lfs object:

ncgalopp@ncgalopp-mac05:~/tmp/cvdl_service $ git checkout add_models
Downloading models/joint/caffe/model.caffemodel (10 MB)
Error downloading object: models/joint/caffe/model.caffemodel (8a1b82c): Smudge error: Error downloading models/joint/caffe/model.caffemodel (8a1b82c9159d13070ece5ad3eabf51aa7fe66de59b9bec19dd6e631e76b10b19): LFS: Repository or object not found: http://ncgalopp-nuc.jf.intel.com:7070/tsd/cvdl_service.git/gitlab-lfs/objects/8a1b82c9159d13070ece5ad3eabf51aa7fe66de59b9bec19dd6e631e76b10b19
Check that it exists and that you have proper access to it

Errors logged to /Users/ncgalopp/tmp/cvdl_service/.git/lfs/logs/20180713T151924.317920918.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: models/joint/caffe/model.caffemodel: smudge filter lfs failed

What is the expected correct behavior?

Clean checkout of the branch.

Output of checks

# gitlab-rake gitlab:lfs:check
Checking integrity of LFS objects
- 1..23: Failures: 7
Done!

Results of GitLab environment info

System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.4.4p296
Gem Version:    2.7.6
Bundler Version:1.16.2
Rake Version:   12.3.1
Redis Version:  3.2.11
Git Version:    2.17.1
Sidekiq Version:5.1.3
Go Version:     unknown

GitLab information
Version:        11.0.3
Revision:       aa62075
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            http://ncgalopp-nuc.jf.intel.com:7070
HTTP Clone URL: http://ncgalopp-nuc.jf.intel.com:7070/some-group/some-project.git
SSH Clone URL:  git@ncgalopp-nuc.jf.intel.com:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        7.1.4
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks
Git:            /opt/gitlab/embedded/bin/git