Sidekiq 'Object not found - failed to find pack entry' error

Recently we started getting reports of failing PostReceive jobs on gitlab.com: new commits pushed by a user would not show up in the UI as 'Jane Doe pushed abc123'.

Looking in the sidekiq logs we find the following error, where 'abc123' is the SHA1 of the commit that was pushed.

WARN: Object not found - failed to find pack entry (abc123)

The error also happens during EmailsOnPushWorker jobs, only then it gets embedded in a JSON blob with retry information.

WARN: {"retry"=>true ... Object not found - failed to find pack entry ...

At the moment we see about 0.5% of PostReceive jobs on gitlab.com failing with this error. We suspect this is related to our use of NFS. We have confirmed with a customer who is operating a large NFS-backed GitLab instance that they also see this message in their logs sporadically. However, it may also be completely unrelated to NFS.

cc @vsizov