Integrity-checking on lfs push

Customer in https://gitlab.zendesk.com/agent/tickets/96017 reported the following error when attempting to clone a project:

Error downloading object: 
Project/path/path/path/lfs.file (99af4b9): Smudge error: Error downloading Project/path/path/path/lfs.file (99af4b902335975c9ce0f2e4327157f309110103d3ad904a8b3f7cd7b7ed9add): Expected OID 99af4b902335975c9ce0f2e4327157f309110103d3ad904a8b3f7cd7b7ed9add, got d4514124a94116c0deb560f92e1ed5d6a1c896792a9b2f5d78a567b9a83c28e8 after 1694368 bytes written

The project appeared to push to the server just fine, with the stub file pointing to OID 99af4b902335975c9ce0f2e4327157f309110103d3ad904a8b3f7cd7b7ed9add, and there was a file in the LFS store named 99af4b902335975c9ce0f2e4327157f309110103d3ad904a8b3f7cd7b7ed9add, but when we grabbed that file from the server and ran git lfs pointer --file=99af4b902335975c9ce0f2e4327157f309110103d3ad904a8b3f7cd7b7ed9add against it the OID came back as d4514124a94116c0deb560f92e1ed5d6a1c896792a9b2f5d78a567b9a83c28e8.

This result explains the clone error, but it doesn't explain how we ended up with an apparently corrupted or truncated file on the server.

There was one more abnormality we found in this case. We checked the GitLab LfsObject and LfsObjectsProject table entries for this lfs object:

irb(main):005:0> o=LfsObject.find_by(oid: "99af4b902335975c9ce0f2e4327157f309110103d3ad904a8b3f7cd7b7ed9add")
=> #<LfsObject id: 17696, oid: "99af4b902335975c9ce0f2e4327157f309110103d3ad904a8b...", size: 1694368, created_at: "2018-04-26 16:07:56", updated_at: "2018-04-26 16:07:56", file: "4b902335975c9ce0f2e4327157f309110103d3ad904a8b3f7c...", file_store: 1>
irb(main):006:0> p=Project.find(LfsObjectsProject.find(o.id).project_id)
=> #<Project id:1036 groupname/unexpected-repo>

The project associated with the lfs object was not the project that the file with this OID was uploaded against. The customer ran this command against both the expected repo and the unexpected repo

git log --all -p -S 99af4b902335975c9ce0f2e4327157f309110103d3ad904a8b3f7cd7b7ed9add

and we saw commits referencing this OID in the expected repo, but not in the unexpected repo.