Error 500 on API request with payloads containing LFS objects

Summary

Every time I try to commit (update or create) a file to the repository which is eligible to LFS filter in .gitattributes and has already been used in the other paths or branches of current project, Gitlab will return 500 Internal Server Error in response and following content will appear in the production.log file:

ActiveRecord::RecordInvalid (Validation failed: Lfs object already exists in project):
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/validations.rb:79:in `raise_record_invalid'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/validations.rb:43:in `save!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:291:in `block in save!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in `transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:220:in `transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:291:in `save!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/has_many_through_association.rb:117:in `save_through_record'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/has_many_through_association.rb:66:in `insert_record'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:574:in `block (2 levels) in concat_records'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:419:in `replace_on_target'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:414:in `add_to_target'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:573:in `block in concat_records'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:571:in `each'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:571:in `concat_records'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/has_many_association.rb:180:in `concat_records'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/has_many_through_association.rb:44:in `concat_records'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:179:in `block in concat'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:194:in `block in transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:220:in `transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:193:in `transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:179:in `concat'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/has_many_through_association.rb:38:in `concat'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_proxy.rb:970:in `<<'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/lfs/file_transformer.rb:63:in `link_lfs_object!'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/lfs/file_transformer.rb:29:in `new_file'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/files/multi_service.rb:18:in `block in actions_after_lfs_transformation'
...

This does not only happen to objects related to the current project but even if I have the same file committed to other project and a different branch than master, the same thing will happen as well. In this case, GET /projects/:id/repository/tree will confirm that there is no file in the given path, but still if I try to commit a new file in that path using POST /projects/:id/repository/commits API, I'll get error 500 and Lfs object already exists in project appears in the production.log.

Steps to reproduce

I'm not sure if these steps are reproducible, but this is what happens in my scenario:

  • create a new project
  • update .gitattributes by adding *.jpg filter=lfs diff=lfs merge=lfs -text to it
  • create a new branch
  • commit a .jpg file using POST /projects/:id/repository/commits to it
  • try to commit the same file either by updating previous one, adding it to a new path, or adding it to another project with the same LFS filter defined in .gitattributes.

What is the current bug behavior?

Returns 500 error

What is the expected correct behavior?

Even if the file already exists in the path, returning error 500 describes nothing to the API user. Showing a proper error message that the following file you are trying to commit exists in the following path already will suffice.

I'm not sure, but if LFS items are already namespaced based on the project, some of this problem should never happen at all.

Relevant logs and/or screenshots

Provided in the summary section.

Output of checks

This bug happens on self-hosted GitLab Community Edition 11.0.4

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.4.4p296
Gem Version:    2.7.6
Bundler Version:1.16.2
Rake Version:   12.3.1
Redis Version:  3.2.11
Git Version:    2.17.1
Sidekiq Version:5.1.3
Go Version:     unknown


GitLab information
Version:        11.0.4
Revision:       edb037c
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://git.zabanshenas.com
HTTP Clone URL: https://git.zabanshenas.com/some-group/some-project.git
SSH Clone URL:  git@git.zabanshenas.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no


GitLab Shell
Version:        7.1.4
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab Shell ...


GitLab Shell version >= 7.1.4 ? ... OK (7.1.4)

Repo base directory exists?

default... yes

Repo storage directories are symlinks?

default... no

Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
1/2 ... ok

1/3 ... ok

1/8 ... ok

1/9 ... ok

1/10 ... ok
2/12 ... ok
5/14 ... ok
2/20 ... ok
1/24 ... ok
2/25 ... ok
2/27 ... ok

1/29 ... ok

1/30 ... ok

1/31 ... ok

2/32 ... repository is empty
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/2 ... yes
1/3 ... yes
1/8 ... yes
1/9 ... yes
1/10 ... yes
2/12 ... yes
5/14 ... yes
2/20 ... yes
1/24 ... yes
2/25 ... yes
2/27 ... yes
1/29 ... yes
1/30 ... yes
1/31 ... yes
2/32 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.4.4)
Git version >= 2.9.5 ? ... yes (2.17.1)
Git user has default SSH configuration? ... yes
Active users: ... 7


Checking GitLab ... Finished
Edited by 🤖 GitLab Bot 🤖