Error 500 on API request with payloads containing LFS objects
Summary
Every time I try to commit (update or create) a file to the repository which is eligible to LFS filter in .gitattributes
and has already been used in the other paths or branches of current project, Gitlab will return 500 Internal Server Error
in response and following content will appear in the production.log
file:
ActiveRecord::RecordInvalid (Validation failed: Lfs object already exists in project):
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/validations.rb:79:in `raise_record_invalid'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/validations.rb:43:in `save!'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:291:in `block in save!'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:220:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:291:in `save!'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/has_many_through_association.rb:117:in `save_through_record'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/has_many_through_association.rb:66:in `insert_record'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:574:in `block (2 levels) in concat_records'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:419:in `replace_on_target'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:414:in `add_to_target'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:573:in `block in concat_records'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:571:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:571:in `concat_records'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/has_many_association.rb:180:in `concat_records'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/has_many_through_association.rb:44:in `concat_records'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:179:in `block in concat'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:194:in `block in transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:220:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:193:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_association.rb:179:in `concat'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/has_many_through_association.rb:38:in `concat'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/associations/collection_proxy.rb:970:in `<<'
/opt/gitlab/embedded/service/gitlab-rails/app/services/lfs/file_transformer.rb:63:in `link_lfs_object!'
/opt/gitlab/embedded/service/gitlab-rails/app/services/lfs/file_transformer.rb:29:in `new_file'
/opt/gitlab/embedded/service/gitlab-rails/app/services/files/multi_service.rb:18:in `block in actions_after_lfs_transformation'
...
This does not only happen to objects related to the current project but even if I have the same file committed to other project and a different branch than master
, the same thing will happen as well. In this case, GET /projects/:id/repository/tree
will confirm that there is no file in the given path, but still if I try to commit a new file in that path using POST /projects/:id/repository/commits
API, I'll get error 500 and Lfs object already exists in project
appears in the production.log
.
Steps to reproduce
I'm not sure if these steps are reproducible, but this is what happens in my scenario:
- create a new project
- update
.gitattributes
by adding*.jpg filter=lfs diff=lfs merge=lfs -text
to it - create a new branch
- commit a
.jpg
file usingPOST /projects/:id/repository/commits
to it - try to commit the same file either by updating previous one, adding it to a new path, or adding it to another project with the same LFS filter defined in
.gitattributes
.
What is the current bug behavior?
Returns 500 error
What is the expected correct behavior?
Even if the file already exists in the path, returning error 500 describes nothing to the API user. Showing a proper error message that the following file you are trying to commit exists in the following path already will suffice.
I'm not sure, but if LFS
items are already namespaced based on the project, some of this problem should never happen at all.
Relevant logs and/or screenshots
Provided in the summary section.
Output of checks
This bug happens on self-hosted GitLab Community Edition 11.0.4
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.4.4p296 Gem Version: 2.7.6 Bundler Version:1.16.2 Rake Version: 12.3.1 Redis Version: 3.2.11 Git Version: 2.17.1 Sidekiq Version:5.1.3 Go Version: unknown
GitLab information Version: 11.0.4 Revision: edb037c Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://git.zabanshenas.com HTTP Clone URL: https://git.zabanshenas.com/some-group/some-project.git SSH Clone URL: git@git.zabanshenas.com:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 7.1.4 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 7.1.4 ? ... OK (7.1.4)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 1/2 ... ok
1/3 ... ok
1/8 ... ok
1/9 ... ok
1/10 ... ok 2/12 ... ok 5/14 ... ok 2/20 ... ok 1/24 ... ok 2/25 ... ok 2/27 ... ok
1/29 ... ok
1/30 ... ok
1/31 ... ok
2/32 ... repository is empty Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OKAccess to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/2 ... yes 1/3 ... yes 1/8 ... yes 1/9 ... yes 1/10 ... yes 2/12 ... yes 5/14 ... yes 2/20 ... yes 1/24 ... yes 2/25 ... yes 2/27 ... yes 1/29 ... yes 1/30 ... yes 1/31 ... yes 2/32 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.4.4) Git version >= 2.9.5 ? ... yes (2.17.1) Git user has default SSH configuration? ... yes Active users: ... 7
Checking GitLab ... Finished