Slow repository pull mirroring from the same GitLab instance when using LFS.
Summary
A customer recently reported that they are encountering slow repository mirroring in the same GitLab instance. They also noticed an increase in Sidekiq jobs (probably due to the slow mirroring).
According to them, they experienced the problem after upgrading to GitLab 12.8.8-ee.
Steps to reproduce
- Create a new repository (e.g.
LFS-repo
). - Initialize LFS, and add a large file.
- Setup a mirror repository on the same instance (e.g.
LFS-repo-mirror
). - Push another LFS object to
LFS-repo
.
Example Project
(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)
(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)
What is the current bug behavior?
Slow mirroring of repository (same GitLab instance) and increased number of Sidekiq jobs. Errors in application_json.log
What is the expected correct behavior?
Fast mirroring as the LFS already exists in the instance.
Relevant logs and/or screenshots
I managed to reproduce the error in my test instance. I managed to reproduce this error in 12.8.8-ee and 12.9.2-ee.
application_json.log
:
{
"severity": "ERROR",
"time": "2020-04-13T03:32:12.892Z",
"correlation_id": "LCRJPhviIt3",
"message": "LFS file with oid a108b502c6dab13005acbc1ca0c42122d12995000b6fbd73e879904366602bee could't be downloaded from https://jdasmarinas-geo-01.do.gitlap.com/root/lfs-test.git/gitlab-lfs/objects/a108b502c6dab13005acbc1ca0c42122d12995000b6fbd73e879904366602bee: Validation failed: Lfs object already exists in repository"
}
Sidekiq dump provided by the customer:
/opt/gitlab/embedded/lib/ruby/2.6.0/net/protocol.rb:217:in `wait_readable'
/opt/gitlab/embedded/lib/ruby/2.6.0/net/protocol.rb:217:in `rbuf_fill'
/opt/gitlab/embedded/lib/ruby/2.6.0/net/protocol.rb:191:in `readuntil'
/opt/gitlab/embedded/lib/ruby/2.6.0/net/protocol.rb:201:in `readline'
/opt/gitlab/embedded/lib/ruby/2.6.0/net/http/response.rb:40:in `read_status_line'
/opt/gitlab/embedded/lib/ruby/2.6.0/net/http/response.rb:29:in `read_new'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/aws-sdk-core-2.11.374/lib/seahorse/client/net_http/patches.rb:29:in `block in new_transport_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/aws-sdk-core-2.11.374/lib/seahorse/client/net_http/patches.rb:26:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/aws-sdk-core-2.11.374/lib/seahorse/client/net_http/patches.rb:26:in `new_transport_request'
/opt/gitlab/embedded/lib/ruby/2.6.0/net/http.rb:1479:in `request'
/opt/gitlab/embedded/lib/ruby/2.6.0/net/http.rb:1472:in `block in request'
/opt/gitlab/embedded/lib/ruby/2.6.0/net/http.rb:920:in `start'
/opt/gitlab/embedded/lib/ruby/2.6.0/net/http.rb:1470:in `request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/httparty-0.16.4/lib/httparty/request.rb:146:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/httparty-0.16.4/lib/httparty.rb:573:in `perform_request'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/http.rb:24:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/httparty-0.16.4/lib/httparty.rb:491:in `get'
/opt/gitlab/embedded/service/gitlab-rails/app/services/projects/lfs_pointers/lfs_download_service.rb:59:in `download_and_save_fil
Results of GitLab environment info
Expand for output related to GitLab environment info
System information
System: CentOS 7.4.1708
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.6.5p114
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.7
Git Version: 2.24.1
Sidekiq Version:5.2.7
Go Version: unknown
GitLab information
Version: 12.8.8-ee
Revision: 6319b8e640d
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 10.9
URL: https://gitlhostname
HTTP Clone URL: https://gitlhostname/some-group/some-project.git
SSH Clone URL: ssh://git@gitlhostname:12051/some-group/some-project.git
Elasticsearch: no
Geo: yes
Geo node: Primary
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 11.0.0
Repository storage paths:
- default: /gitlab/gitlab-repos/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 11.0.0 ? ... OK (11.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /gitlab/gitlab-rails sudo find /gitlab/gitlab-rails -type f -exec chmod 0644 {} ; sudo find /gitlab/gitlab-rails -type d -not -path /gitlab/gitlab-rails -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 19/1 ... yes 21/2 ... yes 42/3 ... yes 45/4 ... yes 22/5 ... yes ... 420/25816 ... yes 2563/25817 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1586796612 sudo mv /gitlab/gitlab-users/.ssh/id_rsa ~/gitlab-check-backup-1586796612 sudo mv /gitlab/gitlab-users/.ssh/id_rsa.pub ~/gitlab-check-backup-1586796612 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. Active users: ... 5752 Is authorized keys file accessible? ... skipped (authorized keys not enabled) Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking Geo ...
GitLab Geo is available ... yes GitLab Geo is enabled ... yes This machine's Geo node name matches a database record ... yes, found a primary node named "https://hostname/" HTTP/HTTPS repository cloning is enabled ... yes Machine clock is synchronized ... yes Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1586796612 sudo mv /gitlab/gitlab-users/.ssh/id_rsa ~/gitlab-check-backup-1586796612 sudo mv /gitlab/gitlab-users/.ssh/id_rsa.pub ~/gitlab-check-backup-1586796612 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. OpenSSH configured to use AuthorizedKeysCommand ... skipped Reason: Cannot access OpenSSH configuration file Try fixing it: This is expected if you are using SELinux. You may want to check configuration manually For more information see: doc/administration/operations/fast_ssh_key_lookup.md GitLab configured to disable writing to authorized_keys file ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking Geo ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
The customer managed to pinpoint this commit: 57ddfcc5
He tested adding return if LfsObject.exists?(oid: lfs_oid)
back and according to them, it fixed the issues they are encountering.
Customer information
- Zendesk link: https://gitlab.zendesk.com/agent/tickets/153172 (internal)
- Salesforce: https://gitlab.my.salesforce.com/00161000004bZPDAA2 (internal)