Gitaly 4:Deadline Exceeded when running gitlab-rake gitlab:git:fsck
Summary
On projects with many objects - e.g. a mirror of https://github.com/madisongh/linux-tegra the rake task https://github.com/madisongh/linux-tegra
fails with - tommy/linux-tegra on default: 4:Deadline Exceeded
Steps to reproduce
Mirror project to a local GitLab install
Run sudo gitlab-rake gitlab:git:fsck
### Gives error ... linux-tegra on default: 4:Deadline Exceeded
Workarounds:
- by increasing Gitaly Default Timeout Period to be 880 (in my case) instead of default 55
- by running
git fsck
directly on your GitLab instance, Rake task will still fail but you can complete fsck operation.
Example Project
Use a mirror of https://github.com/madisongh/linux-tegra
What is the current bug behavior?
Rake task fails and aborts checking of that project plus any others still to ccomplete.
What is the expected correct behavior?
Rake task completes successfully.
Relevant logs and/or screenshots
Failure
# sudo gitlab-rake gitlab:git:fsck
...
- tommy/linux-tegra on default: 4:Deadline Exceeded
Logs
2018-10-30_00:22:01.25263 time="2018-10-30T00:22:01Z" level=debug msg="spawn complete" args="[/opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/
repositories/tommy/linux-tegra.git fsck]" command.exitCode=-1 command.inblock=0 command.maxrss=1677920 command.oublock=0 command.real_time_ms=55047.307085 com
mand.system_time_ms=548 command.user_time_ms=54092 grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=Fsck grpc.request.fullMethod=/gitaly
.RepositoryService/Fsck grpc.request.glRepository=project-208 grpc.request.repoPath=tommy/linux-tegra.git grpc.request.repoStorage=default grpc.request.topLev
elGroup=tommy grpc.service=gitaly.RepositoryService path=/opt/gitlab/embedded/bin/git peer.address=@ pid=27917 span.kind=server system=grpc
2018-10-30_00:22:01.25265 time="2018-10-30T00:22:01Z" level=info msg="finished unary call" grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab
-web grpc.method=Fsck grpc.request.fullMethod=/gitaly.RepositoryService/Fsck grpc.request.glRepository=project-208 grpc.request.repoPath=tommy/linux-tegra.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=tommy grpc.service=gitaly.RepositoryService grpc.time_ms=55047 peer.address=@ span.kind=server sy
stem=grpc
Manually running `git fsck`
/var/opt/gitlab/git-data/repositories/tommy/linux-tegra.git# time git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (4998859/4998859), done.
Checking connectivity: 4998874, done.
real 8m44.346s
user 8m38.384s
sys 0m2.608s
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
sudo gitlab-rake gitlab:env:info
System information System: Ubuntu 16.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.4.5p335 Gem Version: 2.7.6 Bundler Version:1.16.2 Rake Version: 12.3.1 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.1 Go Version: unknown
GitLab information Version: 11.4.3-ee Revision: dce6f33 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.8 URL: https://astrachan-latest.asgks.net HTTP Clone URL: https://astrachan-latest.asgks.net/some-group/some-project.git SSH Clone URL: git@astrachan-latest.asgks.net:some-group/some-project.git Elasticsearch: yes Geo: yes Geo node: Primary Using LDAP: no Using Omniauth: yes Omniauth Providers: saml, bitbucket
GitLab Shell Version: 8.3.3 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...
GitLab Shell version >= 8.3.3 ? ... OK (8.3.3) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 1/13 ... ok ... Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/13 ... yes ... Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.4.5) Git version >= 2.9.5 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 21 Elasticsearch version 5.1 - 5.5? ... no (5.6.8) For more information see: doc/integration/elasticsearch.md
Checking GitLab ... Finished
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)