Skip to content

Gitaly 4:Deadline Exceeded when running gitlab-rake gitlab:git:fsck

Summary

On projects with many objects - e.g. a mirror of https://github.com/madisongh/linux-tegra the rake task https://github.com/madisongh/linux-tegra fails with - tommy/linux-tegra on default: 4:Deadline Exceeded

Steps to reproduce

Mirror project to a local GitLab install

Run sudo gitlab-rake gitlab:git:fsck ### Gives error ... linux-tegra on default: 4:Deadline Exceeded

Workarounds:

  • by increasing Gitaly Default Timeout Period to be 880 (in my case) instead of default 55
  • by running git fsck directly on your GitLab instance, Rake task will still fail but you can complete fsck operation.

Example Project

Use a mirror of https://github.com/madisongh/linux-tegra

What is the current bug behavior?

Rake task fails and aborts checking of that project plus any others still to ccomplete.

What is the expected correct behavior?

Rake task completes successfully.

Relevant logs and/or screenshots

Failure
# sudo gitlab-rake gitlab:git:fsck
...
- tommy/linux-tegra on default: 4:Deadline Exceeded

Logs
2018-10-30_00:22:01.25263 time="2018-10-30T00:22:01Z" level=debug msg="spawn complete" args="[/opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/
repositories/tommy/linux-tegra.git fsck]" command.exitCode=-1 command.inblock=0 command.maxrss=1677920 command.oublock=0 command.real_time_ms=55047.307085 com
mand.system_time_ms=548 command.user_time_ms=54092 grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=Fsck grpc.request.fullMethod=/gitaly
.RepositoryService/Fsck grpc.request.glRepository=project-208 grpc.request.repoPath=tommy/linux-tegra.git grpc.request.repoStorage=default grpc.request.topLev
elGroup=tommy grpc.service=gitaly.RepositoryService path=/opt/gitlab/embedded/bin/git peer.address=@ pid=27917 span.kind=server system=grpc

2018-10-30_00:22:01.25265 time="2018-10-30T00:22:01Z" level=info msg="finished unary call" grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab
-web grpc.method=Fsck grpc.request.fullMethod=/gitaly.RepositoryService/Fsck grpc.request.glRepository=project-208 grpc.request.repoPath=tommy/linux-tegra.git
 grpc.request.repoStorage=default grpc.request.topLevelGroup=tommy grpc.service=gitaly.RepositoryService grpc.time_ms=55047 peer.address=@ span.kind=server sy
stem=grpc

Manually running `git fsck`
/var/opt/gitlab/git-data/repositories/tommy/linux-tegra.git# time git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (4998859/4998859), done.
Checking connectivity: 4998874, done.

real    8m44.346s
user    8m38.384s
sys     0m2.608s

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

sudo gitlab-rake gitlab:env:info


System information
System:         Ubuntu 16.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.4.5p335
Gem Version:    2.7.6
Bundler Version:1.16.2
Rake Version:   12.3.1
Redis Version:  3.2.12
Git Version:    2.18.1
Sidekiq Version:5.2.1
Go Version:     unknown


GitLab information
Version:        11.4.3-ee
Revision:       dce6f33
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
DB Version:     9.6.8
URL:            https://astrachan-latest.asgks.net
HTTP Clone URL: https://astrachan-latest.asgks.net/some-group/some-project.git
SSH Clone URL:  git@astrachan-latest.asgks.net:some-group/some-project.git
Elasticsearch:  yes
Geo:            yes
Geo node:       Primary
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml, bitbucket


GitLab Shell
Version:        8.3.3
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 

sudo gitlab-rake gitlab:check SANITIZE=true


Checking GitLab Shell ...


GitLab Shell version >= 8.3.3 ? ... OK (8.3.3)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
1/13 ... ok
...
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/13 ... yes
...
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.4.5)
Git version >= 2.9.5 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 21
Elasticsearch version 5.1 - 5.5? ... no (5.6.8)
For more information see:
doc/integration/elasticsearch.md


Checking GitLab ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Edited by Alexandr Tanayno