Error 503 accesing repository on Gitlab (Official Debian package)
Summary
I have problem viewing some pages on my gitlab environment, and also, doing push to my repository.
The problem appears loading these pages:
- Project -> Details.
- Repository -> Files.
- Repository -> Commits.
- Repository -> Branches.
- Repository -> Tags.
- Repository -> Contributos.
- Repository -> Graph (with Error 2).
- Repository -> Charts.
Work with gitlab:
- I've working with another person on a concrete repository. These repo are from gitlab 8.13. Also have a remote (on my local clone) from github, and I merge changes from github to my repo. I've do at least 3 of these merges (with more than 600 commits total) and pushed to gitlab, without problem.
- Lot of the merged commits are signed, my ones are signed too.
- The problem appears after one concrete commit. I've do a push to the repo of a simple change (a modification of .gitlab.ci.yml only), and, after it, my partner on the project do another one with some code changes. He didn't a pull before doing it, and generated a double commit. One for their changes, and another one "merge branch master...".
After push these two commits, the problem appears.
Repository has been working without any problem until this last commit. The commit are here:
commit 9827dcf92b7eccdac395697b650bc1ff0f3fadc7 (HEAD -> master, origin/master, origin/HEAD)
Merge: a6a31bdea f2a6999a9
Author: author2
Date: Fri Nov 30 15:04:56 2018 +0100
Merge branch 'master' of git.example.com:Group/Repository
commit a6a31bdea50006b4853b73e61c61bf971fc435b5
Author: author2
Date: Fri Nov 30 15:04:19 2018 +0100
Actualizados los plugins segun el repo
The commit referenced by "merge branch master" are:
commit f2a6999a928a68202634a53b9519661171863b31
Author: Er_Maqui <er_maqui@darkbolt.net>
Date: Fri Nov 30 14:26:29 2018 +0100
Relaxed CI Tests
The problem appears where I'm working on Group/Repository, but, now I cannot enter any repository page on my installation. I can enter on a project page who doesn't have any code yet. I didn't try to push to another repository. Pull / Clone are working correctly.
Steps to reproduce
I don't know how to reproduce, because I didn't know what's the problem. The problem appears when working normally with gitlab.
Example Project
I didn't have a example project now. If it's required, I can try to do one, but on my installation I cannot do it because I cannot enter on any project on it.
What is the current bug behavior?
I cannot access to the pages specified on this bug report. Also, I cannot do a push of my new code.
What is the expected correct behavior?
Gitlab should work correctly, allowing view of the pages and pushing code.
Relevant logs and/or screenshots
Error trying to load a project page
Gitlab::Git::CommandError (4:Deadline Exceeded):
lib/gitlab/git/repository.rb:868:in 'rescue in wrapped_gitaly_errors'
lib/gitlab/git/repository.rb:861:in 'wrapped_gitaly_errors'
app/models/repository.rb:482:in 'method_missing'
lib/gitlab/git/blob.rb:78:in 'batch_lfs_pointers'
lib/extracts_path.rb:147:in 'lfs_blob_ids'
lib/gitlab/i18n.rb:53:in 'with_locale'
lib/gitlab/i18n.rb:59:in 'with_user_locale'
app/controllers/application_controller.rb:417:in 'set_locale'
lib/gitlab/middleware/multipart.rb:101:in 'call'
lib/gitlab/request_profiler/middleware.rb:14:in 'call'
lib/gitlab/middleware/go.rb:17:in 'call'
lib/gitlab/etag_caching/middleware.rb:11:in 'call'
lib/gitlab/middleware/read_only/controller.rb:38:in 'call'
lib/gitlab/middleware/read_only.rb:16:in 'call'
lib/gitlab/middleware/basic_health_check.rb:25:in 'call'
lib/gitlab/request_context.rb:18:in 'call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in 'call'
lib/gitlab/middleware/release_env.rb:10:in 'call'
Completed 503 Service Unavailable in 30455ms (Views: 0.7ms | ActiveRecord: 2.5ms)
Error trying to load repository -> graph page
Gitlab::Git::CommandError (4:Deadline Exceeded):
lib/gitlab/git/repository.rb:868:in 'rescue in wrapped_gitaly_errors'
lib/gitlab/git/repository.rb:861:in 'wrapped_gitaly_errors'
lib/gitlab/git/repository.rb:177:in 'tags'
lib/gitlab/git/repository.rb:449:in 'refs_hash'
app/models/repository.rb:482:in 'method_missing'
lib/gitlab/git/commit.rb:392:in 'refs'
lib/gitlab/git/commit.rb:289:in 'ref_names'
app/models/network/commit.rb:17:in 'method_missing'
app/models/network/graph.rb:134:in 'include_ref?'
app/models/network/graph.rb:123:in 'block in commits_sort_by_ref'
app/models/network/graph.rb:122:in 'sort'
app/models/network/graph.rb:122:in 'commits_sort_by_ref'
app/models/network/graph.rb:66:in 'index_commits'
app/models/network/graph.rb:19:in 'initialize'
app/controllers/projects/network_controller.rb:25:in 'new'
app/controllers/projects/network_controller.rb:25:in 'block (2 levels) in show'
app/controllers/projects/network_controller.rb:17:in 'show'
lib/gitlab/i18n.rb:53:in 'with_locale'
lib/gitlab/i18n.rb:59:in 'with_user_locale'
app/controllers/application_controller.rb:417:in 'set_locale'
lib/gitlab/middleware/multipart.rb:101:in 'call'
lib/gitlab/request_profiler/middleware.rb:14:in 'call'
lib/gitlab/middleware/go.rb:17:in 'call'
lib/gitlab/etag_caching/middleware.rb:11:in 'call'
lib/gitlab/middleware/read_only/controller.rb:38:in 'call'
lib/gitlab/middleware/read_only.rb:16:in 'call'
lib/gitlab/middleware/basic_health_check.rb:25:in 'call'
lib/gitlab/request_context.rb:18:in 'call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in 'call'
lib/gitlab/middleware/release_env.rb:10:in 'call'
Completed 503 Service Unavailable in 30909ms (ActiveRecord: 2.8ms)
Error trying to do a git push on one client
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 974 bytes | 24.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: GitLab: API is not accessible
To git.example.net:Group/Repository.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'gitlab@git.example.net:Group/Repository.git'
Error trying to do a git push on one client (Server side)
GRPC::DeadlineExceeded (4:Deadline Exceeded):
/usr/lib/x86_64-linux-gnu/rubygems-integration/2.5.0/gems/grpc-1.16.1/src/ruby/lib/grpc/generic/active_call.rb:31:in 'check_status'
/usr/lib/x86_64-linux-gnu/rubygems-integration/2.5.0/gems/grpc-1.16.1/src/ruby/lib/grpc/generic/active_call.rb:181:in 'attach_status_results_and_complete_call'
/usr/lib/x86_64-linux-gnu/rubygems-integration/2.5.0/gems/grpc-1.16.1/src/ruby/lib/grpc/generic/active_call.rb:170:in 'receive_and_check_status'
/usr/lib/x86_64-linux-gnu/rubygems-integration/2.5.0/gems/grpc-1.16.1/src/ruby/lib/grpc/generic/active_call.rb:338:in 'each_remote_read_then_finish'
/usr/share/gitlab/lib/gitlab/gitaly_client/blob_service.rb:107:in 'each'
/usr/share/gitlab/lib/gitlab/gitaly_client/blob_service.rb:107:in 'flat_map'
/usr/share/gitlab/lib/gitlab/gitaly_client/blob_service.rb:107:in 'map_lfs_pointers'
/usr/share/gitlab/lib/gitlab/gitaly_client/blob_service.rb:90:in 'get_new_lfs_pointers'
/usr/share/gitlab/lib/gitlab/git/lfs_changes.rb:10:in 'new_pointers'
/usr/share/gitlab/lib/gitlab/checks/lfs_integrity.rb:14:in 'objects_missing?'
/usr/share/gitlab/lib/gitlab/checks/change_access.rb:197:in 'lfs_objects_exist_check'
/usr/share/gitlab/lib/gitlab/checks/change_access.rb:41:in 'exec'
/usr/share/gitlab/lib/gitlab/git_access.rb:277:in 'check_single_change_access'
/usr/share/gitlab/lib/gitlab/git_access.rb:265:in 'block in check_change_access!'
/usr/share/gitlab/lib/gitlab/git_access.rb:260:in 'each'
/usr/share/gitlab/lib/gitlab/git_access.rb:260:in 'with_index'
/usr/share/gitlab/lib/gitlab/git_access.rb:260:in 'check_change_access!'
/usr/share/gitlab/lib/gitlab/git_access.rb:250:in 'check_push_access!'
/usr/share/gitlab/lib/gitlab/git_access.rb:69:in 'check'
/usr/share/gitlab/lib/api/internal.rb:63:in 'block (2 levels) in <class:Internal>'
/usr/lib/ruby/vendor_ruby/grape/endpoint.rb:57:in 'call'
/usr/lib/ruby/vendor_ruby/grape/endpoint.rb:57:in 'block (2 levels) in generate_api_method'
/usr/lib/ruby/vendor_ruby/active_support/notifications.rb:166:in 'instrument'
/usr/lib/ruby/vendor_ruby/grape/endpoint.rb:56:in 'block in generate_api_method'
/usr/lib/ruby/vendor_ruby/grape/endpoint.rb:262:in 'block in run'
/usr/lib/ruby/vendor_ruby/active_support/notifications.rb:166:in 'instrument'
/usr/lib/ruby/vendor_ruby/grape/endpoint.rb:243:in 'run'
/usr/lib/ruby/vendor_ruby/grape/endpoint.rb:313:in 'block in build_stack'
/usr/lib/ruby/vendor_ruby/grape/middleware/base.rb:31:in 'call!'
/usr/lib/ruby/vendor_ruby/grape/middleware/base.rb:24:in 'call'
/usr/lib/ruby/vendor_ruby/grape/middleware/base.rb:31:in 'call!'
/usr/lib/ruby/vendor_ruby/grape/middleware/base.rb:24:in 'call'
/usr/share/rubygems-integration/all/gems/rack-oauth2-1.9.2/lib/rack/oauth2/server/resource.rb:20:in '_call'
/usr/share/rubygems-integration/all/gems/rack-oauth2-1.9.2/lib/rack/oauth2/server/resource/bearer.rb:8:in '_call'
/usr/share/rubygems-integration/all/gems/rack-oauth2-1.9.2/lib/rack/oauth2/server/abstract/handler.rb:17:in 'call'
/usr/lib/ruby/vendor_ruby/grape/middleware/error.rb:38:in 'block in call!'
/usr/lib/ruby/vendor_ruby/grape/middleware/error.rb:37:in 'catch'
/usr/lib/ruby/vendor_ruby/grape/middleware/error.rb:37:in 'call!'
/usr/lib/ruby/vendor_ruby/grape/middleware/base.rb:24:in 'call'
/usr/lib/ruby/vendor_ruby/grape_logging/middleware/request_logger.rb:60:in 'block in call!'
/usr/lib/ruby/vendor_ruby/grape_logging/middleware/request_logger.rb:58:in 'catch'
/usr/lib/ruby/vendor_ruby/grape_logging/middleware/request_logger.rb:58:in 'call!'
/usr/lib/ruby/vendor_ruby/grape/middleware/base.rb:24:in 'call'
/usr/lib/ruby/vendor_ruby/rack/head.rb:13:in 'call'
/usr/lib/ruby/vendor_ruby/grape/endpoint.rb:227:in 'call!'
/usr/lib/ruby/vendor_ruby/grape/endpoint.rb:221:in 'call'
/usr/lib/ruby/vendor_ruby/grape/router/route.rb:72:in 'exec'
/usr/lib/ruby/vendor_ruby/grape/router.rb:121:in 'process_route'
/usr/lib/ruby/vendor_ruby/grape/router.rb:74:in 'block in identity'
/usr/lib/ruby/vendor_ruby/grape/router.rb:93:in 'transaction'
/usr/lib/ruby/vendor_ruby/grape/router.rb:72:in 'identity'
/usr/lib/ruby/vendor_ruby/grape/router.rb:57:in 'block in call'
/usr/lib/ruby/vendor_ruby/grape/router.rb:137:in 'with_optimization'
/usr/lib/ruby/vendor_ruby/grape/router.rb:56:in 'call'
/usr/lib/ruby/vendor_ruby/grape/api.rb:119:in 'call'
/usr/lib/ruby/vendor_ruby/grape/api.rb:45:in 'call!'
/usr/lib/ruby/vendor_ruby/grape/api.rb:40:in 'call'
/usr/lib/ruby/vendor_ruby/action_dispatch/routing/mapper.rb:51:in 'serve'
/usr/lib/ruby/vendor_ruby/action_dispatch/journey/router.rb:43:in 'block in serve'
/usr/lib/ruby/vendor_ruby/action_dispatch/journey/router.rb:30:in 'each'
/usr/lib/ruby/vendor_ruby/action_dispatch/journey/router.rb:30:in 'serve'
/usr/lib/ruby/vendor_ruby/action_dispatch/routing/route_set.rb:817:in 'call'
/usr/share/gitlab/lib/gitlab/middleware/multipart.rb:101:in 'call'
/usr/share/gitlab/lib/gitlab/request_profiler/middleware.rb:14:in 'call'
/usr/share/gitlab/lib/gitlab/middleware/go.rb:17:in 'call'
/usr/share/gitlab/lib/gitlab/etag_caching/middleware.rb:11:in 'call'
/usr/lib/ruby/vendor_ruby/batch_loader/middleware.rb:11:in 'call'
/usr/lib/ruby/vendor_ruby/rack/attack.rb:107:in 'call'
/usr/lib/ruby/vendor_ruby/warden/manager.rb:35:in 'block in call'
/usr/lib/ruby/vendor_ruby/warden/manager.rb:34:in 'catch'
/usr/lib/ruby/vendor_ruby/warden/manager.rb:34:in 'call'
/usr/lib/ruby/vendor_ruby/rack/cors.rb:97:in 'call'
/usr/lib/ruby/vendor_ruby/rack/etag.rb:24:in 'call'
/usr/lib/ruby/vendor_ruby/rack/conditionalget.rb:38:in 'call'
/usr/lib/ruby/vendor_ruby/rack/head.rb:13:in 'call'
/usr/lib/ruby/vendor_ruby/action_dispatch/middleware/params_parser.rb:27:in 'call'
/usr/share/gitlab/lib/gitlab/middleware/read_only/controller.rb:38:in 'call'
/usr/share/gitlab/lib/gitlab/middleware/read_only.rb:16:in 'call'
/usr/lib/ruby/vendor_ruby/action_dispatch/middleware/flash.rb:260:in 'call'
/usr/lib/ruby/vendor_ruby/rack/session/abstract/id.rb:225:in 'context'
/usr/lib/ruby/vendor_ruby/rack/session/abstract/id.rb:220:in 'call'
/usr/lib/ruby/vendor_ruby/action_dispatch/middleware/cookies.rb:560:in 'call'
/usr/lib/ruby/vendor_ruby/active_record/query_cache.rb:36:in 'call'
/usr/lib/ruby/vendor_ruby/active_record/connection_adapters/abstract/connection_pool.rb:653:in 'call'
/usr/lib/ruby/vendor_ruby/action_dispatch/middleware/callbacks.rb:29:in 'block in call'
/usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:88:in '__run_callbacks__'
/usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:778:in '_run_call_callbacks'
/usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:81:in 'run_callbacks'
/usr/lib/ruby/vendor_ruby/action_dispatch/middleware/callbacks.rb:27:in 'call'
/usr/lib/ruby/vendor_ruby/action_dispatch/middleware/remote_ip.rb:78:in 'call'
/usr/lib/ruby/vendor_ruby/action_dispatch/middleware/debug_exceptions.rb:17:in 'call'
/usr/lib/ruby/vendor_ruby/action_dispatch/middleware/show_exceptions.rb:30:in 'call'
/usr/share/gitlab/lib/gitlab/middleware/basic_health_check.rb:25:in 'call'
/usr/lib/ruby/vendor_ruby/rails/rack/logger.rb:38:in 'call_app'
/usr/lib/ruby/vendor_ruby/rails/rack/logger.rb:20:in 'block in call'
/usr/lib/ruby/vendor_ruby/active_support/tagged_logging.rb:68:in 'block in tagged'
/usr/lib/ruby/vendor_ruby/active_support/tagged_logging.rb:26:in 'tagged'
/usr/lib/ruby/vendor_ruby/active_support/tagged_logging.rb:68:in 'tagged'
/usr/lib/ruby/vendor_ruby/rails/rack/logger.rb:20:in 'call'
/usr/share/gitlab/lib/gitlab/request_context.rb:18:in 'call'
/usr/lib/ruby/vendor_ruby/request_store/middleware.rb:9:in 'call'
/usr/lib/ruby/vendor_ruby/action_dispatch/middleware/request_id.rb:21:in 'call'
/usr/lib/ruby/vendor_ruby/rack/methodoverride.rb:22:in 'call'
/usr/lib/ruby/vendor_ruby/rack/runtime.rb:18:in 'call'
/usr/lib/ruby/vendor_ruby/rack/lock.rb:17:in 'call'
/usr/lib/ruby/vendor_ruby/rack/sendfile.rb:113:in 'call'
/usr/share/gitlab/lib/gitlab/metrics/requests_rack_middleware.rb:27:in 'call'
/usr/lib/ruby/vendor_ruby/raven/integrations/rack.rb:51:in 'call'
/usr/lib/ruby/vendor_ruby/rails/engine.rb:518:in 'call'
/usr/lib/ruby/vendor_ruby/rails/application.rb:165:in 'call'
/usr/lib/ruby/vendor_ruby/rails/railtie.rb:194:in 'public_send'
/usr/lib/ruby/vendor_ruby/rails/railtie.rb:194:in 'method_missing'
/usr/share/gitlab/lib/gitlab/middleware/release_env.rb:10:in 'call'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:66:in 'block in call'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:50:in 'each'
/usr/lib/ruby/vendor_ruby/rack/urlmap.rb:50:in 'call'
/usr/lib/ruby/vendor_ruby/unicorn/http_server.rb:606:in 'process_client'
/usr/lib/ruby/vendor_ruby/unicorn/worker_killer.rb:52:in 'process_client'
/usr/lib/ruby/vendor_ruby/unicorn/http_server.rb:701:in 'worker_loop'
/usr/lib/ruby/vendor_ruby/unicorn/http_server.rb:549:in 'spawn_missing_workers'
/usr/lib/ruby/vendor_ruby/unicorn/http_server.rb:142:in 'start'
/usr/bin/unicorn_rails:209:in '<main>'
Output of checks
Expand for output related to GitLab:git:fsck
Check if Gitlab is configured correctly... Performed integrity check for Project/Repo [...] Same message repeated for all repositories The following repositories reported errors:
-
- SomeGroup/SomeRepo on default: 4:Deadline Exceeded
-
- Group1/Repo1 on default: 4:Deadline Exceeded <-- This is the repository where i'm working when gitlab fail appears.
Expand for output related to GitLab:artifacts:check
Check if Gitlab is configured correctly... Checking integrity of Job artifacts Done!
Expand for output related to GitLab:lfs:check
Check if Gitlab is configured correctly... Checking integrity of LFS objects Done!
Expand for output related to GitLab:uploads:check
Check if Gitlab is configured correctly... Checking integrity of Uploads
-
- 1..33: Failures: 0 Done!
Results of GitLab environment info
Expand for output related to GitLab environment info
Check if Gitlab is configured correctly...
System information System: Debian unstable Current User: gitlab Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.1 Rake Version: 12.3.1 Redis Version: 5.0.2 Git Version: 2.20.0. Sidekiq Version:5.2.2 Go Version: unknown
GitLab information Version: 11.4.9 Revision: Unknown Directory: /usr/share/gitlab DB Adapter: postgresql URL: https://localhost HTTP Clone URL: https://localhost/some-group/some-project.git SSH Clone URL: gitlab@localhost:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 8.3.3 Repository storage paths:
-
- default: /home/gitlab/repositories Hooks: /usr/share/gitlab-shell/hooks Git: /usr/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Check if Gitlab is configured correctly... Checking GitLab Shell ...
GitLab Shell version >= 8.3.3 ? ... OK (8.3.3) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by gitlab:root, or gitlab:gitlab? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... Project / Repo ... ok [...] Same message repeated for all repositories, all OK except empty repositories Running /usr/share/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /usr/share/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... yes Init script up-to-date? ... yes Projects have namespace: ... Project / Repo ... yes [...] Same message repeated for all repositories, all yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.5.3) Git version >= 2.9.5 ? ... yes (2.20.0) Git user has default SSH configuration? ... yes Active users: ... 8
Checking GitLab ... Finished
Additional information
Installation:
- The installation itself is a database migrated from debian gitlab package 8.13 directly to 11.1 if I remember correctly.
- I didn't use the installation until 11.3, because I've waiting for some information about another debian bugs (#914159, #914496, and others).
- The installation are a usual one, but I've moved /var/lib/gitlab to /home/gitlab with a link. The references on config files of gitlab, gitlab-shell and gitaly to repositories are pointing to correct folder (/home/gitlab/repositories). That's because I didn't have enough space free on /.
- Gitlab check didn't show any errors. Installation has been working without problem for some time, doing lot of commits (working only on one project).
This bug are filled originally on Debian Bug Tracking System (BTS). Here's the link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=915136.
The bug has been detected on 11.3.10 version, 11.3.11 and 11.4.9 (All of them are gitlab packaged for debian by @pravi, downloaded from unstable debian repository).