CE - Distributed Installation Web-Editor / IDE / Mergerequest not working (Gitaly problem?)
Summary
We have installed a distributed gitlab-ce installation. (Summarize the bug encountered concisely)
Gitaly and Gitlab are seperated instances / docker containers
Steps to reproduce
Open a file in the web editor, or ide and save it
Example Project
no idea, how to send you an example
What is the current bug behavior?
If i click on save in the web Editor or IDE, the save fails after some time (indicates a timeout?) telling me
14: Connect failed
or
14: Deadline exceeded
The same for creating Mergerequests
pushing via SSH or https is no problem
What is the expected correct behavior?
It should save the actual content of the file
Relevant logs and/or screenshots
2019-01-29_15:07:59.31935 time="2019-01-29T15:07:59Z" level=info msg="finished streaming call with code OK" correlation_id=WpJD1m0qPi2 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=TreeEntry grpc.request.deadline="2019-01-29T15:08:29Z" grpc.request.fullMethod=/gitaly.CommitService/TreeEntry grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:07:59Z" grpc.time_ms=2.911 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:07:59.32539 time="2019-01-29T15:07:59Z" level=info msg="finished unary call with code OK" correlation_id=WpJD1m0qPi2 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.deadline="2019-01-29T15:08:29Z" grpc.request.fullMethod=/gitaly.CommitService/FindCommit grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:07:59Z" grpc.time_ms=2.477 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:07:59.32962 time="2019-01-29T15:07:59Z" level=info msg="finished streaming call with code OK" correlation_id=WpJD1m0qPi2 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=GetTreeEntries grpc.request.deadline="2019-01-29T15:08:29Z" grpc.request.fullMethod=/gitaly.CommitService/GetTreeEntries grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:07:59Z" grpc.time_ms=2.429 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:07:59.33468 time="2019-01-29T15:07:59Z" level=info msg="finished streaming call with code OK" correlation_id=WpJD1m0qPi2 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=FindCommits grpc.request.deadline="2019-01-29T15:08:29Z" grpc.request.fullMethod=/gitaly.CommitService/FindCommits grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:07:59Z" grpc.time_ms=2.842 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:07:59.34968 time="2019-01-29T15:07:59Z" level=info msg="finished streaming call with code OK" correlation_id=WpJD1m0qPi2 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=FindCommits grpc.request.deadline="2019-01-29T15:08:29Z" grpc.request.fullMethod=/gitaly.CommitService/FindCommits grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:07:59Z" grpc.time_ms=2.726 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:08:54.35198 time="2019-01-29T15:08:54Z" level=info msg="finished streaming call with code Canceled" correlation_id=WpJD1m0qPi2 error="rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=UserCommitFiles grpc.request.deadline="2019-01-29T15:08:54Z" grpc.request.fullMethod=/gitaly.OperationService/UserCommitFiles grpc.service=gitaly.OperationService grpc.start_time="2019-01-29T15:07:59Z" grpc.time_ms=55000.49 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:09:37.02291 time="2019-01-29T15:09:37Z" level=info msg="#<Thread:0x000000000358e618@/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:82 run> terminated with exception (report_on_exception is true):" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02293 time="2019-01-29T15:09:37Z" level=info msg="/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:66:in `run_batch': call#run_batch failed somehow (GRPC::Core::CallError)" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02293 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:66:in `client_invoke'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02294 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:123:in `block in send_initial_metadata'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02295 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:120:in `synchronize'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02295 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:120:in `send_initial_metadata'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02296 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:208:in `send_status'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02296 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:198:in `send_status'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02297 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:134:in `rescue in run_server_method'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02298 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:117:in `run_server_method'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02299 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:399:in `block in loop_handle_server_calls'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02299 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:138:in `block in loop_execute_jobs'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02300 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:135:in `loop'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02300 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:135:in `loop_execute_jobs'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02301 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:84:in `block (2 levels) in start'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02301 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:83:in `catch'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:37.02302 time="2019-01-29T15:09:37Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:83:in `block in start'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:09:50.32960 time="2019-01-29T15:09:50Z" level=info msg="finished unary call with code OK" correlation_id=Miug4EHnGe7 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.deadline="2019-01-29T15:10:20Z" grpc.request.fullMethod=/gitaly.CommitService/FindCommit grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:09:50Z" grpc.time_ms=2.787 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:09:50.33441 time="2019-01-29T15:09:50Z" level=info msg="finished streaming call with code OK" correlation_id=Miug4EHnGe7 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=TreeEntry grpc.request.deadline="2019-01-29T15:10:20Z" grpc.request.fullMethod=/gitaly.CommitService/TreeEntry grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:09:50Z" grpc.time_ms=2.802 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:09:50.33964 time="2019-01-29T15:09:50Z" level=info msg="finished unary call with code OK" correlation_id=Miug4EHnGe7 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.deadline="2019-01-29T15:10:20Z" grpc.request.fullMethod=/gitaly.CommitService/FindCommit grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:09:50Z" grpc.time_ms=2.432 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:09:50.34356 time="2019-01-29T15:09:50Z" level=info msg="finished streaming call with code OK" correlation_id=Miug4EHnGe7 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=GetTreeEntries grpc.request.deadline="2019-01-29T15:10:20Z" grpc.request.fullMethod=/gitaly.CommitService/GetTreeEntries grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:09:50Z" grpc.time_ms=2.291 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:09:50.34832 time="2019-01-29T15:09:50Z" level=info msg="finished streaming call with code OK" correlation_id=Miug4EHnGe7 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=FindCommits grpc.request.deadline="2019-01-29T15:10:20Z" grpc.request.fullMethod=/gitaly.CommitService/FindCommits grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:09:50Z" grpc.time_ms=2.905 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:09:50.36148 time="2019-01-29T15:09:50Z" level=info msg="finished streaming call with code OK" correlation_id=Miug4EHnGe7 grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=FindCommits grpc.request.deadline="2019-01-29T15:10:20Z" grpc.request.fullMethod=/gitaly.CommitService/FindCommits grpc.request.glRepository=project-5 grpc.request.repoPath=markussiebert/test.git grpc.request.repoStorage=default grpc.request.topLevelGroup=markussiebert grpc.service=gitaly.CommitService grpc.start_time="2019-01-29T15:09:50Z" grpc.time_ms=2.713 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:10:45.36409 time="2019-01-29T15:10:45Z" level=info msg="finished streaming call with code Canceled" correlation_id=Miug4EHnGe7 error="rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=UserCommitFiles grpc.request.deadline="2019-01-29T15:10:45Z" grpc.request.fullMethod=/gitaly.OperationService/UserCommitFiles grpc.service=gitaly.OperationService grpc.start_time="2019-01-29T15:09:50Z" grpc.time_ms=55000.777 peer.address="10.104.80.28:60196" span.kind=server system=grpc
2019-01-29_15:11:15.12445 time="2019-01-29T15:11:15Z" level=info msg="#<Thread:0x00007fa444b3ae58@/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:82 run> terminated with exception (report_on_exception is true):" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12522 time="2019-01-29T15:11:15Z" level=info msg="/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:66:in `run_batch': call#run_batch failed somehow (GRPC::Core::CallError)" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12555 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:66:in `client_invoke'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12592 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:123:in `block in send_initial_metadata'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12640 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:120:in `synchronize'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12641 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:120:in `send_initial_metadata'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12641 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:208:in `send_status'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12642 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:198:in `send_status'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12643 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:134:in `rescue in run_server_method'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12643 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:117:in `run_server_method'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12731 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:399:in `block in loop_handle_server_calls'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12732 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:138:in `block in loop_execute_jobs'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12733 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:135:in `loop'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12734 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:135:in `loop_execute_jobs'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12734 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:84:in `block (2 levels) in start'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12735 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:83:in `catch'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
2019-01-29_15:11:15.12810 time="2019-01-29T15:11:15Z" level=info msg="\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_server.rb:83:in `block in start'" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 794 /tmp/gitaly-ruby708101922/socket.1]" supervisor.name=gitaly-ruby.1
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
Gitlab env info of the web/rails node:System: Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.3 Go Version: unknown GitLab information Version: 11.7.0 Revision: 1d9280e Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.abc.de HTTP Clone URL: https://gitlab.abc.de/some-group/some-project.git SSH Clone URL: git@gitlabssh.abc.de:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 8.4.4 Repository storage paths: - default: /persistent/gitlab/default/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git #### Results of GitLab application Check
Gitlab env info of the gitaly node:
System: Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.3 Go Version: unknown rake aborted! PG::ConnectionBad: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/info.rake:38:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Tasks: TOP => gitlab:env:info (See full trace by running task with --trace)
What do i need a database on gitaly?
Check for web node:
Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 8.4.4 ? ... OK (8.4.4) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/2 ... yes 2/5 ... yes 1/6 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 3 Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Check for gitaly node:
Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 8.4.4 ? ... OK (8.4.4) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... FAIL: 7:permission denied Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks. Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... Exception: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... Exception: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... Exception: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
I can see the red line Gitaly: ... default ... FAIL: 7:permission denied but have no idea how to fix this or its no documentation anywhere.