Gitaly logs missing socket ruby.0 since update to 13.9.1
(I'm not sure whether this is a Gitlab or Omnibus issue, I'll keep track of both.)
Since pulling 13.9.1 via gitlab-ce:latest
, Gitaly logs the following:
003cnil\u003e}. Err: connection error: desc = \"transport: Error while
dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.0: connect:
no such file or directory\". Reconnecting...","pid":257,"system":"system",
"time":"2021-03-02T09:04:38.642Z"}
Indeed there's no such socket:
root@gitlab:/# ls /var/opt/gitlab/gitaly/internal_sockets
internal.sock
A bit more context from the log of Gitaly, fun starts at line 19:
{"level":"info","msg":"Wrapper started","time":"2021-03-02T14:01:53.075Z","wrapper":251}
{"level":"info","msg":"finding gitaly","pid_file":"/var/opt/gitlab/gitaly/gitaly.pid","time":"2021-03-02T14:01:53.092Z","wrapper":251}
{"level":"info","msg":"spawning a process","time":"2021-03-02T14:01:53.111Z","wrapper":251}
{"gitaly":259,"level":"info","msg":"monitoring gitaly","time":"2021-03-02T14:01:53.112Z","wrapper":251}
time="2021-03-02T14:01:53Z" level=info msg="Starting GitalyversionGitaly, version 13.9.1"
time="2021-03-02T14:01:53Z" level=warning msg="git path not configured. Using default path resolution" resolvedPath=/opt/gitlab/embedded/bin/git
time="2021-03-02T14:01:53Z" level=info msg="clearing disk cache object folder" path=/var/opt/gitlab/git-data/repositories
time="2021-03-02T14:01:53Z" level=info msg="moving disk cache object folder to /var/opt/gitlab/git-data/repositories/+gitaly/tmp/diskcache237985151" path=/var/opt/gitlab/git-data/repositories
time="2021-03-02T14:01:53Z" level=info msg="disk cache object folder doesn't exist, no need to remove" path=/var/opt/gitlab/git-data/repositories
time="2021-03-02T14:01:53Z" level=info msg="Starting file walker for /var/opt/gitlab/git-data/repositories/+gitaly/cache" path=/var/opt/gitlab/git-data/repositories/+gitaly/cache
time="2021-03-02T14:01:53Z" level=info msg="Starting file walker for /var/opt/gitlab/git-data/repositories/+gitaly/state" path=/var/opt/gitlab/git-data/repositories/+gitaly/state
time="2021-03-02T14:01:53Z" level=info msg="cleared all cache object files in /var/opt/gitlab/git-data/repositories/+gitaly/tmp/diskcache237985151 after 90.216µs" path=/var/opt/gitlab/git-data/repositories
{"address":"/var/opt/gitlab/gitaly/gitaly.socket","level":"info","msg":"listening at unix address","time":"2021-03-02T14:01:54.099Z"}
{"level":"info","msg":"finished tempdir cleaner walk","storage":"default","time":"2021-03-02T14:01:54.099Z","time_ms":0}
{"address":"/var/opt/gitlab/gitaly/internal_sockets/internal.sock","level":"info","msg":"listening at unix address","time":"2021-03-02T14:01:54.099Z"}
{"address":"localhost:9236","level":"info","msg":"starting prometheus listener","time":"2021-03-02T14:01:54.103Z"}
{"level":"info","msg":"starting RSS monitor","supervisor.name":"gitaly-ruby.0","supervisor.rss_threshold":209715200,"time":"2021-03-02T14:01:54.103Z"}
{"level":"info","msg":"starting RSS monitor","supervisor.name":"gitaly-ruby.1","supervisor.rss_threshold":209715200,"time":"2021-03-02T14:01:54.103Z"}
{"level":"warning","msg":"grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.0 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.0: connect: no such file or directory\". Reconnecting...","pid":259,"system":"system","time":"2021-03-02T14:01:54.104Z"}
{"level":"warning","msg":"grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.1 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: no such file or directory\". Reconnecting...","pid":259,"system":"system","time":"2021-03-02T14:01:54.104Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","supervisor.pid":278,"time":"2021-03-02T14:01:54.104Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","supervisor.pid":279,"time":"2021-03-02T14:01:54.105Z"}
{"level":"info","msg":"PID 279 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:54.410Z"}
{"level":"info","msg":"PID 278 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-03-02T14:01:54.410Z"}
{"level":"info","msg":"/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:119:in `initialize': Permission denied @ rb_sysopen - /var/log/gitlab/gitaly/gitaly_ruby_json.log (Errno::EACCES)","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:56.181Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:119:in `open'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:56.181Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:119:in `build_server_interceptor_chain'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:56.181Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:41:in `main'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:56.181Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:149:in `\u003cmain\u003e'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:56.181Z"}
{"level":"info","msg":"I, [2021-03-02T14:01:56.175016 #279] INFO -- sentry: ** [Raven] Raven 3.0.4 configured not to capture errors: DSN not set","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:56.185Z"}
{"level":"info","msg":"/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:119:in `initialize': Permission denied @ rb_sysopen - /var/log/gitlab/gitaly/gitaly_ruby_json.log (Errno::EACCES)","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-03-02T14:01:56.187Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:119:in `open'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-03-02T14:01:56.187Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:119:in `build_server_interceptor_chain'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-03-02T14:01:56.187Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:41:in `main'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-03-02T14:01:56.187Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:149:in `\u003cmain\u003e'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-03-02T14:01:56.187Z"}
{"error":"exit status 1","level":"warning","msg":"exited","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:56.191Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","supervisor.pid":307,"time":"2021-03-02T14:01:56.191Z"}
{"level":"info","msg":"I, [2021-03-02T14:01:56.180784 #278] INFO -- sentry: ** [Raven] Raven 3.0.4 configured not to capture errors: DSN not set","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-03-02T14:01:56.192Z"}
{"error":"exit status 1","level":"warning","msg":"exited","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-03-02T14:01:56.198Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","supervisor.pid":309,"time":"2021-03-02T14:01:56.199Z"}
{"level":"info","msg":"PID 307 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:56.519Z"}
{"level":"info","msg":"PID 309 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-03-02T14:01:56.566Z"}
{"level":"info","msg":"/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:119:in `initialize': Permission denied @ rb_sysopen - /var/log/gitlab/gitaly/gitaly_ruby_json.log (Errno::EACCES)","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:57.933Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:119:in `open'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:57.933Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:119:in `build_server_interceptor_chain'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","259","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-03-02T14:01:57.933Z"}
I don't know where to go from here.