Wiki not working on 15.0.3+ w/ aarch64 due to broken dependencies for nokogiri

Summary

Upgrading to 15.0.3+ breaks wikis seemingly as a result of 68f56153 bumping the version of nokogiri.

This impacts aarch64 - I originally saw this on Oracle Linux 8 but I have since deployed on Alma 8 (as I noted OL8 is not on the supported os list) and the issue persists.

Steps to reproduce

Deploy 15.0.3+ on an aarch64 environment and attempt to use the built in wiki on a project.

Example Project

What is the current bug behavior?

Presented with

Failed to retrieve page
The page could not be displayed because it timed out. You can view the source or clone the repository

when trying to browse a wiki

What is the expected correct behavior?

The rendered wiki.

Relevant logs and/or screenshots

This can be seen in the gitaly logs at start up:

italy/current:{"level":"info","msg":"  /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so) - /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:09:04.727Z"}
gitaly/current:{"level":"info","msg":"/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/extension.rb:7:in `require_relative': /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so) - /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so (LoadError)","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:09:04.727Z"}
gitaly/current:{"level":"info","msg":"  /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so) - /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:09:05.601Z"}
gitaly/current:{"level":"info","msg":"/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/extension.rb:7:in `require_relative': /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so) - /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so (LoadError)","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:09:05.601Z"}

Suggesting that the precompiled gem we use was linked against GLIBC_2.29 which Alma 8 does not ship with:

[root@xxxxxx gitlab]# rpm -qa | grep glibc
glibc-common-2.28-189.5.el8_6.aarch64
glibc-langpack-en-2.28-189.5.el8_6.aarch64
glibc-gconv-extra-2.28-189.5.el8_6.aarch64
glibc-2.28-189.5.el8_6.aarch64

This issue on the nokogiri github suggests we are not alone.

When trying to view a wiki page this can be seen in the gitaly logs

{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:40.260Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:41.261Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:41.261Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:42.261Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:43.582Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:44.382Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0 /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:44.402Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:46.045Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:46.045Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:47.046Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:47.046Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:48.047Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:48.047Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:49.048Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1 gitaly-ruby \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.1: connect: no such file or directory\". Reconnecting...","pid":358726,"system":"system","time":"2022-07-11T12:15:49.048Z"}
{"correlation_id":"01G7PJZHE354EBQ51F2JAV47M5","error":"rpc error: code = Canceled desc = failed to connect to gitaly-ruby worker: context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"server_stream","grpc.method":"WikiFindPage","grpc.request.deadline":"2022-07-11T12:15:50.009","grpc.request.fullMethod":"/gitaly.WikiService/WikiFindPage","grpc.request.glProjectPath":"infrastructure/wiki.wiki","grpc.request.glRepository":"wiki-8","grpc.request.payload_bytes":143,"grpc.request.repoPath":"@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.wiki.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":0,"grpc.service":"gitaly.WikiService","grpc.start_time":"2022-07-11T12:15:40.259","grpc.time_ms":9740.594,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":358726,"remote_ip":"xxx","span.kind":"server","system":"grpc","time":"2022-07-11T12:15:50.000Z","username":"root"}

More verbose logs of errors relating to GLIBC:

{"error":"exit status 1","level":"warning","msg":"exited","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:24.526Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","supervisor.pid":361775,"time":"2022-07-11T12:27:24.527Z"}
{"level":"info","msg":"PID 361775 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:24.750Z"}
{"level":"info","msg":"","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.388Z"}
{"level":"info","msg":"ERROR: It looks like you're trying to use Nokogiri as a precompiled native gem on a system with glibc \u003c 2.17:","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.388Z"}
{"level":"info","msg":"","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.388Z"}
{"level":"info","msg":"  /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so) - /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.388Z"}
{"level":"info","msg":"","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.388Z"}
{"level":"info","msg":"  If that's the case, then please install Nokogiri via the `ruby` platform gem:","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"      gem install nokogiri --platform=ruby","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"  or:","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"      bundle config set force_ruby_platform true","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"  Please visit https://nokogiri.org/tutorials/installing_nokogiri.html for more help.","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/extension.rb:7:in `require_relative': /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so) - /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/2.7/nokogiri.so (LoadError)","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri/extension.rb:7:in `\u003ctop (required)\u003e'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri.rb:10:in `require_relative'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/nokogiri-1.13.3-aarch64-linux/lib/nokogiri.rb:10:in `\u003ctop (required)\u003e'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sanitize-6.0.0/lib/sanitize.rb:3:in `require'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sanitize-6.0.0/lib/sanitize.rb:3:in `\u003ctop (required)\u003e'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-gollum-lib-4.2.7.10.gitlab.2/lib/gollum-lib.rb:22:in `require'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-gollum-lib-4.2.7.10.gitlab.2/lib/gollum-lib.rb:22:in `\u003ctop (required)\u003e'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/gollum.rb:4:in `require'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/gollum.rb:4:in `\u003ctop (required)\u003e'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git.rb:16:in `require_relative'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git.rb:16:in `\u003ctop (required)\u003e'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/lib/gitaly_server.rb:4:in `require_relative'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/lib/gitaly_server.rb:4:in `\u003ctop (required)\u003e'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:15:in `require_relative'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"level":"info","msg":"\tfrom /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby:15:in `\u003cmain\u003e'","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.389Z"}
{"error":"exit status 1","level":"warning","msg":"exited","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.392Z"}
{"level":"warning","msg":"opening circuit breaker","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","358726","/var/opt/gitlab/gitaly/run/gitaly-358726/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2022-07-11T12:27:25.392Z"}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info


# gitlab-rake gitlab:env:info

System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.4
Bundler Version:2.3.15
Rake Version:	13.0.6
Redis Version:	6.2.6
Sidekiq Version:6.4.0
Go Version:	unknown

GitLab information
Version:	15.0.3-ee
Revision:	b7e551ef451
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	13.6
URL:		https://xxxxx
HTTP Clone URL:	https://xxxxx/some-group/some-project.git
SSH Clone URL:	git@xxxxx:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.3.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

# gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.3.0 ? ... OK (14.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Checking Reply by email ...

IMAP server credentials are correct? ... Checking git@xxxxxxxx yes Mailroom enabled? ... skipped MailRoom running? ... skipped

Checking Reply by email ... Finished

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/1 ... yes 8/2 ... yes 9/3 ... yes 1/4 ... yes 1/5 ... yes 13/6 ... yes 1/7 ... yes 19/8 ... yes 22/11 ... yes 26/12 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.5) Git user has default SSH configuration? ... yes Active users: ... 3 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

Edited Jul 12, 2022 by Brian Williams