"panic: runtime error: invalid memory address or nil pointer dereference" when clone repo via ssh
Summary
Cloning a git repo via ssh fails with the following error:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8fa379]
goroutine 1 [running]:
gitlab.com/gitlab-org/gitlab-shell/internal/logger.Configure(0xc0001a6280)
/var/cache/omnibus/src/gitlab-shell/internal/logger/logger.go:26 +0x129
main.main()
/var/cache/omnibus/src/gitlab-shell/cmd/gitlab-shell/main.go:48 +0x162
Steps to reproduce
On self-hosted gitlab with the latest version:
GitLab13.9.1-ee (8ae438629fa)
GitLab Shell 13.16.1
GitLab Workhorse v13.9.1
GitLab API v4
Ruby 2.7.2p137
Rails 6.0.3.4
PostgreSQL 12.5
Redis 6.0.10
Running in AWS, with the following OS:
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
Installed using the Official Linux package (recommended installation).
User attempts to clone a project via ssh and receives the error message:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8fa379]
goroutine 1 [running]:
gitlab.com/gitlab-org/gitlab-shell/internal/logger.Configure(0xc0001a6280)
/var/cache/omnibus/src/gitlab-shell/internal/logger/logger.go:26 +0x129
main.main()
/var/cache/omnibus/src/gitlab-shell/cmd/gitlab-shell/main.go:48 +0x162
Also reproduced on the gitlab host command line:
$ sudo -u git -- /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8fa379]
goroutine 1 [running]:
gitlab.com/gitlab-org/gitlab-shell/internal/logger.Configure(0xc0001bc280)
/var/cache/omnibus/src/gitlab-shell/internal/logger/logger.go:26 +0x129
main.main()
/var/cache/omnibus/src/gitlab-shell/cmd/gitlab-shell/main.go:48 +0x162
Example Project
What is the current bug behavior?
An attempt to clone a repo via ssh reports an error.
What is the expected correct behavior?
Users are able to clone a repo via ssh. This feature worked on the previous set of gitlab components:
GitLab13.8.0-ee (1ae10d09692)
GitLab Shell 13.15.0
GitLab Workhorse v8.59.0
GitLab API v4
Ruby 2.7.2p137
Rails 6.0.3.4
PostgreSQL 12.4
Redis 5.0.9
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
$ sudo gitlab-rake gitlab:env:info System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.3 Redis Version: 6.0.10 Git Version: 2.29.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.9.1-ee Revision: 8ae438629fa Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.5 URL: https://gitlab.targetrwe.com HTTP Clone URL: https://gitlab.targetrwe.com/some-group/some-project.git SSH Clone URL: git@gitlab.targetrwe.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers: gitlab GitLab Shell Version: 13.16.1 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git $ sudo cat /opt/gitlab/embedded/service/gitlab-shell/config.yml # This file is managed by gitlab-ctl. Manual changes will be # erased! To change the contents below, edit /etc/gitlab/gitlab.rb # and run `sudo gitlab-ctl reconfigure`. # GitLab user. git by default user: git # Url to gitlab instance. Used for api calls. May but need not end with a slash. gitlab_url: "http+unix://%2Fvar%2Fopt%2Fgitlab%2Fgitlab-workhorse%2Fsockets%2Fsocket" gitlab_relative_url_root: http_settings: # user: someone # password: somepass # ca_file: /etc/ssl/cert.pem # ca_path: /etc/pki/tls/certs # self_signed_cert: false # File used as authorized_keys for gitlab user auth_file: "/var/opt/gitlab/.ssh/authorized_keys" # SSL certificate dir where custom certificates can be placed # https://golang.org/pkg/crypto/x509/ ssl_cert_dir: "/opt/gitlab/embedded/ssl/certs/" # Log file. # Default is gitlab-shell.log in the root directory. log_file: "/var/log/gitlab/gitlab-shell/gitlab-shell.log" # Log level. INFO by default log_level: log_format: json # Audit usernames. # Set to true to see real usernames in the logs instead of key ids, which is easier to follow, but # incurs an extra API call on every gitlab-shell command. audit_usernames: migration: {"enabled":true,"features":[]} $ ls -l "/opt/gitlab/embedded/ssl/certs/" total 228 -rw-r--r-- 1 root root 225579 Feb 23 21:33 cacert.pem -rw-r--r--. 1 root root 147 Oct 1 23:39 README $ sudo ls -l "/var/opt/gitlab/.ssh/authorized_keys" -rw-------. 1 git git 3358 Mar 1 18:54 /var/opt/gitlab/.ssh/authorized_keys $ sudo ls -l "/var/log/gitlab/gitlab-shell/gitlab-shell.log" -rw-r--r-- 1 root root 0 Dec 15 19:13 /var/log/gitlab/gitlab-shell/gitlab-shell.log $ sudo ls -l /var/opt/gitlab/git-data/repositories total 0 drwxr-sr-x. 4 git root 30 Mar 2 14:18 +gitaly drwxr-s---. 25 git root 236 Feb 15 16:33 @hashed drwxr-sr-x 3 git git 21 Nov 17 14:52 targetrwe $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 3.8G 0 3.8G 0% /dev tmpfs 3.8G 12K 3.8G 1% /dev/shm tmpfs 3.8G 315M 3.5G 9% /run tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup /dev/nvme0n1p2 100G 32G 69G 32% / tmpfs 767M 0 767M 0% /run/user/1044
Results of GitLab application Check
Expand for output related to the GitLab application check
$ sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.16.1 ? ... OK (13.16.1) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8fa379]
goroutine 1 [running]: gitlab.com/gitlab-org/gitlab-shell/internal/logger.Configure(0xc0001c4280) /var/cache/omnibus/src/gitlab-shell/internal/logger/logger.go:26 +0x129 main.main() /var/cache/omnibus/src/gitlab-shell/cmd/check/main.go:33 +0x122 gitlab-shell self-check failed Try fixing it: Make sure GitLab is running; Check the gitlab-shell configuration file: sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking gitlab.incoming@targetrwe.com yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 36 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 12/5 ... yes 11/6 ... yes 44/8 ... yes 49/9 ... yes 49/10 ... yes 49/11 ... yes 49/12 ... yes 42/14 ... yes 42/15 ... yes 55/17 ... yes 47/18 ... yes 56/19 ... yes 56/20 ... yes 57/23 ... yes 47/24 ... yes 42/25 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.29.0 ? ... yes (2.29.0) Git user has default SSH configuration? ... yes Active users: ... 31 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished