Git over SSH not working: Access denied for user git by PAM account configuration [preauth]
Summary
I am running a gitlab installation on a single-node Kubernetes machine via the gitlab-ce
Docker image.
Using git
over ssh
does not work. sshd
recognizes my key and accepts it, but the connection immediately closes afterwards. Logs in the container seem to indicate the git user is unable to login due to PAM.
Steps to reproduce
(Sanitized information is replaced with angle brackets)
Attempt to clone a repo from the instance over ssh
:
git clone ssh://git@<my-gitlab-url>:<port>/user/repo.git
Check the details section for my full config.
What is the current bug behavior?
git clone ssh://git@<my-gitlab-url>:2323/<my-user>/testing.git
Cloning into 'testing'...
Connection closed by <my-IP> port 2323
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
[Return code 128]
Relevant output of ssh -Tvvv -p 2323 git@<my-gitlab-url>
:
...
debug1: Next authentication method: publickey
debug1: Offering public key: xxxxxxxxxxxxxxxx
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: xxxxxxxxxxxxxxxx
debug3: sign_and_send_pubkey: xxxxxxxxxxxxxxxx
debug3: sign_and_send_pubkey: signing using xxxxxxxxxxxxxxx
debug3: send packet: type 50
Connection closed by <my-IP> port 2323
[Return code 255]
What is the expected correct behavior?
A successful clone. Or in the case of direct ssh
, something similar to what happens with gitlab.com (I don't know if the omnibus image does this as well):
ssh -T git@gitlab.com
Welcome to GitLab, @Renilux!
Relevant logs
Relevant logs
#Line in /var/log/gitlab/sshd/current that was written at the same time as the ssh attempt: 2020-12-27_22:51:30.37117 Access denied for user git by PAM account configuration [preauth]
Details of package version
Provide the package version installation details
ii gitlab-ce 13.7.1-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis)
Environment details
- Operating System: Debian 10 with backported kernel:
Linux <hostname> 5.9.0-0.bpo.2-amd64 #1 SMP Debian 5.9.6-1~bpo10+1 (2020-11-19) x86_64 GNU/Linux
- Installation Target, remove incorrect values:
- Kubernetes bare-metal bootstrapped with
kubeadm
- Kubernetes bare-metal bootstrapped with
- Installation Type, remove incorrect values:
- New Installation
- Is there any other software running on the machine: Not in the docker image. An nginx container is also running in my kubernetes cluster to provide SSL and reverse proxy to GitLab and any other future software I put on the cluster.
- Is this a single or multiple node installation? Single node
- Resources
- CPU limit:
4
- Memory total limit:
8Gi
- CPU limit:
More details:
- Kubernetes Information:
- K8s version: v1.20.1
- Container runtime: CRI-O v1.20.0
- Container networking: Cilium
- Cgroup driver: systemd
- Component Versions from admin area:
- GitLab 13.7.1 (c97c8073a0e)
- GitLab Shell 13.14.0
- GitLab Workhorse v8.58.0
- GitLab API v4
- Ruby 2.7.2p137
- Rails 6.0.3.3
- PostgreSQL 12.4
- Redis 5.0.9
Kubernetes Deployment YAML for gitlab-ce image
apiVersion: apps/v1 kind: Deployment metadata: name: gitlab namespace: gitlab spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: gitlab template: metadata: labels: app: gitlab spec: automountServiceAccountToken: false containers: - name: gitlab image: docker.io/gitlab/gitlab-ce:latest securityContext: capabilities: add: - SYS_CHROOT - MKNOD resources: limits: cpu: "4" memory: 8Gi ports: - name: http containerPort: 80 - name: registry containerPort: 5001 - name: ssh containerPort: 22 hostPort: 2323 volumeMounts: - name: gitlab mountPath: /var/opt/gitlab subPath: data - name: gitlab mountPath: /var/log/gitlab subPath: logs - name: gitlab mountPath: /etc/gitlab subPath: config terminationGracePeriodSeconds: 90 volumes: - name: gitlab hostPath: path: /srv/data/k8s/namespace/gitlab/gitlab type: DirectoryOrCreate
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://`my-domain`' letsencrypt['enable'] = false nginx['listen_port'] = 80 nginx['listen_https'] = falsegitlab_rails['gitlab_shell_ssh_port'] = 2323
registry_external_url 'https://
my-registry-domain
' gitlab_rails['registry_enabled'] = true registry['enable'] = trueregistry_nginx['enable'] = true registry_nginx['listen_port'] = 5001 registry_nginx['listen_https'] = false registry_nginx['proxy_set_headers'] = { "Host" => "$http_host", "X-Real-IP" => "$remote_addr", "X-Forwarded-For" => "$proxy_add_x_forwarded_for", "X-Forwarded-Proto" => "https", "X-Forwarded-Ssl" => "on" }
gitlab_rails['content_security_policy'] = { enabled: true, report_only: false, directives: { default_src: "'self'", script_src: "'self' 'unsafe-inline' 'unsafe-eval' https://www.recaptcha.net https://apis.google.com", frame_ancestor: "'self'", frame_src: "'self' https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com", img_src: "* data: blob:", style_src: "'self' 'unsafe-inline'" } }
Update: Workaround found
I have found a workaround that seems to be fully functional.
In the file /assets/sshd_config
, set UsePAM
to no
.
Then edit the git user account in /etc/shadow
. Enable the account by changing the password field from !
to *
.
This still seems like an issue because this particular workaround requires modifying files outside of the volume mount points described in the installation docs (i.e., not /etc/gitlab
, /var/log/gitlab
, or /var/opt/gitlab
)