Git over SSH not working: Access denied for user git by PAM account configuration [preauth]

Summary

I am running a gitlab installation on a single-node Kubernetes machine via the gitlab-ce Docker image.

Using git over ssh does not work. sshd recognizes my key and accepts it, but the connection immediately closes afterwards. Logs in the container seem to indicate the git user is unable to login due to PAM.

Steps to reproduce

(Sanitized information is replaced with angle brackets)

Attempt to clone a repo from the instance over ssh:

git clone ssh://git@<my-gitlab-url>:<port>/user/repo.git

Check the details section for my full config.

What is the current bug behavior?

git clone ssh://git@<my-gitlab-url>:2323/<my-user>/testing.git
Cloning into 'testing'...
Connection closed by <my-IP> port 2323
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[Return code 128]

Relevant output of ssh -Tvvv -p 2323 git@<my-gitlab-url>:

...
debug1: Next authentication method: publickey
debug1: Offering public key: xxxxxxxxxxxxxxxx
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: xxxxxxxxxxxxxxxx
debug3: sign_and_send_pubkey: xxxxxxxxxxxxxxxx
debug3: sign_and_send_pubkey: signing using xxxxxxxxxxxxxxx
debug3: send packet: type 50
Connection closed by <my-IP> port 2323
[Return code 255]

What is the expected correct behavior?

A successful clone. Or in the case of direct ssh, something similar to what happens with gitlab.com (I don't know if the omnibus image does this as well):

ssh -T git@gitlab.com
Welcome to GitLab, @Renilux!

Relevant logs

#Line in /var/log/gitlab/sshd/current that was written at the same time as the ssh attempt:
2020-12-27_22:51:30.37117 Access denied for user git by PAM account configuration [preauth]

Details of package version

Provide the package version installation details

ii  gitlab-ce                                              13.7.1-ce.0                      amd64                            GitLab Community Edition (including NGINX, Postgres, Redis)

Environment details

Operating System: Debian 10 with backported kernel: Linux <hostname> 5.9.0-0.bpo.2-amd64 #1 SMP Debian 5.9.6-1~bpo10+1 (2020-11-19) x86_64 GNU/Linux
Installation Target, remove incorrect values:
- Kubernetes bare-metal bootstrapped with kubeadm
Installation Type, remove incorrect values:
- New Installation
Is there any other software running on the machine: Not in the docker image. An nginx container is also running in my kubernetes cluster to provide SSL and reverse proxy to GitLab and any other future software I put on the cluster.
Is this a single or multiple node installation? Single node
Resources
- CPU limit: 4
- Memory total limit: 8Gi

More details:

Kubernetes Information:
- K8s version: v1.20.1
- Container runtime: CRI-O v1.20.0
- Container networking: Cilium
- Cgroup driver: systemd
Component Versions from admin area:
- GitLab 13.7.1 (c97c8073a0e)
- GitLab Shell 13.14.0
- GitLab Workhorse v8.58.0
- GitLab API v4
- Ruby 2.7.2p137
- Rails 6.0.3.3
- PostgreSQL 12.4
- Redis 5.0.9

Kubernetes Deployment YAML for gitlab-ce image

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab
  namespace: gitlab
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: gitlab
  template:
    metadata:
      labels:
        app: gitlab
    spec:
      automountServiceAccountToken: false
      containers:
        - name: gitlab
          image: docker.io/gitlab/gitlab-ce:latest
          securityContext:
            capabilities:
              add:
                - SYS_CHROOT
                - MKNOD
          resources:
            limits:
              cpu: "4"
              memory: 8Gi
          ports:
            - name: http
              containerPort: 80
            - name: registry
              containerPort: 5001
            - name: ssh
              containerPort: 22
              hostPort: 2323
          volumeMounts:
            - name: gitlab
              mountPath: /var/opt/gitlab
              subPath: data
            - name: gitlab
              mountPath: /var/log/gitlab
              subPath: logs
            - name: gitlab
              mountPath: /etc/gitlab
              subPath: config
      terminationGracePeriodSeconds: 90
      volumes:
        - name: gitlab
          hostPath:
            path: /srv/data/k8s/namespace/gitlab/gitlab
            type: DirectoryOrCreate

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

external_url 'https://`my-domain`'
letsencrypt['enable'] = false
nginx['listen_port'] = 80
nginx['listen_https'] = false
gitlab_rails['gitlab_shell_ssh_port'] = 2323
registry_external_url 'https://my-registry-domain'
gitlab_rails['registry_enabled'] = true
registry['enable'] = true
registry_nginx['enable'] = true
registry_nginx['listen_port'] = 5001
registry_nginx['listen_https'] = false
registry_nginx['proxy_set_headers'] = {
"Host" => "$http_host",
"X-Real-IP" => "$remote_addr",
"X-Forwarded-For" => "$proxy_add_x_forwarded_for",
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
}
gitlab_rails['content_security_policy'] = {
enabled: true,
report_only: false,
directives: {
default_src: "'self'",
script_src: "'self' 'unsafe-inline' 'unsafe-eval' https://www.recaptcha.net https://apis.google.com",
frame_ancestor: "'self'",
frame_src: "'self' https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com",
img_src: "* data: blob:",
style_src: "'self' 'unsafe-inline'"
}
}

Update: Workaround found

I have found a workaround that seems to be fully functional.

In the file /assets/sshd_config, set UsePAM to no.

Then edit the git user account in /etc/shadow. Enable the account by changing the password field from ! to *.

This still seems like an issue because this particular workaround requires modifying files outside of the volume mount points described in the installation docs (i.e., not /etc/gitlab, /var/log/gitlab, or /var/opt/gitlab)

Edited Dec 28, 2020 by Renilux