Send SSH client IP to /authorized_certs

Fred Reininkrequested to merge
shell-audit-certificate-based-authentication-2 into main

What does this MR do and why?

Addresses "Gap 2" in https://gitlab.com/gitlab-org/gitlab/-/work_items/611349#gap-2-failed-authentications-are-not-audited

gitlab-sshd now sends the SSH client's IP address when it resolves an SSH certificate through GitLab. GitLab records an audit event when certificate authentication fails, and that event needs the client's address, not GitLab Shell's.

  • authorizedcerts.Client.GetByKey accepts a raw remoteAddr argument, strips the port, and sends a valid IP as check_ip to /api/v4/internal/authorized_certs.
  • handleUserCertificate and resolveCertificateViaAPI pass the SSH connection's remote address through to the client, where parsing and validation happen.
  • When PROXY protocol is enabled, the address is the client address from the PROXY header, the same address used for check_ip on /allowed.
  • check_ip is left out when the address is unknown or invalid, including Unix socket paths. GitLab versions that don't know the parameter ignore it.
  • Client tests cover IPv4 and IPv6 with and without ports, zoned IPv6, an empty address, and a Unix socket path. Callback tests verify that IPv4 and IPv6 client IPs are sent when /authorized_certs rejects the certificate. A real-server PROXY protocol test asserting check_ip is deferred to a follow-up.

References

How to set up and validate locally

  1. Attempt SSH authentication with a valid certificate signed by an unregistered CA.
  2. Verify that the resulting audit event records the SSH client's IP.

Authentication is expected to fail: the successful outcome of this test is an audit event with the correct client address.

Prerequisites

  • A working Linux GDK running GitLab EE with an Ultimate development license.
  • GDK's gitlab-sshd configured to listen on 127.0.0.1:2222, without PROXY protocol.
  • The ability to bind the SSH client to 127.0.0.2. Using a distinct loopback address lets this test distinguish the SSH client from GitLab Shell's Rails-request address.
  • Clean working trees in GDK's active gitlab/ and gitlab-shell/ checkouts.

1. Check out and build both MRs

Run from your GDK root directory. Record both repositories' current branches for cleanup and keep this terminal open for subsequent commands:

export GDK_ROOT="$PWD"
mkdir -p "$GDK_ROOT/tmp"
export CERT_DIR="$(mktemp -d "$GDK_ROOT/tmp/cert-client-ip.XXXXXX")"

cd "$GDK_ROOT/gitlab"
git fetch https://gitlab.com/gitlab-org/gitlab.git merge-requests/259775/head
git switch --detach FETCH_HEAD
bundle install

cd "$GDK_ROOT/gitlab-shell"
git fetch https://gitlab.com/gitlab-org/gitlab-shell.git merge-requests/1578/head
git switch --detach FETCH_HEAD
make compile

cd "$GDK_ROOT"
gdk restart rails-web sidekiq sshd

These must be the checkouts used by GDK's services. make compile builds both GitLab Shell and gitlab-sshd. Neither MR requires a frontend build or database migration.

2. Create an SSH certificate

In the same terminal:

export CERT_ID="cert-ip-check-$(date -u +%Y%m%d%H%M%S)"

ssh-keygen -t ed25519 -N '' -f "$CERT_DIR/ca"
ssh-keygen -t ed25519 -N '' -f "$CERT_DIR/client"

ssh-keygen \
  -s "$CERT_DIR/ca" \
  -I "$CERT_ID" \
  -n git \
  -V '-5m:+1h' \
  "$CERT_DIR/client.pub"

printf 'Certificate identity: %s\n' "$CERT_ID"

Record the printed identity. Do not register this CA in GitLab or add it to sshd.trusted_user_ca_keys, and do not register the client public key as a GitLab SSH key. The valid certificate must reach /authorized_certs and be rejected because its CA is unknown.

3. Attempt authentication

ssh -F /dev/null \
  -b 127.0.0.2 \
  -p 2222 \
  -i "$CERT_DIR/client" \
  -o CertificateFile="$CERT_DIR/client-cert.pub" \
  -o IdentitiesOnly=yes \
  -o IdentityAgent=none \
  -o PreferredAuthentications=publickey \
  -o BatchMode=yes \
  -o UserKnownHostsFile="$CERT_DIR/known_hosts" \
  -o StrictHostKeyChecking=accept-new \
  -T git@127.0.0.1

Expect Permission denied (publickey) and a nonzero exit status. This is intentional. The command sends the test certificate from 127.0.0.2 and avoids reusing keys from your SSH agent.

4. Verify the audit event

From your GDK root, open a Rails console:

gdk rails console

Run the following, replacing the identity with the value printed in step 2:

identity = 'paste-the-certificate-identity-from-step-2'
event = AuditEvents::InstanceAuditEvent
  .where(event_name: 'ssh_certificate_authentication_failed', author_name: identity)
  .order(:id).last
raise 'Audit event not found' unless event

pp event.attributes
pp event.details

raise 'Unexpected client IP' unless event.ip_address.to_s == '127.0.0.2'
raise 'Unexpected failure reason' unless event.details.with_indifferent_access[:reason] == 'certificate_not_found'

Expect an ssh_certificate_authentication_failed event for the test identity, with reason certificate_not_found and ip_address equal to 127.0.0.2, not 127.0.0.1. The event is recorded synchronously, so no audit streaming receiver is needed.

  • A certificate authentication attempt rejected by /authorized_certs produces an audit event containing the SSH client's IP address.

image

Cleanup

  • Exit the Rails console.
  • Return both repositories to their previous branches and run make compile in gitlab-shell.
  • From your GDK root, run gdk restart rails-web sidekiq sshd.
  • Remove the temporary certificate directory after checking its path with echo "$CERT_DIR".

Either merge order is safe: older Rails versions ignore check_ip, and the companion Rails change falls back to the request IP when it is missing.

Edited by Fred Reinink

Merge request reports