Distrust compromised eMudhra CA certificates

Jason Plumrequested to merge
2467-emudhra-ca-distrust into master

What does this MR do?

Removes trust in the compromised eMudhra CA certificates (https://emudhra.we-have-a-problem.com/) from the UBI and Debian images, at build time and when the certificates container runs.

  • UBI: ca-certificates-2025.2.80_v9.0.305-91.el9 is the newest package in the UBI 9.8 repositories. It still trusts the anchor emSign ECC Root CA - G3, which accepts 6 of the 12 compromised test chains. The new hardening/003_ca_distrust.sh puts hardening/ca-distrust/*.pem in the p11-kit blocklist and extracts the trust store again. All UBI Dockerfiles run the hardening scripts, so this also covers kubectl, gitlab-kas, gitlab-agentk, gitlab-zoekt and cfssl-self-sign. These images build on ubi-micro and install their own ca-certificates. At runtime, update-ca-trust extract in bundle-certificates-ubi applies the blocklist, also to custom CAs that users mount.
  • Debian: 20250419+deb13u1 already drops these roots. But update-ca-certificates adds every certificate in /usr/local/share/ca-certificates, so a custom CA mount can bring them back. The new /scripts/ca-distrust removes the certificates that match a fingerprint in the list, from the files, hash links and bundles in /etc/ssl/certs. It runs in gitlab-base at build time and at the end of bundle-certificates.

The list holds 21 certificates: 4 roots, the emSign Root TLS CA - G3 cross-sign, and the 16 compromised intermediates. Each one has its SHA-256 fingerprint in a comment. The roots must be in the list: a blocklist with only the intermediates still trusts 6/12 chains in the PEM bundles that Go, curl and Ruby read.

The advisory's Linux workarounds block 2 roots by label, and the Debian ! lines only apply to package certificates. Neither one blocks a CA that a user mounts. With them, a mounted emSign Root TLS CA - G3 (UBI) or mounted roots (Debian) are still trusted.

Distrusting emSign ECC Root CA - G3 also distrusts legitimate sites under that root, the same as the advisory and the Debian fix.

Related issues

Closes https://gitlab.com/gitlab-org/build/CNG/-/work_items/2467+s

Related https://gitlab.com/gitlab-org/release/tasks/-/work_items/28133+s

Test plan

Local builds of the real gitlab-base, certificates and kubectl Dockerfiles (UBI 9.8, Debian trixie, arm64). gomplate, gitlab-logger and kubectl are stub images, because they do not change the CA store. The checks use openssl verify -purpose sslserver with an isolated store. The 12 chains are from chains.zip on the advisory page. The legitimate chains are from gitlab.com, registry.access.redhat.com and deb.debian.org. A control store that holds only the 4 roots trusts all 12 chains.

Store Compromised chains trusted Legit chains Private CA
UBI images on master (gitlab-base, certificates, kubectl) 6/12 3/3 –
UBI images on this branch, also with FIPS_MODE=1 0/12 3/3 –
UBI certificates output, user mounts the 4 roots, master 6/12 3/3 OK
UBI certificates output, user mounts the 4 roots, this branch 0/12 3/3 OK
Debian certificates output, user mounts the 4 roots, filter off 12/12 3/3 OK
Debian certificates output, same mounts, this branch (bundle and hash directory) 0/12 3/3 OK

The UBI rows apply to all three output files: ca-bundle.crt, ca-certificates.crt and ca-bundle.trust.crt. Go 1.27.2 crypto/x509, with SSL_CERT_FILE and SSL_CERT_DIR, gives the same results. Go 1.27.3 and later rejects these chains itself. The Debian filter removes only the listed entries and keeps the order of the bundle.

GitLab chart deployment

GitLab chart 10.5.1 on k3d (arm64), with PostgreSQL, Redis and object storage from gitlab-dev-stack. The custom CAs (the 4 roots, 1 private CA, and 1 mixed bundle that holds a second private CA and emSign ECC Root CA - G3) are set in global.certificates.customCAs. The probes make TLS connections from 8 containers (webservice, workhorse, sidekiq, toolbox, gitaly, gitlab-shell, registry, pages) and from Ruby Net::HTTP to the 12 live test hosts of the disclosure, 3 legitimate hosts, and 2 in-cluster servers signed by the private CAs.

certificates image Compromised hosts trusted Legit hosts Private CAs
v19.5.1 (Debian apps) 12/12 3/3 OK
This MR (Debian apps) 0/12 3/3 OK
v19.5.1-ubi (UBI apps) 6/12 3/3 OK
This MR, FIPS (UBI apps) 0/12 3/3 OK

The chart runs the init container as uid 1000 and mounts custom CAs with mode 0440. The first version of ca-distrust could not write into the read-only copy of a filtered custom CA bundle, and the pods stayed in Init:CrashLoopBackOff. The script now replaces the file.

On UBI apps that are not rebuilt, extracted/openssl/ca-bundle.trust.crt and extracted/java/cacerts in the app image still hold emSign ECC Root CA - G3, because the chart does not share them from the init container. App images built from this MR's gitlab-base do not.

Container size: in images that run the step, the UBI hardening layer grows from 20.5 kB to 2.64 MB. Images built on gitlab-base skip it and add 20.5 kB.

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Integration tests added to GitLab QA
  • The impact any change in container size has should be evaluated
  • New dependencies are managed with GitLab forked renovatebot

🤖 Generated with Claude Code

Edited by Robert Marshall

Merge request reports