Use the system CA bundle for gRPC default roots

What does this MR do and why?

Points gRPC's default CA roots at Gitlab::X509::Certificate.ca_certs_bundle — the same CA bundle every other gRPC client in the codebase uses — instead of gRPC's own vendored etc/roots.pem, in config/initializers/grpc.rb. (ca_certs_bundle honors the SSL_CERT_FILE / SSL_CERT_DIR overrides and merges the cert directory, and is exactly what Gitaly/KAS/Spamcheck/IAM already pass explicitly.) This mirrors the intent of config/initializers/excon.rb.

gRPC ships its own bundled CA roots and, in Ruby, installs them as the default via GRPC::Core::ChannelCredentials.set_default_roots_pem when the gem loads. That bundle updates on gRPC's own release schedule and can lag the system trust store. Concretely, the gem we vendor (grpc 1.81.1) predates the eMudhra/emSign CA distrust security update (https://github.com/grpc/grpc/commit/8b32af3bf498c2ba9667410eaa8c836e700e97e6), so gRPC clients using default credentials would verify against a stale root set.

With this change, both Omnibus and CNG gRPC TLS (for any client using default credentials — e.g. the Cells Topology Service client and the Google Artifact Registry client) honor the system/embedded cacert.pem and any admin-added trusted-certs. Gitaly, KAS, Spamcheck, and IAM already pass an explicit CA (Gitlab::X509::Certificate.ca_certs_bundle), so they are unaffected; this is defense in depth and consistency for the default-credential clients.

How to test

The change makes gRPC default-credential clients verify server certs against GitLab's system CA bundle, so they reject any cert that does not chain to a CA in that bundle (e.g. a distrusted root such as the removed emSign roots).

1. Confirm the roots gRPC now uses are the system bundle (Rails console)

bundle = Gitlab::X509::Certificate.ca_certs_bundle
bundle.scan(/BEGIN CERTIFICATE/).size            # matches the embedded cacert.pem cert count
bundle.include?('emSign ECC Root CA - G3')       # => false once the eMudhra distrust is applied

2. Generate a cert chained to an "emSign" root and prove the trust decision

This stands in for a server whose cert chains to a distrusted root. Run anywhere with openssl + the grpc gem:

mkdir -p /tmp/emsign-test && cd /tmp/emsign-test
SUBJ='/C=IN/OU=emSign PKI/O=eMudhra Technologies Limited/CN=emSign ECC Root CA - G3'

# Self-signed "emSign" root CA
openssl req -x509 -newkey rsa:2048 -nodes -keyout ca.key -out ca.crt -days 5 -subj "$SUBJ"

# Server cert (CN/SAN localhost) signed by that CA
printf 'subjectAltName=DNS:localhost,IP:127.0.0.1\n' > san.cnf
openssl req -newkey rsa:2048 -nodes -keyout server.key -out server.csr -subj '/CN=localhost'
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt -days 5 -extfile san.cnf

The server cert is trusted iff the emSign CA is in the trust bundle:

cat "$(ruby -ropenssl -e 'print OpenSSL::X509::DEFAULT_CERT_FILE')" ca.crt > with_emsign.pem

openssl verify -CAfile "$(ruby -ropenssl -e 'print OpenSSL::X509::DEFAULT_CERT_FILE')" server.crt
#   => server.crt: ... unable to get local issuer certificate  (REJECTED — this is the patched state)

openssl verify -CAfile with_emsign.pem server.crt
#   => server.crt: OK                                          (ACCEPTED when emSign is trusted)

3. Prove a gRPC default-credential client rejects the distrusted cert

Start a local gRPC server presenting the emSign-signed cert:

# server.rb — run in one terminal:  ruby server.rb
require 'grpc'
creds = GRPC::Core::ServerCredentials.new(
  nil, [{ private_key: File.read('/tmp/emsign-test/server.key'),
          cert_chain:  File.read('/tmp/emsign-test/server.crt') }], false)
s = GRPC::RpcServer.new
s.add_http2_port('127.0.0.1:50443', creds)
puts 'server up on 127.0.0.1:50443'
s.run_till_terminated_or_interrupted([1, 'int', 'SIGQUIT'])

Then point a default-credential client at it with the system bundle (emSign absent):

# client.rb — run in another terminal:  ruby client.rb
require 'grpc'; require 'openssl'
GRPC::Core::ChannelCredentials.set_default_roots_pem(File.read(OpenSSL::X509::DEFAULT_CERT_FILE))
stub = GRPC::ClientStub.new('localhost:50443', GRPC::Core::ChannelCredentials.new)
begin
  stub.request_response('/grpc.health.v1.Health/Check', '', ->(m) { m }, ->(m) { m },
                        deadline: Time.now + 4)
  puts 'ACCEPTED (handshake ok)'
rescue StandardError => e
  puts "REJECTED :: #{e.message[0, 120].tr("\n", ' ')}"
end

Expected output on a patched instance (emSign not in the bundle):

REJECTED :: 14:failed to connect to all addresses; ... Ssl handshake failed (TSI_PROTOCOL_FAILURE): SSL_ERROR_SSL

Add the emSign CA to the trust store (set_default_roots_pem(File.read('with_emsign.pem')), or in Omnibus drop ca.crt in /etc/gitlab/trusted-certs and gitlab-ctl reconfigure) and the same handshake succeeds — confirming the client follows the system bundle, not gRPC's vendored roots.pem. Reverting this MR makes the client follow roots.pem instead. GRPC_DEFAULT_SSL_ROOTS_FILE_PATH still takes precedence if set.

Automated

bundle exec rspec spec/initializers/grpc_spec.rb asserts the initializer pins the default roots to ca_certs_bundle (and no-ops when the bundle is blank).

MR acceptance checklist

  • Behavior verified: gRPC default roots resolve to the system bundle (GRPC_DEFAULT_SSL_ROOTS_FILE_PATH still takes precedence if set).

Changelog: changed

Edited by Stan Hu

Merge request reports

Loading
Loading