Use the system CA bundle for gRPC default roots
What does this MR do and why?
Points gRPC's default CA roots at Gitlab::X509::Certificate.ca_certs_bundle — the same CA bundle every other gRPC client in the codebase uses — instead of gRPC's own vendored etc/roots.pem, in config/initializers/grpc.rb. (ca_certs_bundle honors the SSL_CERT_FILE / SSL_CERT_DIR overrides and merges the cert directory, and is exactly what Gitaly/KAS/Spamcheck/IAM already pass explicitly.) This mirrors the intent of config/initializers/excon.rb.
gRPC ships its own bundled CA roots and, in Ruby, installs them as the default via GRPC::Core::ChannelCredentials.set_default_roots_pem when the gem loads. That bundle updates on gRPC's own release schedule and can lag the system trust store. Concretely, the gem we vendor (grpc 1.81.1) predates the eMudhra/emSign CA distrust security update (https://github.com/grpc/grpc/commit/8b32af3bf498c2ba9667410eaa8c836e700e97e6), so gRPC clients using default credentials would verify against a stale root set.
With this change, both Omnibus and CNG gRPC TLS (for any client using default credentials — e.g. the Cells Topology Service client and the Google Artifact Registry client) honor the system/embedded cacert.pem and any admin-added trusted-certs. Gitaly, KAS, Spamcheck, and IAM already pass an explicit CA (Gitlab::X509::Certificate.ca_certs_bundle), so they are unaffected; this is defense in depth and consistency for the default-credential clients.
How to test
The change makes gRPC default-credential clients verify server certs against GitLab's system CA bundle, so they reject any cert that does not chain to a CA in that bundle (e.g. a distrusted root such as the removed emSign roots).
1. Confirm the roots gRPC now uses are the system bundle (Rails console)
bundle = Gitlab::X509::Certificate.ca_certs_bundle
bundle.scan(/BEGIN CERTIFICATE/).size # matches the embedded cacert.pem cert count
bundle.include?('emSign ECC Root CA - G3') # => false once the eMudhra distrust is applied2. Generate a cert chained to an "emSign" root and prove the trust decision
This stands in for a server whose cert chains to a distrusted root. Run anywhere with openssl + the grpc gem:
mkdir -p /tmp/emsign-test && cd /tmp/emsign-test
SUBJ='/C=IN/OU=emSign PKI/O=eMudhra Technologies Limited/CN=emSign ECC Root CA - G3'
# Self-signed "emSign" root CA
openssl req -x509 -newkey rsa:2048 -nodes -keyout ca.key -out ca.crt -days 5 -subj "$SUBJ"
# Server cert (CN/SAN localhost) signed by that CA
printf 'subjectAltName=DNS:localhost,IP:127.0.0.1\n' > san.cnf
openssl req -newkey rsa:2048 -nodes -keyout server.key -out server.csr -subj '/CN=localhost'
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out server.crt -days 5 -extfile san.cnfThe server cert is trusted iff the emSign CA is in the trust bundle:
cat "$(ruby -ropenssl -e 'print OpenSSL::X509::DEFAULT_CERT_FILE')" ca.crt > with_emsign.pem
openssl verify -CAfile "$(ruby -ropenssl -e 'print OpenSSL::X509::DEFAULT_CERT_FILE')" server.crt
# => server.crt: ... unable to get local issuer certificate (REJECTED — this is the patched state)
openssl verify -CAfile with_emsign.pem server.crt
# => server.crt: OK (ACCEPTED when emSign is trusted)3. Prove a gRPC default-credential client rejects the distrusted cert
Start a local gRPC server presenting the emSign-signed cert:
# server.rb — run in one terminal: ruby server.rb
require 'grpc'
creds = GRPC::Core::ServerCredentials.new(
nil, [{ private_key: File.read('/tmp/emsign-test/server.key'),
cert_chain: File.read('/tmp/emsign-test/server.crt') }], false)
s = GRPC::RpcServer.new
s.add_http2_port('127.0.0.1:50443', creds)
puts 'server up on 127.0.0.1:50443'
s.run_till_terminated_or_interrupted([1, 'int', 'SIGQUIT'])Then point a default-credential client at it with the system bundle (emSign absent):
# client.rb — run in another terminal: ruby client.rb
require 'grpc'; require 'openssl'
GRPC::Core::ChannelCredentials.set_default_roots_pem(File.read(OpenSSL::X509::DEFAULT_CERT_FILE))
stub = GRPC::ClientStub.new('localhost:50443', GRPC::Core::ChannelCredentials.new)
begin
stub.request_response('/grpc.health.v1.Health/Check', '', ->(m) { m }, ->(m) { m },
deadline: Time.now + 4)
puts 'ACCEPTED (handshake ok)'
rescue StandardError => e
puts "REJECTED :: #{e.message[0, 120].tr("\n", ' ')}"
endExpected output on a patched instance (emSign not in the bundle):
REJECTED :: 14:failed to connect to all addresses; ... Ssl handshake failed (TSI_PROTOCOL_FAILURE): SSL_ERROR_SSLAdd the emSign CA to the trust store (set_default_roots_pem(File.read('with_emsign.pem')), or in Omnibus drop ca.crt in /etc/gitlab/trusted-certs and gitlab-ctl reconfigure) and the same handshake succeeds — confirming the client follows the system bundle, not gRPC's vendored roots.pem. Reverting this MR makes the client follow roots.pem instead. GRPC_DEFAULT_SSL_ROOTS_FILE_PATH still takes precedence if set.
Automated
bundle exec rspec spec/initializers/grpc_spec.rb asserts the initializer pins the default roots to ca_certs_bundle (and no-ops when the bundle is blank).
MR acceptance checklist
- Behavior verified: gRPC default roots resolve to the system bundle (
GRPC_DEFAULT_SSL_ROOTS_FILE_PATHstill takes precedence if set).
Changelog: changed