GitLab Dependency grpc-1.30.2 for Ruby 2.7.4 affected by DST Root CA X3 certificate expiration
Summary
As noted by LetsEncrypt in https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/, the following certificate has expired:
# Issuer: CN=DST Root CA X3 O=Digital Signature Trust Co.
# Subject: CN=DST Root CA X3 O=Digital Signature Trust Co.
# Label: "DST Root CA X3"
# Serial: 91299735575339953335919266965803778155
# MD5 Fingerprint: 41:03:52:dc:0f:f7:50:1b:16:f0:02:8e:ba:6f:45:c5
# SHA1 Fingerprint: da:c9:02:4f:54:d8:f6:df:94:93:5f:b1:73:26:38:ca:6a:d7:7c:13
# SHA256 Fingerprint: 06:87:26:03:31:a7:24:03:d9:09:f1:05:e6:9b:cf:0d:32:e1:bd:24:93:ff:c6:d9:20:6d:11:bc:d6:77:07:39
#-----BEGIN CERTIFICATE-----
However, said certificate is used by the GRPC Ruby gem version we're currently using (1.30.2
) as shown in https://github.com/grpc/grpc/blob/v1.30.1/etc/roots.pem#L777
As of today, GitLab relies on this GRPC Ruby gem for GRPC communication as per https://gitlab.com/gitlab-org/gitlab/-/blame/master/Gemfile#L482
This has nothing to do with the GRPC ruby gem downgrade from
f427fdfa, as that version is also affected.
Downgrade grpc from 1.38.0 to 1.30.2 The newer shared library contains CPU instructions that are incompatible with some customer machines.
We need to ignore deprecation warnings from related files as well until this is resolved.
Changelog: fixed
This certificate has only been removed from /etc/roots.pem
four days ago from master https://github.com/grpc/grpc/pull/27539/files and it's unlikely to be cherry picked onto past releases https://github.com/grpc/grpc/issues/27532#issuecomment-933704887
Steps to reproduce
In the case of Spamcheck (https://gitlab.com/gitlab-com/gl-security/engineering-and-research/automation-team/spam/spamcheck/), which was affected by this (https://gitlab.com/gitlab-com/gl-security/engineering-and-research/automation-team/spam/spamcheck/-/issues/150):
- install the latest GitLab/GDK version
- run Spamcheck locally using docker-compose (
build/up
) and protect the service with a LetsEncrypt certificate - point GitLab to it under http://gdk.test:3000/admin/application_settings/reporting#js-spam-settings
- attempt to post a public comment.
- check
gitlab/log/exceptions_json.log
{"severity":"ERROR","time":"2021-10-04T20:13:52.009Z","correlation_id":"01FH6EYT118Z19HF1XJAJE5183","exception.class":"NoMethodError","exception.message":"private method
eval' called for nil:NilClass","exception.backtrace":["lib/gitlab/spamcheck/client.rb:42:in
issue_spam?'","app/services/spam/spam_verdict_service.rb:75:inspamcheck_verdict'","app/services/spam/spam_verdict_service.rb:21:in
block in execute'","app/services/spam/spam_verdict_service.rb:20:inexecute'","app/services/spam/spam_action_service.rb:66:in
perform_spam_service_check'","app/services/spam/spam_action_service.rb:33:inexecute'","app/services/issues/create_service.rb:30:in
before_create'","app/services/issuable_base_service.rb:210:increate'","app/services/issues/create_service.rb:21:in
execute'","ee/app/services/ee/issues/create_service.rb:17:inexecute'","app/controllers/projects/issues_controller.rb:141:in
create'","ee/lib/gitlab/ip_address_state.rb:10:inwith'","ee/app/controllers/ee/application_controller.rb:44:in
set_current_ip_address'","app/controllers/application_controller.rb:485:inset_current_admin'","lib/gitlab/session.rb:11:in
with_session'","app/controllers/application_controller.rb:476:inset_session_storage'","lib/gitlab/i18n.rb:105:in
with_locale'","lib/gitlab/i18n.rb:111:inwith_user_locale'","app/controllers/application_controller.rb:470:in
set_locale'","app/controllers/application_controller.rb:464:inset_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in
call'","lib/gitlab/middleware/rails_queue_duration.rb:33:incall'","lib/gitlab/metrics/rack_middleware.rb:16:in
block in call'","lib/gitlab/metrics/web_transaction.rb:21:inrun'","lib/gitlab/metrics/rack_middleware.rb:16:in
call'","lib/gitlab/middleware/speedscope.rb:13:incall'","lib/gitlab/request_profiler/middleware.rb:17:in
call'","lib/gitlab/query_limiting/middleware.rb:17:inblock in call'","lib/gitlab/query_limiting/transaction.rb:40:in
run'","lib/gitlab/query_limiting/middleware.rb:16:incall'","lib/gitlab/jira/middleware.rb:19:in
call'","lib/gitlab/middleware/go.rb:20:incall'","lib/gitlab/etag_caching/middleware.rb:21:in
call'","lib/gitlab/middleware/multipart.rb:172:incall'","lib/gitlab/middleware/read_only/controller.rb:50:in
call'","lib/gitlab/middleware/read_only.rb:18:incall'","lib/gitlab/middleware/same_site_cookies.rb:27:in
call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:incall'","lib/gitlab/middleware/basic_health_check.rb:25:in
call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:incall'","lib/gitlab/middleware/request_context.rb:21:in
call'","config/initializers/fix_local_cache_middleware.rb:11:incall'","lib/gitlab/middleware/static.rb:11:in
call'","lib/gitlab/webpack/dev_server_middleware.rb:34:inperform_request'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in
call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:incall'","lib/gitlab/metrics/requests_rack_middleware.rb:74:in
call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"user.username":"root","tags.program":"web","tags.locale":"en","tags.feature_category":"issue_tracking","tags.correlation_id":"01FH6EYT118Z19HF1XJAJE5183"}
What is the current bug behavior?
GRPC connections with default channel credentials such as https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/spamcheck/client.rb#L34 fail due to certificate validation:
Output of checks
This bug happens on GitLab.com
Results of GitLab environment info
$ bundle exec rake gitlab:env:info
System information
System: Ubuntu 20.04
Proxy: no
Using RVM: no
Ruby Version: 2.7.4p191
Gem Version: 3.1.6
Bundler Version:2.1.4
Rake Version: 13.0.6
Redis Version: 6.0.15
Git Version: 2.33.0
Sidekiq Version:6.2.2
Go Version: go1.16.8 linux/amd64
GitLab information
Version: 14.4.0-pre
Revision: 515c5104e8a
DB Adapter: PostgreSQL
DB Version: 12.6
URL: http://gdk.test:3000
HTTP Clone URL: http://gdk.test:3000/some-group/some-project.git
SSH Clone URL: ssh://git@gdk.test:2222/some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: google_oauth2
GitLab Shell
Version: 13.21.1
Repository storage paths:
Git: /usr/bin/git
Possible fixes
Remove certificate da:c9:02:4f:54:d8:f6:df:94:93:5f:b1:73:26:38:ca:6a:d7:7c:13
from ~/.asdf/installs/ruby/2.7.4/lib/ruby/gems/2.7.0/gems/grpc-1.30.2-x86_64-linux/etc/roots.pem
or upgrade the Ruby GRPC gem to its latest version.
cc @stanhu @mkaeppler @gitlab-com/gl-security/engineering-and-research/automation-team
cc @nnelson @jarv @rehab @cmcfarland for your involvement with gitlab-com/gl-infra/production#5430 (closed)