Active Record Decryption Error since upgrading to 17.10

Summary

After upgrading to 17.10, many customers are starting to see errors with active record decryption on self-managed.

"exception.class": "ActiveRecord::Encryption::Errors::Decryption",
  "exception.message": "ActiveRecord::Encryption::Errors::Decryption",
  "exception.backtrace": [
    "activerecord (7.0.8.7) lib/active_record/encryption/encryptor.rb:58:in `rescue in decrypt'",
    "activerecord (7.0.8.7) lib/active_record/encryption/encryptor.rb:52:in `decrypt'",

Mainly we have seen this in the context of ODCI/OAUTH. At least 2 of our customers faced it when trying to use the functionality https://docs.gitlab.com/ci/cloud_services/aws/.

Steps to reproduce

Follow the doc https://docs.gitlab.com/ci/cloud_services/aws/ to retrieve temporary credentials.

What is the current bug behavior?

The error ActiveRecord::Encryption::Errors::Decryption is happening when authenticating with AWS.

What is the expected correct behavior?

Authentication with AWS should work successfully.

Relevant logs and/or screenshots

Full Trace

"exception.class":"ActiveRecord::Encryption::Errors::Decryption","exception.message":"ActiveRecord::Encryption::Errors::Decryption","exception.backtrace":\["activerecord (7.0.8.7) lib/active_record/encryption/encryptor.rb:58:in `rescue in decrypt'","activerecord (7.0.8.7) lib/active_record/encryption/encryptor.rb:52:in `decrypt'","activerecord (7.0.8.7) lib/active_record/encryption/encrypted_attribute_type.rb:73:in `block in decrypt'","activerecord (7.0.8.7) lib/active_record/encryption/scheme.rb:67:in `with_context'","activerecord (7.0.8.7) lib/active_record/encryption/encrypted_attribute_type.rb:15:in `with_context'","activerecord (7.0.8.7) lib/active_record/encryption/encrypted_attribute_type.rb:72:in `decrypt'","activerecord (7.0.8.7) lib/active_record/encryption/encrypted_attribute_type.rb:31:in `deserialize'","activemodel (7.0.8.7) lib/active_model/attribute_set/builder.rb:52:in `block in fetch_value'","activemodel (7.0.8.7) lib/active_model/attribute_set/builder.rb:46:in `fetch'","activemodel (7.0.8.7) lib/active_model/attribute_set/builder.rb:46:in `fetch_value'","activerecord (7.0.8.7) lib/active_record/attribute_methods/read.rb:38:in `_read_attribute'","activemodel (7.0.8.7) lib/active_model/attribute_methods.rb:271:in `secret_key'","activerecord (7.0.8.7) lib/active_record/relation/delegation.rb:88:in `each'","activerecord (7.0.8.7) lib/active_record/relation/delegation.rb:88:in `each'","ee/app/models/cloud_connector/keys.rb:19:in `map'","ee/app/models/cloud_connector/keys.rb:19:in `all_as_pem'","ee/app/controllers/ee/jwks_controller.rb:11:in `load_keys'","app/controllers/jwks_controller.rb:17:in `payload'","app/controllers/jwks_controller.rb:11:in `keys'","actionpack (7.0.8.7) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'","actionpack (7.0.8.7) lib/abstract_controller/base.rb:215:in `process_action'","actionpack (7.0.8.7) lib/action_controller/metal/rendering.rb:165:in `process_action'","actionpack (7.0.8.7) lib/abstract_controller/callbacks.rb:234:in `block in process_action'","activesupport (7.0.8.7) lib/active_support/callbacks.rb:118:in `block in run_callbacks'","marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'","activesupport (7.0.8.7) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","sentry-rails (5.22.1) lib/sentry/rails/controller_transaction.rb:34:in `block in sentry_around_action'","sentry-ruby (5.22.1) lib/sentry/hub.rb:108:in `with_child_span'","sentry-ruby (5.22.1) lib/sentry-ruby.rb:503:in `with_child_span'","sentry-rails (5.22.1) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action'","activesupport (7.0.8.7) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","activesupport (7.0.8.7) lib/active_support/callbacks.rb:138:in `run_callbacks'","actionpack (7.0.8.7) lib/abstract_controller/callbacks.rb:233:in `process_action'","actionpack (7.0.8.7) lib/action_controller/metal/rescue.rb:23:in `process_action'","actionpack (7.0.8.7) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'","activesupport (7.0.8.7) lib/active_support/notifications.rb:206:in `block in instrument'","activesupport (7.0.8.7) lib/active_support/notifications/instrumenter.rb:24:in `instrument'","activesupport (7.0.8.7) lib/active_support/notifications.rb:206:in `instrument'","actionpack (7.0.8.7) lib/action_controller/metal/instrumentation.rb:66:in `process_action'","actionpack (7.0.8.7) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'","activerecord (7.0.8.7) lib/active_record/railties/controller_runtime.rb:27:in `process_action'","actionpack (7.0.8.7) lib/abstract_controller/base.rb:151:in `process'","actionpack (7.0.8.7) lib/action_controller/metal.rb:188:in `dispatch'","actionpack (7.0.8.7) lib/action_controller/metal.rb:251:in `dispatch'","actionpack (7.0.8.7) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'","actionpack (7.0.8.7) lib/action_dispatch/routing/route_set.rb:32:in `serve'","actionpack (7.0.8.7) lib/action_dispatch/journey/router.rb:50:in `block in serve'","actionpack (7.0.8.7) lib/action_dispatch/journey/router.rb:32:in `each'","actionpack (7.0.8.7) lib/action_dispatch/journey/router.rb:32:in `serve'","actionpack (7.0.8.7) lib/action_dispatch/routing/route_set.rb:852:in `call'","gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'","omniauth (2.1.2) lib/omniauth/strategy.rb:470:in `call_app!'","omniauth-saml (2.2.2) lib/omniauth/strategies/saml.rb:86:in `other_phase'","omniauth (2.1.2) lib/omniauth/strategy.rb:195:in `call!'","omniauth (2.1.2) lib/omniauth/strategy.rb:169:in `call'","omniauth (2.1.2) lib/omniauth/strategy.rb:202:in `call!'","omniauth (2.1.2) lib/omniauth/strategy.rb:169:in `call'","flipper (0.28.3) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'","flipper (0.28.3) lib/flipper/middleware/memoizer.rb:37:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'","lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/middleware/go.rb:21:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:83:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/ci/job_token/middleware.rb:11:in `call'","batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'","apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'","warden (1.2.9) lib/warden/manager.rb:36:in `block in call'","warden (1.2.9) lib/warden/manager.rb:34:in `catch'","warden (1.2.9) lib/warden/manager.rb:34:in `call'","rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'","rack (2.2.11) lib/rack/tempfile_reaper.rb:15:in `call'","rack (2.2.11) lib/rack/etag.rb:27:in `call'","rack (2.2.11) lib/rack/conditional_get.rb:27:in `call'","rack (2.2.11) lib/rack/head.rb:12:in `call'","actionpack (7.0.8.7) lib/action_dispatch/http/permissions_policy.rb:38:in `call'","actionpack (7.0.8.7) lib/action_dispatch/http/content_security_policy.rb:39:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'","rack (2.2.11) lib/rack/session/abstract/id.rb:266:in `context'","rack (2.2.11) lib/rack/session/abstract/id.rb:260:in `call'","actionpack (7.0.8.7) lib/action_dispatch/middleware/cookies.rb:704:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","actionpack (7.0.8.7) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'","activesupport (7.0.8.7) lib/active_support/callbacks.rb:99:in `run_callbacks'","actionpack (7.0.8.7) lib/action_dispatch/middleware/callbacks.rb:26:in `call'","sentry-rails (5.22.1) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'","actionpack (7.0.8.7) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'","lib/gitlab/middleware/path_traversal_check.rb:35:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","sentry-ruby (5.22.1) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'","sentry-ruby (5.22.1) lib/sentry/hub.rb:269:in `with_session_tracking'","sentry-ruby (5.22.1) lib/sentry-ruby.rb:416:in `with_session_tracking'","sentry-ruby (5.22.1) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'","sentry-ruby (5.22.1) lib/sentry/hub.rb:59:in `with_scope'","sentry-ruby (5.22.1) lib/sentry-ruby.rb:396:in `with_scope'","sentry-ruby (5.22.1) lib/sentry/rack/capture_exceptions.rb:20:in `call'","actionpack (7.0.8.7) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'","railties (7.0.8.7) lib/rails/rack/logger.rb:25:in `block in call'","activesupport (7.0.8.7) lib/active_support/tagged_logging.rb:99:in `block in tagged'","activesupport (7.0.8.7) lib/active_support/tagged_logging.rb:37:in `tagged'","activesupport (7.0.8.7) lib/active_support/tagged_logging.rb:99:in `tagged'","railties (7.0.8.7) lib/rails/rack/logger.rb:25:in `call'","actionpack (7.0.8.7) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","request_store (1.7.0) lib/request_store/middleware.rb:19:in `call'","rack (2.2.11) lib/rack/method_override.rb:24:in `call'","rack (2.2.11) lib/rack/runtime.rb:22:in `call'","rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'","rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'","rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","actionpack (7.0.8.7) lib/action_dispatch/middleware/executor.rb:14:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","rack (2.2.11) lib/rack/sendfile.rb:110:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'","gitlab-labkit (0.37.0) lib/labkit/middleware/rack.rb:22:in `block in call'","gitlab-labkit (0.37.0) lib/labkit/context.rb:35:in `with_context'","gitlab-labkit (0.37.0) lib/labkit/middleware/rack.rb:21:in `call'","actionpack (7.0.8.7) lib/action_dispatch/middleware/request_id.rb:26:in `call'","actionpack (7.0.8.7) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'","railties (7.0.8.7) lib/rails/engine.rb:530:in `call'","railties (7.0.8.7) lib/rails/railtie.rb:226:in `public_send'","railties (7.0.8.7) lib/rails/railtie.rb:226:in `method_missing'","lib/gitlab/middleware/release_env.rb:12:in `call'","rack (2.2.11) lib/rack/urlmap.rb:74:in `block in call'","rack (2.2.11) lib/rack/urlmap.rb:58:in `each'","rack (2.2.11) lib/rack/urlmap.rb:58:in `call'","puma (6.5.0) lib/puma/configuration.rb:279:in `call'","puma (6.5.0) lib/puma/request.rb:99:in `block in handle_request'","puma (6.5.0) lib/puma/thread_pool.rb:389:in `with_force_shutdown'","puma (6.5.0) lib/puma/request.rb:98:in `handle_request'","puma (6.5.0) lib/puma/server.rb:468:in `process_client'","puma (6.5.0) lib/puma/server.rb:249:in `block in run'","puma (6.5.0) lib/puma/thread_pool.rb:166:in `block in spawn_thread'"\],"exception.cause_class":"ActiveRecord::Encryption::Errors::Decryption

Fix

Steps to properly fix this can be found at https://docs.gitlab.com/update/versions/gitlab_17_changes/#unify-new-encryption-secrets.

Additional information

• The issue started happening when customer has upgraded to GitLab 17.10.
• Customers that experience it use GitLab Omnibus.

Workaround

See this comment. Note that removing the keys will make Duo non-functional, so only apply it if you are not using Duo Self-hosted.