Active Records encryption failure results in "Erorr 500" and "Unable to save settings" instead of proper error messages/option to recover
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
Incorrect handling of active_record_encryption_primary_key, active_record_encryption_deterministic_key and/or active_record_encryption_key_derivation_salt in a self hosted instance can lead to undecryptable settings, which I have so far only encountered for the Docker Hub login data of the Dependency Proxy. Such problems result in generic HTTP 500 or Unable to save settings errors that are only recoverable from through direct PostgreSQL database manipulations.
Steps to reproduce
- Set up https://docs.gitlab.com/user/packages/dependency_proxy/
- Log into Docker Hub (under "Settings -> Packages and Registries")
- Change the above secrets - this happens automatically if they are not persisted correctly, and GitLab will silently generate new ones!
- Restart GitLab
Example Project
Not appropriate since it's a server setup issue.
What is the current bug behavior?
The "Packages and Registries" settings menu will be broken (all settings disabled, unable to make changes, unable to save settings) and the proxy will be unusable (responds with Error response from daemon: received unexpected HTTP status: 500 Internal Server Error to attempts to pull).
What is the expected correct behavior?
In my opinion, there are two issues:
- a warning message should indicate what the actual problem is (which setting is unreadable and why - the settings are stored perfectly fine, so it should be possible to specifically mention that the problem is the decryption)
- there should be a way to recover from this scenario that does not involve database manipulation, which is dangerous and error prone. The most common scenario would likely be that the user would want to overwrite the unreadable settings with new settings encrypted with the current secrets.
Relevant logs and/or screenshots
A typical error log entry in the server logs:
{
"method": "GET",
"path": "/v2/company/dependency_proxy/containers/alpine/manifests/latest",
"format": "json",
"controller": "Groups::DependencyProxyForContainersController",
"action": "manifest",
"status": 500,
"time": "2025-05-26T12:30:17.740Z",
"params": [
{
"key": "group_id",
"value": "company"
},
{
"key": "image",
"value": "alpine"
},
{
"key": "tag",
"value": "latest"
}
],
"correlation_id": "01JW6950GAGX52CJ8X0FEN4AJB",
"meta.caller_id": "Groups::DependencyProxyForContainersController#manifest",
"meta.feature_category": "virtual_registry",
"meta.organization_id": 1,
"meta.remote_ip": "myip",
"meta.user": "group_3_bot_5a23f182fd28836377bb61fe605641a7",
"meta.user_id": 64,
"meta.root_namespace": "company",
"meta.client_id": "user/64",
"remote_ip": "myip",
"user_id": 64,
"username": "group_3_bot_5a23f182fd28836377bb61fe605641a7",
"ua": "docker/27.3.1 go/go1.22.7 git-commit/41ca978 kernel/5.15.146.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \\(linux\\))",
"request_urgency": "low",
"target_duration_s": 5,
"db_count": 14,
"db_write_count": 0,
"db_cached_count": 2,
"db_txn_count": 0,
"db_replica_txn_count": 0,
"db_primary_txn_count": 0,
"db_replica_count": 0,
"db_primary_count": 14,
"db_replica_write_count": 0,
"db_primary_write_count": 0,
"db_replica_cached_count": 0,
"db_primary_cached_count": 2,
"db_replica_wal_count": 0,
"db_primary_wal_count": 0,
"db_replica_wal_cached_count": 0,
"db_primary_wal_cached_count": 0,
"db_replica_txn_max_duration_s": 0.0,
"db_primary_txn_max_duration_s": 0.0,
"db_replica_txn_duration_s": 0.0,
"db_primary_txn_duration_s": 0.0,
"db_replica_duration_s": 0.0,
"db_primary_duration_s": 0.008,
"db_main_txn_count": 0,
"db_ci_txn_count": 0,
"db_main_replica_txn_count": 0,
"db_ci_replica_txn_count": 0,
"db_main_count": 14,
"db_ci_count": 0,
"db_main_replica_count": 0,
"db_ci_replica_count": 0,
"db_main_write_count": 0,
"db_ci_write_count": 0,
"db_main_replica_write_count": 0,
"db_ci_replica_write_count": 0,
"db_main_cached_count": 2,
"db_ci_cached_count": 0,
"db_main_replica_cached_count": 0,
"db_ci_replica_cached_count": 0,
"db_main_wal_count": 0,
"db_ci_wal_count": 0,
"db_main_replica_wal_count": 0,
"db_ci_replica_wal_count": 0,
"db_main_wal_cached_count": 0,
"db_ci_wal_cached_count": 0,
"db_main_replica_wal_cached_count": 0,
"db_ci_replica_wal_cached_count": 0,
"db_main_txn_max_duration_s": 0.0,
"db_ci_txn_max_duration_s": 0.0,
"db_main_replica_txn_max_duration_s": 0.0,
"db_ci_replica_txn_max_duration_s": 0.0,
"db_main_txn_duration_s": 0.0,
"db_ci_txn_duration_s": 0.0,
"db_main_replica_txn_duration_s": 0.0,
"db_ci_replica_txn_duration_s": 0.0,
"db_main_duration_s": 0.008,
"db_ci_duration_s": 0.0,
"db_main_replica_duration_s": 0.0,
"db_ci_replica_duration_s": 0.0,
"cpu_s": 0.110439,
"pid": 1064,
"worker_id": "puma_3",
"rate_limiting_gates": [],
"exception.class": "ActiveRecord::Encryption::Errors::Decryption",
"exception.message": "ActiveRecord::Encryption::Errors::Decryption",
"exception.backtrace": [
"activerecord (7.1.5.1) lib/active_record/encryption/encryptor.rb:58:in `rescue in decrypt'",
"activerecord (7.1.5.1) lib/active_record/encryption/encryptor.rb:52:in `decrypt'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:90:in `block in decrypt'",
"activerecord (7.1.5.1) lib/active_record/encryption/scheme.rb:69:in `with_context'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:15:in `with_context'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:85:in `decrypt'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:36:in `deserialize'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:104:in `block in try_to_deserialize_with_previous_encrypted_types'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:103:in `each'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:103:in `with_index'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:103:in `try_to_deserialize_with_previous_encrypted_types'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:98:in `rescue in decrypt'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:84:in `decrypt'",
"activerecord (7.1.5.1) lib/active_record/encryption/encrypted_attribute_type.rb:36:in `deserialize'",
"activemodel (7.1.5.1) lib/active_model/attribute_set/builder.rb:52:in `block in fetch_value'",
"activemodel (7.1.5.1) lib/active_model/attribute_set/builder.rb:46:in `fetch'",
"activemodel (7.1.5.1) lib/active_model/attribute_set/builder.rb:46:in `fetch_value'",
"activerecord (7.1.5.1) lib/active_record/attribute_methods/read.rb:51:in `_read_attribute'",
"activemodel (7.1.5.1) lib/active_model/attribute_methods.rb:289:in `identity'",
"activerecord (7.1.5.1) lib/active_record/attribute_methods/query.rb:14:in `public_send'",
"activerecord (7.1.5.1) lib/active_record/attribute_methods/query.rb:14:in `query_attribute'",
"activemodel (7.1.5.1) lib/active_model/attribute_methods.rb:289:in `identity?'",
"app/models/dependency_proxy/group_setting.rb:15:in `authorization_header'",
"app/services/dependency_proxy/request_token_service.rb:13:in `execute'",
"app/controllers/groups/dependency_proxy_for_containers_controller.rb:181:in `ensure_token_granted!'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:403:in `block in make_lambda'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:183:in `block (2 levels) in halting_and_conditional'",
"actionpack (7.1.5.1) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:184:in `block in halting_and_conditional'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:598:in `block in invoke_before'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:598:in `each'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:598:in `invoke_before'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:119:in `block in run_callbacks'",
"app/controllers/application_controller.rb:494:in `set_current_admin'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"lib/gitlab/session.rb:11:in `with_session'",
"app/controllers/application_controller.rb:485:in `set_session_storage'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"lib/gitlab/i18n.rb:114:in `with_locale'",
"app/controllers/application_controller.rb:478:in `set_locale'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"app/controllers/application_controller.rb:465:in `set_current_context'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"lib/gitlab/ip_address_state.rb:11:in `with'",
"app/controllers/application_controller.rb:471:in `set_current_ip_address'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:34:in `block in sentry_around_action'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:138:in `with_child_span'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:515:in `with_child_span'",
"sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:141:in `run_callbacks'",
"actionpack (7.1.5.1) lib/abstract_controller/callbacks.rb:258:in `process_action'",
"actionpack (7.1.5.1) lib/action_controller/metal/rescue.rb:25:in `process_action'",
"actionpack (7.1.5.1) lib/action_controller/metal/instrumentation.rb:74:in `block in process_action'",
"activesupport (7.1.5.1) lib/active_support/notifications.rb:206:in `block in instrument'",
"activesupport (7.1.5.1) lib/active_support/notifications/instrumenter.rb:58:in `instrument'",
"activesupport (7.1.5.1) lib/active_support/notifications.rb:206:in `instrument'",
"actionpack (7.1.5.1) lib/action_controller/metal/instrumentation.rb:73:in `process_action'",
"actionpack (7.1.5.1) lib/action_controller/metal/params_wrapper.rb:261:in `process_action'",
"activerecord (7.1.5.1) lib/active_record/railties/controller_runtime.rb:32:in `process_action'",
"actionpack (7.1.5.1) lib/abstract_controller/base.rb:160:in `process'",
"actionview (7.1.5.1) lib/action_view/rendering.rb:40:in `process'",
"actionpack (7.1.5.1) lib/action_controller/metal.rb:227:in `dispatch'",
"actionpack (7.1.5.1) lib/action_controller/metal.rb:307:in `block in dispatch'",
"lib/gitlab/middleware/action_controller_static_context.rb:23:in `call'",
"actionpack (7.1.5.1) lib/action_controller/metal.rb:307:in `dispatch'",
"actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'",
"actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:32:in `serve'",
"actionpack (7.1.5.1) lib/action_dispatch/journey/router.rb:51:in `block in serve'",
"config/initializers/action_dispatch_journey_router.rb:53:in `block in find_routes'",
"config/initializers/action_dispatch_journey_router.rb:25:in `map!'",
"config/initializers/action_dispatch_journey_router.rb:25:in `find_routes'",
"actionpack (7.1.5.1) lib/action_dispatch/journey/router.rb:32:in `serve'",
"actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:882:in `call'",
"gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'",
"flipper (0.28.3) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'",
"flipper (0.28.3) lib/flipper/middleware/memoizer.rb:37:in `call'",
"lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'",
"lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'",
"lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/middleware/go.rb:21:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:83:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/ci/job_token/middleware.rb:11:in `call'",
"batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'",
"rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'",
"apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'",
"warden (1.2.9) lib/warden/manager.rb:36:in `block in call'",
"warden (1.2.9) lib/warden/manager.rb:34:in `catch'",
"warden (1.2.9) lib/warden/manager.rb:34:in `call'",
"rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'",
"rack (2.2.13) lib/rack/tempfile_reaper.rb:15:in `call'",
"rack (2.2.13) lib/rack/etag.rb:27:in `call'",
"rack (2.2.13) lib/rack/conditional_get.rb:27:in `call'",
"rack (2.2.13) lib/rack/head.rb:12:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/http/permissions_policy.rb:36:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/http/content_security_policy.rb:36:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
"rack (2.2.13) lib/rack/session/abstract/id.rb:266:in `context'",
"rack (2.2.13) lib/rack/session/abstract/id.rb:260:in `call'",
"lib/gitlab/middleware/secure_headers.rb:11:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/cookies.rb:689:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:101:in `run_callbacks'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/callbacks.rb:28:in `call'",
"sentry-rails (5.23.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'",
"lib/gitlab/middleware/path_traversal_check.rb:35:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:299:in `with_session_tracking'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:428:in `with_session_tracking'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:89:in `with_scope'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:408:in `with_scope'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'",
"railties (7.1.5.1) lib/rails/rack/logger.rb:24:in `block in call'",
"activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:139:in `block in tagged'",
"activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:39:in `tagged'",
"activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:139:in `tagged'",
"activesupport (7.1.5.1) lib/active_support/broadcast_logger.rb:241:in `method_missing'",
"railties (7.1.5.1) lib/rails/rack/logger.rb:24:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/remote_ip.rb:92:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:15:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"request_store (1.7.0) lib/request_store/middleware.rb:19:in `call'",
"rack (2.2.13) lib/rack/method_override.rb:24:in `call'",
"rack (2.2.13) lib/rack/runtime.rb:22:in `call'",
"rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'",
"rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'",
"rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:44:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/executor.rb:14:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'",
"gitlab-labkit (0.37.0) lib/labkit/middleware/rack.rb:22:in `block in call'",
"gitlab-labkit (0.37.0) lib/labkit/context.rb:35:in `with_context'",
"gitlab-labkit (0.37.0) lib/labkit/middleware/rack.rb:21:in `call'",
"rack (2.2.13) lib/rack/sendfile.rb:110:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/request_id.rb:28:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"railties (7.1.5.1) lib/rails/engine.rb:536:in `call'",
"railties (7.1.5.1) lib/rails/railtie.rb:226:in `public_send'",
"railties (7.1.5.1) lib/rails/railtie.rb:226:in `method_missing'",
"lib/gitlab/middleware/release_env.rb:12:in `call'",
"rack (2.2.13) lib/rack/urlmap.rb:74:in `block in call'",
"rack (2.2.13) lib/rack/urlmap.rb:58:in `each'",
"rack (2.2.13) lib/rack/urlmap.rb:58:in `call'",
"puma (6.5.0) lib/puma/configuration.rb:279:in `call'",
"puma (6.5.0) lib/puma/request.rb:99:in `block in handle_request'",
"puma (6.5.0) lib/puma/thread_pool.rb:389:in `with_force_shutdown'",
"puma (6.5.0) lib/puma/request.rb:98:in `handle_request'",
"puma (6.5.0) lib/puma/server.rb:468:in `process_client'",
"puma (6.5.0) lib/puma/server.rb:249:in `block in run'",
"puma (6.5.0) lib/puma/thread_pool.rb:166:in `block in spawn_thread'"
],
"exception.cause_class": "ActiveRecord::Encryption::Errors::Decryption",
"db_duration_s": 0.01957,
"view_duration_s": 0.0,
"duration_s": 0.12097
}Output of checks
Results of GitLab environment info
Results of GitLab application Check
Above not relevant, happens in fresh setups with default settings.
Possible fixes
Can't provide code fixes for the root problem. The following fixes the symptoms, with the obvious data loss being your settings (which were undecryptable anyways) and which primarily include your Docker Hub login, as well as possibly cached images, which should be ephemeral anyways:
delete from dependency_proxy_group_settings;and in some cases, it seems the image/manifest storage is also affected:
delete from dependency_proxy_blobs;
delete from dependency_proxy_blob_states;
delete from dependency_proxy_manifest_states;
delete from dependency_proxy_manifests;
delete from dependency_proxy_group_settings;I am uncertain whether this may leave orphaned files from images there were cached behind.
Edited by 🤖 GitLab Bot 🤖