Error with prometheus-client-mmap-0.28 "exception data slice is nil"
Problem
I spotted this by accident in an internal review-app on the ruby 3.1 review-app:
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:54:in `sync': unmapped file (IOError)
# [...]
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:38:in 'rescue in initialize': exception data slice is nil at pos 32856 while processing metrics file /metrics/histogram_puma_master-0.db (Prometheus::Client::ParsingError)
Expand full stacktrace
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"bundler: failed to load command: puma (/srv/gitlab/vendor/bundle/ruby/3.1.0/bin/puma)"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:54:in `sync': unmapped file (IOError)"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:54:in `close'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:117:in `unsafe_initialize_file'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:107:in `block in initialize_file'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:106:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:106:in `initialize_file'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:31:in `block in increment'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:30:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:30:in `increment'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:27:in `observe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:65:in `block in observe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/metric.rb:76:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/metric.rb:76:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:65:in `observe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_base.rb:147:in `instance_observe_duration'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `block in instrument_call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `each'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `ensure in instrument_call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:63:in `instrument_call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:270:in `block in send_command'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:269:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:269:in `send_command'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/commands/strings.rb:191:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:336:in `block (2 levels) in read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:336:in `block in read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:468:in `failsafe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:335:in `read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/strategy/local_cache.rb:118:in `read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:331:in `read_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:328:in `block in fetch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:327:in `fetch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:72:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:73:in `block in get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:809:in `block in save_block_result_to_cache'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:808:in `save_block_result_to_cache'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:338:in `fetch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:72:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/adapters/memoizable.rb:69:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:234:in `gate_values'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:108:in `block in enabled?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:379:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/instrumenters/noop.rb:5:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:376:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:105:in `enabled?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:259:in `block in current_feature_value'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:274:in `with_feature'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:255:in `current_feature_value'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:102:in `enabled?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/redis/multi_store.rb:204:in `use_primary_and_secondary_stores?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/redis/multi_store.rb:257:in `close'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:63:in `block (2 levels) in after_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:147:in `shutdown_connections'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:96:in `block in shutdown'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:92:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:92:in `shutdown'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:160:in `reload'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:60:in `block in after_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:53:in `each'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:53:in `after_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:74:in `_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/fork_tracker.rb:7:in `_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:99:in `fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:99:in `spawn_worker'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:81:in `block in spawn_workers'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:74:in `times'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:74:in `spawn_workers'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:163:in `check_workers'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:436:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/launcher.rb:194:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cli.rb:75:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/bin/puma:10:in `<top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/bin/puma:25:in `load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/bin/puma:25:in `<top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `kernel_load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:23:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:492:in `exec'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:34:in `dispatch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:28:in `start'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/gems/3.1.0/gems/bundler-2.4.19/exe/bundle:37:in `block in <top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/gems/3.1.0/gems/bundler-2.4.19/exe/bundle:29:in `<top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/bin/bundle:5:in `load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/bin/bundle:5:in `<main>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:38:in `rescue in initialize': exception data slice is nil at pos 32856 while processing metrics file /metrics/histogram_puma_master-0.db (Prometheus::Client::ParsingError)"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:27:in `initialize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:121:in `new'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:121:in `unsafe_initialize_file'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:107:in `block in initialize_file'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:106:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:106:in `initialize_file'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:31:in `block in increment'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:30:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:30:in `increment'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:26:in `observe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:65:in `block in observe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/metric.rb:76:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/metric.rb:76:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:65:in `observe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_base.rb:147:in `instance_observe_duration'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `block in instrument_call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `each'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `instrument_call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:139:in `block in connect'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:344:in `with_reconnect'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:114:in `connect'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/config/initializers/zz_metrics.rb:45:in `connect'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:417:in `ensure_connected'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:269:in `block in process'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:356:in `logging'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/redis.rb:76:in `block in logging'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/redis.rb:17:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/hub.rb:100:in `with_child_span'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry-ruby.rb:446:in `with_child_span'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/redis.rb:16:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/redis.rb:76:in `logging'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:268:in `process'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:161:in `call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:42:in `instrument_call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:270:in `block in send_command'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:269:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:269:in `send_command'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/commands/strings.rb:191:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:336:in `block (2 levels) in read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:336:in `block in read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:468:in `failsafe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:335:in `read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/strategy/local_cache.rb:118:in `read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:331:in `read_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:328:in `block in fetch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:327:in `fetch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:72:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:73:in `block in get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:809:in `block in save_block_result_to_cache'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:808:in `save_block_result_to_cache'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:338:in `fetch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:72:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/adapters/memoizable.rb:69:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:234:in `gate_values'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:108:in `block in enabled?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:379:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/instrumenters/noop.rb:5:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:376:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:105:in `enabled?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:259:in `block in current_feature_value'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:274:in `with_feature'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:255:in `current_feature_value'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:102:in `enabled?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/redis/multi_store.rb:204:in `use_primary_and_secondary_stores?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/redis/multi_store.rb:257:in `close'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:63:in `block (2 levels) in after_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:147:in `shutdown_connections'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:96:in `block in shutdown'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:92:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:92:in `shutdown'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:160:in `reload'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:60:in `block in after_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:53:in `each'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:53:in `after_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:74:in `_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/fork_tracker.rb:7:in `_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:99:in `fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:99:in `spawn_worker'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:81:in `block in spawn_workers'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:74:in `times'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:74:in `spawn_workers'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:163:in `check_workers'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:436:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/launcher.rb:194:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cli.rb:75:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/bin/puma:10:in `<top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/bin/puma:25:in `load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/bin/puma:25:in `<top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `kernel_load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:23:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:492:in `exec'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:34:in `dispatch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:28:in `start'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/gems/3.1.0/gems/bundler-2.4.19/exe/bundle:37:in `block in <top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/gems/3.1.0/gems/bundler-2.4.19/exe/bundle:29:in `<top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/bin/bundle:5:in `load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/bin/bundle:5:in `<main>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/helper/entry_parser.rb:72:in `block in entries': data slice is nil at pos 32856 (Prometheus::Client::Helper::EntryParser::ParsingError)"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:72:in `each'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:72:in `each'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:72:in `map'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:72:in `read_all_positions'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:34:in `initialize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:121:in `new'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:121:in `unsafe_initialize_file'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:107:in `block in initialize_file'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:106:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:106:in `initialize_file'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:31:in `block in increment'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:30:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:30:in `increment'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:26:in `observe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:65:in `block in observe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/metric.rb:76:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/metric.rb:76:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:65:in `observe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_base.rb:147:in `instance_observe_duration'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `block in instrument_call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `each'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `instrument_call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:139:in `block in connect'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:344:in `with_reconnect'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:114:in `connect'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/config/initializers/zz_metrics.rb:45:in `connect'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:417:in `ensure_connected'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:269:in `block in process'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:356:in `logging'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/redis.rb:76:in `block in logging'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/redis.rb:17:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/hub.rb:100:in `with_child_span'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry-ruby.rb:446:in `with_child_span'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/redis.rb:16:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.8.0/lib/sentry/redis.rb:76:in `logging'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:268:in `process'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:161:in `call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:42:in `instrument_call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:270:in `block in send_command'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:269:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:269:in `send_command'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/commands/strings.rb:191:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:336:in `block (2 levels) in read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:336:in `block in read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:468:in `failsafe'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:335:in `read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/strategy/local_cache.rb:118:in `read_serialized_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:331:in `read_entry'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:328:in `block in fetch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:327:in `fetch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:72:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:73:in `block in get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:809:in `block in save_block_result_to_cache'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:808:in `save_block_result_to_cache'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:338:in `fetch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:72:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/adapters/memoizable.rb:69:in `get'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:234:in `gate_values'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:108:in `block in enabled?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:379:in `block in instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/instrumenters/noop.rb:5:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:376:in `instrument'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:105:in `enabled?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:259:in `block in current_feature_value'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:274:in `with_feature'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:255:in `current_feature_value'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/feature.rb:102:in `enabled?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/redis/multi_store.rb:204:in `use_primary_and_secondary_stores?'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/lib/gitlab/redis/multi_store.rb:257:in `close'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:63:in `block (2 levels) in after_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:147:in `shutdown_connections'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:96:in `block in shutdown'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:92:in `synchronize'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:92:in `shutdown'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:160:in `reload'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:60:in `block in after_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:53:in `each'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:53:in `after_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:74:in `_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/fork_tracker.rb:7:in `_fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:99:in `fork'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:99:in `spawn_worker'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:81:in `block in spawn_workers'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:74:in `times'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:74:in `spawn_workers'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:163:in `check_workers'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:436:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/launcher.rb:194:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cli.rb:75:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/bin/puma:10:in `<top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/bin/puma:25:in `load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/vendor/bundle/ruby/3.1.0/bin/puma:25:in `<top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `kernel_load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:23:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:492:in `exec'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:34:in `dispatch'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:28:in `start'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/gems/3.1.0/gems/bundler-2.4.19/exe/bundle:37:in `block in <top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/site_ruby/3.1.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /usr/lib/ruby/gems/3.1.0/gems/bundler-2.4.19/exe/bundle:29:in `<top (required)>'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/bin/bundle:5:in `load'"}
{"component": "gitlab","subcomponent":"puma.stderr","time":"2023-09-29T09:08:49Z","message":"\tfrom /srv/gitlab/bin/bundle:5:in `<main>'"}
{"component": "gitlab","subcomponent":"application_json","severity":"WARN","time":"2023-09-29T09:08:50.121Z","message":"writing value to /metrics/histogram_puma_master-0.db failed with unmapped file"}
{"component": "gitlab","subcomponent":"application_json","severity":"DEBUG","time":"2023-09-29T09:08:50.122Z","message":"/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:46:in `upsert_entry'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_dict.rb:46:in `write_value'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:139:in `write_value'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:34:in `block in increment'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:30:in `synchronize'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/mmaped_value.rb:30:in `increment'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:26:in `observe'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:65:in `block in observe'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/metric.rb:76:in `synchronize'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/metric.rb:76:in `synchronize'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/prometheus-client-mmap-0.28.0-x86_64-linux/lib/prometheus/client/histogram.rb:65:in `observe'\n/srv/gitlab/lib/gitlab/instrumentation/redis_base.rb:147:in `instance_observe_duration'\n/srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `block in instrument_call'\n/srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `each'\n/srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:56:in `ensure in instrument_call'\n/srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:63:in `instrument_call'\n/srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:270:in `block in send_command'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:269:in `synchronize'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis.rb:269:in `send_command'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/commands/strings.rb:191:in `get'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:336:in `block (2 levels) in read_serialized_entry'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:336:in `block in read_serialized_entry'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:468:in `failsafe'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:335:in `read_serialized_entry'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/strategy/local_cache.rb:118:in `read_serialized_entry'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache/redis_cache_store.rb:331:in `read_entry'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:328:in `block in fetch'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `block in instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `block in instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:327:in `fetch'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:72:in `get'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:73:in `block in get'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:809:in `block in save_block_result_to_cache'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `block in instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `block in instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:783:in `instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:808:in `save_block_result_to_cache'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/cache.rb:338:in `fetch'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-active_support_cache_store-0.26.2/lib/flipper/adapters/active_support_cache_store.rb:72:in `get'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/adapters/memoizable.rb:69:in `get'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:234:in `gate_values'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:108:in `block in enabled?'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:379:in `block in instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/instrumenters/noop.rb:5:in `instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:376:in `instrument'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/flipper-0.26.2/lib/flipper/feature.rb:105:in `enabled?'\n/srv/gitlab/lib/feature.rb:259:in `block in current_feature_value'\n/srv/gitlab/lib/feature.rb:274:in `with_feature'\n/srv/gitlab/lib/feature.rb:255:in `current_feature_value'\n/srv/gitlab/lib/feature.rb:102:in `enabled?'\n/srv/gitlab/lib/gitlab/redis/multi_store.rb:204:in `use_primary_and_secondary_stores?'\n/srv/gitlab/lib/gitlab/redis/multi_store.rb:257:in `close'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:63:in `block (2 levels) in after_fork'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:147:in `shutdown_connections'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:96:in `block in shutdown'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:92:in `synchronize'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:92:in `shutdown'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:160:in `reload'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:60:in `block in after_fork'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:53:in `each'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:53:in `after_fork'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:74:in `_fork'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/fork_tracker.rb:7:in `_fork'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:99:in `fork'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:99:in `spawn_worker'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:81:in `block in spawn_workers'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:74:in `times'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:74:in `spawn_workers'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:163:in `check_workers'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cluster.rb:436:in `run'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/launcher.rb:194:in `run'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/lib/puma/cli.rb:75:in `run'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/puma-6.3.1/bin/puma:10:in `\u003ctop (required)\u003e'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/bin/puma:25:in `load'\n/srv/gitlab/vendor/bundle/ruby/3.1.0/bin/puma:25:in `\u003ctop (required)\u003e'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `load'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `kernel_load'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:23:in `run'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:492:in `exec'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:34:in `dispatch'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:28:in `start'\n/usr/lib/ruby/gems/3.1.0/gems/bundler-2.4.19/exe/bundle:37:in `block in \u003ctop (required)\u003e'\n/usr/lib/ruby/site_ruby/3.1.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'\n/usr/lib/ruby/gems/3.1.0/gems/bundler-2.4.19/exe/bundle:29:in `\u003ctop (required)\u003e'\n/srv/gitlab/bin/bundle:5:in `load'\n/srv/gitlab/bin/bundle:5:in `\u003cmain\u003e'"}
Update
In the last hour or so, I don't see those errors, and the webservice is showing healthy logs. Maybe the problem was fixed in the meantime? (gitlab-org/gitlab!132581 (merged) seems loosely related)
Things to figure out
Mainly why we didn't receive an exception in sentry for this issue? This error should have been visible in https://new-sentry.gitlab.net/organizations/gitlab/projects/gitlab-review-apps/?issuesType=all&project=19.
Edited by David Dieulivol