Redis::CommandError: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE
- After upgrading to v11.11.3 our self-hosted GitLab instance keep showing 500 pages to users.
- Following is the error printed in
gitlab-ctl tail
2019-06-14_11:05:06.02289 2019-06-14T11:05:06.022Z 21697 TID-owjmuu0ct WARN: Redis::CommandError: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
2019-06-14_11:05:06.02290 2019-06-14T11:05:06.022Z 21697 TID-owjmuu0ct WARN: /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis/client.rb:121:in `call'
2019-06-14_11:05:06.02291 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2019-06-14_11:05:06.02292 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis/client.rb:104:in `block in connect'
2019-06-14_11:05:06.02293 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis/client.rb:293:in `with_reconnect'
2019-06-14_11:05:06.02294 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis/client.rb:100:in `connect'
2019-06-14_11:05:06.02295 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/zz_metrics.rb:202:in `connect'
2019-06-14_11:05:06.02296 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis/client.rb:364:in `ensure_connected'
2019-06-14_11:05:06.02300 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis/client.rb:221:in `block in process'
2019-06-14_11:05:06.02301 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis/client.rb:306:in `logging'
2019-06-14_11:05:06.02302 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis/client.rb:220:in `process'
2019-06-14_11:05:06.02302 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis/client.rb:120:in `call'
2019-06-14_11:05:06.02303 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2019-06-14_11:05:06.02304 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis.rb:494:in `block in del'
2019-06-14_11:05:06.02305 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis.rb:58:in `block in synchronize'
2019-06-14_11:05:06.02306 /opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
2019-06-14_11:05:06.02307 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis.rb:58:in `synchronize'
2019-06-14_11:05:06.02308 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-3.3.5/lib/redis.rb:493:in `del'
2019-06-14_11:05:06.02309 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-namespace-1.6.0/lib/redis/namespace.rb:442:in `call_with_namespace'
2019-06-14_11:05:06.02313 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/redis-namespace-1.6.0/lib/redis/namespace.rb:328:in `method_missing'
2019-06-14_11:05:06.02314 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status.rb:43:in `block in unset'
2019-06-14_11:05:06.02315 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq.rb:97:in `block in redis'
2019-06-14_11:05:06.02316 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
2019-06-14_11:05:06.02317 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
2019-06-14_11:05:06.02318 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
2019-06-14_11:05:06.02319 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
2019-06-14_11:05:06.02320 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
2019-06-14_11:05:06.02321 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq.rb:94:in `redis'
2019-06-14_11:05:06.02325 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status.rb:42:in `unset'
2019-06-14_11:05:06.02326 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:9:in `call'
2019-06-14_11:05:06.02327 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2019-06-14_11:05:06.02328 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/correlation_logger.rb:10:in `block in call'
2019-06-14_11:05:06.02329 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/gitlab-labkit-0.2.0/lib/labkit/correlation/correlation_id.rb:18:in `use_id'
2019-06-14_11:05:06.02330 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/correlation_logger.rb:9:in `call'
2019-06-14_11:05:06.02331 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2019-06-14_11:05:06.02332 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'
2019-06-14_11:05:06.02332 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2019-06-14_11:05:06.02333 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `call'
2019-06-14_11:05:06.02337 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2019-06-14_11:05:06.02340 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:18:in `call'
2019-06-14_11:05:06.02341 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2019-06-14_11:05:06.02342 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sentry-raven-2.9.0/lib/raven/integrations/sidekiq.rb:9:in `call'
2019-06-14_11:05:06.02343 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2019-06-14_11:05:06.02344 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
2019-06-14_11:05:06.02345 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
2019-06-14_11:05:06.02346 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
2019-06-14_11:05:06.02347 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
2019-06-14_11:05:06.02351 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
2019-06-14_11:05:06.02353 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'
2019-06-14_11:05:06.02353 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.1.7/lib/active_support/execution_wrapper.rb:85:in `wrap'
2019-06-14_11:05:06.02354 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.1.7/lib/active_support/reloader.rb:68:in `block in wrap'
2019-06-14_11:05:06.02354 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.1.7/lib/active_support/execution_wrapper.rb:85:in `wrap'
2019-06-14_11:05:06.02354 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.1.7/lib/active_support/reloader.rb:67:in `wrap'
2019-06-14_11:05:06.02355 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'
2019-06-14_11:05:06.02355 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
2019-06-14_11:05:06.02355 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
2019-06-14_11:05:06.02356 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
2019-06-14_11:05:06.02357 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
2019-06-14_11:05:06.02357 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
2019-06-14_11:05:06.02357 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
2019-06-14_11:05:06.02357 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
2019-06-14_11:05:06.02358 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
2019-06-14_11:05:06.02358 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
2019-06-14_11:05:06.02358 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
2019-06-14_11:05:06.02358 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
2019-06-14_11:05:06.02360 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
2019-06-14_11:05:06.02360 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
2019-06-14_11:05:06.02360 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
2019-06-14_11:05:06.02360 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
Edited by damith