Redis::CommandError: ERR unknown command
Summary
Redis fails with Redis::CommandError: ERR unknown command
. This has been reported by two customers:
-
https://gitlab.zendesk.com/agent/tickets/52937
8.14.3-ee.0
-
https://gitlab.zendesk.com/agent/tickets/53273
8.14.4-ee
Steps to reproduce
Unknown. Intermittent failures when Sidekiq workers process jobs.
Relevant logs and/or screenshots
- GitLab v8.14.3-ee - Omnibus
- Redis 3.2.5
Sidekiq logs
UpdateMergeRequestsWorker
2016-12-14_09:52:24.93127 2016-12-14T09:52:24.931Z 48547 TID-owlpk16pg WARN: {"class":"UpdateMergeRequestsWorker","args":
[43,148,"f97cc70f73e9ca30e3a9a733eff3859b3ce38115","0a41ed52c2c1b64b20553d8cff3bd9415807e95a","refs/heads/release-9068-223-
kyc"],"retry":3,"queue":"update_merge_requests","jid":"4c6604de1de1272ce7b8eeb3","created_at":1481709028.1302595,
"enqueued_at":1481709144.8713293,"error_message":"ERR unknown command
'<86>Dec'","error_class":"Redis::CommandError","failed_at":1481709028.338645,"retry_count":2,"retried_at":1481709144.9306111}
ProcessCommitWorker
{"class":"ProcessCommitWorker","args":[43,136,{"id":"1e1ef9e6dfafb2effec158a690a2238a5511213f","message":"Review + fix and refacto
headers\n","parent_ids":["6e9595036d429f87b71f45307d1094b0dfea5e3f"],"authored_date":"2016-11-30 11:52:42
+0100","author_name":"Example","author_email":"example@example.com","committed_date":"2016-11-30 11:52:42
+0100","committer_name":"Example,"committer_email":"example@example.com"},false],"retry":3,"queue":"process_commit","jid":"8ab27b0a0c3592e2bd
390e74","created_at":1481643113.1915205,"enqueued_at":1481643113.1915545,"error_message":"ERR unknown command
'<86>Dec'","error_class":"Redis::CommandError","failed_at":1481643113.711671,"retry_count":0}
Full stacktrace
2016-12-14_09:52:24.93129 2016-12-14T09:52:24.931Z 48547 TID-owlpk16pg WARN: Redis::CommandError: ERR unknown command '<86>Dec'
2016-12-14_09:52:24.93140 2016-12-14T09:52:24.931Z 48547 TID-owlpk16pg WARN: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-12-14_09:52:24.93141 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis.rb:822:in `block in get'
2016-12-14_09:52:24.93142 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis.rb:57:in `block in synchronize'
2016-12-14_09:52:24.93142 /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2016-12-14_09:52:24.93143 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis.rb:57:in `synchronize'
2016-12-14_09:52:24.93143 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis.rb:821:in `get'
2016-12-14_09:52:24.93145 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.2.0/lib/redis/store/interface.rb:5:in `get'
2016-12-14_09:52:24.93145 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.2.0/lib/redis/store/marshalling.rb:17:in `get'
2016-12-14_09:52:24.93146 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.2.0/lib/redis/store/namespace.rb:23:in `block in get'
2016-12-14_09:52:24.93146 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.2.0/lib/redis/store/namespace.rb:84:in `namespace'
2016-12-14_09:52:24.93147 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.2.0/lib/redis/store/namespace.rb:23:in `get'
2016-12-14_09:52:24.93148 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-activesupport-5.0.1/lib/active_support/cache/redis_store.rb:238:in `block in read_entry'
2016-12-14_09:52:24.93148 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:64:in `block (2 levels) in with'
2016-12-14_09:52:24.93148 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:63:in `handle_interrupt'
2016-12-14_09:52:24.93149 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:63:in `block in with'
2016-12-14_09:52:24.93149 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:60:in `handle_interrupt'
2016-12-14_09:52:24.93150 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:60:in `with'
2016-12-14_09:52:24.93150 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-activesupport-5.0.1/lib/active_support/cache/redis_store.rb:218:in `with'
2016-12-14_09:52:24.93150 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-activesupport-5.0.1/lib/active_support/cache/redis_store.rb:238:in `read_entry'
2016-12-14_09:52:24.93151 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/cache.rb:558:in `block in find_cached_entry'
2016-12-14_09:52:24.93151 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/cache.rb:547:in `block in instrument'
2016-12-14_09:52:24.93152 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:164:in `block in instrument'
2016-12-14_09:52:24.93152 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2016-12-14_09:52:24.93153 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:164:in `instrument'
2016-12-14_09:52:24.93153 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/cache.rb:547:in `instrument'
2016-12-14_09:52:24.93153 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/cache.rb:556:in `find_cached_entry'
2016-12-14_09:52:24.93154 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/cache.rb:293:in `fetch'
2016-12-14_09:52:24.93155 /opt/gitlab/embedded/service/gitlab-rails/lib/repository_cache.rb:20:in `fetch'
2016-12-14_09:52:24.93155 /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:1309:in `cache_method_output'
2016-12-14_09:52:24.93155 /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:54:in `block in cache_method'
2016-12-14_09:52:24.93156 /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:912:in `avatar_in_git'
2016-12-14_09:52:24.93156 /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:918:in `avatar_url'
2016-12-14_09:52:24.93156 /opt/gitlab/embedded/service/gitlab-rails/app/models/ee/geo_aware_avatar.rb:11:in `avatar_url'
2016-12-14_09:52:24.93157 /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:1130:in `hook_attrs'
2016-12-14_09:52:24.93157 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/data_builder/push.rb:57:in `build'
2016-12-14_09:52:24.93157 /opt/gitlab/embedded/service/gitlab-rails/app/workers/update_merge_requests_worker.rb:14:in `perform'
2016-12-14_09:52:24.93158 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:155:in `execute_job'
2016-12-14_09:52:24.93159 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:136:in `block (3 levels) in process'
2016-12-14_09:52:24.93159 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
2016-12-14_09:52:24.93160 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
2016-12-14_09:52:24.93160 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2016-12-14_09:52:24.93161 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call'
2016-12-14_09:52:24.93161 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2016-12-14_09:52:24.93161 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
2016-12-14_09:52:24.93162 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2016-12-14_09:52:24.93162 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
2016-12-14_09:52:24.93163 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2016-12-14_09:52:24.93163 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
2016-12-14_09:52:24.93164 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/logging.rb:32:in `with_context'
2016-12-14_09:52:24.93164 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
2016-12-14_09:52:24.93166 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2016-12-14_09:52:24.93166 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
2016-12-14_09:52:24.93166 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:131:in `block (2 levels) in process'
2016-12-14_09:52:24.93167 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:171:in `stats'
2016-12-14_09:52:24.93167 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:130:in `block in process'
2016-12-14_09:52:24.93168 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq.rb:35:in `block in <module:Sidekiq>'
2016-12-14_09:52:24.93169 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:122:in `process'
2016-12-14_09:52:24.93169 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:81:in `process_one'
2016-12-14_09:52:24.93169 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:69:in `run'
2016-12-14_09:52:24.93170 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/util.rb:17:in `watchdog'
2016-12-14_09:52:24.93170 /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/util.rb:25:in `block in safe_thread'
Edited by 🤖 GitLab Bot 🤖