Docker image shutdown order wrong
Summary
When shutting down a docker container, postgresql and redis shut down before sidekiq/gitlab-rails/unicorn. This can lead to unexpected data loss if sidekiq or other services are still operating when the DB vanishes.
Steps to reproduce
Run the container using
docker run -e GITLAB_OMNIBUS_CONFIG="external_url 'http://git.mydomain.de/'; " -p 8090:80 gitlab/gitlab-ce:9.5.4-ce.0
and once it is healthy, terminate it using
docker stop CONTAINERID
What is the current bug behavior?
postgresql and redis are terminated at the same time as the other runit services.
What is the expected correct behavior?
First nginx+ssh is stopped to prevent new requests from coming in, then unicorn, rails and sidekiq (the latter with a grace period at the end to allow completion of jobs), and only when all three have exited, postgresql and redis get terminated.
Relevant logs and/or screenshots
SIGTERM signal received, try to gracefully shutdown all services...
==> /var/log/gitlab/gitaly/current <==
2017-09-22_15:05:57.52255 time="2017-09-22T15:05:57Z" level=fatal msg="shutting down" error="received signal "terminated""
ok: down: gitaly: 0s, normally up
==> /var/log/gitlab/gitlab-monitor/current <==
2017-09-22_15:05:57.94438 == Sinatra has ended his set (crowd applauds)
2017-09-22_15:05:57.94492 [2017-09-22 15:05:57] INFO going to shutdown ...
2017-09-22_15:05:57.94498 [2017-09-22 15:05:57] INFO WEBrick::HTTPServer#start done.
==> /var/log/gitlab/nginx/current <==
ok: down: gitlab-monitor: 1s, normally up
ok: down: gitlab-workhorse: 0s, normally up
==> /var/log/gitlab/logrotate/current <==
2017-09-22_15:05:58.78844 Received TERM from runit, sending to process group (-PID)
ok: down: logrotate: 1s, normally up
ok: down: nginx: 1s, normally up, want up
ok: down: node-exporter: 0s, normally up
ok: down: postgres-exporter: 1s, normally up
==> /var/log/gitlab/postgresql/current <==
2017-09-22_15:06:00.05578 received TERM from runit, sending INT instead to force quit connections
2017-09-22_15:06:00.05641 LOG: received fast shutdown request
2017-09-22_15:06:00.05643 LOG: aborting any active transactions
2017-09-22_15:06:00.05679 FATAL: terminating connection due to administrator command
2017-09-22_15:06:00.05680 FATAL: terminating connection due to administrator command
2017-09-22_15:06:00.05681 LOG: autovacuum launcher shutting down
2017-09-22_15:06:00.05814 LOG: shutting down
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/-/metrics" for 127.0.0.1 at 2017-09-22 15:06:00 +0000
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
FATAL: terminating connection due to administrator command
==> /var/log/gitlab/postgresql/current <==
2017-09-22_15:06:00.13161 FATAL: the database system is shutting down
2017-09-22_15:06:00.13269 FATAL: the database system is shutting down
==> /var/log/gitlab/gitlab-rails/production.log <==
PG::ConnectionBad (FATAL: the database system is shutting down
):
lib/gitlab/request_context.rb:18:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
==> /var/log/gitlab/postgresql/current <==
2017-09-22_15:06:00.15492 LOG: database system is shut down
ok: down: postgresql: 0s, normally up
==> /var/log/gitlab/prometheus/current <==
2017-09-22_15:06:00.47740 time="2017-09-22T15:06:00Z" level=warning msg="Received SIGTERM, exiting gracefully..." source="main.go:233"
2017-09-22_15:06:00.47744 time="2017-09-22T15:06:00Z" level=info msg="See you next time!" source="main.go:240"
2017-09-22_15:06:00.47757 time="2017-09-22T15:06:00Z" level=info msg="Stopping target manager..." source="targetmanager.go:75"
2017-09-22_15:06:00.47801 time="2017-09-22T15:06:00Z" level=info msg="Stopping rule manager..." source="manager.go:388"
2017-09-22_15:06:00.47813 time="2017-09-22T15:06:00Z" level=info msg="Rule manager stopped." source="manager.go:394"
2017-09-22_15:06:00.47813 time="2017-09-22T15:06:00Z" level=info msg="Stopping notification handler..." source="notifier.go:409"
2017-09-22_15:06:00.47832 time="2017-09-22T15:06:00Z" level=info msg="Stopping local storage..." source="storage.go:450"
2017-09-22_15:06:00.47842 time="2017-09-22T15:06:00Z" level=info msg="Stopping maintenance loop..." source="storage.go:452"
2017-09-22_15:06:00.47864 time="2017-09-22T15:06:00Z" level=info msg="Maintenance loop stopped." source="storage.go:1439"
2017-09-22_15:06:00.47865 time="2017-09-22T15:06:00Z" level=info msg="Stopping series quarantining..." source="storage.go:456"
2017-09-22_15:06:00.47873 time="2017-09-22T15:06:00Z" level=info msg="Series quarantining stopped." source="storage.go:1888"
2017-09-22_15:06:00.47883 time="2017-09-22T15:06:00Z" level=info msg="Stopping chunk eviction..." source="storage.go:460"
2017-09-22_15:06:00.47891 time="2017-09-22T15:06:00Z" level=info msg="Chunk eviction stopped." source="storage.go:1134"
2017-09-22_15:06:00.47899 time="2017-09-22T15:06:00Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
2017-09-22_15:06:00.52158 time="2017-09-22T15:06:00Z" level=info msg="Done checkpointing in-memory metrics and chunks in 42.508857ms." source="persistence.go:665"
2017-09-22_15:06:00.52162 time="2017-09-22T15:06:00Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1526"
2017-09-22_15:06:00.52353 time="2017-09-22T15:06:00Z" level=info msg="Done checkpointing fingerprint mappings in 1.879588ms." source="persistence.go:1549"
2017-09-22_15:06:00.52390 time="2017-09-22T15:06:00Z" level=info msg="Local storage stopped." source="storage.go:477"
ok: down: prometheus: 0s, normally up
==> /var/log/gitlab/redis/current <==
2017-09-22_15:06:00.89936 569:signal-handler (1506092760) Received SIGTERM scheduling shutdown...
2017-09-22_15:06:00.92051 569:M 22 Sep 15:06:00.920 # User requested shutdown...
2017-09-22_15:06:00.92053 569:M 22 Sep 15:06:00.920 * Saving the final RDB snapshot before exiting.
2017-09-22_15:06:00.92481 569:M 22 Sep 15:06:00.924 * DB saved on disk
2017-09-22_15:06:00.92482 569:M 22 Sep 15:06:00.924 * Removing the pid file.
2017-09-22_15:06:00.92492 569:M 22 Sep 15:06:00.924 * Removing the unix socket file.
2017-09-22_15:06:00.92528 569:M 22 Sep 15:06:00.925 # Redis is now ready to exit, bye bye...
==> /var/log/gitlab/sidekiq/current <==
2017-09-22_15:06:00.92659 2017-09-22T15:06:00.926Z 778 TID-or5cmhljs ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.92969 2017-09-22T15:06:00.926Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.93519 2017-09-22T15:06:00.933Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.93520 2017-09-22T15:06:00.933Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.93521 2017-09-22T15:06:00.933Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.93521 2017-09-22T15:06:00.933Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.93523 2017-09-22T15:06:00.933Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.93523 2017-09-22T15:06:00.933Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.93523 2017-09-22T15:06:00.933Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.93531 2017-09-22T15:06:00.933Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.93532 2017-09-22T15:06:00.933Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.93533 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.93533 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.93533 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.93534 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.93534 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.93535 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.93551 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.93552 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.93553 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.93554 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.93554 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.93554 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.93555 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.93574 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.93576 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.93576 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.93576 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.93577 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.93590 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.93591 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.93591 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.93591 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.93592 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.93592 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.93592 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.93593 2017-09-22T15:06:00.934Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.93593 2017-09-22T15:06:00.935Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.93593 2017-09-22T15:06:00.935Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.93593 2017-09-22T15:06:00.935Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.93594 2017-09-22T15:06:00.935Z 778 TID-or5cmhljs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.93768 2017-09-22T15:06:00.928Z 778 TID-or5cmj38k ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.93769 2017-09-22T15:06:00.929Z 778 TID-or5cmiewg ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.93769 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.93769 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.93770 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.93770 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.93770 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.93770 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.93771 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.93771 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.93771 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.93772 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.93773 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.93773 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.93773 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.93774 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.93774 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.93774 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.93774 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.93775 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.93775 2017-09-22T15:06:00.936Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.93775 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.93776 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.93776 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.93776 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.93776 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.93777 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.93777 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.93777 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.93778 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.93778 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.93778 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.93779 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.93779 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.93780 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.93780 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.93780 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.94075 2017-09-22T15:06:00.928Z 778 TID-or5cmj2zo ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.94076 2017-09-22T15:06:00.937Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.94076 2017-09-22T15:06:00.937Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.94109 2017-09-22T15:06:00.937Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.94111 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.94111 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.94111 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.94111 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.94112 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.94112 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.94113 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.94113 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.94113 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.94113 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.94114 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.94114 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.94115 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.94115 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.94115 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.94115 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.94116 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.94117 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.94117 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.94117 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.94117 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.94118 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.94118 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.94118 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.94119 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.94119 2017-09-22T15:06:00.938Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.94119 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.94120 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.94120 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.94120 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.94120 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.94121 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.94121 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.94121 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.94122 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.94122 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.94122 2017-09-22T15:06:00.939Z 778 TID-or5cmj2zo ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.94123 2017-09-22T15:06:00.932Z 778 TID-or5cmih6o ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.94123 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.94123 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.94124 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.94124 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.94124 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.94124 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.94125 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.94125 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.94125 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.94126 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.94126 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.94126 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.94126 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.94127 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.94127 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.94128 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.94128 2017-09-22T15:06:00.939Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.94128 2017-09-22T15:06:00.940Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.94128 2017-09-22T15:06:00.940Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.94128 2017-09-22T15:06:00.940Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.94129 2017-09-22T15:06:00.940Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.94129 2017-09-22T15:06:00.940Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.94130 2017-09-22T15:06:00.940Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.94130 2017-09-22T15:06:00.940Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.94130 2017-09-22T15:06:00.940Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.94131 2017-09-22T15:06:00.940Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.95203 2017-09-22T15:06:00.931Z 778 TID-or5cmic78 ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95204 2017-09-22T15:06:00.941Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95205 2017-09-22T15:06:00.941Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95205 2017-09-22T15:06:00.941Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95205 2017-09-22T15:06:00.941Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.95205 2017-09-22T15:06:00.941Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.95206 2017-09-22T15:06:00.927Z 778 TID-or5cmiehg ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95206 2017-09-22T15:06:00.930Z 778 TID-or5cmj1iw ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95209 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95210 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95210 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95210 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.95210 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.95211 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.95211 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.95211 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.95212 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.95212 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.95212 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.95212 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.95213 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.95213 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.95215 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.95216 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.95216 2017-09-22T15:06:00.942Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.95216 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.95217 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.95217 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.95217 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.95218 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.95224 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.95225 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.95225 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.95225 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.95226 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.95226 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.95227 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.95227 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.95227 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.95227 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.95228 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.95228 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.95228 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.95229 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.95229 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.95229 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.95230 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.95230 2017-09-22T15:06:00.943Z 778 TID-or5cmj1iw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.95231 2017-09-22T15:06:00.930Z 778 TID-or5cmj29k ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95231 2017-09-22T15:06:00.943Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95231 2017-09-22T15:06:00.943Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95232 2017-09-22T15:06:00.943Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95232 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.95232 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.95232 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.95233 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.95233 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.95234 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.95234 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.95234 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.95234 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.95235 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.95235 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.95235 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.95236 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.95236 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.95236 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.95236 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.95237 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.95237 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.95238 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.95238 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.95238 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.95239 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.95239 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.95239 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.95240 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.95240 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.95240 2017-09-22T15:06:00.944Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.95240 2017-09-22T15:06:00.945Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.95241 2017-09-22T15:06:00.945Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.95241 2017-09-22T15:06:00.945Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.95242 2017-09-22T15:06:00.945Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.95242 2017-09-22T15:06:00.945Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.95242 2017-09-22T15:06:00.945Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.95242 2017-09-22T15:06:00.945Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.95243 2017-09-22T15:06:00.945Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.95243 2017-09-22T15:06:00.945Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.95243 2017-09-22T15:06:00.945Z 778 TID-or5cmj29k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.95247 2017-09-22T15:06:00.931Z 778 TID-or5cmj0oc ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95247 2017-09-22T15:06:00.945Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95247 2017-09-22T15:06:00.945Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95248 2017-09-22T15:06:00.945Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95248 2017-09-22T15:06:00.945Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.95248 2017-09-22T15:06:00.945Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.95249 2017-09-22T15:06:00.945Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.95249 2017-09-22T15:06:00.930Z 778 TID-or5cmidp4 ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95249 2017-09-22T15:06:00.945Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95250 2017-09-22T15:06:00.945Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95250 2017-09-22T15:06:00.945Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95250 2017-09-22T15:06:00.945Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.95250 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.95251 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.95251 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.95253 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.95253 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.95253 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.95253 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.95254 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.95254 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.95255 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.95255 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.95255 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.95255 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.95256 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.95256 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.95256 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.95257 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.95257 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.95258 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.95258 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.95258 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.95258 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.95259 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.95259 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.95259 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.95260 2017-09-22T15:06:00.946Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.95273 2017-09-22T15:06:00.947Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.95274 2017-09-22T15:06:00.947Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.95274 2017-09-22T15:06:00.947Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.95275 2017-09-22T15:06:00.947Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.95276 2017-09-22T15:06:00.947Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.95276 2017-09-22T15:06:00.947Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.95276 2017-09-22T15:06:00.947Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.95276 2017-09-22T15:06:00.947Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.95277 2017-09-22T15:06:00.947Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.95277 2017-09-22T15:06:00.947Z 778 TID-or5cmidp4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.95277 2017-09-22T15:06:00.928Z 778 TID-or5cmigl0 ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95278 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95278 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95279 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95279 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.95279 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.95279 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.95280 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.95280 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.95281 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.95281 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.95281 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.95281 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.95282 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.95282 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.95282 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.95283 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.95283 2017-09-22T15:06:00.947Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.95283 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.95283 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.95284 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.95284 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.95284 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.95285 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.95285 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.95285 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.95286 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.95286 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.95287 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.95287 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.95287 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.95287 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.95288 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.95288 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.95288 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.95289 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.95289 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.95289 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.95289 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.95290 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.95290 2017-09-22T15:06:00.948Z 778 TID-or5cmigl0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.95291 2017-09-22T15:06:00.928Z 778 TID-or5cmj1rs ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95291 2017-09-22T15:06:00.948Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95291 2017-09-22T15:06:00.948Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95291 2017-09-22T15:06:00.948Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95292 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.95292 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.95292 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.95293 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.95293 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.95293 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.95294 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.95294 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.95294 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.95294 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.95298 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.95298 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.95298 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.95298 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.95299 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.95299 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.95299 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.95300 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.95300 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.95300 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.95300 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.95301 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.95301 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.95302 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.95302 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.95302 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.95303 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.95303 2017-09-22T15:06:00.949Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.95303 2017-09-22T15:06:00.950Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.95304 2017-09-22T15:06:00.950Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.95304 2017-09-22T15:06:00.950Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.95304 2017-09-22T15:06:00.950Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.95306 2017-09-22T15:06:00.950Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.95306 2017-09-22T15:06:00.950Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.95306 2017-09-22T15:06:00.950Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.95307 2017-09-22T15:06:00.950Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.95307 2017-09-22T15:06:00.950Z 778 TID-or5cmj1rs ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.95308 2017-09-22T15:06:00.927Z 778 TID-or5cmj3z8 ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95308 2017-09-22T15:06:00.950Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95308 2017-09-22T15:06:00.950Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95308 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95309 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.95309 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.95309 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.95310 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.95310 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.95310 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.95345 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.95346 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.95347 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.95348 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.95348 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.95348 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.95349 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.95349 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.95349 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.95349 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.95350 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.95350 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.95351 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.95351 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.95351 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.95351 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.95352 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.95352 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.95352 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.95353 2017-09-22T15:06:00.932Z 778 TID-or5cmj20o ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95353 2017-09-22T15:06:00.953Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95353 2017-09-22T15:06:00.953Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95353 2017-09-22T15:06:00.927Z 778 TID-or5cmicuk ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95443 2017-09-22T15:06:00.954Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95444 2017-09-22T15:06:00.954Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95445 2017-09-22T15:06:00.931Z 778 TID-or5cmig4c ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95452 2017-09-22T15:06:00.954Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95473 2017-09-22T15:06:00.936Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95481 2017-09-22T15:06:00.928Z 778 TID-or5cmhluw ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95502 2017-09-22T15:06:00.951Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.95508 2017-09-22T15:06:00.927Z 778 TID-or5cmj2q8 ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95528 2017-09-22T15:06:00.933Z 778 TID-or5cmj2j0 ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95534 2017-09-22T15:06:00.941Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95540 2017-09-22T15:06:00.955Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95545 2017-09-22T15:06:00.955Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95550 2017-09-22T15:06:00.955Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.95555 2017-09-22T15:06:00.955Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.95560 2017-09-22T15:06:00.955Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.95565 2017-09-22T15:06:00.955Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.95577 2017-09-22T15:06:00.955Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.95599 2017-09-22T15:06:00.954Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95615 2017-09-22T15:06:00.941Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.95634 2017-09-22T15:06:00.953Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95656 2017-09-22T15:06:00.954Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95657 2017-09-22T15:06:00.929Z 778 TID-or5cmj4io ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95685 2017-09-22T15:06:00.954Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95686 2017-09-22T15:06:00.956Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95687 2017-09-22T15:06:00.954Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.95694 2017-09-22T15:06:00.956Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.95724 2017-09-22T15:06:00.955Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95725 2017-09-22T15:06:00.957Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95726 2017-09-22T15:06:00.957Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95726 2017-09-22T15:06:00.929Z 778 TID-or5cmifh0 ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95733 2017-09-22T15:06:00.957Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95738 2017-09-22T15:06:00.957Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.95743 2017-09-22T15:06:00.957Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.95781 2017-09-22T15:06:00.930Z 778 TID-or5cmibg0 ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.95782 2017-09-22T15:06:00.957Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.95783 2017-09-22T15:06:00.955Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.95789 2017-09-22T15:06:00.957Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.96172 2017-09-22T15:06:00.956Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.96173 2017-09-22T15:06:00.957Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.96174 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.96174 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.96174 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.96175 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.96175 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.96175 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.96176 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.96176 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.96176 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.96177 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.96177 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.96177 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.96178 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.96179 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.96179 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.96179 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.96179 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.96180 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.96180 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.96180 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.96181 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.96181 2017-09-22T15:06:00.958Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.96181 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.96181 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.96182 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.96182 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.96182 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.96183 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.96183 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.96183 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.96183 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.96184 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.96184 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.96184 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.96185 2017-09-22T15:06:00.959Z 778 TID-or5cmj20o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.96185 2017-09-22T15:06:00.933Z 778 TID-or5cmida4 ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.96186 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.96186 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.96186 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.96187 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.96187 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.96187 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.96187 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.96188 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.96188 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.96188 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.96189 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.96189 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.96189 2017-09-22T15:06:00.959Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.96189 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.96190 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.96190 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.96190 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.96191 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.96191 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.96191 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.96191 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.96192 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.96192 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.96192 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.96193 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.96193 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.96194 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.96194 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.96194 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.96194 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.96195 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.96195 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.96195 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.96196 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.96196 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.96196 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.96196 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.96197 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.96197 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.96197 2017-09-22T15:06:00.960Z 778 TID-or5cmida4 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.96198 2017-09-22T15:06:00.931Z 778 TID-or5cmj12s ERROR: Error fetching job: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
2017-09-22_15:06:00.96198 2017-09-22T15:06:00.960Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.96198 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.96198 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.96199 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.96199 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.96200 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.96200 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.96200 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.96201 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.96201 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.96201 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.96202 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.96202 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.96202 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.96202 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.96203 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.96203 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.96359 2017-09-22T15:06:00.956Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.96360 2017-09-22T15:06:00.963Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.96360 2017-09-22T15:06:00.963Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.96360 2017-09-22T15:06:00.963Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.96361 2017-09-22T15:06:00.963Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.96361 2017-09-22T15:06:00.957Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.96597 2017-09-22T15:06:00.956Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.96603 2017-09-22T15:06:00.965Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.96609 2017-09-22T15:06:00.966Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.96681 2017-09-22T15:06:00.956Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.96682 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.96683 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.96683 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.96684 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.96684 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.96684 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.96685 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.96685 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.96686 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.96686 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.96686 2017-09-22T15:06:00.955Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.96735 2017-09-22T15:06:00.963Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.96736 2017-09-22T15:06:00.966Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.96736 2017-09-22T15:06:00.967Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.96736 2017-09-22T15:06:00.967Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.96736 2017-09-22T15:06:00.967Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.96737 2017-09-22T15:06:00.967Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.96886 2017-09-22T15:06:00.954Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `__connect_nonblock'
2017-09-22_15:06:00.96887 2017-09-22T15:06:00.966Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.96965 2017-09-22T15:06:00.956Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.96966 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.96966 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.96967 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.96967 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.96968 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.96968 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.96969 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.96969 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.96969 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.96970 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.97043 2017-09-22T15:06:00.945Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.97044 2017-09-22T15:06:00.970Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.97044 2017-09-22T15:06:00.966Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.97091 2017-09-22T15:06:00.968Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.97092 2017-09-22T15:06:00.970Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.97093 2017-09-22T15:06:00.957Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.97113 2017-09-22T15:06:00.937Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.97119 2017-09-22T15:06:00.971Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.97124 2017-09-22T15:06:00.971Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.97133 2017-09-22T15:06:00.971Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.97210 2017-09-22T15:06:00.963Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.97211 2017-09-22T15:06:00.969Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.97211 2017-09-22T15:06:00.971Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.97212 2017-09-22T15:06:00.971Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.97212 2017-09-22T15:06:00.971Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.97212 2017-09-22T15:06:00.971Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.97212 2017-09-22T15:06:00.971Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.97213 2017-09-22T15:06:00.971Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.97214 2017-09-22T15:06:00.971Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.97214 2017-09-22T15:06:00.971Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.97215 2017-09-22T15:06:00.971Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.97215 2017-09-22T15:06:00.971Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.97215 2017-09-22T15:06:00.972Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.97252 2017-09-22T15:06:00.957Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.97254 2017-09-22T15:06:00.972Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.97256 2017-09-22T15:06:00.971Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.97290 2017-09-22T15:06:00.957Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.97352 2017-09-22T15:06:00.968Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.97354 2017-09-22T15:06:00.973Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.97356 2017-09-22T15:06:00.972Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.97381 2017-09-22T15:06:00.970Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.97384 2017-09-22T15:06:00.973Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.97385 2017-09-22T15:06:00.973Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.97441 2017-09-22T15:06:00.970Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.97444 2017-09-22T15:06:00.974Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.97445 2017-09-22T15:06:00.974Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.97446 2017-09-22T15:06:00.974Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.97448 2017-09-22T15:06:00.974Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.97449 2017-09-22T15:06:00.974Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.97457 2017-09-22T15:06:00.972Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.97460 2017-09-22T15:06:00.967Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.97479 2017-09-22T15:06:00.972Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.97505 2017-09-22T15:06:00.972Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.97520 2017-09-22T15:06:00.974Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.97523 2017-09-22T15:06:00.973Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.97524 2017-09-22T15:06:00.973Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.97578 2017-09-22T15:06:00.970Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.97592 2017-09-22T15:06:00.975Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.97595 2017-09-22T15:06:00.975Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.97596 2017-09-22T15:06:00.975Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.97597 2017-09-22T15:06:00.975Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.97599 2017-09-22T15:06:00.975Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.97606 2017-09-22T15:06:00.975Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.97608 2017-09-22T15:06:00.975Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.97609 2017-09-22T15:06:00.966Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/socket.rb:1207:in `connect_nonblock'
2017-09-22_15:06:00.97610 2017-09-22T15:06:00.955Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.97630 2017-09-22T15:06:00.976Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.97670 2017-09-22T15:06:00.976Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.97684 2017-09-22T15:06:00.976Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.97686 2017-09-22T15:06:00.976Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.97688 2017-09-22T15:06:00.976Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.97689 2017-09-22T15:06:00.976Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.97691 2017-09-22T15:06:00.974Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.97699 2017-09-22T15:06:00.974Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.97750 2017-09-22T15:06:00.973Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.97751 2017-09-22T15:06:00.977Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.97751 2017-09-22T15:06:00.977Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.97752 2017-09-22T15:06:00.977Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.97752 2017-09-22T15:06:00.977Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.97752 2017-09-22T15:06:00.975Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:257:in `connect'
2017-09-22_15:06:00.97788 2017-09-22T15:06:00.977Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:308:in `connect'
2017-09-22_15:06:00.97790 2017-09-22T15:06:00.977Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection'
2017-09-22_15:06:00.97792 2017-09-22T15:06:00.971Z 778 TID-or5cmiewg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.97793 2017-09-22T15:06:00.975Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.97810 2017-09-22T15:06:00.976Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.97881 2017-09-22T15:06:00.974Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.97882 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.97882 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.97883 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.97883 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.97883 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.97883 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.97884 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.97884 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.97885 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.97885 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.97885 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.98270 2017-09-22T15:06:00.977Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.98271 2017-09-22T15:06:00.940Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.98271 2017-09-22T15:06:00.978Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.98272 2017-09-22T15:06:00.975Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.98302 2017-09-22T15:06:00.982Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.98303 2017-09-22T15:06:00.982Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.98305 2017-09-22T15:06:00.982Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.98306 2017-09-22T15:06:00.982Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.98314 2017-09-22T15:06:00.982Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.98317 2017-09-22T15:06:00.974Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.98346 2017-09-22T15:06:00.983Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.98373 2017-09-22T15:06:00.983Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.98374 2017-09-22T15:06:00.983Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.98374 2017-09-22T15:06:00.983Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.98374 2017-09-22T15:06:00.975Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.98395 2017-09-22T15:06:00.983Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.98397 2017-09-22T15:06:00.983Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.98399 2017-09-22T15:06:00.983Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.98406 2017-09-22T15:06:00.977Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.98421 2017-09-22T15:06:00.984Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.98438 2017-09-22T15:06:00.976Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect'
2017-09-22_15:06:00.98448 2017-09-22T15:06:00.984Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.98494 2017-09-22T15:06:00.961Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.98497 2017-09-22T15:06:00.984Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.98499 2017-09-22T15:06:00.984Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.98500 2017-09-22T15:06:00.984Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.98501 2017-09-22T15:06:00.984Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.98508 2017-09-22T15:06:00.984Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.98510 2017-09-22T15:06:00.956Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.98531 2017-09-22T15:06:00.978Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.98544 2017-09-22T15:06:00.982Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.98583 2017-09-22T15:06:00.983Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.98585 2017-09-22T15:06:00.985Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.98587 2017-09-22T15:06:00.985Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.98596 2017-09-22T15:06:00.984Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.98629 2017-09-22T15:06:00.970Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.98631 2017-09-22T15:06:00.986Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.98647 2017-09-22T15:06:00.982Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.98681 2017-09-22T15:06:00.984Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.98684 2017-09-22T15:06:00.986Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.98685 2017-09-22T15:06:00.986Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.98692 2017-09-22T15:06:00.986Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.98693 2017-09-22T15:06:00.986Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.98696 2017-09-22T15:06:00.986Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.98698 2017-09-22T15:06:00.985Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.98722 2017-09-22T15:06:00.987Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.98737 2017-09-22T15:06:00.984Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.98755 2017-09-22T15:06:00.987Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.98755 2017-09-22T15:06:00.987Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.98756 2017-09-22T15:06:00.985Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.98764 2017-09-22T15:06:00.975Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.98777 2017-09-22T15:06:00.986Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.98808 2017-09-22T15:06:00.986Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.98809 2017-09-22T15:06:00.987Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.98809 2017-09-22T15:06:00.986Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.98827 2017-09-22T15:06:00.988Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.98828 2017-09-22T15:06:00.988Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.98828 2017-09-22T15:06:00.988Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.98829 2017-09-22T15:06:00.988Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.98857 2017-09-22T15:06:00.985Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.98858 2017-09-22T15:06:00.988Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.98858 2017-09-22T15:06:00.988Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.98859 2017-09-22T15:06:00.987Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.98865 2017-09-22T15:06:00.988Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.98892 2017-09-22T15:06:00.987Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.98894 2017-09-22T15:06:00.988Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.98894 2017-09-22T15:06:00.988Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.98895 2017-09-22T15:06:00.987Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.98912 2017-09-22T15:06:00.985Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.98924 2017-09-22T15:06:00.988Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.98974 2017-09-22T15:06:00.978Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect'
2017-09-22_15:06:00.98975 2017-09-22T15:06:00.989Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect'
2017-09-22_15:06:00.98975 2017-09-22T15:06:00.989Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected'
2017-09-22_15:06:00.98975 2017-09-22T15:06:00.989Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'
2017-09-22_15:06:00.98976 2017-09-22T15:06:00.977Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.98976 2017-09-22T15:06:00.989Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.99000 2017-09-22T15:06:00.987Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.99001 2017-09-22T15:06:00.988Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.99002 2017-09-22T15:06:00.985Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.99013 2017-09-22T15:06:00.990Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.99013 2017-09-22T15:06:00.990Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.99041 2017-09-22T15:06:00.988Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.99042 2017-09-22T15:06:00.990Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.99042 2017-09-22T15:06:00.990Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.99085 2017-09-22T15:06:00.988Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.99088 2017-09-22T15:06:00.990Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.99090 2017-09-22T15:06:00.990Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.99091 2017-09-22T15:06:00.990Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.99093 2017-09-22T15:06:00.990Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.99111 2017-09-22T15:06:00.990Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.99112 2017-09-22T15:06:00.990Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.99127 2017-09-22T15:06:00.990Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.99129 2017-09-22T15:06:00.988Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:00.99144 2017-09-22T15:06:00.988Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.99155 2017-09-22T15:06:00.991Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.99168 2017-09-22T15:06:00.991Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.99177 2017-09-22T15:06:00.990Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.99180 2017-09-22T15:06:00.991Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.99244 2017-09-22T15:06:00.991Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.99254 2017-09-22T15:06:00.989Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.99262 2017-09-22T15:06:00.991Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.99265 2017-09-22T15:06:00.990Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.99288 2017-09-22T15:06:00.989Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'
2017-09-22_15:06:00.99296 2017-09-22T15:06:00.989Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.99299 2017-09-22T15:06:00.989Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.99308 2017-09-22T15:06:00.992Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.99316 2017-09-22T15:06:00.992Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:00.99388 2017-09-22T15:06:00.992Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:00.99399 2017-09-22T15:06:00.992Z 778 TID-or5cmj12s ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:00.99406 2017-09-22T15:06:00.977Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.99409 2017-09-22T15:06:00.992Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.99418 2017-09-22T15:06:00.992Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.99427 2017-09-22T15:06:00.989Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.99430 2017-09-22T15:06:00.991Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'
2017-09-22_15:06:00.99440 2017-09-22T15:06:00.992Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'
2017-09-22_15:06:00.99509 2017-09-22T15:06:00.992Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-22_15:06:00.99520 2017-09-22T15:06:00.991Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.99528 2017-09-22T15:06:00.992Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.99532 2017-09-22T15:06:00.992Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.99541 2017-09-22T15:06:00.992Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:00.99548 2017-09-22T15:06:00.983Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.99552 2017-09-22T15:06:00.993Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.99562 2017-09-22T15:06:00.992Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.99570 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.99633 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.99643 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.99651 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.99654 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.99663 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:00.99672 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:00.99675 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:00.99684 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:00.99692 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:00.99696 2017-09-22T15:06:00.990Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:00.99777 2017-09-22T15:06:00.994Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:00.99787 2017-09-22T15:06:00.994Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:00.99795 2017-09-22T15:06:00.994Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:00.99798 2017-09-22T15:06:00.992Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:209:in `block in call_with_timeout'
2017-09-22_15:06:00.99807 2017-09-22T15:06:00.994Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:280:in `with_socket_timeout'
2017-09-22_15:06:00.99815 2017-09-22T15:06:00.994Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:208:in `call_with_timeout'
2017-09-22_15:06:00.99818 2017-09-22T15:06:00.991Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.99827 2017-09-22T15:06:00.992Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:00.99836 2017-09-22T15:06:00.994Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:00.99840 2017-09-22T15:06:00.994Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:00.99910 2017-09-22T15:06:00.994Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:00.99918 2017-09-22T15:06:00.994Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:00.99921 2017-09-22T15:06:00.994Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:00.99931 2017-09-22T15:06:00.994Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:00.99938 2017-09-22T15:06:00.994Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:00.99942 2017-09-22T15:06:00.994Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:00.99952 2017-09-22T15:06:00.995Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:01.00018 2017-09-22T15:06:00.994Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:01.00028 2017-09-22T15:06:00.995Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:01.00036 2017-09-22T15:06:00.995Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:01.00039 2017-09-22T15:06:00.995Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:01.00048 2017-09-22T15:06:00.991Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:01.00056 2017-09-22T15:06:00.995Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:01.00059 2017-09-22T15:06:00.995Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:01.00068 2017-09-22T15:06:00.978Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:01.00076 2017-09-22T15:06:00.995Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:01.00080 2017-09-22T15:06:00.993Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:01.00146 2017-09-22T15:06:00.991Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:01.00157 2017-09-22T15:06:00.991Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:01.00178 2017-09-22T15:06:00.990Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:01.00187 2017-09-22T15:06:00.996Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:01.00197 2017-09-22T15:06:00.996Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:01.00212 2017-09-22T15:06:00.996Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.00232 2017-09-22T15:06:00.995Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:01.00253 2017-09-22T15:06:00.996Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:01.00259 2017-09-22T15:06:00.996Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:01.00268 2017-09-22T15:06:00.996Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:01.00271 2017-09-22T15:06:00.993Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1137:in `block in _bpop'
2017-09-22_15:06:01.00276 2017-09-22T15:06:00.996Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-22_15:06:01.00285 2017-09-22T15:06:00.996Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:01.00288 2017-09-22T15:06:00.996Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:01.00293 2017-09-22T15:06:00.996Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:01.00302 2017-09-22T15:06:00.996Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:01.00305 2017-09-22T15:06:00.996Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.00310 2017-09-22T15:06:00.989Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.00320 2017-09-22T15:06:00.997Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.00323 2017-09-22T15:06:00.997Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.00328 2017-09-22T15:06:00.997Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.00337 2017-09-22T15:06:00.997Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.00341 2017-09-22T15:06:00.997Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.00348 2017-09-22T15:06:00.997Z 778 TID-or5cmj3z8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.00355 2017-09-22T15:06:00.995Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:01.00364 2017-09-22T15:06:00.997Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:01.00395 2017-09-22T15:06:00.997Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:01.00399 2017-09-22T15:06:00.997Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:01.00404 2017-09-22T15:06:00.994Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-22_15:06:01.00408 2017-09-22T15:06:00.997Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
2017-09-22_15:06:01.00414 2017-09-22T15:06:00.997Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:01.00417 2017-09-22T15:06:00.998Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:01.00423 2017-09-22T15:06:00.996Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:01.00427 2017-09-22T15:06:00.998Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:01.00432 2017-09-22T15:06:00.998Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:01.00436 2017-09-22T15:06:00.996Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:01.00442 2017-09-22T15:06:00.998Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:01.00445 2017-09-22T15:06:00.995Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:01.00449 2017-09-22T15:06:00.987Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.00451 2017-09-22T15:06:00.998Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.00454 2017-09-22T15:06:00.998Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.00456 2017-09-22T15:06:00.998Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.00459 2017-09-22T15:06:00.998Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.00461 2017-09-22T15:06:00.998Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.00466 2017-09-22T15:06:00.998Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.00469 2017-09-22T15:06:00.998Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.00473 2017-09-22T15:06:00.998Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.00477 2017-09-22T15:06:00.995Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:01.00481 2017-09-22T15:06:00.999Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:01.00487 2017-09-22T15:06:00.999Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:01.00490 2017-09-22T15:06:00.999Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:01.00494 2017-09-22T15:06:00.998Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:01.00496 2017-09-22T15:06:00.999Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:01.00500 2017-09-22T15:06:00.999Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:01.00503 2017-09-22T15:06:00.994Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.00505 2017-09-22T15:06:00.997Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:01.00510 2017-09-22T15:06:00.995Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1134:in `_bpop'
2017-09-22_15:06:01.00512 2017-09-22T15:06:00.999Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:1179:in `brpop'
2017-09-22_15:06:01.00516 2017-09-22T15:06:00.999Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:01.00519 2017-09-22T15:06:00.999Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:01.00521 2017-09-22T15:06:00.999Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:01.00524 2017-09-22T15:06:00.999Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:01.00528 2017-09-22T15:06:01.000Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:01.00531 2017-09-22T15:06:01.000Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:01.00535 2017-09-22T15:06:01.000Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.00537 2017-09-22T15:06:00.998Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:01.00539 2017-09-22T15:06:01.000Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:01.00543 2017-09-22T15:06:01.000Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:01.00547 2017-09-22T15:06:01.000Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.00549 2017-09-22T15:06:00.999Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:01.00554 2017-09-22T15:06:01.000Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:01.00556 2017-09-22T15:06:01.000Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:01.00560 2017-09-22T15:06:01.000Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:01.00565 2017-09-22T15:06:01.000Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:01.00568 2017-09-22T15:06:00.998Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:01.00573 2017-09-22T15:06:01.000Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.00575 2017-09-22T15:06:00.998Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in `call_with_namespace'
2017-09-22_15:06:01.00579 2017-09-22T15:06:01.001Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in `method_missing'
2017-09-22_15:06:01.00582 2017-09-22T15:06:00.999Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:01.00586 2017-09-22T15:06:01.001Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.00589 2017-09-22T15:06:01.000Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.00592 2017-09-22T15:06:01.001Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.00594 2017-09-22T15:06:00.996Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:01.00596 2017-09-22T15:06:01.001Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:01.00599 2017-09-22T15:06:01.001Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:01.00601 2017-09-22T15:06:01.001Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:01.00603 2017-09-22T15:06:00.997Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.00607 2017-09-22T15:06:01.002Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.00610 2017-09-22T15:06:00.999Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:01.00612 2017-09-22T15:06:00.993Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.00617 2017-09-22T15:06:01.001Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
2017-09-22_15:06:01.00620 2017-09-22T15:06:01.001Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.00623 2017-09-22T15:06:00.999Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.00625 2017-09-22T15:06:01.002Z 778 TID-or5cmih6o ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.00628 2017-09-22T15:06:01.001Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.00631 2017-09-22T15:06:01.001Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.00635 2017-09-22T15:06:01.003Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.00637 2017-09-22T15:06:01.002Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.00641 2017-09-22T15:06:01.001Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.00644 2017-09-22T15:06:01.000Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.00649 2017-09-22T15:06:01.003Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.00652 2017-09-22T15:06:01.002Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.00657 2017-09-22T15:06:01.002Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.00660 2017-09-22T15:06:01.003Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.00662 2017-09-22T15:06:01.002Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.00664 2017-09-22T15:06:01.002Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.00668 2017-09-22T15:06:00.995Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:01.00672 2017-09-22T15:06:01.005Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.00674 2017-09-22T15:06:01.000Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.00681 2017-09-22T15:06:01.003Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.00702 2017-09-22T15:06:01.003Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.00732 2017-09-22T15:06:00.998Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.00733 2017-09-22T15:06:01.003Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.00734 2017-09-22T15:06:01.003Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.00739 2017-09-22T15:06:01.007Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.00744 2017-09-22T15:06:01.007Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.00749 2017-09-22T15:06:01.007Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.00757 2017-09-22T15:06:01.007Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.00764 2017-09-22T15:06:01.007Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.00795 2017-09-22T15:06:01.006Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.00796 2017-09-22T15:06:01.006Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.00797 2017-09-22T15:06:01.006Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.00803 2017-09-22T15:06:01.007Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.00809 2017-09-22T15:06:01.008Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.00834 2017-09-22T15:06:01.007Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.00839 2017-09-22T15:06:01.002Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:95:in `block in redis'
2017-09-22_15:06:01.00847 2017-09-22T15:06:01.003Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.00859 2017-09-22T15:06:01.003Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.00867 2017-09-22T15:06:01.005Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.00888 2017-09-22T15:06:01.003Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.00893 2017-09-22T15:06:01.007Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.00911 2017-09-22T15:06:01.007Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.00921 2017-09-22T15:06:01.007Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.00937 2017-09-22T15:06:01.009Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.00957 2017-09-22T15:06:01.007Z 778 TID-or5cmj38k ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.00958 2017-09-22T15:06:01.007Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.00987 2017-09-22T15:06:01.008Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.00989 2017-09-22T15:06:01.008Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
2017-09-22_15:06:01.00989 2017-09-22T15:06:01.008Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.00995 2017-09-22T15:06:01.009Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.01000 2017-09-22T15:06:01.009Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.01005 2017-09-22T15:06:01.010Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.01010 2017-09-22T15:06:01.010Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.01016 2017-09-22T15:06:01.010Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01051 2017-09-22T15:06:01.010Z 778 TID-or5cmicuk ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.01052 2017-09-22T15:06:01.008Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01053 2017-09-22T15:06:01.008Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.01053 2017-09-22T15:06:01.009Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.01053 2017-09-22T15:06:01.010Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.01054 2017-09-22T15:06:01.009Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.01059 2017-09-22T15:06:01.010Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.01064 2017-09-22T15:06:01.010Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.01069 2017-09-22T15:06:01.010Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.01074 2017-09-22T15:06:01.010Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01094 2017-09-22T15:06:01.010Z 778 TID-or5cmiehg ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.01098 2017-09-22T15:06:01.009Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.01102 2017-09-22T15:06:01.008Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.01121 2017-09-22T15:06:01.010Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.01141 2017-09-22T15:06:01.009Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
2017-09-22_15:06:01.01152 2017-09-22T15:06:01.010Z 778 TID-or5cmj0oc ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.01157 2017-09-22T15:06:01.008Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.01163 2017-09-22T15:06:01.011Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.01168 2017-09-22T15:06:01.011Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.01174 2017-09-22T15:06:01.011Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.01201 2017-09-22T15:06:01.010Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.01206 2017-09-22T15:06:01.011Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.01209 2017-09-22T15:06:01.011Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.01213 2017-09-22T15:06:01.011Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01227 2017-09-22T15:06:01.011Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.01234 2017-09-22T15:06:01.012Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.01256 2017-09-22T15:06:01.008Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.01270 2017-09-22T15:06:01.012Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.01278 2017-09-22T15:06:01.012Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01290 2017-09-22T15:06:01.011Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
2017-09-22_15:06:01.01300 2017-09-22T15:06:01.012Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
2017-09-22_15:06:01.01310 2017-09-22T15:06:01.012Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
2017-09-22_15:06:01.01313 2017-09-22T15:06:01.012Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.01321 2017-09-22T15:06:01.013Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.01329 2017-09-22T15:06:01.013Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.01334 2017-09-22T15:06:01.013Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.01345 2017-09-22T15:06:01.013Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.01349 2017-09-22T15:06:01.013Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.01363 2017-09-22T15:06:01.013Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01367 2017-09-22T15:06:01.013Z 778 TID-or5cmj2j0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.01380 2017-09-22T15:06:01.010Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01383 2017-09-22T15:06:01.013Z 778 TID-or5cmic78 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.01388 2017-09-22T15:06:01.009Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:92:in `redis'
2017-09-22_15:06:01.01399 2017-09-22T15:06:01.010Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.01402 2017-09-22T15:06:01.013Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/fetch.rb:36:in `retrieve_work'
2017-09-22_15:06:01.01414 2017-09-22T15:06:01.013Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:91:in `get_one'
2017-09-22_15:06:01.01417 2017-09-22T15:06:01.013Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:101:in `fetch'
2017-09-22_15:06:01.01422 2017-09-22T15:06:01.013Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.01432 2017-09-22T15:06:01.013Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.01436 2017-09-22T15:06:01.011Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.01443 2017-09-22T15:06:01.013Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01452 2017-09-22T15:06:01.013Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-22_15:06:01.01462 2017-09-22T15:06:01.014Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'
2017-09-22_15:06:01.01463 2017-09-22T15:06:01.014Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01464 2017-09-22T15:06:01.012Z 778 TID-or5cmig4c ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.01464 2017-09-22T15:06:01.014Z 778 TID-or5cmifh0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.01464 2017-09-22T15:06:01.014Z 778 TID-or5cmhluw ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.01468 2017-09-22T15:06:01.013Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01468 2017-09-22T15:06:01.014Z 778 TID-or5cmj4io ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.01469 2017-09-22T15:06:01.012Z 778 TID-or5cmibg0 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-09-22_15:06:01.01469 2017-09-22T15:06:01.012Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'
2017-09-22_15:06:01.01469 2017-09-22T15:06:01.014Z 778 TID-or5cmj2q8 ERROR: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'
ok: down: redis: 1s, normally up
ok: down: redis-exporter: 0s, normally up
2017-09-22_15:06:01.74346 2017-09-22T15:06:01.743Z 778 TID-or5fe2rco INFO: Shutting down
2017-09-22_15:06:01.74366 2017-09-22T15:06:01.743Z 778 TID-or5cmhlc0 INFO: Scheduler exiting...
2017-09-22_15:06:01.74396 2017-09-22T15:06:01.743Z 778 TID-or5fe2rco INFO: Terminating quiet workers
2017-09-22_15:06:01.74424 2017-09-22T15:06:01.744Z 778 TID-or5cmo40w INFO: Scheduler exiting...
2017-09-22_15:06:02.24486 2017-09-22T15:06:02.244Z 778 TID-or5fe2rco INFO: Bye!
ok: down: sidekiq: 0s, normally up
==> /var/log/gitlab/sshd/current <==
2017-09-22_15:06:02.58573 Received signal 15; terminating.
ok: down: sshd: 1s, normally up
==> /var/log/gitlab/unicorn/current <==
2017-09-22_15:06:03.00872 Received TERM from runit, sending to process group (-PID)
2017-09-22_15:06:03.01007 Terminated
2017-09-22_15:06:03.01031 forwarding TERM to unicorn master 759
2017-09-22_15:06:03.01040 forwarding CONT to unicorn master 759
2017-09-22_15:06:03.01073 wrapper for unicorn master 759 exiting
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
I, [2017-09-22T15:06:03.118851 #759] INFO -- : reaped #<Process::Status: pid 891 exit 0> worker=0
I, [2017-09-22T15:06:03.118995 #759] INFO -- : reaped #<Process::Status: pid 894 exit 0> worker=1
I, [2017-09-22T15:06:03.119088 #759] INFO -- : reaped #<Process::Status: pid 897 exit 0> worker=2
I, [2017-09-22T15:06:03.119301 #759] INFO -- : master complete
ok: down: unicorn: 0s, normally up
Output of checks
(checks not applicable at the time the bug appears, but it's a fresh instance anyway)
Results of GitLab environment info
(checks not applicable at the time the bug appears, but it's a fresh instance anyway)
Results of GitLab application Check
(checks not applicable at the time the bug appears, but it's a fresh instance anyway)
Possible fixes
Use proper expression of services in the runit config files.