Redis showing: Cannot allocate memory

Summary

After a while our self hosted gitlab instance stop working and show a 500 error because redis can't allocate memory.

Note:

  • The runner is not in the same machine
  • The VM habe 4 Gb of ram
  • No other appr besides gitlab is running in the server (only the apps required by gitlab-ce)
  • This happend to us before the update too
  • Uname : Linux git.vauxoo.com 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • lsb_release:
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:        16.04
Codename:       xenial

Steps to reproduce

We just leave it and after a while it stop working, only 3 users are using the instance (we still have it in beta stage)

Expected behavior

That gitlab-ce keep working

Actual behavior

The instance stop working, I have to restartit so it can keep working

Relevant logs and/or screenshots

It is consuming all the ram (htop screenshot):

htop

/var/log/gitlab/redis/current

2016-09-14_22:38:56.07592 25141:M 14 Sep 22:38:56.075 * 1 changes in 900 seconds. Saving...
2016-09-14_22:38:56.07594 25141:M 14 Sep 22:38:56.075 # Can't save in background: fork: Cannot allocate memory
2016-09-14_22:39:02.08860 25141:M 14 Sep 22:39:02.088 * 1 changes in 900 seconds. Saving...
2016-09-14_22:39:02.08867 25141:M 14 Sep 22:39:02.088 # Can't save in background: fork: Cannot allocate memory
2016-09-14_22:39:08.10205 25141:M 14 Sep 22:39:08.102 * 1 changes in 900 seconds. Saving...
2016-09-14_22:39:08.10212 25141:M 14 Sep 22:39:08.102 # Can't save in background: fork: Cannot allocate memory
2016-09-14_22:39:14.01696 25141:M 14 Sep 22:39:14.016 * 1 changes in 900 seconds. Saving...
2016-09-14_22:39:14.01702 25141:M 14 Sep 22:39:14.017 # Can't save in background: fork: Cannot allocate memory
2016-09-14_22:39:20.02897 25141:M 14 Sep 22:39:20.028 * 1 changes in 900 seconds. Saving...
2016-09-14_22:39:20.02903 25141:M 14 Sep 22:39:20.029 # Can't save in background: fork: Cannot allocate memory

/var/log/gitlab/gitlab-rails/sidekiq.log

2016-09-14_22:53:53.20906 2016-09-14T22:53:53.209Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:53:58.20977 2016-09-14T22:53:58.209Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:01.34954 2016-09-14T22:54:01.349Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:54:01.34958 2016-09-14T22:54:01.349Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:54:01.35439 2016-09-14T22:54:01.354Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:54:01.35442 2016-09-14T22:54:01.354Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:54:03.21048 2016-09-14T22:54:03.210Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:08.21125 2016-09-14T22:54:08.211Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:13.21194 2016-09-14T22:54:13.211Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:18.21266 2016-09-14T22:54:18.212Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:22.13033 2016-09-14T22:54:22.130Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:54:22.13035 2016-09-14T22:54:22.130Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:54:22.13506 2016-09-14T22:54:22.135Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:54:22.13509 2016-09-14T22:54:22.135Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:54:23.21342 2016-09-14T22:54:23.213Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:28.21415 2016-09-14T22:54:28.214Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:33.21484 2016-09-14T22:54:33.214Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:35.42024 2016-09-14T22:54:35.420Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:54:35.42026 2016-09-14T22:54:35.420Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:54:35.42521 2016-09-14T22:54:35.425Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:54:35.42524 2016-09-14T22:54:35.425Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:54:38.21549 2016-09-14T22:54:38.215Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:43.21614 2016-09-14T22:54:43.216Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:48.21684 2016-09-14T22:54:48.216Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:53.21752 2016-09-14T22:54:53.217Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:54:57.82412 2016-09-14T22:54:57.824Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:54:57.82414 2016-09-14T22:54:57.824Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:54:57.82851 2016-09-14T22:54:57.828Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:54:57.82852 2016-09-14T22:54:57.828Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:54:58.21839 2016-09-14T22:54:58.218Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:55:03.21915 2016-09-14T22:55:03.219Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:55:08.21987 2016-09-14T22:55:08.219Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:55:08.85246 2016-09-14T22:55:08.852Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:55:08.85248 2016-09-14T22:55:08.852Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:55:08.85666 2016-09-14T22:55:08.856Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:55:08.85668 2016-09-14T22:55:08.856Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:55:13.22054 2016-09-14T22:55:13.220Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:55:18.22122 2016-09-14T22:55:18.221Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:55:23.22193 2016-09-14T22:55:23.221Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:55:28.17606 2016-09-14T22:55:28.176Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:55:28.17609 2016-09-14T22:55:28.176Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:55:28.18074 2016-09-14T22:55:28.180Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:55:28.18076 2016-09-14T22:55:28.180Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:55:28.22253 2016-09-14T22:55:28.222Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:55:33.22323 2016-09-14T22:55:33.223Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:55:35.90783 2016-09-14T22:55:35.907Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:55:35.90786 2016-09-14T22:55:35.907Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:55:35.91198 2016-09-14T22:55:35.911Z 14459 TID-ybkvo ERROR: CRON JOB: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2016-09-14_22:55:35.91201 2016-09-14T22:55:35.911Z 14459 TID-ybkvo ERROR: CRON JOB: /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
2016-09-14_22:55:38.22396 2016-09-14T22:55:38.223Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.
2016-09-14_22:55:43.22466 2016-09-14T22:55:43.224Z 14459 TID-yt7yg ERROR: heartbeat: EXECABORT Transaction discarded because of previous errors.

/var/log/gitlab/gitlab-shell/gitlab-shell.log

I, [2016-09-14T22:28:58.262857 #7036]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.03308
I, [2016-09-14T22:28:58.263010 #7036]  INFO -- : gitlab-shell: executing git command <git-receive-pack /var/opt/gitlab/git-data/repositories/vauxoo-dev/orchest.git> for user with key key-4.
I, [2016-09-14T22:28:59.396158 #7045]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04000
E, [2016-09-14T22:28:59.397101 #7045] ERROR -- : GitLab: An unexpected error occurred in writing to Redis: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
I, [2016-09-14T22:29:41.869274 #7147]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.03215
I, [2016-09-14T22:29:41.869421 #7147]  INFO -- : gitlab-shell: executing git command <git-receive-pack /var/opt/gitlab/git-data/repositories/vauxoo-dev/orchest.git> for user with key key-4.
I, [2016-09-14T22:29:42.244884 #7155]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.03133
E, [2016-09-14T22:29:42.245674 #7155] ERROR -- : GitLab: An unexpected error occurred in writing to Redis: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
I, [2016-09-14T22:54:15.546594 #9017]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 2.19090
I, [2016-09-14T22:54:15.547140 #9017]  INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/vauxoo/deployv.git> for user with key key-4.

/var/log/gitlab/unicorn/unicorn_stderr.log

W, [2016-09-14T22:03:46.319338 #2647]  WARN -- : #<Unicorn::HttpServer:0x00000001e32820>: worker (pid: 2647) exceeds memory limit (505832448.0 bytes > 488530952 bytes)
W, [2016-09-14T22:03:46.319456 #2647]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 2647) alive: 255 sec (trial 1)
I, [2016-09-14T22:03:46.475246 #25616]  INFO -- : reaped #<Process::Status: pid 2647 exit 0> worker=1
I, [2016-09-14T22:03:46.481907 #4151]  INFO -- : worker=1 spawned pid=4151
I, [2016-09-14T22:03:46.482226 #4151]  INFO -- : worker=1 ready
W, [2016-09-14T22:12:37.029517 #4151]  WARN -- : #<Unicorn::HttpServer:0x00000001e32820>: worker (pid: 4151) exceeds memory limit (531759104.0 bytes > 488345125 bytes)
W, [2016-09-14T22:12:37.030492 #4151]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 4151) alive: 530 sec (trial 1)
I, [2016-09-14T22:12:37.269258 #25616]  INFO -- : reaped #<Process::Status: pid 4151 exit 0> worker=1
I, [2016-09-14T22:12:37.276932 #5222]  INFO -- : worker=1 spawned pid=5222
I, [2016-09-14T22:12:37.277238 #5222]  INFO -- : worker=1 ready

Output of checks

Results of GitLab application Check

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:check SANITIZE=true)

Checking GitLab Shell ...

GitLab Shell version >= 3.4.0 ? ... OK (3.4.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
4/1 ... ok
5/2 ... ok
5/3 ... ok
6/4 ... ok
6/5 ... ok
6/6 ... ok
7/7 ... ok
7/9 ... ok
7/10 ... ok
6/11 ... ok
7/12 ... rake aborted!
Redis::CommandError: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis/client.rb:114:in `call'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis.rb:729:in `block in setex'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis.rb:57:in `block in synchronize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis.rb:57:in `synchronize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-3.2.2/lib/redis.rb:728:in `setex'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/interface.rb:17:in `setex'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:13:in `block in setex'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:29:in `_marshal'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:13:in `setex'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:9:in `block in setex'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:74:in `namespace'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:9:in `setex'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/ttl.rb:6:in `set'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:5:in `block in set'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:29:in `_marshal'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:5:in `set'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:5:in `block in set'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:74:in `namespace'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:5:in `set'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:223:in `block in write_entry'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `with'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:223:in `write_entry'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:60:in `block in write'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/cache.rb:547:in `block in instrument'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:166:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/cache.rb:547:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:58:in `write'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/cache.rb:588:in `save_block_result_to_cache'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/cache.rb:299:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/repository_cache.rb:20:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:39:in `exists?'
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:840:in `empty_repo?'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:479:in `block in check_repos_hooks_directory_is_link'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/batches.rb:51:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/batches.rb:51:in `block in find_each'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/batches.rb:124:in `find_in_batches'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/batches.rb:50:in `find_each'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:9:in `find_each'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:475:in `check_repos_hooks_directory_is_link'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:348:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
Tasks: TOP => gitlab:check => gitlab:gitlab_shell:check
(See full trace by running task with --trace)

Results of GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:env:info)

System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.3.1p112
Gem Version:    2.6.6
Bundler Version:2.3.0
Rake Version:   10.5.0
Sidekiq Version:4.1.4

GitLab information
Version:        8.11.4
Revision:       b871b76
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://git.vauxoo.com
HTTP Clone URL: https://git.vauxoo.com/some-group/some-project.git
SSH Clone URL:  git@git.vauxoo.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: github

GitLab Shell
Version:        3.4.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:            /opt/gitlab/embedded/bin/git