sidekiq workers in docker image get often stuck

Summary

Using the official gitlab-ce image with redis and pgsql in another docker container, sidekiq gets stuck and does not process any work, leading to a detachment between the GUI and git repository state.

Steps to reproduce

Install the docker image with shared pgsql/redis, import some couple projects and e.g. start a merge request merge. The projects and PRs need not be huge.

What is the current bug behavior?

One of the sidekiq workers gets stuck, api/v3/sidekiq/queue_metrics results in a HTTP 500, no work gets done (e.g. webhooks of projects are not being run, permissions not updated,...)

Only after kill -9 on the sidekiq process all jobs seem to be run again and successfully disappear.

What is the expected correct behavior?

The API endpoint should in all cases stay reachable. The sidekiq log should print a WARN when a job gets stuck (e.g. longer than 60 seconds), and ideally hard-reset the sidekiq process.

Relevant logs and/or screenshots

This here is an extremely long running job - it would be helpful if the sidekiq log at least could state what the job is supposed to be doing so one can investigate the cause for locking bugs.

2017-02-27_10:11:24.11633 2017-02-27T10:11:24.116Z 1575 TID-otufg9jn0 ProcessCommitWorker JID-2c530abd6116f7e8d82d0b2c INFO: start
2017-02-27_10:26:52.23486 2017-02-27T10:26:52.234Z 1575 TID-otufg9jn0 ProcessCommitWorker JID-2c530abd6116f7e8d82d0b2c INFO: done: 928.118 sec

Output of checks

All checks run fine.

Results of GitLab environment info

System information
System:
Current User:  	git
Using RVM:     	no
Ruby Version:  	2.3.3p222
Gem Version:   	2.6.6
Bundler Version:1.13.7
Rake Version:  	10.5.0
Redis Version: 	3.2.5
Sidekiq Version:4.2.7

GitLab information
Version:       	8.17.0
Revision:      	77254ab
Directory:     	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:    	postgresql
URL:   		http://git.xxx.de
HTTP Clone URL:	http://git.xxx.de/some-group/some-project.git
SSH Clone URL: 	[git@git.xxx.de:yyy]:some-group/some-project.git
Using LDAP:    	yes
Using Omniauth:	no

GitLab Shell
Version:       	4.1.1
Repository storage paths:
- default:     	/gitlab-data/git-data/repositories
Hooks: 		/opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:   		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 4.1.1 ? ... OK (4.1.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
4/1 ... ok
4/3 ... ok
4/4 ... ok
4/5 ... ok
4/6 ... ok
4/7 ... ok
4/8 ... ok
4/9 ... ok
4/10 ... ok
4/11 ... ok
4/12 ... ok
1/13 ... ok
4/14 ... ok
4/15 ... ok
21/16 ... ok
21/17 ... repository is empty
26/18 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /gitlab-data/home/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
xxxxxxx
Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
4/1 ... yes
4/3 ... yes
4/4 ... yes
4/5 ... yes
4/6 ... yes
4/7 ... yes
4/8 ... yes
4/9 ... yes
4/10 ... yes
4/11 ... yes
4/12 ... yes
1/13 ... yes
4/14 ... yes
4/15 ... yes
21/16 ... yes
21/17 ... yes
26/18 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.10.2)
Active users: 44

Checking GitLab ... Finished