Gitaly generates zombie process after upgrade to 9.5.0

Summary

Gitaly generates a zombie child process after upgrade to 9.5.0.

My supervision (Shinken) raised alarms just after the upgrade. This behavior appears on two instances of Gitlab-CE, one on Ubuntu 14.04 and the other one on Ubuntu 16.04.

Steps to reproduce

On an Omnibus install:

sudo gitlab-ctl stop 
sudo gitlab-ctl start
ps axf |grep -A5 -B5 Z

What is the current bug behavior?

Gitaly generates a zombie child process right after gitlab-ctl start

What is the expected correct behavior?

No zombie

Relevant logs and/or screenshots

See the sixth line with a state of Z, process is [bundle] <defunct>:

# ps axf |grep -A5 -B5 Z
 1123 ?        S      0:00  |   \_ svlogd -tt /var/log/gitlab/postgres-exporter
 3866 ?        Ssl    0:00  |   \_ /opt/gitlab/embedded/bin/postgres_exporter -web.listen-address=localhost:9187 -extend.query-path=/var/opt/gitlab/postgres-exporter/queries.yaml
 1103 ?        Ss     0:00  \_ runsv gitaly
 1118 ?        S      0:00  |   \_ svlogd -tt /var/log/gitlab/gitaly
 3823 ?        Ssl    0:00  |   \_ /opt/gitlab/embedded/bin/gitaly /var/opt/gitlab/gitaly/config.toml
 3827 ?        Z      0:02  |       \_ [bundle] <defunct>
 1104 ?        Ss     0:00  \_ runsv mattermost
 1137 ?        S      0:00  |   \_ svlogd -tt /var/log/gitlab/mattermost
 3848 ?        Ssl    0:00  |   \_ /opt/gitlab/embedded/bin/mattermost --config /var/opt/gitlab/mattermost/config.json
 1105 ?        Ss     0:00  \_ runsv gitlab-workhorse
 1120 ?        S      2:14  |   \_ svlogd -tt /var/log/gitlab/gitlab-workhorse

I couldn't find related data with a gitlab-ctl tail. I also looked in the issues of Gitlab-CE, gitaly and omnibus (including the tags bug and regression), but found nothing relevant.

Output of checks

Results of GitLab environment info --> Ubuntu 14.04

Expand for output related to GitLab environment info 
System information
System:		Ubuntu 14.04
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	12.0.0
Redis Version:	3.2.5
Git Version:	2.13.5
Sidekiq Version:5.0.4
Go Version:	unknown

GitLab information
Version:	9.5.0
Revision:	77bfdac
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://gitlab.taqamaqa.com
HTTP Clone URL:	https://gitlab.taqamaqa.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.taqamaqa.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: github


GitLab Shell
Version:	5.8.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab environment info --> Ubuntu 16.04

Expand for output related to GitLab environment info 
System information
System:		Ubuntu 16.04
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	12.0.0
Redis Version:	3.2.5
Git Version:	2.13.5
Sidekiq Version:5.0.4
Go Version:	unknown

GitLab information
Version:	9.5.0
Revision:	77bfdac
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://gitlab.adullact.net
HTTP Clone URL:	https://gitlab.adullact.net/some-group/some-project.git
SSH Clone URL:	git@gitlab.adullact.net:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no


GitLab Shell
Version:	5.8.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check --> Ubuntu 14.04

Expand for output related to the GitLab application check 
Checking GitLab Shell ...

GitLab Shell version >= 5.8.0 ? ... OK (5.8.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
2/3 ... ok
2/4 ... ok
2/5 ... ok
2/7 ... ok
2/8 ... ok
2/9 ... ok
2/10 ... ok
3/11 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Reply by email ...


Reply by email is disabled in config/gitlab.yml


Checking Reply by email ... Finished


Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/3 ... yes
2/4 ... yes
2/5 ... yes
2/7 ... yes
2/8 ... yes
2/9 ... yes
2/10 ... yes
3/11 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.3)
Git version >= 2.7.3 ? ... yes (2.13.5)
Active users: ... 5


Checking GitLab ... Finished

Results of GitLab application Check --> Ubuntu 16.04

Expand for output related to the GitLab application check 
Checking GitLab Shell ...

GitLab Shell version >= 5.8.0 ? ... OK (5.8.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
43/4 ... repository is empty
110/5 ... ok
77/6 ... repository is empty
5/7 ... ok
110/8 ... ok
70/9 ... ok
110/10 ... ok
4/12 ... ok
4/13 ... ok
110/14 ... ok
110/15 ... ok
191/17 ... repository is empty
110/18 ... ok
110/19 ... ok
110/20 ... repository is empty
217/21 ... ok
60/22 ... ok
220/23 ... repository is empty
219/24 ... ok
70/25 ... repository is empty
2/26 ... ok
3/27 ... ok
110/28 ... ok
110/29 ... ok
110/31 ... ok
161/32 ... ok
110/33 ... ok
110/34 ... ok
161/36 ... repository is empty
345/37 ... ok
416/39 ... ok
2/40 ... ok
110/41 ... repository is empty
110/44 ... ok
4/46 ... ok
426/47 ... ok
110/48 ... ok
426/49 ... ok
426/50 ... ok
6/51 ... ok
110/52 ... ok
72/53 ... ok
2/54 ... repository is empty
551/56 ... repository is empty
569/58 ... ok
110/59 ... ok
70/60 ... ok
110/61 ... ok
110/63 ... ok
110/64 ... ok
110/65 ... ok
110/66 ... ok
110/67 ... ok
110/68 ... ok
110/69 ... ok
110/70 ... ok
110/71 ... ok
627/72 ... ok
627/73 ... ok
627/74 ... ok
627/75 ... ok
627/76 ... ok
627/77 ... ok
627/78 ... ok
627/79 ... ok
627/80 ... ok
627/81 ... ok
578/82 ... ok
110/83 ... ok
110/84 ... ok
110/85 ... ok
110/86 ... ok
110/87 ... ok
110/88 ... ok
5/89 ... ok
711/90 ... ok
392/91 ... ok
5/92 ... repository is empty
548/93 ... ok
2/94 ... ok
548/95 ... ok
743/96 ... ok
2/97 ... ok
110/98 ... ok
2/99 ... repository is empty
110/101 ... ok
802/102 ... repository is empty
802/103 ... ok
855/104 ... ok
110/106 ... ok
110/108 ... ok
359/109 ... ok
110/110 ... ok
110/112 ... ok
2/113 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Reply by email ...


Reply by email is disabled in config/gitlab.yml


Checking Reply by email ... Finished


Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
43/4 ... yes
110/5 ... yes
77/6 ... yes
5/7 ... yes
110/8 ... yes
70/9 ... yes
110/10 ... yes
4/12 ... yes
4/13 ... yes
110/14 ... yes
110/15 ... yes
191/17 ... yes
110/18 ... yes
110/19 ... yes
110/20 ... yes
217/21 ... yes
60/22 ... yes
220/23 ... yes
219/24 ... yes
70/25 ... yes
2/26 ... yes
3/27 ... yes
110/28 ... yes
110/29 ... yes
110/31 ... yes
161/32 ... yes
110/33 ... yes
110/34 ... yes
161/36 ... yes
345/37 ... yes
416/39 ... yes
2/40 ... yes
110/41 ... yes
110/44 ... yes
4/46 ... yes
426/47 ... yes
110/48 ... yes
426/49 ... yes
426/50 ... yes
6/51 ... yes
110/52 ... yes
72/53 ... yes
2/54 ... yes
551/56 ... yes
569/58 ... yes
110/59 ... yes
70/60 ... yes
110/61 ... yes
110/63 ... yes
110/64 ... yes
110/65 ... yes
110/66 ... yes
110/67 ... yes
110/68 ... yes
110/69 ... yes
110/70 ... yes
110/71 ... yes
627/72 ... yes
627/73 ... yes
627/74 ... yes
627/75 ... yes
627/76 ... yes
627/77 ... yes
627/78 ... yes
627/79 ... yes
627/80 ... yes
627/81 ... yes
578/82 ... yes
110/83 ... yes
110/84 ... yes
110/85 ... yes
110/86 ... yes
110/87 ... yes
110/88 ... yes
5/89 ... yes
711/90 ... yes
392/91 ... yes
5/92 ... yes
548/93 ... yes
2/94 ... yes
548/95 ... yes
743/96 ... yes
2/97 ... yes
110/98 ... yes
2/99 ... yes
110/101 ... yes
802/102 ... yes
802/103 ... yes
855/104 ... yes
110/106 ... yes
110/108 ... yes
359/109 ... yes
110/110 ... yes
110/112 ... yes
2/113 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.3)
Git version >= 2.7.3 ? ... yes (2.13.5)
Active users: ... 952


Checking GitLab ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)