Gitaly pod shows a lot of git zombie processes
Hi,
We are a customer with a premium gitlab license, after a non-voluntary restart of the gitaly pod today, I realized that gitaly is spawning a lot of git zombie processes, at least 1 or 2 per minute. This is happening on a small Kubernetes deployment with 2 GEO sites. Both sites are showing the same behavior, gitaly is running with pretty much the default values provided by the helm chart.
Eventually the cgroup is exhausted and the pod is restarted causing some downtime. Is there any known cause for this behavior?:
git@production-gitlab-gitaly-0:/$ ps auxwww | grep git | grep defunct | wc -l
552git@production-gitlab-gitaly-0:/var/log/gitaly$ ps auxwww
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
git 1 12.7 3.0 3405872 235464 ? Ssl 14:39 13:14 /usr/local/bin/gitaly /etc/gitaly/config.toml
git 16 0.7 0.0 1227888 7436 ? Sl 14:39 0:48 /usr/local/bin/gitlab-logger --json /var/log/gitaly
git 419 0.1 0.0 0 0 ? Z 14:40 0:09 [git] <defunct>
git 651 0.0 0.0 0 0 ? Z 14:40 0:01 [git] <defunct>
git 1649 0.1 0.0 0 0 ? Z 14:42 0:08 [git] <defunct>
git 1851 0.0 0.0 0 0 ? Z 14:42 0:01 [git] <defunct>
git 3606 0.1 0.0 0 0 ? Z 14:44 0:07 [git] <defunct>
git 3699 0.0 0.0 0 0 ? Z 14:44 0:01 [git] <defunct>
git 4715 0.0 0.0 0 0 ? Z 14:45 0:01 [git] <defunct>
git 5366 0.1 0.0 0 0 ? Z 14:46 0:06 [git] <defunct>
git 6473 0.0 0.0 0 0 ? Z 14:47 0:01 [git] <defunct>
git 6498 0.1 0.0 0 0 ? Z 14:47 0:06 [git] <defunct>
git 7284 0.1 0.0 0 0 ? Z 14:48 0:06 [git] <defunct>
git 7353 0.0 0.0 0 0 ? Z 14:48 0:01 [git] <defunct>
git 8508 0.1 0.0 0 0 ? Z 14:50 0:06 [git] <defunct>
git 8568 0.0 0.0 0 0 ? Z 14:50 0:01 [git] <defunct>
git 8895 0.1 0.0 0 0 ? Z 14:51 0:06 [git] <defunct>
git 8967 0.0 0.0 0 0 ? Z 14:51 0:01 [git] <defunct>
git 10858 0.0 0.0 0 0 ? Z 14:53 0:01 [git] <defunct>
git 10901 0.1 0.0 0 0 ? Z 14:53 0:06 [git] <defunct>
git 11428 0.1 0.0 0 0 ? Z 14:54 0:06 [git] <defunct>
git 12786 0.0 0.0 0 0 ? Z 14:55 0:01 [git] <defunct>
git 12883 0.1 0.0 0 0 ? Z 14:55 0:06 [git] <defunct>
git 15756 0.1 0.0 0 0 ? Z 14:57 0:06 [git] <defunct>
git 15911 0.0 0.0 0 0 ? Z 14:57 0:01 [git] <defunct>
git 16911 0.0 0.0 0 0 ? Z 14:59 0:01 [git] <defunct>
git 16939 0.1 0.0 0 0 ? Z 14:59 0:07 [git] <defunct>
git 17529 0.1 0.0 0 0 ? Z 15:00 0:06 [git] <defunct>
git 17901 0.0 0.0 0 0 ? Z 15:00 0:01 [git] <defunct>
git 19218 0.1 0.0 0 0 ? Z 15:02 0:06 [git] <defunct>
git 19350 0.0 0.0 0 0 ? Z 15:02 0:01 [git] <defunct>
git 19812 0.0 0.0 0 0 ? Z 15:03 0:01 [git] <defunct>
git 19973 0.1 0.0 0 0 ? Z 15:03 0:06 [git] <defunct>
git 21191 0.1 0.0 0 0 ? Z 15:04 0:06 [git] <defunct>
git 21279 0.0 0.0 0 0 ? Z 15:04 0:01 [git] <defunct>
git 22928 0.0 0.0 0 0 ? Z 15:06 0:01 [git] <defunct>
git 22946 0.1 0.0 0 0 ? Z 15:06 0:06 [git] <defunct>
git 23417 0.1 0.0 0 0 ? Z 15:07 0:06 [git] <defunct>
git 25351 0.0 0.0 0 0 ? Z 15:08 0:01 [git] <defunct>
git 26636 0.1 0.0 0 0 ? Z 15:09 0:06 [git] <defunct>
git 26744 0.0 0.0 0 0 ? Z 15:09 0:01 [git] <defunct>
git 27411 0.0 0.0 0 0 ? Z 15:10 0:01 [git] <defunct>
git 27439 0.1 0.0 0 0 ? Z 15:10 0:06 [git] <defunct>
git 28357 0.0 0.0 0 0 ? Z 15:11 0:01 [git] <defunct>
git 28382 0.1 0.0 0 0 ? Z 15:11 0:07 [git] <defunct>
git 29082 0.0 0.0 0 0 ? Z 15:12 0:01 [git] <defunct>
git 29109 0.1 0.0 0 0 ? Z 15:12 0:07 [git] <defunct>
git 29425 0.0 0.0 0 0 ? Z 15:13 0:01 [git] <defunct>
git 30205 0.1 0.0 0 0 ? Z 15:14 0:06 [git] <defunct>
git 30797 0.0 0.0 0 0 ? Z 15:15 0:01 [git] <defunct>
git 30823 0.1 0.0 0 0 ? Z 15:15 0:06 [git] <defunct>
git 31596 0.0 0.0 0 0 ? Z 15:16 0:01 [git] <defunct>
git 32498 0.0 0.0 0 0 ? Z 15:17 0:01 [git] <defunct>
git 32545 0.1 0.0 0 0 ? Z 15:17 0:07 [git] <defunct>
git 33882 0.0 0.0 0 0 ? Z 15:18 0:01 [git] <defunct>
git 33946 0.1 0.0 0 0 ? Z 15:18 0:07 [git] <defunct>
git 34974 0.0 0.0 0 0 ? Z 15:19 0:01 [git] <defunct>
git 34992 0.1 0.0 0 0 ? Z 15:19 0:06 [git] <defunct>
git 35557 0.1 0.0 0 0 ? Z 15:20 0:06 [git] <defunct>
git 36267 0.0 0.0 0 0 ? Z 15:21 0:01 [git] <defunct>
git 36327 0.2 0.0 0 0 ? Z 15:21 0:09 [git] <defunct>
git 37029 0.0 0.0 0 0 ? Z 15:22 0:02 [git] <defunct>
git 37524 0.2 0.0 0 0 ? Z 15:23 0:10 [git] <defunct>
git 37611 0.0 0.0 0 0 ? Z 15:23 0:01 [git] <defunct>
git 37861 0.1 0.0 0 0 ? Z 15:24 0:06 [git] <defunct>
git 38663 0.0 0.0 0 0 ? Z 15:25 0:01 [git] <defunct>
git 38687 0.2 0.0 0 0 ? Z 15:25 0:07 [git] <defunct>
git 39084 0.0 0.0 0 0 ? Z 15:26 0:01 [git] <defunct>
git 39623 0.2 0.0 0 0 ? Z 15:27 0:08 [git] <defunct>
git 40052 0.0 0.0 0 0 ? Z 15:27 0:01 [git] <defunct>
git 40528 0.0 0.0 0 0 ? Z 15:28 0:01 [git] <defunct>
git 40540 0.1 0.0 0 0 ? Z 15:28 0:06 [git] <defunct>
git 40829 0.2 0.0 0 0 ? Z 15:29 0:07 [git] <defunct>
git 40882 0.0 0.0 0 0 ? Z 15:29 0:01 [git] <defunct>
git 41307 0.0 0.0 0 0 ? Z 15:30 0:01 [git] <defunct>
git 41321 0.2 0.0 0 0 ? Z 15:30 0:06 [git] <defunct>
git 42046 0.0 0.0 0 0 ? Z 15:31 0:01 [git] <defunct>
git 42066 0.2 0.0 0 0 ? Z 15:31 0:06 [git] <defunct>
git 43010 0.0 0.0 0 0 ? Z 15:32 0:01 [git] <defunct>
git 43052 0.2 0.0 0 0 ? Z 15:32 0:08 [git] <defunct>
git 43831 0.2 0.0 0 0 ? Z 15:33 0:06 [git] <defunct>
git 44956 0.0 0.0 0 0 ? Z 15:34 0:02 [git] <defunct>
git 45260 0.3 0.0 0 0 ? Z 15:34 0:09 [git] <defunct>
git 45826 0.0 0.0 0 0 ? Z 15:35 0:01 [git] <defunct>
git 45935 0.2 0.0 0 0 ? Z 15:35 0:08 [git] <defunct>
git 46896 0.0 0.0 0 0 ? Z 15:36 0:01 [git] <defunct>
git 47315 0.2 0.0 0 0 ? Z 15:37 0:07 [git] <defunct>
git 48321 0.0 0.0 0 0 ? Z 15:38 0:01 [git] <defunct>
git 48900 0.2 0.0 0 0 ? Z 15:39 0:06 [git] <defunct>
git 49051 0.0 0.0 0 0 ? Z 15:39 0:01 [git] <defunct>
git 50922 0.3 0.0 0 0 ? Z 15:41 0:08 [git] <defunct>
git 51176 0.0 0.0 0 0 ? Z 15:41 0:01 [git] <defunct>
git 52554 0.0 0.0 0 0 ? Z 15:42 0:01 [git] <defunct>
git 52601 0.3 0.0 0 0 ? Z 15:42 0:07 [git] <defunct>
git 54571 0.4 0.0 0 0 ? Z 15:44 0:09 [git] <defunct>
git 54911 0.0 0.0 0 0 ? Z 15:44 0:01 [git] <defunct>
git 56463 0.4 0.0 0 0 ? Z 15:46 0:09 [git] <defunct>
git 56577 0.0 0.0 0 0 ? Z 15:46 0:01 [git] <defunct>
git 57172 0.0 0.0 0 0 ? Z 15:47 0:01 [git] <defunct>
git 57208 0.3 0.0 0 0 ? Z 15:47 0:08 [git] <defunct>
git 57612 0.0 0.0 0 0 ? Z 15:48 0:01 [git] <defunct>
git 58192 0.3 0.0 0 0 ? Z 15:49 0:07 [git] <defunct>
git 58244 0.0 0.0 0 0 ? Z 15:49 0:01 [git] <defunct>
git 58392 0.0 0.0 4720 3820 pts/0 Ss 15:49 0:00 /bin/bash
git 59156 0.4 0.0 0 0 ? Z 15:51 0:07 [git] <defunct>
git 59453 0.0 0.0 0 0 ? Z 15:51 0:01 [git] <defunct>
git 59835 0.0 0.0 0 0 ? Z 15:52 0:01 [git] <defunct>
git 59854 0.3 0.0 0 0 ? Z 15:52 0:06 [git] <defunct>
git 61064 0.0 0.0 0 0 ? Z 15:53 0:01 [git] <defunct>
git 61076 0.3 0.0 0 0 ? Z 15:53 0:06 [git] <defunct>
git 61613 0.0 0.0 0 0 ? Z 15:54 0:01 [git] <defunct>
git 62329 0.5 0.0 0 0 ? Z 15:55 0:08 [git] <defunct>
git 63515 0.1 0.0 0 0 ? Z 15:56 0:01 [git] <defunct>
git 64503 0.5 0.0 0 0 ? Z 15:57 0:08 [git] <defunct>
git 64590 0.0 0.0 0 0 ? Z 15:57 0:01 [git] <defunct>
git 64891 0.5 0.0 0 0 ? Z 15:58 0:08 [git] <defunct>
git 65546 0.1 0.0 0 0 ? Z 15:59 0:02 [git] <defunct>
git 66375 0.1 0.0 0 0 ? Z 16:00 0:01 [git] <defunct>
git 66393 0.7 0.0 0 0 ? Z 16:00 0:09 [git] <defunct>
git 66644 0.1 0.0 0 0 ? Z 16:01 0:01 [git] <defunct>
git 66654 0.6 0.0 0 0 ? Z 16:01 0:08 [git] <defunct>
git 67348 0.1 0.0 0 0 ? Z 16:02 0:02 [git] <defunct>
git 67861 0.7 0.0 0 0 ? Z 16:03 0:09 [git] <defunct>
git 68843 0.1 0.0 0 0 ? Z 16:03 0:01 [git] <defunct>
git 69823 0.7 0.0 0 0 ? Z 16:04 0:08 [git] <defunct>
git 71029 0.1 0.0 0 0 ? Z 16:05 0:01 [git] <defunct>
git 71615 0.7 0.0 0 0 ? Z 16:06 0:08 [git] <defunct>
git 72230 0.1 0.0 0 0 ? Z 16:07 0:01 [git] <defunct>
git 72253 0.7 0.0 0 0 ? Z 16:07 0:07 [git] <defunct>
git 72824 0.7 0.0 0 0 ? Z 16:08 0:07 [git] <defunct>
git 72910 0.1 0.0 0 0 ? Z 16:08 0:01 [git] <defunct>
git 73174 0.1 0.0 0 0 ? Z 16:09 0:01 [git] <defunct>
git 73995 0.9 0.0 0 0 ? Z 16:10 0:07 [git] <defunct>
git 74220 0.1 0.0 0 0 ? Z 16:10 0:01 [git] <defunct>
git 74475 0.1 0.0 0 0 ? Z 16:11 0:01 [git] <defunct>
git 74514 1.4 0.0 0 0 ? Z 16:11 0:10 [git] <defunct>
git 74842 0.3 0.0 0 0 ? Z 16:12 0:02 [git] <defunct>
git 75411 1.3 0.0 0 0 ? Z 16:13 0:08 [git] <defunct>
git 76272 0.3 0.0 0 0 ? Z 16:13 0:02 [git] <defunct>
git 77089 1.5 0.0 0 0 ? Z 16:14 0:08 [git] <defunct>
git 77471 0.2 0.0 0 0 ? Z 16:15 0:01 [git] <defunct>
git 77511 1.4 0.0 0 0 ? Z 16:15 0:07 [git] <defunct>
git 77954 0.2 0.0 0 0 ? Z 16:16 0:01 [git] <defunct>
git 78633 2.0 0.0 0 0 ? Z 16:17 0:07 [git] <defunct>
git 79265 0.4 0.0 0 0 ? Z 16:18 0:01 [git] <defunct>
git 79416 2.2 0.0 0 0 ? Z 16:18 0:06 [git] <defunct>
git 80034 2.6 0.0 0 0 ? Z 16:19 0:06 [git] <defunct>
git 80300 0.4 0.0 0 0 ? Z 16:19 0:01 [git] <defunct>
git 80668 0.6 0.0 0 0 ? Z 16:20 0:01 [git] <defunct>
git 80693 3.6 0.0 0 0 ? Z 16:20 0:06 [git] <defunct>
git 81070 0.9 0.0 0 0 ? Z 16:21 0:01 [git] <defunct>
git 81431 1.6 0.0 0 0 ? Z 16:22 0:01 [git] <defunct>
git 81462 9.8 0.0 0 0 ? Z 16:22 0:07 [git] <defunct>
git 82123 74.6 0.0 0 0 ? Z 16:23 0:06 [git] <defunct>
git 82124 0.0 0.0 8492 4876 ? S 16:23 0:00 /tmp/gitaly-582375387/git-exec-1796046532.d/git --git-dir /home/git/repositories/@hashed/cb/be/cbbe2e41fff1a2f04968bdaeedff3b78085afca2dc4623870bdc2dff3aac6747.git -c gc.auto=0 -c maintenance.auto=0 -c core.autocrlf=input -c core.useReplaceRefs=false -c core.fsync=objects,derived-metadata,reference -c core.fsyncMethod=fsync -c core.packedRefsTimeout=10000 -c core.filesRefLockTimeout=1000 -c core.bigFileThreshold=50m cat-file --use-mailmap -Z --batch-command --buffer --end-of-options
git 82148 0.0 0.0 9560 4956 ? S 16:23 0:00 /tmp/gitaly-582375387/git-exec-1796046532.d/git --git-dir /home/git/repositories/@hashed/50/18/501883ce12ba20f1a0de68f06b8c84d3d148124f66ab52a38359317c78ab82b1.git -c gc.auto=0 -c maintenance.auto=0 -c core.autocrlf=input -c core.useReplaceRefs=false -c core.fsync=objects,derived-metadata,reference -c core.fsyncMethod=fsync -c core.packedRefsTimeout=10000 -c core.filesRefLockTimeout=1000 -c core.bigFileThreshold=50m cat-file --use-mailmap -Z --batch-command --buffer --end-of-options
git 82151 0.0 0.0 29776 5208 ? S 16:23 0:00 /tmp/gitaly-582375387/git-exec-1796046532.d/git --git-dir /home/git/repositories/@hashed/95/53/9553627933b214db60798fe40d2b4f8497781d024f53d62dc1b12469b7d53784.git -c gc.auto=0 -c maintenance.auto=0 -c core.autocrlf=input -c core.useReplaceRefs=false -c core.fsync=objects,derived-metadata,reference -c core.fsyncMethod=fsync -c core.packedRefsTimeout=10000 -c core.filesRefLockTimeout=1000 -c core.bigFileThreshold=50m cat-file --use-mailmap -Z --batch-command --buffer --end-of-options
git 82153 0.0 0.0 893684 6936 ? S 16:23 0:00 /tmp/gitaly-582375387/git-exec-1796046532.d/git --git-dir /home/git/repositories/@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git -c gc.auto=0 -c maintenance.auto=0 -c core.autocrlf=input -c core.useReplaceRefs=false -c core.fsync=objects,derived-metadata,reference -c core.fsyncMethod=fsync -c core.packedRefsTimeout=10000 -c core.filesRefLockTimeout=1000 -c core.bigFileThreshold=50m cat-file --use-mailmap -Z --batch-command --buffer --end-of-options
git 82157 100 0.0 8492 4200 pts/0 R+ 16:23 0:00 ps auxwwwI also opened a support request at https://support.gitlab.com/hc/en-us/requests/513408 .
Thanks in advance.
Edited by Jason Plum