Gitlab CE unstable after update to 9.x (9.3.1 currently)
Summary
Hi, we've updated our Gitlab CE installation to the most recent version. After some time of gitlab uptime we experience errors like out of memory exceptions, no notifications being sent or activity on projects being displayed. The obvious observation is that gitlab-shell processess are stuck for some reason. See the top and ps commands output below and notice 12730 and 12754 processes.
- It looks like command with key-52 is usually being stuck *
Steps to reproduce
No clear way on our end how to reproduce it.
Example Project
Did not see that on gitlab.com.
What is the current bug behavior?
Gitlab is unstable and unusable.
What is the expected correct behavior?
No lags, no out of memory exceptions, no notifications being blocked.
Relevant logs and/or screenshots
[root@gitlab ~]# top
top - 17:53:06 up 8 days, 7:29, 1 user, load average: 2.26, 2.48, 2.58
Tasks: 274 total, 3 running, 271 sleeping, 0 stopped, 0 zombie
%Cpu(s): 28.5 us, 0.4 sy, 0.0 ni, 71.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8011104 total, 1007800 free, 4769284 used, 2234020 buff/cache
KiB Swap: 4063228 total, 3904440 free, 158788 used. 2249044 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12730 git 20 0 93164 14780 4376 R 100.0 0.2 134:00.14 gitlab-shell
12754 git 20 0 96152 15932 4504 R 100.0 0.2 133:58.90 gitlab-shell
5608 git 20 0 1928448 607600 51144 S 22.3 7.6 0:17.97 bundle
5650 gitlab-+ 20 0 2222776 38400 27728 S 3.3 0.5 0:00.59 postgres
662 git 20 0 1375584 509996 11004 S 2.3 6.4 1:37.51 bundle
28253 gitlab-+ 20 0 126748 57672 1440 S 1.3 0.7 8:42.38 redis-server
4553 git 20 0 1819904 518540 47896 S 0.7 6.5 0:16.23 bundle
29388 git 20 0 946920 21964 4068 S 0.7 0.3 2:22.61 gitlab-workhors
703 root 20 0 268820 1752 1416 S 0.3 0.0 11:57.17 vmtoolsd
6385 root 20 0 130152 1928 1260 R 0.3 0.0 0:00.05 top
21814 root 20 0 0 0 0 S 0.3 0.0 0:02.30 kworker/5:0
29418 gitlab-+ 20 0 1371220 83388 5524 S 0.3 1.0 3:51.48 prometheus
1 root 20 0 56652 4084 2452 S 0.0 0.1 57:19.80 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:01.85 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:24.90 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:42.00 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/4
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/5
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/6
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/7
17 root 20 0 0 0 0 S 0.0 0.0 32:11.89 rcu_sched
18 root 20 0 0 0 0 S 0.0 0.0 6:00.93 rcuos/0
19 root 20 0 0 0 0 S 0.0 0.0 7:52.96 rcuos/1
20 root 20 0 0 0 0 S 0.0 0.0 6:03.25 rcuos/2
21 root 20 0 0 0 0 S 0.0 0.0 7:08.01 rcuos/3
22 root 20 0 0 0 0 S 0.0 0.0 6:04.65 rcuos/4
23 root 20 0 0 0 0 S 0.0 0.0 8:01.09 rcuos/5
24 root 20 0 0 0 0 S 0.0 0.0 6:18.67 rcuos/6
25 root 20 0 0 0 0 S 0.0 0.0 6:19.53 rcuos/7
26 root rt 0 0 0 0 S 0.0 0.0 0:10.12 watchdog/0
27 root rt 0 0 0 0 S 0.0 0.0 0:35.00 watchdog/1
28 root rt 0 0 0 0 S 0.0 0.0 2:15.60 migration/1
29 root 20 0 0 0 0 S 0.0 0.0 2:14.64 ksoftirqd/1
31 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
32 root rt 0 0 0 0 S 0.0 0.0 0:36.51 watchdog/2
33 root rt 0 0 0 0 S 0.0 0.0 1:29.73 migration/2
34 root 20 0 0 0 0 S 0.0 0.0 3:01.66 ksoftirqd/2Looks like the stuck processes are not killed even when we run gitlab-ctl restart
[root@gitlab ~]# ps -ef --forest
UID PID PPID C STIME TTY TIME CMD
root 2 0 0 Jun19 ? 00:00:01 [kthreadd]
root 3 2 0 Jun19 ? 00:00:24 \_ [ksoftirqd/0]
root 5 2 0 Jun19 ? 00:00:00 \_ [kworker/0:0H]
root 7 2 0 Jun19 ? 00:00:42 \_ [migration/0]
root 8 2 0 Jun19 ? 00:00:00 \_ [rcu_bh]
root 9 2 0 Jun19 ? 00:00:00 \_ [rcuob/0]
root 10 2 0 Jun19 ? 00:00:00 \_ [rcuob/1]
root 11 2 0 Jun19 ? 00:00:00 \_ [rcuob/2]
root 12 2 0 Jun19 ? 00:00:00 \_ [rcuob/3]
root 13 2 0 Jun19 ? 00:00:00 \_ [rcuob/4]
root 14 2 0 Jun19 ? 00:00:00 \_ [rcuob/5]
root 15 2 0 Jun19 ? 00:00:00 \_ [rcuob/6]
root 16 2 0 Jun19 ? 00:00:00 \_ [rcuob/7]
root 17 2 0 Jun19 ? 00:32:13 \_ [rcu_sched]
root 18 2 0 Jun19 ? 00:06:01 \_ [rcuos/0]
root 19 2 0 Jun19 ? 00:07:53 \_ [rcuos/1]
root 20 2 0 Jun19 ? 00:06:03 \_ [rcuos/2]
root 21 2 0 Jun19 ? 00:07:08 \_ [rcuos/3]
root 22 2 0 Jun19 ? 00:06:05 \_ [rcuos/4]
root 23 2 0 Jun19 ? 00:08:01 \_ [rcuos/5]
root 24 2 0 Jun19 ? 00:06:19 \_ [rcuos/6]
root 25 2 0 Jun19 ? 00:06:19 \_ [rcuos/7]
root 26 2 0 Jun19 ? 00:00:10 \_ [watchdog/0]
root 27 2 0 Jun19 ? 00:00:35 \_ [watchdog/1]
root 28 2 0 Jun19 ? 00:02:15 \_ [migration/1]
root 29 2 0 Jun19 ? 00:02:14 \_ [ksoftirqd/1]
root 31 2 0 Jun19 ? 00:00:00 \_ [kworker/1:0H]
root 32 2 0 Jun19 ? 00:00:36 \_ [watchdog/2]
root 33 2 0 Jun19 ? 00:01:29 \_ [migration/2]
root 34 2 0 Jun19 ? 00:03:01 \_ [ksoftirqd/2]
root 36 2 0 Jun19 ? 00:00:00 \_ [kworker/2:0H]
root 37 2 0 Jun19 ? 00:00:09 \_ [watchdog/3]
root 38 2 0 Jun19 ? 00:00:28 \_ [migration/3]
root 39 2 0 Jun19 ? 00:00:21 \_ [ksoftirqd/3]
root 41 2 0 Jun19 ? 00:00:00 \_ [kworker/3:0H]
root 42 2 0 Jun19 ? 00:00:12 \_ [watchdog/4]
root 43 2 0 Jun19 ? 00:00:47 \_ [migration/4]
root 44 2 0 Jun19 ? 00:00:44 \_ [ksoftirqd/4]
root 46 2 0 Jun19 ? 00:00:00 \_ [kworker/4:0H]
root 47 2 0 Jun19 ? 00:00:13 \_ [watchdog/5]
root 48 2 0 Jun19 ? 00:01:13 \_ [migration/5]
root 49 2 0 Jun19 ? 00:00:46 \_ [ksoftirqd/5]
root 51 2 0 Jun19 ? 00:00:00 \_ [kworker/5:0H]
root 52 2 0 Jun19 ? 00:00:11 \_ [watchdog/6]
root 53 2 0 Jun19 ? 00:01:44 \_ [migration/6]
root 54 2 0 Jun19 ? 00:00:39 \_ [ksoftirqd/6]
root 56 2 0 Jun19 ? 00:00:00 \_ [kworker/6:0H]
root 57 2 0 Jun19 ? 00:00:18 \_ [watchdog/7]
root 58 2 0 Jun19 ? 00:01:05 \_ [migration/7]
root 59 2 0 Jun19 ? 00:01:27 \_ [ksoftirqd/7]
root 61 2 0 Jun19 ? 00:00:00 \_ [kworker/7:0H]
root 62 2 0 Jun19 ? 00:00:00 \_ [khelper]
root 63 2 0 Jun19 ? 00:00:00 \_ [kdevtmpfs]
root 64 2 0 Jun19 ? 00:00:00 \_ [netns]
root 65 2 0 Jun19 ? 00:00:00 \_ [writeback]
root 66 2 0 Jun19 ? 00:00:00 \_ [kintegrityd]
root 67 2 0 Jun19 ? 00:00:00 \_ [bioset]
root 68 2 0 Jun19 ? 00:00:00 \_ [kblockd]
root 69 2 0 Jun19 ? 00:00:00 \_ [khubd]
root 70 2 0 Jun19 ? 00:00:00 \_ [md]
root 73 2 0 Jun19 ? 00:00:00 \_ [khungtaskd]
root 74 2 0 Jun19 ? 00:03:06 \_ [kswapd0]
root 75 2 0 Jun19 ? 00:00:00 \_ [ksmd]
root 76 2 0 Jun19 ? 00:01:46 \_ [khugepaged]
root 77 2 0 Jun19 ? 00:00:00 \_ [fsnotify_mark]
root 78 2 0 Jun19 ? 00:00:00 \_ [crypto]
root 86 2 0 Jun19 ? 00:00:00 \_ [kthrotld]
root 89 2 0 Jun19 ? 00:00:00 \_ [kmpath_rdacd]
root 92 2 0 Jun19 ? 00:00:00 \_ [kpsmoused]
root 112 2 0 Jun19 ? 00:00:00 \_ [deferwq]
root 136 2 0 Jun19 ? 00:00:10 \_ [kauditd]
root 325 2 0 Jun19 ? 00:00:00 \_ [ata_sff]
root 330 2 0 Jun19 ? 00:00:00 \_ [mpt_poll_0]
root 331 2 0 Jun19 ? 00:00:00 \_ [mpt/0]
root 337 2 0 Jun19 ? 00:00:00 \_ [scsi_eh_0]
root 338 2 0 Jun19 ? 00:00:00 \_ [scsi_tmf_0]
root 340 2 0 Jun19 ? 00:00:00 \_ [scsi_eh_1]
root 341 2 0 Jun19 ? 00:00:00 \_ [scsi_tmf_1]
root 342 2 0 Jun19 ? 00:00:00 \_ [scsi_eh_2]
root 343 2 0 Jun19 ? 00:00:00 \_ [scsi_tmf_2]
root 344 2 0 Jun19 ? 00:00:00 \_ [ttm_swap]
root 416 2 0 Jun19 ? 00:00:00 \_ [kdmflush]
root 417 2 0 Jun19 ? 00:00:00 \_ [bioset]
root 426 2 0 Jun19 ? 00:00:00 \_ [kdmflush]
root 427 2 0 Jun19 ? 00:00:00 \_ [bioset]
root 444 2 0 Jun19 ? 00:00:00 \_ [xfsalloc]
root 445 2 0 Jun19 ? 00:00:00 \_ [xfs_mru_cache]
root 446 2 0 Jun19 ? 00:00:00 \_ [xfs-buf/dm-0]
root 447 2 0 Jun19 ? 00:00:00 \_ [xfs-data/dm-0]
root 448 2 0 Jun19 ? 00:00:00 \_ [xfs-conv/dm-0]
root 449 2 0 Jun19 ? 00:00:00 \_ [xfs-cil/dm-0]
root 450 2 0 Jun19 ? 00:01:45 \_ [kworker/2:1H]
root 451 2 0 Jun19 ? 00:17:03 \_ [xfsaild/dm-0]
root 452 2 0 Jun19 ? 00:00:11 \_ [kworker/3:1H]
root 457 2 0 Jun19 ? 00:00:24 \_ [kworker/0:1H]
root 467 2 0 Jun19 ? 00:01:57 \_ [kworker/1:1H]
root 508 2 0 Jun19 ? 00:00:21 \_ [kworker/4:1H]
root 509 2 0 Jun19 ? 00:00:16 \_ [kworker/5:1H]
root 510 2 0 Jun19 ? 00:00:22 \_ [kworker/6:1H]
root 511 2 0 Jun19 ? 00:00:39 \_ [kworker/7:1H]
root 636 2 0 Jun19 ? 00:00:00 \_ [kdmflush]
root 637 2 0 Jun19 ? 00:00:00 \_ [bioset]
root 641 2 0 Jun19 ? 00:00:00 \_ [xfs-buf/sda1]
root 642 2 0 Jun19 ? 00:00:00 \_ [xfs-data/sda1]
root 643 2 0 Jun19 ? 00:00:00 \_ [xfs-conv/sda1]
root 644 2 0 Jun19 ? 00:00:00 \_ [xfs-cil/sda1]
root 645 2 0 Jun19 ? 00:00:00 \_ [xfsaild/sda1]
root 650 2 0 Jun19 ? 00:00:00 \_ [xfs-buf/dm-2]
root 651 2 0 Jun19 ? 00:00:00 \_ [xfs-data/dm-2]
root 652 2 0 Jun19 ? 00:00:00 \_ [xfs-conv/dm-2]
root 653 2 0 Jun19 ? 00:00:00 \_ [xfs-cil/dm-2]
root 654 2 0 Jun19 ? 00:00:00 \_ [xfsaild/dm-2]
root 2689 2 0 Jun19 ? 00:00:00 \_ [cfg80211]
root 1199 2 0 16:25 ? 00:00:00 \_ [kworker/3:2]
root 9327 2 0 16:45 ? 00:00:00 \_ [kworker/1:1]
root 20260 2 0 17:11 ? 00:00:00 \_ [kworker/7:1]
root 21257 2 0 17:14 ? 00:00:00 \_ [kworker/4:1]
root 21814 2 0 17:15 ? 00:00:02 \_ [kworker/5:0]
root 26138 2 0 17:24 ? 00:00:00 \_ [kworker/u16:1]
root 26745 2 0 17:26 ? 00:00:00 \_ [kworker/6:2]
root 27338 2 0 17:27 ? 00:00:00 \_ [kworker/2:2]
root 30413 2 0 17:33 ? 00:00:00 \_ [kworker/0:0]
root 31766 2 0 17:36 ? 00:00:00 \_ [kworker/2:1]
root 1777 2 0 17:42 ? 00:00:00 \_ [kworker/1:0]
root 3811 2 0 17:46 ? 00:00:00 \_ [kworker/4:0]
root 4726 2 0 17:48 ? 00:00:00 \_ [kworker/7:2]
root 5119 2 0 17:49 ? 00:00:00 \_ [kworker/u16:0]
root 5373 2 0 17:50 ? 00:00:00 \_ [kworker/6:1]
root 5934 2 0 17:51 ? 00:00:00 \_ [kworker/1:2]
root 6082 2 0 17:52 ? 00:00:00 \_ [kworker/0:2]
root 6351 2 0 17:52 ? 00:00:00 \_ [kworker/3:0]
root 6604 2 0 17:53 ? 00:00:00 \_ [kworker/5:1]
root 7198 2 0 17:55 ? 00:00:00 \_ [kworker/4:2]
root 7214 2 0 17:55 ? 00:00:00 \_ [kworker/6:0]
root 7608 2 0 17:56 ? 00:00:00 \_ [kworker/1:3]
root 7611 2 0 17:56 ? 00:00:00 \_ [kworker/2:0]
root 8002 2 0 17:57 ? 00:00:00 \_ [kworker/0:1]
root 8945 2 0 17:59 ? 00:00:00 \_ [kworker/7:0]
root 1 0 0 Jun19 ? 00:57:22 /usr/lib/systemd/systemd --switched-root --system --deserialize 24
root 531 1 0 Jun19 ? 00:26:33 /usr/lib/systemd/systemd-journald
root 546 1 0 Jun19 ? 00:00:00 /usr/sbin/lvmetad -f
root 554 1 0 Jun19 ? 00:00:00 /usr/lib/systemd/systemd-udevd
root 672 1 0 Jun19 ? 00:01:25 /sbin/auditd -n
root 694 1 0 Jun19 ? 00:03:57 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
root 695 1 0 Jun19 ? 00:00:17 runsvdir -P /opt/gitlab/service log: .................................................................................................
root 706 695 0 Jun19 ? 00:00:00 \_ runsv redis
root 743 706 0 Jun19 ? 00:00:05 | \_ svlogd -tt /var/log/gitlab/redis
gitlab-+ 28253 706 1 09:44 ? 00:08:49 | \_ /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
root 707 695 0 Jun19 ? 00:00:00 \_ runsv postgresql
root 720 707 0 Jun19 ? 00:00:05 | \_ svlogd -tt /var/log/gitlab/postgresql
gitlab-+ 28251 707 0 09:44 ? 00:00:05 | \_ /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
gitlab-+ 28257 28251 0 09:44 ? 00:00:21 | \_ postgres: checkpointer process
gitlab-+ 28258 28251 0 09:44 ? 00:00:01 | \_ postgres: writer process
gitlab-+ 28259 28251 0 09:44 ? 00:00:06 | \_ postgres: wal writer process
gitlab-+ 28260 28251 0 09:44 ? 00:00:01 | \_ postgres: autovacuum launcher process
gitlab-+ 28261 28251 0 09:44 ? 00:00:23 | \_ postgres: stats collector process
gitlab-+ 29485 28251 0 09:46 ? 00:00:34 | \_ postgres: gitlab-psql postgres [local] idle
gitlab-+ 1310 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1311 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1321 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1324 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1343 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1344 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1387 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1469 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1470 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1474 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1481 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1492 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1499 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1512 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1519 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 1522 28251 0 17:42 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 3999 28251 0 17:47 ? 00:00:01 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 4102 28251 0 17:47 ? 00:00:02 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 4300 28251 0 17:47 ? 00:00:01 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 4615 28251 0 17:48 ? 00:00:01 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 4631 28251 0 17:48 ? 00:00:01 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 5972 28251 0 17:51 ? 00:00:01 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 6925 28251 0 17:54 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 7671 28251 0 17:56 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8377 28251 0 17:58 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8695 28251 0 17:58 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8696 28251 0 17:58 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8697 28251 0 17:58 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8796 28251 0 17:59 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8797 28251 0 17:59 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8798 28251 0 17:59 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8879 28251 0 17:59 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8880 28251 0 17:59 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8885 28251 0 17:59 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8947 28251 0 17:59 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8948 28251 0 17:59 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 8949 28251 0 17:59 ? 00:00:00 | \_ postgres: gitlab gitlabhq_production [local] idle
root 711 695 0 Jun19 ? 00:00:00 \_ runsv unicorn
root 722 711 0 Jun19 ? 00:00:00 | \_ svlogd -tt /var/log/gitlab/unicorn
git 29494 711 0 09:46 ? 00:00:18 | \_ /bin/bash /opt/gitlab/embedded/bin/gitlab-unicorn-wrapper
git 8950 29494 0 17:59 ? 00:00:00 | \_ sleep 1
root 712 695 0 Jun19 ? 00:00:00 \_ runsv sidekiq
root 749 712 0 Jun19 ? 00:00:12 | \_ svlogd -tt /var/log/gitlab/sidekiq
git 662 712 10 17:40 ? 00:01:56 | \_ sidekiq 5.0.0 gitlab-rails [0 of 25 busy]
root 713 695 0 Jun19 ? 00:00:00 \_ runsv nginx
root 740 713 0 Jun19 ? 00:00:00 | \_ svlogd -tt /var/log/gitlab/nginx
root 29454 713 0 09:46 ? 00:00:00 | \_ nginx: master process /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx
gitlab-+ 29459 29454 0 09:46 ? 00:00:09 | \_ nginx: worker process
gitlab-+ 29460 29454 0 09:46 ? 00:00:07 | \_ nginx: worker process
gitlab-+ 29461 29454 0 09:46 ? 00:00:07 | \_ nginx: worker process
gitlab-+ 29462 29454 0 09:46 ? 00:00:09 | \_ nginx: worker process
gitlab-+ 29463 29454 0 09:46 ? 00:00:07 | \_ nginx: worker process
gitlab-+ 29464 29454 0 09:46 ? 00:00:10 | \_ nginx: worker process
gitlab-+ 29465 29454 0 09:46 ? 00:00:09 | \_ nginx: worker process
gitlab-+ 29466 29454 0 09:46 ? 00:00:09 | \_ nginx: worker process
gitlab-+ 29467 29454 0 09:46 ? 00:00:00 | \_ nginx: cache manager process
root 714 695 0 Jun19 ? 00:00:00 \_ runsv logrotate
root 732 714 0 Jun19 ? 00:00:00 | \_ svlogd -tt /var/log/gitlab/logrotate
root 3826 714 0 17:46 ? 00:00:00 | \_ /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper
root 7832 3826 0 17:56 ? 00:00:00 | \_ sleep 3000
root 715 695 0 Jun19 ? 00:00:00 \_ runsv gitlab-workhorse
root 745 715 0 Jun19 ? 00:01:22 | \_ svlogd -tt /var/log/gitlab/gitlab-workhorse
git 29388 715 0 09:46 ? 00:02:24 | \_ /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listenUmask 0 -listenAddr /var/opt/gitlab/gitlab-workhorse/sock
root 719 695 0 Jun19 ? 00:00:00 \_ runsv gitaly
root 744 719 0 Jun19 ? 00:00:00 | \_ svlogd -tt /var/log/gitlab/gitaly
git 29380 719 0 09:46 ? 00:00:04 | \_ /opt/gitlab/embedded/bin/gitaly /var/opt/gitlab/gitaly/config.toml
root 725 695 0 Jun19 ? 00:00:00 \_ runsv prometheus
root 753 725 0 Jun19 ? 00:00:00 | \_ svlogd -tt /var/log/gitlab/prometheus
gitlab-+ 29418 725 0 09:46 ? 00:03:54 | \_ /opt/gitlab/embedded/bin/prometheus -web.listen-address=localhost:9090 -storage.local.path=/var/opt/gitlab/prometheus/data -st
root 726 695 0 Jun19 ? 00:00:00 \_ runsv node-exporter
root 764 726 0 Jun19 ? 00:00:00 | \_ svlogd -tt /var/log/gitlab/node-exporter
gitlab-+ 29458 726 0 09:46 ? 00:01:06 | \_ /opt/gitlab/embedded/bin/node_exporter -web.listen-address=localhost:9100 -collector.textfile.directory=/var/opt/gitlab/node-e
root 727 695 0 Jun19 ? 00:00:00 \_ runsv redis-exporter
root 741 727 0 Jun19 ? 00:00:00 | \_ svlogd -tt /var/log/gitlab/redis-exporter
gitlab-+ 29475 727 0 09:46 ? 00:00:17 | \_ /opt/gitlab/embedded/bin/redis_exporter -web.listen-address=localhost:9121 -redis.addr=unix:///var/opt/gitlab/redis/redis.sock
root 731 695 0 Jun19 ? 00:00:00 \_ runsv postgres-exporter
root 754 731 0 Jun19 ? 00:00:00 | \_ svlogd -tt /var/log/gitlab/postgres-exporter
gitlab-+ 29470 731 0 09:46 ? 00:00:33 | \_ /opt/gitlab/embedded/bin/postgres_exporter -web.listen-address=localhost:9187 -extend.query-path=/var/opt/gitlab/postgres-expo
root 734 695 0 Jun19 ? 00:00:00 \_ runsv gitlab-monitor
root 757 734 0 Jun19 ? 00:00:13 \_ svlogd -tt /var/log/gitlab/gitlab-monitor
git 29426 734 0 09:46 ? 00:02:14 \_ /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/gitlab-mon web -c /var/opt/gitlab/gitlab-monitor/gitlab-monitor.yml
root 698 1 0 Jun19 ? 00:00:55 /usr/sbin/irqbalance --foreground
root 701 1 0 Jun19 ? 00:08:18 /usr/sbin/rsyslogd -n
root 702 1 0 Jun19 ? 00:01:23 /usr/bin/python -Es /usr/sbin/tuned -l -P
root 703 1 0 Jun19 ? 00:11:57 /usr/bin/vmtoolsd
root 709 1 0 Jun19 ? 00:19:08 /usr/lib/systemd/systemd-logind
dbus 710 1 0 Jun19 ? 00:25:49 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 716 1 0 Jun19 ? 00:00:18 /usr/sbin/crond -n
ntp 772 1 0 Jun19 ? 00:00:04 /usr/sbin/ntpd -u ntp:ntp -g
root 931 1 0 Jun19 ? 00:11:29 /usr/sbin/NetworkManager --no-daemon
polkitd 1137 1 0 Jun19 ? 00:08:17 /usr/lib/polkit-1/polkitd --no-debug
root 1421 1 0 Jun19 ? 00:03:05 /usr/sbin/sshd -D
root 6036 1421 0 17:52 ? 00:00:00 \_ sshd: root@pts/0
root 6157 6036 0 17:52 pts/0 00:00:00 | \_ -bash
root 8951 6157 0 17:59 pts/0 00:00:00 | \_ ps -ef --forest
root 7767 1421 0 17:56 ? 00:00:00 \_ sshd: git [priv]
git 7778 7767 0 17:56 ? 00:00:00 | \_ sshd: git@notty
git 7779 7778 0 17:56 ? 00:00:00 | \_ /opt/gitlab/embedded/bin/git-upload-pack /var/opt/gitlab/git-data/repositories/Itslearning/main.git
root 8278 1421 0 17:57 ? 00:00:00 \_ sshd: git [priv]
git 8282 8278 0 17:57 ? 00:00:00 \_ sshd: git@notty
git 8283 8282 0 17:57 ? 00:00:00 \_ /opt/gitlab/embedded/bin/git-upload-pack /var/opt/gitlab/git-data/repositories/Itslearning/main.git
root 2589 1 0 Jun19 ? 00:00:12 /usr/libexec/postfix/master -w
postfix 2610 2589 0 Jun19 ? 00:00:01 \_ qmgr -l -t unix -u
postfix 12947 2589 0 16:54 ? 00:00:00 \_ pickup -l -t unix -u
root 19759 1 0 Jun22 tty1 00:00:00 /sbin/agetty --noclear tty1
git 29522 1 0 09:46 ? 00:00:29 unicorn master -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru
git 3978 29522 4 17:47 ? 00:00:31 \_ unicorn worker[5] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config
git 4086 29522 4 17:47 ? 00:00:35 \_ unicorn worker[1] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config
git 4182 29522 3 17:47 ? 00:00:25 \_ unicorn worker[6] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config
git 4549 29522 2 17:47 ? 00:00:20 \_ unicorn worker[4] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config
git 4553 29522 3 17:47 ? 00:00:27 \_ unicorn worker[8] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config
git 5956 29522 4 17:51 ? 00:00:20 \_ unicorn worker[7] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config
git 6909 29522 4 17:54 ? 00:00:13 \_ unicorn worker[2] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config
git 7649 29522 4 17:56 ? 00:00:07 \_ unicorn worker[0] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config
git 8363 29522 11 17:58 ? 00:00:10 \_ unicorn worker[3] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config
git 12730 1 99 15:39 ? 02:20:34 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-47
git 12754 1 99 15:39 ? 02:20:32 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-52Output of checks
This happens on our inhouse installation of gitlab CE.
Results of GitLab environment info
Server OS is CentOS.
Expand for output related to GitLab environment info
[root@gitlab ~]# sudo gitlab-rake gitlab:env:info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Git Version: 2.13.0 Sidekiq Version:5.0.0 Go Version: unknown
GitLab information Version: 9.3.1 Revision: b2dd17b Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://gitlab.ilr.local HTTP Clone URL: http://gitlab.ilr.local/some-group/some-project.git SSH Clone URL: git@gitlab.ilr.local:some-group/some-project.git Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 5.0.5 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
[root@gitlab ~]# sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab Shell ...
GitLab Shell version >= 5.0.5 ? ... OK (5.0.5) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 4/4 ... ok 32/5 ... ok 32/8 ... ok 56/14 ... ok 56/15 ... ok 56/16 ... ok 56/17 ... ok 56/18 ... ok 56/19 ... ok 69/20 ... ok 76/21 ... ok 81/25 ... ok 35/28 ... ok 82/32 ... ok 82/33 ... ok 82/34 ... ok 82/35 ... ok 82/36 ... ok 82/37 ... ok 82/39 ... ok 82/40 ... ok 82/41 ... ok 82/42 ... ok 82/43 ... ok 82/44 ... ok 82/45 ... ok 82/46 ... ok 4/47 ... ok 81/48 ... ok 88/49 ... ok 32/52 ... ok 87/53 ... ok 32/55 ... ok 32/56 ... ok 4/57 ... ok 32/59 ... ok 32/60 ... ok 4/61 ... ok 4/62 ... ok 56/63 ... ok 56/65 ... ok 32/66 ... ok 4/68 ... ok 97/69 ... ok 32/71 ... ok 32/72 ... ok 4/73 ... ok 28/78 ... repository is empty 4/80 ... ok 4/81 ... ok 4/82 ... ok 56/83 ... ok 56/84 ... ok 32/86 ... ok 4/87 ... ok 4/88 ... ok 32/89 ... ok 56/90 ... ok 81/91 ... ok 151/93 ... ok 56/95 ... ok 151/96 ... repository is empty 81/97 ... ok 81/98 ... ok 81/99 ... ok 4/100 ... ok 36/101 ... ok 32/102 ... ok 105/103 ... ok 32/104 ... ok 4/105 ... ok 4/106 ... ok 81/107 ... ok 81/109 ... ok 17/110 ... ok 32/111 ... ok 22/113 ... ok 4/114 ... ok 4/115 ... ok 81/116 ... ok 81/117 ... ok 164/118 ... repository is empty 164/119 ... repository is empty 164/120 ... repository is empty 164/121 ... repository is empty 164/122 ... repository is empty 164/123 ... repository is empty 81/124 ... ok 163/125 ... ok 35/126 ... repository is empty 35/127 ... ok 175/132 ... ok 35/133 ... ok 181/134 ... ok 4/135 ... ok 4/136 ... ok 4/137 ... ok 4/139 ... ok 4/141 ... ok 190/142 ... ok 4/143 ... ok 190/144 ... ok 190/145 ... ok 32/146 ... ok 32/147 ... ok 193/148 ... ok 35/149 ... repository is empty 196/150 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) CONTENT REMOVED - PERSONAL SENSITIVE DATA Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/4 ... yes 32/5 ... yes 32/8 ... yes 56/14 ... yes 56/15 ... yes 56/16 ... yes 56/17 ... yes 56/18 ... yes 56/19 ... yes 69/20 ... yes 76/21 ... yes 81/25 ... yes 35/28 ... yes 82/32 ... yes 82/33 ... yes 82/34 ... yes 82/35 ... yes 82/36 ... yes 82/37 ... yes 82/39 ... yes 82/40 ... yes 82/41 ... yes 82/42 ... yes 82/43 ... yes 82/44 ... yes 82/45 ... yes 82/46 ... yes 4/47 ... yes 81/48 ... yes 88/49 ... yes 32/52 ... yes 87/53 ... yes 32/55 ... yes 32/56 ... yes 4/57 ... yes 32/59 ... yes 32/60 ... yes 4/61 ... yes 4/62 ... yes 56/63 ... yes 56/65 ... yes 32/66 ... yes 4/68 ... yes 97/69 ... yes 32/71 ... yes 32/72 ... yes 4/73 ... yes 28/78 ... yes 4/80 ... yes 4/81 ... yes 4/82 ... yes 56/83 ... yes 56/84 ... yes 32/86 ... yes 4/87 ... yes 4/88 ... yes 32/89 ... yes 56/90 ... yes 81/91 ... yes 151/93 ... yes 56/95 ... yes 151/96 ... yes 81/97 ... yes 81/98 ... yes 81/99 ... yes 4/100 ... yes 36/101 ... yes 32/102 ... yes 105/103 ... yes 32/104 ... yes 4/105 ... yes 4/106 ... yes 81/107 ... yes 81/109 ... yes 17/110 ... yes 32/111 ... yes 22/113 ... yes 4/114 ... yes 4/115 ... yes 81/116 ... yes 81/117 ... yes 164/118 ... yes 164/119 ... yes 164/120 ... yes 164/121 ... yes 164/122 ... yes 164/123 ... yes 81/124 ... yes 163/125 ... yes 35/126 ... yes 35/127 ... yes 175/132 ... yes 35/133 ... yes 181/134 ... yes 4/135 ... yes 4/136 ... yes 4/137 ... yes 4/139 ... yes 4/141 ... yes 190/142 ... yes 4/143 ... yes 190/144 ... yes 190/145 ... yes 32/146 ... yes 32/147 ... yes 193/148 ... yes 35/149 ... yes 196/150 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.3) Git version >= 2.7.3 ? ... yes (2.13.0) Active users: ... 172
Checking GitLab ... Finished
Possible fixes
No idea on fixes, don't know gitlab internals sorry.