omnibus instance fails to restart
Summary
After the recent upgrade to 15.4.x, GitLab no longer manages to start up automatically (e.g. after a reboot).
I'm currently running GitLab-15.4.2 / Community Edition (omnibus). I didn't have any problems of this kind with GitLab-15.3.x.
Steps to reproduce
apt update
apt upgrade
# upgrade gitlab-ce from 15.3.2-ce.0 to 15.4.2-ce.0
reboot
GitLab is being run on a VM on a somewhat oldish host. Currently the assignments are:
type | assignment |
---|---|
CPU | AMD Opteron(tm) Processor 6128 |
#CPUs | 4 |
RAM | 8 GB |
(I know that the CPU is old, and there have been problems in the past (e.g. #368656 (closed)), but this time it seems to be different: there are no segfaults to be found in any logs, and it's possible to bring up the system manually)
What is the current bug behavior?
After a reboot, the GitLab instance is no longer available (timeout on the webinterface)
-
gitlab-ctl status
shows that all services are up and running. -
htop
shows thatpuma
keeps consuming 100% of the available CPUs
i waited for about 30 minutes (without noticing any change), after which i decided that the system will not be able to come up on its own.
What is the expected correct behavior?
after a reboot (or triggered manually), GitLab takes a while to restart (5-10 minutes) and is then available without administrator intervention.
Relevant logs and/or screenshots
gitlab-ctl tail puma
prints the following
{"timestamp":"2022-10-13T06:37:06.037Z","pid":3662,"message":"PumaWorkerKiller: Consuming 4263.06640625 mb with master and 4 workers."}
{"timestamp":"2022-10-13T06:37:26.038Z","pid":3662,"message":"PumaWorkerKiller: Consuming 4264.65234375 mb with master and 4 workers."}
{"timestamp":"2022-10-13T06:37:46.039Z","pid":3662,"message":"PumaWorkerKiller: Consuming 4262.71484375 mb with master and 4 workers."}
{"timestamp":"2022-10-13T06:38:06.040Z","pid":3662,"message":"PumaWorkerKiller: Consuming 4262.56640625 mb with master and 4 workers."}
{"timestamp":"2022-10-13T06:38:26.041Z","pid":3662,"message":"PumaWorkerKiller: Consuming 4266.11328125 mb with master and 4 workers."}
{"timestamp":"2022-10-13T06:38:46.043Z","pid":3662,"message":"PumaWorkerKiller: Consuming 4265.05859375 mb with master and 4 workers."}
originally i thought that this indicates that PumaWorkerKiller keeps killing puma because it is out-of-memory. however, now that I have managed to get my instance up and running (see below) again, it keeps printing these lines every 30 or so seconds, so I guess they are just informal.
i couldn't find anything else that appears to be relevant.
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 10 Current User: git Using RVM: no Ruby Version: 2.7.5p203 Gem Version: 3.1.6 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.7 Sidekiq Version:6.4.2 Go Version: unknown GitLab information Version: 15.4.2 Revision: ef309cf1466 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.10 URL: https://git.example.com HTTP Clone URL: https://git.example.com/some-group/some-project.git SSH Clone URL: git@git.example.com:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: gitlab, github, google_oauth2 GitLab Shell Version: 14.10.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.10.0 ? ... OK (14.10.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking gitlab-reply yes Mailroom enabled? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Anonymous. No
bind_dn
orpassword
configured LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 10/8 ... yes 13/9 ... yes 175/10 ... yes 13/11 ... yes 14/12 ... yes 4/14 ... yes 4/15 ... yes 10/16 ... yes 10/17 ... yes 17/19 ... yes 10/20 ... yes 4/21 ... yes 4/22 ... yes 4/23 ... yes 4/24 ... yes 4/25 ... yes 4/26 ... yes 4/27 ... yes 4/28 ... yes 4/29 ... yes 4/30 ... yes 4/31 ... yes 4/32 ... yes 4/33 ... yes 4/34 ... yes 4/35 ... yes 4/36 ... yes 4/37 ... yes 4/38 ... yes 24/41 ... yes 4/42 ... yes 25/44 ... yes 10/45 ... yes 10/46 ... yes 31/47 ... yes 35/48 ... yes 35/49 ... yes 4/50 ... yes 4/51 ... yes 4/52 ... yes 4/53 ... yes 10/54 ... yes 38/55 ... yes 38/56 ... yes 38/57 ... yes 10/58 ... yes 17/59 ... yes 17/60 ... yes 10/61 ... yes 25/62 ... yes 10/63 ... yes 17/64 ... yes 10/65 ... yes 10/66 ... yes 10/69 ... yes 25/70 ... yes 10/71 ... yes 10/72 ... yes 4/74 ... yes 25/75 ... yes 128/76 ... yes 20/77 ... yes 48/78 ... yes 47/79 ... yes 18/80 ... yes 42/81 ... yes 18/83 ... yes 18/85 ... yes 18/86 ... yes 18/87 ... yes 46/88 ... yes 50/89 ... yes 31/90 ... yes 56/91 ... yes 50/92 ... yes 61/95 ... yes 18/96 ... yes 50/98 ... yes 17/101 ... yes 65/102 ... yes 10/103 ... yes 10/104 ... yes 67/105 ... yes 4/106 ... yes 17/107 ... yes 10/108 ... yes 25/112 ... yes 10/113 ... yes 10/117 ... yes 10/118 ... yes 80/119 ... yes 104/120 ... yes 10/121 ... yes 80/122 ... yes 83/123 ... yes 17/124 ... yes 80/125 ... yes 10/127 ... yes 27/128 ... yes 27/129 ... yes 31/130 ... yes 27/131 ... yes 80/132 ... yes 25/133 ... yes 10/135 ... yes 32/136 ... yes 104/137 ... yes 31/138 ... yes 80/139 ... yes 25/140 ... yes 63/141 ... yes 31/142 ... yes 80/144 ... yes 81/145 ... yes 10/146 ... yes 61/147 ... yes 10/148 ... yes 30/150 ... yes 89/151 ... yes 80/152 ... yes 78/153 ... yes 78/154 ... yes 28/155 ... yes 80/156 ... yes 30/157 ... yes 30/158 ... yes 30/159 ... yes 30/161 ... yes 71/163 ... yes 25/164 ... yes 27/165 ... yes 14/168 ... yes 4/169 ... yes 4/170 ... yes 61/171 ... yes 80/172 ... yes 4/174 ... yes 80/175 ... yes 99/176 ... yes 99/177 ... yes 81/179 ... yes 80/180 ... yes 15/182 ... yes 25/183 ... yes 10/184 ... yes 80/185 ... yes 86/192 ... yes 15/193 ... yes 13/194 ... yes 10/195 ... yes 15/196 ... yes 100/197 ... yes 80/198 ... yes 31/199 ... yes 17/200 ... yes 17/201 ... yes 27/202 ... yes 10/203 ... yes 27/204 ... yes 4/205 ... yes 27/206 ... yes 10/207 ... yes 28/208 ... yes 86/212 ... yes 15/213 ... yes 113/214 ... yes 27/215 ... yes 114/216 ... yes 30/217 ... yes 81/218 ... yes 25/219 ... yes 10/220 ... yes 65/221 ... yes 80/223 ... yes 25/224 ... yes 27/225 ... yes 18/226 ... yes 104/228 ... yes 78/229 ... yes 81/230 ... yes 18/231 ... yes 18/232 ... yes 123/233 ... yes 86/234 ... yes 27/235 ... yes 30/236 ... yes 65/238 ... yes 31/239 ... yes 10/242 ... yes 86/243 ... yes 128/244 ... yes 27/245 ... yes 25/246 ... yes 130/247 ... yes 25/248 ... yes 10/249 ... yes 80/250 ... yes 80/251 ... yes 4/252 ... yes 10/254 ... yes 10/255 ... yes 15/256 ... yes 80/257 ... yes 10/258 ... yes 131/259 ... yes 53/260 ... yes 135/262 ... yes 10/263 ... yes 30/265 ... yes 30/266 ... yes 27/267 ... yes 13/268 ... yes 80/269 ... yes 25/270 ... yes 10/271 ... yes 86/272 ... yes 86/273 ... yes 86/274 ... yes 99/275 ... yes 30/276 ... yes 10/277 ... yes 27/278 ... yes 4/279 ... yes 30/280 ... yes 30/281 ... yes 30/282 ... yes 15/283 ... yes 71/284 ... yes 15/286 ... yes 80/288 ... yes 80/289 ... yes 27/291 ... yes 93/292 ... yes 53/293 ... yes 86/294 ... yes 27/295 ... yes 15/296 ... yes 153/297 ... yes 27/298 ... yes 65/299 ... yes 28/300 ... yes 10/301 ... yes 56/302 ... yes 157/303 ... yes 157/304 ... yes 157/305 ... yes 18/306 ... yes 153/307 ... yes 161/308 ... yes 50/309 ... yes 10/310 ... yes 161/311 ... yes 10/312 ... yes 78/313 ... yes 78/314 ... yes 56/315 ... yes 78/316 ... yes 81/317 ... yes 27/318 ... yes 10/320 ... yes 80/321 ... yes 166/322 ... yes 30/323 ... yes 10/325 ... yes 10/326 ... yes 47/327 ... yes 78/328 ... yes 30/329 ... yes 80/330 ... yes 31/331 ... yes 175/332 ... yes 80/334 ... yes 80/335 ... yes 31/337 ... yes 31/338 ... yes 161/339 ... yes 10/340 ... yes 55/343 ... yes 80/345 ... yes 196/347 ... yes 86/348 ... yes 17/349 ... yes 205/350 ... yes 27/351 ... yes 138/352 ... yes 196/354 ... yes 80/355 ... yes 196/356 ... yes 81/357 ... yes 32/358 ... yes 26/359 ... yes 80/360 ... yes 17/361 ... yes 15/362 ... yes 47/364 ... yes 206/365 ... yes 73/366 ... yes 17/367 ... yes 31/368 ... yes 17/369 ... yes 27/370 ... yes 232/371 ... yes 56/372 ... yes 27/374 ... yes 27/375 ... yes 27/376 ... yes 78/377 ... yes 203/378 ... yes 17/379 ... yes 17/380 ... yes 196/381 ... yes 27/382 ... yes 30/383 ... yes 203/384 ... yes 80/385 ... yes 141/386 ... yes 27/387 ... yes 31/388 ... yes 17/389 ... yes 80/390 ... yes 175/391 ... yes 135/393 ... yes 10/394 ... yes 25/395 ... yes 4/396 ... yes 196/397 ... yes 196/398 ... yes 15/399 ... yes 27/400 ... yes 264/401 ... yes 56/402 ... yes 28/403 ... yes 28/404 ... yes 78/405 ... yes 4/406 ... yes 27/407 ... yes 30/408 ... yes 27/410 ... yes 196/411 ... yes 81/412 ... yes 17/413 ... yes 81/414 ... yes 99/415 ... yes 314/416 ... yes 314/417 ... yes 314/418 ... yes 10/419 ... yes 10/420 ... yes 80/421 ... yes 9/422 ... yes 9/423 ... yes 9/424 ... yes 9/425 ... yes 9/426 ... yes 9/427 ... yes 9/428 ... yes 9/429 ... yes 9/430 ... yes 9/431 ... yes 9/432 ... yes 9/433 ... yes 9/434 ... yes 9/435 ... yes 9/436 ... yes 9/437 ... yes 9/438 ... yes 9/439 ... yes 9/440 ... yes 9/441 ... yes 9/442 ... yes 9/443 ... yes 9/444 ... yes 9/445 ... yes 9/446 ... yes 9/447 ... yes 9/448 ... yes 9/449 ... yes 9/450 ... yes 9/456 ... yes 9/457 ... yes 9/458 ... yes 9/459 ... yes 9/460 ... yes 9/461 ... yes 9/462 ... yes 9/463 ... yes 9/464 ... yes 9/465 ... yes 9/466 ... yes 9/467 ... yes 9/468 ... yes 9/469 ... yes 9/470 ... yes 9/471 ... yes 56/472 ... yes 80/473 ... yes 9/474 ... yes 138/475 ... yes 9/477 ... yes 9/478 ... yes 9/479 ... yes 9/480 ... yes 9/481 ... yes 9/482 ... yes 9/483 ... yes 9/484 ... yes 4/485 ... yes 207/486 ... yes 319/487 ... yes 30/488 ... yes 141/489 ... yes 32/490 ... yes 32/491 ... yes 9/492 ... yes 9/493 ... yes 9/494 ... yes 9/495 ... yes 28/496 ... yes 9/497 ... yes 4/498 ... yes 246/500 ... yes 150/501 ... yes 25/502 ... yes 4/503 ... yes 55/504 ... yes 127/505 ... yes 4/506 ... yes 9/507 ... yes 10/508 ... yes 9/509 ... yes 25/510 ... yes 9/511 ... yes 9/512 ... yes 9/513 ... yes 9/514 ... yes 341/515 ... yes 30/516 ... yes 4/517 ... yes 330/518 ... yes 330/519 ... yes 9/520 ... yes 9/521 ... yes 9/522 ... yes 15/523 ... yes 9/524 ... yes 28/525 ... yes 360/526 ... yes 157/527 ... yes 17/528 ... yes 9/530 ... yes 367/531 ... yes 367/532 ... yes 9/533 ... yes 10/534 ... yes 196/535 ... yes 4/536 ... yes 25/538 ... yes 10/539 ... yes 110/543 ... yes 25/544 ... yes 9/545 ... yes 31/546 ... yes 17/547 ... yes 14/548 ... yes 17/549 ... yes 14/550 ... yes 394/551 ... yes 27/552 ... yes 9/553 ... yes 10/554 ... yes 31/556 ... yes 330/557 ... yes 28/558 ... yes 9/559 ... yes 73/560 ... yes 61/561 ... yes 438/568 ... yes 14/569 ... yes 9/570 ... yes 9/571 ... yes 9/572 ... yes 31/573 ... yes 436/574 ... yes 436/575 ... yes 436/576 ... yes 439/577 ... yes 440/579 ... yes 441/580 ... yes 78/581 ... yes 56/582 ... yes 81/583 ... yes 10/584 ... yes 15/586 ... yes 9/587 ... yes 9/588 ... yes 10/589 ... yes 10/590 ... yes 10/591 ... yes 9/593 ... yes 9/594 ... yes 80/595 ... yes 31/596 ... yes 141/597 ... yes 28/598 ... yes 9/599 ... yes 4/600 ... yes 196/601 ... yes 9/602 ... yes 10/603 ... yes 80/604 ... yes 15/606 ... yes 51/607 ... yes 81/608 ... yes 10/609 ... yes 78/610 ... yes 18/611 ... yes 65/612 ... yes 15/613 ... yes 9/614 ... yes 14/616 ... yes 289/617 ... yes 485/619 ... yes 9/620 ... yes 17/621 ... yes 81/622 ... yes 17/623 ... yes 81/624 ... yes 25/625 ... yes 28/626 ... yes 48/627 ... yes 28/628 ... yes 32/629 ... yes 31/630 ... yes 30/631 ... yes 9/632 ... yes 78/633 ... yes 110/634 ... yes 238/635 ... yes 238/636 ... yes 15/637 ... yes 157/638 ... yes 14/639 ... yes 9/640 ... yes 9/641 ... yes 74/642 ... yes 9/644 ... yes 130/646 ... yes 141/647 ... yes 501/648 ... yes 17/649 ... yes 503/650 ... yes 503/651 ... yes 81/652 ... yes 30/654 ... yes 9/655 ... yes 74/657 ... yes 81/659 ... yes 17/661 ... yes 10/662 ... yes 4/663 ... yes 157/664 ... yes 330/666 ... yes 9/668 ... yes 196/669 ... yes 17/670 ... yes 196/671 ... yes 510/672 ... yes 17/673 ... yes 9/674 ... yes 141/675 ... yes 27/676 ... yes 9/677 ... yes 51/678 ... yes 31/679 ... yes 83/680 ... yes 99/681 ... yes 9/682 ... yes 25/683 ... yes 81/684 ... yes 524/685 ... yes 73/686 ... yes 9/687 ... yes 485/688 ... yes 81/689 ... yes 503/690 ... yes 9/691 ... yes 9/692 ... yes 73/693 ... yes 81/694 ... yes 532/695 ... yes 10/696 ... yes 9/697 ... yes 17/698 ... yes 17/699 ... yes 9/700 ... yes 238/701 ... yes 51/702 ... yes 17/703 ... yes 289/704 ... yes 4/705 ... yes 78/707 ... yes 289/708 ... yes 529/709 ... yes 130/710 ... yes 497/711 ... yes 15/712 ... yes 31/713 ... yes 73/714 ... yes 73/715 ... yes 341/716 ... yes 53/717 ... yes 15/718 ... yes 81/719 ... yes 53/720 ... yes 497/721 ... yes 558/723 ... yes 558/724 ... yes 558/725 ... yes 314/727 ... yes 53/728 ... yes 17/730 ... yes 503/731 ... yes 17/732 ... yes 4/733 ... yes 17/734 ... yes 17/735 ... yes 17/736 ... yes 17/737 ... yes 17/738 ... yes 17/739 ... yes 4/740 ... yes 10/741 ... yes 330/742 ... yes 81/743 ... yes 588/744 ... yes 580/745 ... yes 289/746 ... yes 28/748 ... yes 9/749 ... yes 529/750 ... yes 9/751 ... yes 588/752 ... yes 9/753 ... yes 9/754 ... yes 9/755 ... yes 14/756 ... yes 594/760 ... yes 594/761 ... yes 289/762 ... yes 128/763 ... yes 529/764 ... yes 17/765 ... yes 81/766 ... yes 17/767 ... yes 25/769 ... yes 605/771 ... yes 25/772 ... yes 605/773 ... yes 81/774 ... yes 17/776 ... yes 504/777 ... yes 31/778 ... yes 9/780 ... yes 17/781 ... yes 9/782 ... yes 621/784 ... yes 81/785 ... yes 340/786 ... yes 9/787 ... yes 161/792 ... yes 10/793 ... yes 15/794 ... yes 529/795 ... yes 9/796 ... yes 1448/797 ... yes 196/798 ... yes 18/799 ... yes 4/800 ... yes 31/801 ... yes 31/802 ... yes 497/804 ... yes 10/805 ... yes 161/807 ... yes 679/808 ... yes 17/809 ... yes 688/810 ... yes 31/811 ... yes 9/812 ... yes 485/813 ... yes 246/814 ... yes 25/815 ... yes 246/816 ... yes 485/817 ... yes 485/818 ... yes 724/822 ... yes 497/823 ... yes 31/827 ... yes 712/829 ... yes 289/832 ... yes 330/833 ... yes 330/834 ... yes 717/836 ... yes 17/837 ... yes 289/838 ... yes 51/839 ... yes 497/840 ... yes 150/842 ... yes 10/843 ... yes 485/844 ... yes 289/846 ... yes 314/847 ... yes 724/848 ... yes 726/849 ... yes 128/850 ... yes 524/851 ... yes 330/852 ... yes 330/853 ... yes 594/855 ... yes 289/857 ... yes 27/858 ... yes 130/859 ... yes 77/860 ... yes 10/861 ... yes 9/862 ... yes 15/863 ... yes 740/864 ... yes 485/866 ... yes 743/869 ... yes 748/871 ... yes 65/873 ... yes 17/874 ... yes 314/875 ... yes 497/876 ... yes 51/877 ... yes 524/878 ... yes 553/879 ... yes 757/880 ... yes 758/881 ... yes 524/882 ... yes 78/883 ... yes 524/884 ... yes 330/885 ... yes 485/886 ... yes 14/887 ... yes 10/888 ... yes 61/889 ... yes 10/890 ... yes 14/891 ... yes 772/892 ... yes 497/893 ... yes 9/894 ... yes 485/895 ... yes 14/896 ... yes 196/897 ... yes 330/898 ... yes 246/899 ... yes 524/900 ... yes 9/901 ... yes 289/903 ... yes 330/904 ... yes 10/905 ... yes 28/906 ... yes 524/907 ... yes 10/908 ... yes 4/909 ... yes 330/910 ... yes 524/911 ... yes 524/912 ... yes 31/914 ... yes 10/915 ... yes 594/916 ... yes 10/917 ... yes 14/918 ... yes 14/919 ... yes 141/920 ... yes 10/921 ... yes 4/922 ... yes 9/923 ... yes 524/924 ... yes 157/926 ... yes 51/927 ... yes 341/928 ... yes 800/930 ... yes 800/931 ... yes 78/932 ... yes 662/933 ... yes 800/934 ... yes 15/935 ... yes 65/936 ... yes 9/937 ... yes 9/938 ... yes 141/939 ... yes 800/940 ... yes 25/941 ... yes 10/942 ... yes 524/943 ... yes 524/944 ... yes 31/945 ... yes 9/946 ... yes 9/947 ... yes 9/948 ... yes 289/949 ... yes 835/951 ... yes 690/952 ... yes 51/953 ... yes 10/954 ... yes 9/955 ... yes 723/956 ... yes 330/957 ... yes 18/958 ... yes 10/959 ... yes 17/960 ... yes 25/961 ... yes 25/962 ... yes 894/966 ... yes 4/967 ... yes 157/968 ... yes 757/969 ... yes 799/971 ... yes 25/972 ... yes 31/973 ... yes 18/974 ... yes 51/975 ... yes 800/976 ... yes 883/977 ... yes 31/978 ... yes 895/979 ... yes 895/980 ... yes 809/981 ... yes 289/982 ... yes 9/983 ... yes 902/984 ... yes 9/985 ... yes 61/986 ... yes 15/987 ... yes 207/989 ... yes 919/990 ... yes 4/992 ... yes 31/993 ... yes 9/994 ... yes 575/995 ... yes 28/998 ... yes 289/1000 ... yes 524/1001 ... yes 919/1002 ... yes 919/1003 ... yes 919/1004 ... yes 919/1005 ... yes 919/1006 ... yes 919/1007 ... yes 919/1008 ... yes 919/1009 ... yes 919/1010 ... yes 919/1011 ... yes 289/1012 ... yes 161/1013 ... yes 28/1014 ... yes 28/1015 ... yes 28/1016 ... yes 497/1017 ... yes 30/1018 ... yes 988/1019 ... yes 839/1020 ... yes 48/1021 ... yes 62/1022 ... yes 74/1025 ... yes 993/1027 ... yes 14/1028 ... yes 1002/1029 ... yes 524/1031 ... yes 74/1032 ... yes 1015/1034 ... yes 1026/1035 ... yes 1018/1036 ... yes 10/1037 ... yes 10/1038 ... yes 1040/1039 ... yes 1041/1041 ... yes 1041/1042 ... yes 723/1043 ... yes 827/1044 ... yes 289/1045 ... yes 805/1046 ... yes 25/1047 ... yes 497/1048 ... yes 1041/1050 ... yes 110/1051 ... yes 17/1052 ... yes 524/1053 ... yes 524/1054 ... yes 31/1055 ... yes 9/1056 ... yes 31/1057 ... yes 9/1058 ... yes 9/1059 ... yes 805/1060 ... yes 800/1061 ... yes 800/1062 ... yes 497/1063 ... yes 28/1064 ... yes 289/1065 ... yes 524/1066 ... yes 4/1067 ... yes 110/1068 ... yes 4/1069 ... yes 289/1070 ... yes 800/1071 ... yes 24/1072 ... yes 17/1073 ... yes 25/1075 ... yes 524/1078 ... yes 9/1079 ... yes 1041/1080 ... yes 1166/1081 ... yes 196/1082 ... yes 1041/1083 ... yes 51/1084 ... yes 9/1085 ... yes 9/1086 ... yes 150/1087 ... yes 196/1088 ... yes 1175/1089 ... yes 83/1090 ... yes 1175/1091 ... yes 289/1092 ... yes 10/1093 ... yes 31/1094 ... yes 524/1095 ... yes 289/1096 ... yes 650/1097 ... yes 956/1099 ... yes 485/1101 ... yes 800/1102 ... yes 157/1103 ... yes 196/1104 ... yes 1208/1105 ... yes 1208/1106 ... yes 1208/1107 ... yes 1208/1108 ... yes 1208/1109 ... yes 485/1110 ... yes 196/1111 ... yes 65/1112 ... yes 485/1114 ... yes 758/1115 ... yes 17/1116 ... yes 17/1117 ... yes 65/1118 ... yes 4/1119 ... yes 1223/1121 ... yes 1230/1122 ... yes 65/1126 ... yes 662/1127 ... yes 485/1128 ... yes 1323/1129 ... yes 65/1131 ... yes 1321/1132 ... yes 1242/1133 ... yes 1243/1135 ... yes 1245/1136 ... yes 1245/1137 ... yes 1245/1138 ... yes 1245/1139 ... yes 572/1140 ... yes 1245/1141 ... yes 1243/1142 ... yes 1245/1143 ... yes 1245/1144 ... yes 524/1145 ... yes 1245/1146 ... yes 524/1147 ... yes 4/1148 ... yes 10/1149 ... yes 10/1150 ... yes 1208/1151 ... yes 1270/1152 ... yes 485/1153 ... yes 1208/1154 ... yes 497/1155 ... yes 17/1156 ... yes 524/1157 ... yes 14/1158 ... yes 128/1159 ... yes 25/1160 ... yes 38/1161 ... yes 1041/1162 ... yes 1037/1163 ... yes 341/1164 ... yes 485/1166 ... yes 17/1167 ... yes 988/1168 ... yes 659/1169 ... yes 477/1170 ... yes 1311/1171 ... yes 789/1172 ... yes 827/1173 ... yes 837/1174 ... yes 1016/1175 ... yes 1300/1176 ... yes 1038/1177 ... yes 1316/1178 ... yes 1313/1180 ... yes 1310/1181 ... yes 1312/1182 ... yes 207/1183 ... yes 1321/1184 ... yes 1323/1185 ... yes 17/1186 ... yes 757/1187 ... yes 150/1188 ... yes 800/1189 ... yes 1230/1190 ... yes 1331/1194 ... yes 1331/1196 ... yes 485/1200 ... yes 485/1201 ... yes 246/1202 ... yes 757/1204 ... yes 758/1205 ... yes 17/1206 ... yes 17/1207 ... yes 17/1208 ... yes 51/1209 ... yes 645/1210 ... yes 1356/1211 ... yes 679/1212 ... yes 809/1213 ... yes 10/1214 ... yes 25/1215 ... yes 25/1216 ... yes 485/1217 ... yes 1382/1222 ... yes 1382/1223 ... yes 1382/1224 ... yes 1382/1225 ... yes 1323/1226 ... yes 1390/1227 ... yes 83/1228 ... yes 4/1229 ... yes 83/1230 ... yes 1392/1231 ... yes 51/1232 ... yes 127/1233 ... yes 127/1234 ... yes 196/1235 ... yes 1320/1236 ... yes 25/1237 ... yes 175/1238 ... yes 1433/1239 ... yes 1321/1240 ... yes 1230/1241 ... yes 1230/1243 ... yes 1230/1244 ... yes 1230/1247 ... yes 1433/1248 ... yes 1230/1249 ... yes 1230/1250 ... yes 485/1251 ... yes 1230/1252 ... yes 1230/1253 ... yes 1448/1254 ... yes 1321/1255 ... yes 28/1256 ... yes 141/1257 ... yes 497/1258 ... yes 196/1259 ... yes 196/1260 ... yes 141/1261 ... yes 141/1262 ... yes 497/1263 ... yes 17/1264 ... yes 580/1265 ... yes 28/1266 ... yes 128/1267 ... yes 1382/1268 ... yes 150/1269 ... yes 560/1270 ... yes 1321/1271 ... yes 1433/1272 ... yes 196/1273 ... yes 1323/1274 ... yes 31/1275 ... yes 150/1276 ... yes 1382/1277 ... yes 1382/1278 ... yes 31/1279 ... yes 1433/1280 ... yes 1352/1281 ... yes 723/1283 ... yes 497/1285 ... yes 681/1286 ... yes 196/1287 ... yes 1433/1288 ... yes 141/1289 ... yes 1527/1290 ... yes 1527/1291 ... yes 667/1292 ... yes 10/1293 ... yes 65/1294 ... yes 689/1295 ... yes 1552/1296 ... yes 1527/1297 ... yes 1245/1298 ... yes 723/1299 ... yes 789/1300 ... yes 1527/1301 ... yes 560/1303 ... yes 17/1304 ... yes 1245/1305 ... yes 14/1306 ... yes 17/1307 ... yes 2700/1308 ... yes 723/1309 ... yes 1527/1311 ... yes 289/1312 ... yes 553/1313 ... yes 17/1314 ... yes 524/1315 ... yes 31/1316 ... yes 1504/1317 ... yes 690/1318 ... yes 1230/1319 ... yes 28/1320 ... yes 1527/1321 ... yes 497/1322 ... yes 17/1323 ... yes 289/1324 ... yes 1527/1325 ... yes 10/1326 ... yes 2765/1327 ... yes 2767/1328 ... yes 477/1329 ... yes 1433/1330 ... yes 18/1331 ... yes 1527/1332 ... yes 18/1333 ... yes 497/1334 ... yes 497/1335 ... yes 1421/1336 ... yes 497/1337 ... yes 695/1338 ... yes 10/1339 ... yes 25/1340 ... yes 131/1341 ... yes 497/1342 ... yes 553/1343 ... yes 51/1344 ... yes 131/1345 ... yes 2822/1346 ... yes 497/1347 ... yes 2822/1348 ... yes 1527/1349 ... yes 1433/1350 ... yes 827/1351 ... yes 10/1352 ... yes 643/1353 ... yes 2823/1354 ... yes 2823/1355 ... yes 723/1356 ... yes 1352/1357 ... yes 809/1358 ... yes 575/1359 ... yes 497/1360 ... yes 2862/1362 ... yes 1433/1363 ... yes 2728/1364 ... yes 1402/1365 ... yes 2728/1366 ... yes 1552/1367 ... yes 553/1368 ... yes 4/1369 ... yes 688/1370 ... yes 30/1371 ... yes 497/1372 ... yes 17/1373 ... yes 1245/1374 ... yes 10/1375 ... yes 10/1376 ... yes 1245/1377 ... yes 289/1378 ... yes 17/1379 ... yes 10/1380 ... yes 2918/1381 ... yes 30/1382 ... yes 2728/1383 ... yes 2728/1384 ... yes 2821/1385 ... yes 2918/1386 ... yes 1321/1387 ... yes 2821/1388 ... yes 2862/1389 ... yes 1331/1390 ... yes 1331/1391 ... yes 1331/1392 ... yes 2862/1394 ... yes 2728/1395 ... yes 485/1396 ... yes 17/1397 ... yes 341/1398 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.5) Git user has default SSH configuration? ... yes Active users: ... 666 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
Probably it's enough to just throw more resources on the instance. Unfortunately, this is not always easy to do.
Manually starting the system
What is interesting, is that I can manually bring up the instance by starting jobs individually (and wait between jobs that appear to consume a log of resources until they have somewhat settled).
This is how I got the system up and running in the end:
gitlab-ctl stop
gitlab-ctl start postgresql
gitlab-ctl start puma
# wait
gitlab-ctl start redis
gitlab-ctl start postgres-exporter
gitlab-ctl start redis-exporter
gitlab-ctl start sidekiq
# wait
gitlab-ctl start gitaly
# wait
gitlab-ctl start gitlab-workhorse
# wait
gitlab-ctl start alertmanager
gitlab-ctl start gitlab-kas
gitlab-ctl start logrotate
gitlab-ctl start mailroom
gitlab-ctl start registry
gitlab-ctl start gitlab-exporter
gitlab-ctl start grafana
gitlab-ctl start node-exporter
gitlab-ctl start gitlab-pages
gitlab-ctl start mattermost
# wait
gitlab-ctl start nginx
Of course this is somewhat suboptimal^Wunusable, as it relies on a human performing the steps, instead of gitlab-ctl
doing everything for me.