Registry process repeatedly turns off at random intervals
Summary
The registry
process of my Gitlab Omnibus deployment periodically shuts down for periods of approximately 10 minutes at random intervals between 2 and 60 minutes. This interruption causes CI attempting to access the registry to fail with 502 Bad Gateway errors from NGINX.
Steps to reproduce
Unfortunately, I have been unable to reproduce this issue. I have deployed a fresh gitlab install onto an Ubuntu 16.04 VM and monitored the registry service via watch gitlab-ctl status registry
for at least an hour (the time interval mentioned in the "upload purge" log event) after each step attempting to reproduce the issue:
- After fresh install of 13.4.3, enable the registry and wait: the process stays up
- Upgrade postgres to 12.4 and restart gitlab and wait: the process stays up
- Run
gitlab-ctl registry-garbage-collect
and then wait: the process restarts immediately after this task then doesn't fail out again - Run
gitlab-ctl registry-garbage-collect -m
and then wait: the process restarts immediately after this task then doesn't fail out again
What is the current bug behavior?
The registry process is randomly shutting off and then restarting.
What is the expected correct behavior?
The registry process should continue to run alongside the other Gitlab components without interruption.
Relevant logs and/or screenshots
==> /var/log/gitlab/registry/current <==
The following log sequence of events appears whenever the registry process restarts:
2020-10-14_11:17:04.96252 time="2020-10-14T07:17:04.962214924-04:00" level=info msg="redis not configured" environment=production go.version=go1.14.7 instance.id=masked 044 service=registry version=v2.10.1-gitlab
2020-10-14_11:17:04.96302 time="2020-10-14T07:17:04.962405925-04:00" level=info msg="Starting upload purge in 40m0s" environment=production go.version=go1.14.7 instance.id=masked service=registry version=v2.10.1-gitlab
2020-10-14_11:17:04.99036 time="2020-10-14T07:17:04.990271219-04:00" level=info msg="using inmemory blob descriptor cache" environment=production go.version=go1.14.7 instance.id=masked service=registry version=v2.10.1-gitlab
2020-10-14_11:17:04.99169 time="2020-10-14T07:17:04.991590328-04:00" level=info msg="listening on 127.0.0.1:5000" environment=production go.version=go1.14.7 instance.id=masked service=registry version=v2.10.1-gitlab
The time value (X) in the second message "Starting upload purge in X" seems to be a rough forecast of when the registry process will next die. Just before the registry process dies, I see this sequence of events repeat itself:
2020-10-14_12:53:04.46454 time="2020-10-14T08:53:04.464327864-04:00" level=info msg="PurgeUploads starting: olderThan=2020-10-07 08:53:04.464248363 -0400 EDT m=-602639.958736471, actuallyDelete=true"
2020-10-14_12:53:04.47472 time="2020-10-14T08:53:04.474585532-04:00" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"
2020-10-14_12:53:04.47475 time="2020-10-14T08:53:04.474652732-04:00" level=info msg="Starting upload purge in 24h0m0s" environment=production go.version=go1.14.7 instance.id=masked service=registry version=v2.10.1-gitlab
Results of GitLab environment info
Expand for output related to GitLab environment info
``` System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.28.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.4.3 Revision: 6689552c479 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.4 URL: https://something.somedomain.com HTTP Clone URL: https://something.somedomain.com/some-group/some-project.git SSH Clone URL: git@something.somedomain.com:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.7.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git ```
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 13.7.0 ? ... OK (13.7.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 84 users of 100 limit. Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 78/1 ... yes 5/2 ... yes 2/3 ... yes 47/4 ... yes 47/5 ... yes 2/6 ... yes 47/7 ... yes 47/8 ... yes 47/9 ... yes 47/10 ... yes 47/11 ... yes 2/12 ... yes 47/13 ... yes 47/14 ... yes 2/15 ... yes 47/16 ... yes 47/17 ... yes 47/18 ... yes 47/19 ... yes 3/20 ... yes 75/21 ... yes 5/22 ... yes 5/23 ... yes 39/24 ... yes 39/25 ... yes 41/26 ... yes 41/27 ... yes 39/28 ... yes 5/29 ... yes 41/30 ... yes 41/31 ... yes 31/32 ... yes 31/33 ... yes 31/34 ... yes 32/35 ... yes 32/36 ... yes 32/37 ... yes 32/38 ... yes 32/39 ... yes 32/40 ... yes 32/41 ... yes 32/42 ... yes 31/43 ... yes 32/44 ... yes 32/45 ... yes 7/46 ... yes 7/47 ... yes 7/48 ... yes 7/49 ... yes 7/50 ... yes 7/51 ... yes 8/52 ... yes 9/53 ... yes 32/54 ... yes 32/55 ... yes 32/56 ... yes 32/57 ... yes 32/58 ... yes 9/59 ... yes 9/60 ... yes 9/61 ... yes 9/62 ... yes 9/63 ... yes 32/64 ... yes 32/65 ... yes 32/66 ... yes 32/67 ... yes 9/68 ... yes 9/69 ... yes 9/70 ... yes 32/71 ... yes 9/72 ... yes 10/73 ... yes 10/74 ... yes 10/75 ... yes 10/76 ... yes 10/77 ... yes 10/78 ... yes 10/79 ... yes 10/80 ... yes 10/81 ... yes 47/82 ... yes 47/84 ... yes 11/85 ... yes 54/86 ... yes 54/87 ... yes 53/88 ... yes 53/89 ... yes 32/90 ... yes 17/91 ... yes 17/92 ... yes 17/93 ... yes 17/94 ... yes 17/95 ... yes 11/96 ... yes 10/97 ... yes 11/98 ... yes 78/100 ... yes 75/101 ... yes 11/102 ... yes 11/103 ... yes 53/104 ... yes 2/108 ... yes 20/109 ... yes 10/110 ... yes 20/111 ... yes 20/112 ... yes 20/113 ... yes 55/114 ... yes 20/115 ... yes 52/116 ... yes 34/117 ... yes 20/118 ... yes 20/119 ... yes 20/120 ... yes 6/121 ... yes 20/122 ... yes 20/123 ... yes 20/124 ... yes 22/126 ... yes 22/127 ... yes 58/128 ... yes 22/129 ... yes 58/130 ... yes 58/131 ... yes 28/134 ... yes 28/135 ... yes 22/137 ... yes 20/138 ... yes 6/141 ... yes 42/142 ... yes 42/143 ... yes 22/148 ... yes 6/152 ... yes 28/153 ... yes 34/154 ... yes 22/155 ... yes 28/156 ... yes 36/159 ... yes 36/160 ... yes 36/161 ... yes 36/162 ... yes 36/163 ... yes 36/164 ... yes 36/165 ... yes 36/166 ... yes 36/167 ... yes 78/169 ... yes 11/170 ... yes 11/171 ... yes 37/172 ... yes 20/175 ... yes 42/176 ... yes 42/177 ... yes 70/178 ... yes 45/179 ... yes 46/180 ... yes 47/181 ... yes 47/182 ... yes 47/183 ... yes 47/184 ... yes 47/185 ... yes 2/186 ... yes 56/188 ... yes 6/189 ... yes 57/190 ... yes 57/191 ... yes 6/195 ... yes 31/196 ... yes 6/197 ... yes 125/198 ... yes 6/199 ... yes 2/200 ... yes 78/201 ... yes 37/202 ... yes 78/203 ... yes 60/204 ... yes 59/205 ... yes 73/206 ... yes 35/208 ... yes 35/209 ... yes 31/210 ... yes 35/211 ... yes 31/212 ... yes 35/214 ... yes 35/215 ... yes 35/216 ... yes 23/217 ... yes 13/218 ... yes 78/219 ... yes 31/220 ... yes 21/222 ... yes 23/223 ... yes 10/224 ... yes 10/225 ... yes 64/226 ... yes 64/227 ... yes 64/228 ... yes 64/229 ... yes 65/231 ... yes 6/232 ... yes 64/234 ... yes 65/235 ... yes 65/236 ... yes 65/237 ... yes 65/238 ... yes 13/240 ... yes 65/241 ... yes 70/242 ... yes 70/243 ... yes 13/244 ... yes 23/245 ... yes 70/246 ... yes 22/247 ... yes 70/248 ... yes 72/249 ... yes 72/250 ... yes 72/251 ... yes 72/253 ... yes 72/254 ... yes 72/255 ... yes 72/256 ... yes 72/257 ... yes 72/258 ... yes 72/259 ... yes 72/260 ... yes 72/261 ... yes 22/262 ... yes 22/263 ... yes 64/264 ... yes 65/265 ... yes 64/266 ... yes 64/267 ... yes 64/268 ... yes 35/269 ... yes 64/270 ... yes 73/272 ... yes 64/273 ... yes 64/274 ... yes 10/275 ... yes 73/276 ... yes 73/277 ... yes 35/278 ... yes 64/279 ... yes 73/280 ... yes 74/281 ... yes 74/282 ... yes 116/283 ... yes 116/286 ... yes 76/287 ... yes 77/288 ... yes 77/289 ... yes 77/290 ... yes 35/291 ... yes 35/292 ... yes 70/293 ... yes 64/294 ... yes 64/295 ... yes 73/296 ... yes 37/297 ... yes 35/298 ... yes 78/299 ... yes 35/300 ... yes 35/301 ... yes 35/302 ... yes 35/303 ... yes 35/304 ... yes 35/305 ... yes 35/306 ... yes 35/307 ... yes 35/308 ... yes 35/309 ... yes 35/310 ... yes 35/311 ... yes 37/312 ... yes 37/313 ... yes 22/314 ... yes 22/315 ... yes 13/316 ... yes 13/317 ... yes 13/318 ... yes 13/319 ... yes 80/321 ... yes 80/324 ... yes 13/325 ... yes 57/326 ... yes 85/327 ... yes 59/328 ... yes 13/329 ... yes 82/331 ... yes 13/332 ... yes 21/333 ... yes 63/335 ... yes 63/336 ... yes 70/337 ... yes 85/338 ... yes 92/339 ... yes 63/340 ... yes 89/341 ... yes 92/342 ... yes 83/343 ... yes 28/345 ... yes 63/346 ... yes 125/347 ... yes 82/348 ... yes 92/350 ... yes 85/351 ... yes 85/352 ... yes 85/353 ... yes 63/354 ... yes 63/355 ... yes 63/356 ... yes 63/357 ... yes 63/358 ... yes 63/359 ... yes 22/360 ... yes 22/361 ... yes 83/362 ... yes 83/363 ... yes 63/364 ... yes 63/365 ... yes 63/366 ... yes 63/367 ... yes 63/368 ... yes 63/369 ... yes 63/370 ... yes 63/371 ... yes 115/372 ... yes 115/373 ... yes 115/374 ... yes 115/375 ... yes 94/376 ... yes 115/377 ... yes 94/378 ... yes 95/379 ... yes 96/380 ... yes 96/381 ... yes 96/382 ... yes 22/383 ... yes 70/384 ... yes 94/385 ... yes 97/386 ... yes 59/387 ... yes 59/388 ... yes 125/389 ... yes 116/390 ... yes 116/391 ... yes 92/393 ... yes 97/394 ... yes 104/395 ... yes 102/397 ... yes 92/398 ... yes 103/399 ... yes 21/400 ... yes 103/401 ... yes 102/402 ... yes 10/403 ... yes 72/404 ... yes 72/405 ... yes 72/406 ... yes 72/407 ... yes 102/408 ... yes 72/409 ... yes 72/410 ... yes 74/411 ... yes 102/412 ... yes 22/413 ... yes 115/415 ... yes 94/416 ... yes 72/417 ... yes 6/418 ... yes 115/419 ... yes 115/420 ... yes 22/421 ... yes 22/423 ... yes 70/424 ... yes 130/425 ... yes 130/426 ... yes 130/427 ... yes 11/428 ... yes 74/429 ... yes 106/430 ... yes 120/431 ... yes 22/432 ... yes 113/433 ... yes 113/434 ... yes 70/435 ... yes 113/436 ... yes 114/437 ... yes 106/438 ... yes 106/439 ... yes 108/440 ... yes 130/441 ... yes 130/442 ... yes 114/443 ... yes 113/444 ... yes 115/445 ... yes 114/447 ... yes 72/448 ... yes 114/449 ... yes 74/450 ... yes 74/451 ... yes 74/452 ... yes 114/453 ... yes 103/455 ... yes 16/456 ... yes 125/459 ... yes 113/460 ... yes 100/462 ... yes 94/463 ... yes 113/464 ... yes 113/465 ... yes 117/466 ... yes 117/467 ... yes 10/468 ... yes 23/470 ... yes 110/471 ... yes 108/473 ... yes 108/474 ... yes 120/475 ... yes 114/476 ... yes 74/478 ... yes 23/479 ... yes 114/483 ... yes 114/484 ... yes 22/485 ... yes 10/486 ... yes 120/487 ... yes 92/488 ... yes 114/490 ... yes 125/491 ... yes 102/492 ... yes 122/493 ... yes 121/494 ... yes 92/496 ... yes 119/497 ... yes 123/498 ... yes 123/500 ... yes 123/501 ... yes 123/502 ... yes 121/503 ... yes 114/504 ... yes 124/505 ... yes 125/506 ... yes 125/507 ... yes 125/508 ... yes 125/509 ... yes 114/511 ... yes 114/512 ... yes 120/514 ... yes 124/515 ... yes 22/516 ... yes 127/518 ... yes 13/519 ... yes 92/520 ... yes 74/521 ... yes 120/522 ... yes 101/523 ... yes 119/524 ... yes 128/525 ... yes 128/526 ... yes 127/527 ... yes 128/528 ... yes 128/529 ... yes 128/530 ... yes 128/531 ... yes 128/532 ... yes 128/533 ... yes 125/534 ... yes 120/535 ... yes 114/536 ... yes 114/537 ... yes 97/538 ... yes 106/539 ... yes 108/540 ... yes 101/541 ... yes 127/542 ... yes 108/543 ... yes 108/544 ... yes 108/545 ... yes 108/546 ... yes 23/547 ... yes 104/548 ... yes 129/550 ... yes 114/551 ... yes 108/552 ... yes 62/555 ... yes 62/556 ... yes 133/558 ... yes 113/559 ... yes 134/560 ... yes 134/561 ... yes 62/562 ... yes 135/563 ... yes 133/564 ... yes 133/565 ... yes 108/566 ... yes 133/567 ... yes 122/569 ... yes 122/570 ... yes 122/571 ... yes 143/572 ... yes 122/573 ... yes 23/574 ... yes 137/575 ... yes 142/576 ... yes 138/577 ... yes 133/578 ... yes 138/579 ... yes 141/581 ... yes 136/582 ... yes 134/583 ... yes 138/584 ... yes 124/585 ... yes 142/588 ... yes 131/589 ... yes 133/590 ... yes 147/591 ... yes 141/592 ... yes 141/593 ... yes 147/594 ... yes 138/595 ... yes 139/596 ... yes 139/597 ... yes 133/598 ... yes 98/599 ... yes 108/600 ... yes 108/601 ... yes 133/602 ... yes 148/603 ... yes 148/604 ... yes 148/605 ... yes 148/606 ... yes 148/607 ... yes 108/608 ... yes 149/609 ... yes 149/610 ... yes 149/611 ... yes 149/612 ... yes 149/613 ... yes 149/614 ... yes 148/615 ... yes 150/616 ... yes 125/617 ... yes 142/618 ... yes 108/619 ... yes 134/620 ... yes 134/621 ... yes 151/622 ... yes 134/623 ... yes 108/624 ... yes 125/625 ... yes 133/628 ... yes 10/629 ... yes 139/631 ... yes 139/632 ... yes 139/633 ... yes 139/634 ... yes 133/637 ... yes 139/638 ... yes 133/639 ... yes 139/640 ... yes 131/641 ... yes 148/642 ... yes 139/643 ... yes 161/644 ... yes 133/645 ... yes 134/646 ... yes 161/647 ... yes 133/648 ... yes 133/649 ... yes 133/650 ... yes 133/651 ... yes 133/652 ... yes 133/654 ... yes 133/656 ... yes 101/657 ... yes 139/660 ... yes 139/661 ... yes 133/662 ... yes 149/663 ... yes 158/664 ... yes 158/665 ... yes 158/666 ... yes 158/667 ... yes 23/670 ... yes 133/671 ... yes 133/672 ... yes 133/673 ... yes 138/674 ... yes 138/675 ... yes 138/676 ... yes 133/677 ... yes 133/678 ... yes 133/679 ... yes 133/680 ... yes 160/681 ... yes 94/682 ... yes 22/684 ... yes 161/685 ... yes 161/686 ... yes 161/687 ... yes 139/688 ... yes 139/689 ... yes 139/690 ... yes 139/691 ... yes 138/692 ... yes 138/693 ... yes 138/694 ... yes 134/695 ... yes 134/696 ... yes 134/697 ... yes 137/698 ... yes 6/699 ... yes 108/701 ... yes 133/702 ... yes 164/704 ... yes 163/705 ... yes 108/706 ... yes 164/707 ... yes 164/708 ... yes 164/709 ... yes 133/710 ... yes 163/711 ... yes 163/713 ... yes 108/714 ... yes 164/715 ... yes 163/716 ... yes 163/717 ... yes 133/718 ... yes 159/719 ... yes 108/720 ... yes 108/721 ... yes 108/722 ... yes 133/723 ... yes 173/724 ... yes 161/725 ... yes 133/726 ... yes 161/727 ... yes 161/728 ... yes 161/729 ... yes 161/730 ... yes 108/731 ... yes 133/732 ... yes 133/733 ... yes 133/734 ... yes 133/735 ... yes 164/736 ... yes 10/737 ... yes 10/738 ... yes 158/740 ... yes 158/741 ... yes 158/742 ... yes 164/743 ... yes 1/751 ... yes 164/755 ... yes 164/756 ... yes 164/757 ... yes 164/760 ... yes 133/761 ... yes 133/762 ... yes 106/764 ... yes 182/765 ... yes 182/766 ... yes 182/767 ... yes 131/768 ... yes 133/769 ... yes 133/770 ... yes 104/771 ... yes 133/772 ... yes 133/773 ... yes 133/774 ... yes 140/775 ... yes 57/776 ... yes 104/778 ... yes 147/779 ... yes 147/780 ... yes 147/781 ... yes 31/782 ... yes 142/783 ... yes 189/784 ... yes 189/785 ... yes 62/786 ... yes 189/787 ... yes 190/788 ... yes 190/789 ... yes 189/790 ... yes 192/791 ... yes 192/792 ... yes 134/793 ... yes 133/794 ... yes 143/795 ... yes 193/796 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.24.0 ? ... yes (2.28.0) Git user has default SSH configuration? ... yes Active users: ... 62 Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Possible fixes
Context: Issue started occurring a few days ago when I took several actions that changed Gitlab Omnibus, and I've attempted to rule out causes of the issue.
- Updated Gitlab from 13.4.2 to 13.4.3: Possible issue candidate
-- No problems with running
sudo apt-get upgrade gitlab-ce
; I ran this before the postgres update - Updated Postgres from 11.7 to 12.4 as per this documentation: Possible issue candidate
-- The first attempt to run
gitlab-ctl pg-upgrade -V 12 --tmp-dir /var/opt/gitlab/postgresql/data
only succeeded in upgrading PostgreSQL to 11.9; the postgres symlinks still had11
in their paths andcat /var/opt/gitlab/postgresql/data/PG_VERSION
returned11
-- I then rangitlab-ctl pg-upgrade -V 12 --tmp-dir /var/opt/gitlab/postgresql
and then it succeeded to update the symlinks to have12
in their paths andcat /var/opt/gitlab/postgresql/data/PG_VERSION
returned12
- Manually ran
sudo gitlab-ctl registry-garbage-collect
as per this documentation - Manually ran
sudo gitlab-ctl registry-garbage-collect -m
as per this documentation - Setup cronjob in /etc/cron.d as per this documentation
- Changed registry storage backend to use an S3 bucket as per this documentation: Not believed to be issue candidate -- I have validated the IAM permission scope of the user used in the gitlab.rb file. Running gitlab-ctl reconfigure does not error out and Gitlab is able to read the images in the S3 bucket and write images to the bucket so long as the registry process is alive -- I have reverted this change to use local filesystem registry storage and the issue persists
My other attempts at rectifying this issue:
- Uploading an empty file to the S3 bucket root as suggested here and here. My S3 bucket was not empty because I manually used
aws s3 sync
to populate the bucket prior to runninggitlab-ctl reconfigure
, but I figured it was worth a shot. My issue persists. - Including
registry['health_storagedriver_enabled'] = false
in gitlab.rb as per this MR and runninggitlab-ctl reconfigure
; the issue persists - Including
registry['debug_addr'] = "localhost:5001"
andregistry['log_level'] = 'debug'
in gitlab.rb and running gitlab-ctl reconfigure; no new insights are revealed from curling that debug endpoint or in theregistry/current
log
I've been leaning towards either the Postgres upgrade process or upgrade to 13.4.3 possibly introducing this undesired behavior (somehow), but my attempts to reproduce the issue have thus-far failed, so it's hard to believe either of those are the true culprit. I can't expect Gitlab to squandar time on this issue when I can't even replicate it, but perhaps some other people have experienced something similar.