Registry process repeatedly turns off at random intervals

Summary

The registry process of my Gitlab Omnibus deployment periodically shuts down for periods of approximately 10 minutes at random intervals between 2 and 60 minutes. This interruption causes CI attempting to access the registry to fail with 502 Bad Gateway errors from NGINX.

Steps to reproduce

Unfortunately, I have been unable to reproduce this issue. I have deployed a fresh gitlab install onto an Ubuntu 16.04 VM and monitored the registry service via watch gitlab-ctl status registry for at least an hour (the time interval mentioned in the "upload purge" log event) after each step attempting to reproduce the issue:

After fresh install of 13.4.3, enable the registry and wait: the process stays up
Upgrade postgres to 12.4 and restart gitlab and wait: the process stays up
Run gitlab-ctl registry-garbage-collect and then wait: the process restarts immediately after this task then doesn't fail out again
Run gitlab-ctl registry-garbage-collect -m and then wait: the process restarts immediately after this task then doesn't fail out again

What is the current bug behavior?

The registry process is randomly shutting off and then restarting.

What is the expected correct behavior?

The registry process should continue to run alongside the other Gitlab components without interruption.

Relevant logs and/or screenshots

==> /var/log/gitlab/registry/current <==

The following log sequence of events appears whenever the registry process restarts:

2020-10-14_11:17:04.96252 time="2020-10-14T07:17:04.962214924-04:00" level=info msg="redis not configured" environment=production go.version=go1.14.7 instance.id=masked 044 service=registry version=v2.10.1-gitlab
2020-10-14_11:17:04.96302 time="2020-10-14T07:17:04.962405925-04:00" level=info msg="Starting upload purge in 40m0s" environment=production go.version=go1.14.7 instance.id=masked service=registry version=v2.10.1-gitlab
2020-10-14_11:17:04.99036 time="2020-10-14T07:17:04.990271219-04:00" level=info msg="using inmemory blob descriptor cache" environment=production go.version=go1.14.7 instance.id=masked service=registry version=v2.10.1-gitlab
2020-10-14_11:17:04.99169 time="2020-10-14T07:17:04.991590328-04:00" level=info msg="listening on 127.0.0.1:5000" environment=production go.version=go1.14.7 instance.id=masked service=registry version=v2.10.1-gitlab

The time value (X) in the second message "Starting upload purge in X" seems to be a rough forecast of when the registry process will next die. Just before the registry process dies, I see this sequence of events repeat itself:

2020-10-14_12:53:04.46454 time="2020-10-14T08:53:04.464327864-04:00" level=info msg="PurgeUploads starting: olderThan=2020-10-07 08:53:04.464248363 -0400 EDT m=-602639.958736471, actuallyDelete=true"
2020-10-14_12:53:04.47472 time="2020-10-14T08:53:04.474585532-04:00" level=info msg="Purge uploads finished.  Num deleted=0, num errors=0"
2020-10-14_12:53:04.47475 time="2020-10-14T08:53:04.474652732-04:00" level=info msg="Starting upload purge in 24h0m0s" environment=production go.version=go1.14.7 instance.id=masked service=registry version=v2.10.1-gitlab

Results of GitLab environment info

Expand for output related to GitLab environment info


```
System information
System:		Ubuntu 16.04
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.28.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.4.3
Revision:	6689552c479
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.4
URL:		https://something.somedomain.com
HTTP Clone URL:	https://something.somedomain.com/some-group/some-project.git
SSH Clone URL:	git@something.somedomain.com:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	13.7.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git
```

Results of GitLab application Check

Expand for output related to the GitLab application check


Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.7.0 ? ... OK (13.7.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
	User output sanitized. Found 84 users of 100 limit.

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
78/1 ... yes
5/2 ... yes
2/3 ... yes
47/4 ... yes
47/5 ... yes
2/6 ... yes
47/7 ... yes
47/8 ... yes
47/9 ... yes
47/10 ... yes
47/11 ... yes
2/12 ... yes
47/13 ... yes
47/14 ... yes
2/15 ... yes
47/16 ... yes
47/17 ... yes
47/18 ... yes
47/19 ... yes
3/20 ... yes
75/21 ... yes
5/22 ... yes
5/23 ... yes
39/24 ... yes
39/25 ... yes
41/26 ... yes
41/27 ... yes
39/28 ... yes
5/29 ... yes
41/30 ... yes
41/31 ... yes
31/32 ... yes
31/33 ... yes
31/34 ... yes
32/35 ... yes
32/36 ... yes
32/37 ... yes
32/38 ... yes
32/39 ... yes
32/40 ... yes
32/41 ... yes
32/42 ... yes
31/43 ... yes
32/44 ... yes
32/45 ... yes
7/46 ... yes
7/47 ... yes
7/48 ... yes
7/49 ... yes
7/50 ... yes
7/51 ... yes
8/52 ... yes
9/53 ... yes
32/54 ... yes
32/55 ... yes
32/56 ... yes
32/57 ... yes
32/58 ... yes
9/59 ... yes
9/60 ... yes
9/61 ... yes
9/62 ... yes
9/63 ... yes
32/64 ... yes
32/65 ... yes
32/66 ... yes
32/67 ... yes
9/68 ... yes
9/69 ... yes
9/70 ... yes
32/71 ... yes
9/72 ... yes
10/73 ... yes
10/74 ... yes
10/75 ... yes
10/76 ... yes
10/77 ... yes
10/78 ... yes
10/79 ... yes
10/80 ... yes
10/81 ... yes
47/82 ... yes
47/84 ... yes
11/85 ... yes
54/86 ... yes
54/87 ... yes
53/88 ... yes
53/89 ... yes
32/90 ... yes
17/91 ... yes
17/92 ... yes
17/93 ... yes
17/94 ... yes
17/95 ... yes
11/96 ... yes
10/97 ... yes
11/98 ... yes
78/100 ... yes
75/101 ... yes
11/102 ... yes
11/103 ... yes
53/104 ... yes
2/108 ... yes
20/109 ... yes
10/110 ... yes
20/111 ... yes
20/112 ... yes
20/113 ... yes
55/114 ... yes
20/115 ... yes
52/116 ... yes
34/117 ... yes
20/118 ... yes
20/119 ... yes
20/120 ... yes
6/121 ... yes
20/122 ... yes
20/123 ... yes
20/124 ... yes
22/126 ... yes
22/127 ... yes
58/128 ... yes
22/129 ... yes
58/130 ... yes
58/131 ... yes
28/134 ... yes
28/135 ... yes
22/137 ... yes
20/138 ... yes
6/141 ... yes
42/142 ... yes
42/143 ... yes
22/148 ... yes
6/152 ... yes
28/153 ... yes
34/154 ... yes
22/155 ... yes
28/156 ... yes
36/159 ... yes
36/160 ... yes
36/161 ... yes
36/162 ... yes
36/163 ... yes
36/164 ... yes
36/165 ... yes
36/166 ... yes
36/167 ... yes
78/169 ... yes
11/170 ... yes
11/171 ... yes
37/172 ... yes
20/175 ... yes
42/176 ... yes
42/177 ... yes
70/178 ... yes
45/179 ... yes
46/180 ... yes
47/181 ... yes
47/182 ... yes
47/183 ... yes
47/184 ... yes
47/185 ... yes
2/186 ... yes
56/188 ... yes
6/189 ... yes
57/190 ... yes
57/191 ... yes
6/195 ... yes
31/196 ... yes
6/197 ... yes
125/198 ... yes
6/199 ... yes
2/200 ... yes
78/201 ... yes
37/202 ... yes
78/203 ... yes
60/204 ... yes
59/205 ... yes
73/206 ... yes
35/208 ... yes
35/209 ... yes
31/210 ... yes
35/211 ... yes
31/212 ... yes
35/214 ... yes
35/215 ... yes
35/216 ... yes
23/217 ... yes
13/218 ... yes
78/219 ... yes
31/220 ... yes
21/222 ... yes
23/223 ... yes
10/224 ... yes
10/225 ... yes
64/226 ... yes
64/227 ... yes
64/228 ... yes
64/229 ... yes
65/231 ... yes
6/232 ... yes
64/234 ... yes
65/235 ... yes
65/236 ... yes
65/237 ... yes
65/238 ... yes
13/240 ... yes
65/241 ... yes
70/242 ... yes
70/243 ... yes
13/244 ... yes
23/245 ... yes
70/246 ... yes
22/247 ... yes
70/248 ... yes
72/249 ... yes
72/250 ... yes
72/251 ... yes
72/253 ... yes
72/254 ... yes
72/255 ... yes
72/256 ... yes
72/257 ... yes
72/258 ... yes
72/259 ... yes
72/260 ... yes
72/261 ... yes
22/262 ... yes
22/263 ... yes
64/264 ... yes
65/265 ... yes
64/266 ... yes
64/267 ... yes
64/268 ... yes
35/269 ... yes
64/270 ... yes
73/272 ... yes
64/273 ... yes
64/274 ... yes
10/275 ... yes
73/276 ... yes
73/277 ... yes
35/278 ... yes
64/279 ... yes
73/280 ... yes
74/281 ... yes
74/282 ... yes
116/283 ... yes
116/286 ... yes
76/287 ... yes
77/288 ... yes
77/289 ... yes
77/290 ... yes
35/291 ... yes
35/292 ... yes
70/293 ... yes
64/294 ... yes
64/295 ... yes
73/296 ... yes
37/297 ... yes
35/298 ... yes
78/299 ... yes
35/300 ... yes
35/301 ... yes
35/302 ... yes
35/303 ... yes
35/304 ... yes
35/305 ... yes
35/306 ... yes
35/307 ... yes
35/308 ... yes
35/309 ... yes
35/310 ... yes
35/311 ... yes
37/312 ... yes
37/313 ... yes
22/314 ... yes
22/315 ... yes
13/316 ... yes
13/317 ... yes
13/318 ... yes
13/319 ... yes
80/321 ... yes
80/324 ... yes
13/325 ... yes
57/326 ... yes
85/327 ... yes
59/328 ... yes
13/329 ... yes
82/331 ... yes
13/332 ... yes
21/333 ... yes
63/335 ... yes
63/336 ... yes
70/337 ... yes
85/338 ... yes
92/339 ... yes
63/340 ... yes
89/341 ... yes
92/342 ... yes
83/343 ... yes
28/345 ... yes
63/346 ... yes
125/347 ... yes
82/348 ... yes
92/350 ... yes
85/351 ... yes
85/352 ... yes
85/353 ... yes
63/354 ... yes
63/355 ... yes
63/356 ... yes
63/357 ... yes
63/358 ... yes
63/359 ... yes
22/360 ... yes
22/361 ... yes
83/362 ... yes
83/363 ... yes
63/364 ... yes
63/365 ... yes
63/366 ... yes
63/367 ... yes
63/368 ... yes
63/369 ... yes
63/370 ... yes
63/371 ... yes
115/372 ... yes
115/373 ... yes
115/374 ... yes
115/375 ... yes
94/376 ... yes
115/377 ... yes
94/378 ... yes
95/379 ... yes
96/380 ... yes
96/381 ... yes
96/382 ... yes
22/383 ... yes
70/384 ... yes
94/385 ... yes
97/386 ... yes
59/387 ... yes
59/388 ... yes
125/389 ... yes
116/390 ... yes
116/391 ... yes
92/393 ... yes
97/394 ... yes
104/395 ... yes
102/397 ... yes
92/398 ... yes
103/399 ... yes
21/400 ... yes
103/401 ... yes
102/402 ... yes
10/403 ... yes
72/404 ... yes
72/405 ... yes
72/406 ... yes
72/407 ... yes
102/408 ... yes
72/409 ... yes
72/410 ... yes
74/411 ... yes
102/412 ... yes
22/413 ... yes
115/415 ... yes
94/416 ... yes
72/417 ... yes
6/418 ... yes
115/419 ... yes
115/420 ... yes
22/421 ... yes
22/423 ... yes
70/424 ... yes
130/425 ... yes
130/426 ... yes
130/427 ... yes
11/428 ... yes
74/429 ... yes
106/430 ... yes
120/431 ... yes
22/432 ... yes
113/433 ... yes
113/434 ... yes
70/435 ... yes
113/436 ... yes
114/437 ... yes
106/438 ... yes
106/439 ... yes
108/440 ... yes
130/441 ... yes
130/442 ... yes
114/443 ... yes
113/444 ... yes
115/445 ... yes
114/447 ... yes
72/448 ... yes
114/449 ... yes
74/450 ... yes
74/451 ... yes
74/452 ... yes
114/453 ... yes
103/455 ... yes
16/456 ... yes
125/459 ... yes
113/460 ... yes
100/462 ... yes
94/463 ... yes
113/464 ... yes
113/465 ... yes
117/466 ... yes
117/467 ... yes
10/468 ... yes
23/470 ... yes
110/471 ... yes
108/473 ... yes
108/474 ... yes
120/475 ... yes
114/476 ... yes
74/478 ... yes
23/479 ... yes
114/483 ... yes
114/484 ... yes
22/485 ... yes
10/486 ... yes
120/487 ... yes
92/488 ... yes
114/490 ... yes
125/491 ... yes
102/492 ... yes
122/493 ... yes
121/494 ... yes
92/496 ... yes
119/497 ... yes
123/498 ... yes
123/500 ... yes
123/501 ... yes
123/502 ... yes
121/503 ... yes
114/504 ... yes
124/505 ... yes
125/506 ... yes
125/507 ... yes
125/508 ... yes
125/509 ... yes
114/511 ... yes
114/512 ... yes
120/514 ... yes
124/515 ... yes
22/516 ... yes
127/518 ... yes
13/519 ... yes
92/520 ... yes
74/521 ... yes
120/522 ... yes
101/523 ... yes
119/524 ... yes
128/525 ... yes
128/526 ... yes
127/527 ... yes
128/528 ... yes
128/529 ... yes
128/530 ... yes
128/531 ... yes
128/532 ... yes
128/533 ... yes
125/534 ... yes
120/535 ... yes
114/536 ... yes
114/537 ... yes
97/538 ... yes
106/539 ... yes
108/540 ... yes
101/541 ... yes
127/542 ... yes
108/543 ... yes
108/544 ... yes
108/545 ... yes
108/546 ... yes
23/547 ... yes
104/548 ... yes
129/550 ... yes
114/551 ... yes
108/552 ... yes
62/555 ... yes
62/556 ... yes
133/558 ... yes
113/559 ... yes
134/560 ... yes
134/561 ... yes
62/562 ... yes
135/563 ... yes
133/564 ... yes
133/565 ... yes
108/566 ... yes
133/567 ... yes
122/569 ... yes
122/570 ... yes
122/571 ... yes
143/572 ... yes
122/573 ... yes
23/574 ... yes
137/575 ... yes
142/576 ... yes
138/577 ... yes
133/578 ... yes
138/579 ... yes
141/581 ... yes
136/582 ... yes
134/583 ... yes
138/584 ... yes
124/585 ... yes
142/588 ... yes
131/589 ... yes
133/590 ... yes
147/591 ... yes
141/592 ... yes
141/593 ... yes
147/594 ... yes
138/595 ... yes
139/596 ... yes
139/597 ... yes
133/598 ... yes
98/599 ... yes
108/600 ... yes
108/601 ... yes
133/602 ... yes
148/603 ... yes
148/604 ... yes
148/605 ... yes
148/606 ... yes
148/607 ... yes
108/608 ... yes
149/609 ... yes
149/610 ... yes
149/611 ... yes
149/612 ... yes
149/613 ... yes
149/614 ... yes
148/615 ... yes
150/616 ... yes
125/617 ... yes
142/618 ... yes
108/619 ... yes
134/620 ... yes
134/621 ... yes
151/622 ... yes
134/623 ... yes
108/624 ... yes
125/625 ... yes
133/628 ... yes
10/629 ... yes
139/631 ... yes
139/632 ... yes
139/633 ... yes
139/634 ... yes
133/637 ... yes
139/638 ... yes
133/639 ... yes
139/640 ... yes
131/641 ... yes
148/642 ... yes
139/643 ... yes
161/644 ... yes
133/645 ... yes
134/646 ... yes
161/647 ... yes
133/648 ... yes
133/649 ... yes
133/650 ... yes
133/651 ... yes
133/652 ... yes
133/654 ... yes
133/656 ... yes
101/657 ... yes
139/660 ... yes
139/661 ... yes
133/662 ... yes
149/663 ... yes
158/664 ... yes
158/665 ... yes
158/666 ... yes
158/667 ... yes
23/670 ... yes
133/671 ... yes
133/672 ... yes
133/673 ... yes
138/674 ... yes
138/675 ... yes
138/676 ... yes
133/677 ... yes
133/678 ... yes
133/679 ... yes
133/680 ... yes
160/681 ... yes
94/682 ... yes
22/684 ... yes
161/685 ... yes
161/686 ... yes
161/687 ... yes
139/688 ... yes
139/689 ... yes
139/690 ... yes
139/691 ... yes
138/692 ... yes
138/693 ... yes
138/694 ... yes
134/695 ... yes
134/696 ... yes
134/697 ... yes
137/698 ... yes
6/699 ... yes
108/701 ... yes
133/702 ... yes
164/704 ... yes
163/705 ... yes
108/706 ... yes
164/707 ... yes
164/708 ... yes
164/709 ... yes
133/710 ... yes
163/711 ... yes
163/713 ... yes
108/714 ... yes
164/715 ... yes
163/716 ... yes
163/717 ... yes
133/718 ... yes
159/719 ... yes
108/720 ... yes
108/721 ... yes
108/722 ... yes
133/723 ... yes
173/724 ... yes
161/725 ... yes
133/726 ... yes
161/727 ... yes
161/728 ... yes
161/729 ... yes
161/730 ... yes
108/731 ... yes
133/732 ... yes
133/733 ... yes
133/734 ... yes
133/735 ... yes
164/736 ... yes
10/737 ... yes
10/738 ... yes
158/740 ... yes
158/741 ... yes
158/742 ... yes
164/743 ... yes
1/751 ... yes
164/755 ... yes
164/756 ... yes
164/757 ... yes
164/760 ... yes
133/761 ... yes
133/762 ... yes
106/764 ... yes
182/765 ... yes
182/766 ... yes
182/767 ... yes
131/768 ... yes
133/769 ... yes
133/770 ... yes
104/771 ... yes
133/772 ... yes
133/773 ... yes
133/774 ... yes
140/775 ... yes
57/776 ... yes
104/778 ... yes
147/779 ... yes
147/780 ... yes
147/781 ... yes
31/782 ... yes
142/783 ... yes
189/784 ... yes
189/785 ... yes
62/786 ... yes
189/787 ... yes
190/788 ... yes
190/789 ... yes
189/790 ... yes
192/791 ... yes
192/792 ... yes
134/793 ... yes
133/794 ... yes
143/795 ... yes
193/796 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 62
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Context: Issue started occurring a few days ago when I took several actions that changed Gitlab Omnibus, and I've attempted to rule out causes of the issue.

Updated Gitlab from 13.4.2 to 13.4.3: Possible issue candidate -- No problems with running sudo apt-get upgrade gitlab-ce; I ran this before the postgres update
Updated Postgres from 11.7 to 12.4 as per this documentation: Possible issue candidate -- The first attempt to run gitlab-ctl pg-upgrade -V 12 --tmp-dir /var/opt/gitlab/postgresql/data only succeeded in upgrading PostgreSQL to 11.9; the postgres symlinks still had 11 in their paths and cat /var/opt/gitlab/postgresql/data/PG_VERSION returned 11 -- I then ran gitlab-ctl pg-upgrade -V 12 --tmp-dir /var/opt/gitlab/postgresql and then it succeeded to update the symlinks to have 12 in their paths and cat /var/opt/gitlab/postgresql/data/PG_VERSION returned 12
Manually ran sudo gitlab-ctl registry-garbage-collect as per this documentation
Manually ran sudo gitlab-ctl registry-garbage-collect -m as per this documentation
Setup cronjob in /etc/cron.d as per this documentation
Changed registry storage backend to use an S3 bucket as per this documentation: Not believed to be issue candidate -- I have validated the IAM permission scope of the user used in the gitlab.rb file. Running gitlab-ctl reconfigure does not error out and Gitlab is able to read the images in the S3 bucket and write images to the bucket so long as the registry process is alive -- I have reverted this change to use local filesystem registry storage and the issue persists

My other attempts at rectifying this issue:

Uploading an empty file to the S3 bucket root as suggested here and here. My S3 bucket was not empty because I manually used aws s3 sync to populate the bucket prior to running gitlab-ctl reconfigure, but I figured it was worth a shot. My issue persists.
Including registry['health_storagedriver_enabled'] = false in gitlab.rb as per this MR and running gitlab-ctl reconfigure; the issue persists
Including registry['debug_addr'] = "localhost:5001" and registry['log_level'] = 'debug' in gitlab.rb and running gitlab-ctl reconfigure; no new insights are revealed from curling that debug endpoint or in the registry/current log

I've been leaning towards either the Postgres upgrade process or upgrade to 13.4.3 possibly introducing this undesired behavior (somehow), but my attempts to reproduce the issue have thus-far failed, so it's hard to believe either of those are the true culprit. I can't expect Gitlab to squandar time on this issue when I can't even replicate it, but perhaps some other people have experienced something similar.