Occasional 502 error on a GET request to internal image provider

Summary

GitLab internals produce an occasional badgateway: failed to receive response: context canceled error when accessing a /uploads/-/system/.../*.png resource.

Steps to reproduce

N/A

Example Project

N/A

What is the current bug behavior?

During normal day to day operation our monitoring system detects an occasional burst of 502 error replies from gitlab-workhorse exporter. With no apparent pattern or identifiable cause they all seem to originate from a local request to an image under /uploads/-/system/... path. Using this very URL from the log entry we can verify that an image can be read without problems.

What is the expected correct behavior?

The image exists so there should be no reason for a 502 error to happen.

Relevant logs and/or screenshots

(Host address redacted for security reasons)

{"correlation_id":"01FF07SC6YDK3BQQSPQZA16NJ0","duration_ms":0,"error":"badgateway: failed to receive response: context canceled","level":"error","method":"GET","msg":"","time":"2021-09-07T13:40:02Z","uri":"/uploads/-/system/appearance/favicon/1/fav.xxx.gitlab.png"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FF07SC6YDK3BQQSPQZA16NJ0","duration_ms":0,"host":"xxxxxxxx","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":502,"system":"http","time":"2021-09-07T13:40:02Z","ttfb_ms":0,"uri":"/uploads/-/system/appearance/favicon/1/fav.xxx.gitlab.png","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0","written_bytes":2940}

{"correlation_id":"01FF07SRV76T4S8YGTZNW9RJ1K","duration_ms":0,"error":"badgateway: failed to receive response: context canceled","level":"error","method":"GET","msg":"","time":"2021-09-07T13:40:15Z","uri":"/uploads/-/system/group/avatar/7/general.png?width=15"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FF07SRV76T4S8YGTZNW9RJ1K","duration_ms":0,"host":"xxxxxxxx","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://xxxxxxxx/general/projectABC/-/merge_requests","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":502,"system":"http","time":"2021-09-07T13:40:15Z","ttfb_ms":0,"uri":"/uploads/-/system/group/avatar/7/general.png?width=15","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0","written_bytes":2940}

{"correlation_id":"01FF07V9J1DJCCT6GB61TS1EPQ","duration_ms":0,"error":"badgateway: failed to receive response: context canceled","level":"error","method":"GET","msg":"","time":"2021-09-07T13:41:05Z","uri":"/uploads/-/system/project/avatar/56/system-quill.png?width=32"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FF07V9J1DJCCT6GB61TS1EPQ","duration_ms":0,"host":"xxxxxxxx","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://xxxxxxxx/general/projectABC/-/merge_requests/1","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":502,"system":"http","time":"2021-09-07T13:41:05Z","ttfb_ms":0,"uri":"/uploads/-/system/project/avatar/56/projectABC.png?width=32","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0","written_bytes":2940}

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.14
Git Version:    2.32.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        14.2.3
Revision:       2324101daeb
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://xxxxxxxx
HTTP Clone URL: https://xxxxxxxx/some-group/some-project.git
SSH Clone URL:  ssh://git@xxxxxxxx:8822/some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.19.1
Repository storage paths:   /var/opt/gitlab/git-data/repositories
GitLab Shell path:          /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.19.1 ? ... OK (13.19.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
7/2 ... yes
10/3 ... yes
21/6 ... yes
22/7 ... yes
19/8 ... yes
32/15 ... yes
19/17 ... yes
32/18 ... yes
37/19 ... yes
40/20 ... yes
32/21 ... yes
46/22 ... yes
47/25 ... yes
70/26 ... yes
74/27 ... yes
70/28 ... yes
12/29 ... yes
77/31 ... yes
77/32 ... yes
125/33 ... yes
13/34 ... yes
117/36 ... yes
40/37 ... yes
151/38 ... yes
115/39 ... yes
159/40 ... yes
115/41 ... yes
59/42 ... yes
19/43 ... yes
32/44 ... yes
47/45 ... yes
227/46 ... yes
115/47 ... yes
369/50 ... yes
10/51 ... yes
117/52 ... yes
37/55 ... yes
7/56 ... yes
7/57 ... yes
369/58 ... yes
3/61 ... yes
369/62 ... yes
404/63 ... yes
3/64 ... yes
342/67 ... yes
419/68 ... yes
117/69 ... yes
419/70 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.31.0 ? ... yes (2.32.0)
Git user has default SSH configuration? ... yes
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished