Grafana avatars 404
Summary
Using the builtin grafana gitlabdomain.com/-/grafana/
, the avatar images are broken.
Steps to reproduce
I first noticed the issue when signing into Grafana using my GitLab account.
At first I thought this had to do with it trying to use my GitLab avatar or perhaps an issue with Gravatar. I then enabled the login form and signed in with the admin
account and the issue persisted.
What is the current bug behavior?
https://gitlabdomain.com/-/grafana/avatar/46d229b033af06a191ff2267bca9ae56
returns 404.
What is the expected correct behavior?
The avatar path should return the correct image, or at least a placeholder.
Relevant logs
Relevant logs
==> /var/log/gitlab/nginx/gitlab_error.log <== 2020/08/01 00:22:46 [error] 6405#0: *67529 open() "/var/www/letsencrypt/.well-known/assetlinks.json" failed (2: No such file or directory), client: <redacted_ip_1>, server: <gitlab_domain>, request: "GET /.well-known/assetlinks.json HTTP/1.1", host: "<gitlab_domain>" 2020/08/01 13:01:15 [error] 6405#0: *114131 connect() failed (111: Connection refused) while connecting to upstream, client: <my_browser_ip>, server: <gitlab_domain>, request: "GET /-/grafana/public/build/runtime.d3489276abe5fda2bd26.js HTTP/2.0", upstream: "http://[::1]:3001/public/build/runtime.d3489276abe5fda2bd26.js", host: "<gitlab_domain>", referrer: "https://<gitlab_domain>/-/grafana/" 2020/08/01 13:01:32 [error] 6405#0: *114131 connect() failed (111: Connection refused) while connecting to upstream, client: <my_browser_ip>, server: <gitlab_domain>, request: "GET /-/grafana/public/build/prometheusPlugin.d3489276abe5fda2bd26.js HTTP/2.0", upstream: "http://[::1]:3001/public/build/prometheusPlugin.d3489276abe5fda2bd26.js", host: "<gitlab_domain>", referrer: "https://<gitlab_domain>/-/grafana/d/wsSteMemz/gitlab-omnibus-overview" 2020/08/01 13:01:52 [error] 6405#0: *114131 connect() failed (111: Connection refused) while connecting to upstream, client: <my_browser_ip>, server: <gitlab_domain>, request: "GET /-/grafana/public/build/app.d3489276abe5fda2bd26.js.map HTTP/2.0", upstream: "http://[::1]:3001/public/build/app.d3489276abe5fda2bd26.js.map", host: "<gitlab_domain>" 2020/08/01 13:02:08 [error] 6405#0: *114131 connect() failed (111: Connection refused) while connecting to upstream, client: <my_browser_ip>, server: <gitlab_domain>, request: "GET /-/grafana/api/user/auth-tokens HTTP/2.0", upstream: "http://[::1]:3001/api/user/auth-tokens", host: "<gitlab_domain>", referrer: "https://<gitlab_domain>/-/grafana/profile" 2020/08/01 13:11:01 [error] 6405#0: *114795 connect() failed (111: Connection refused) while connecting to upstream, client: <my_browser_ip>, server: <gitlab_domain>, request: "GET /-/grafana/api/user HTTP/2.0", upstream: "http://[::1]:3001/api/user", host: "<gitlab_domain>", referrer: "https://<gitlab_domain>/-/grafana/profile"
==> /var/log/gitlab/grafana/current <== 2020-08-01_01:06:47.87908 t=2020-07-31T20:06:47-0500 lvl=info msg="Path Data" logger=settings path=/var/opt/gitlab/grafana/data 2020-08-01_01:06:47.87908 t=2020-07-31T20:06:47-0500 lvl=info msg="Path Logs" logger=settings path=/var/log/gitlab/grafana 2020-08-01_01:06:47.87908 t=2020-07-31T20:06:47-0500 lvl=info msg="Path Plugins" logger=settings path=/var/opt/gitlab/grafana/data/plugins 2020-08-01_01:06:47.87908 t=2020-07-31T20:06:47-0500 lvl=info msg="Path Provisioning" logger=settings path=/var/opt/gitlab/grafana/provisioning 2020-08-01_01:06:47.87909 t=2020-07-31T20:06:47-0500 lvl=info msg="App mode production" logger=settings 2020-08-01_01:06:47.87937 t=2020-07-31T20:06:47-0500 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2020-08-01_01:06:47.87955 t=2020-07-31T20:06:47-0500 lvl=info msg="Starting DB migration" logger=migrator 2020-08-01_01:06:47.88248 t=2020-07-31T20:06:47-0500 lvl=info msg="Starting plugin search" logger=plugins 2020-08-01_01:06:47.89060 t=2020-07-31T20:06:47-0500 lvl=info msg="HTTP Server Listen" logger=http.server address=127.0.0.1:3001 protocol=http subUrl=/-/grafana socket= 2020-08-01_01:06:59.75167 t=2020-07-31T20:06:59-0500 lvl=info msg="Successful Login" logger=http.server User=admin@localhost
==> /var/log/gitlab/nginx/gitlab_access.log <== 127.0.0.1 - - [01/Aug/2020:13:16:12 -0500] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 13.2.2 (refs/pipelines/172580057; go1.13.8; linux/amd64)" - <my_browser_ip> - - [01/Aug/2020:13:16:12 -0500] "GET /-/grafana/avatar/46d229b033af06a191ff2267bca9ae56 HTTP/2.0" 404 1564 "https://<gitlab_domain>/-/grafana/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36" 1.99
==> /var/log/gitlab/gitlab-workhorse/current <== {"correlation_id":"GcBSNfa4TG7","duration_ms":1,"host":"<gitlab_domain>","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":204,"system":"http","time":"2020-08-01T13:16:13-05:00","uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 13.2.2 (refs/pipelines/172580057; go1.13.8; linux/amd64)","written_bytes":0}
==> /var/log/gitlab/nginx/gitlab_access.log <== 127.0.0.1 - - [01/Aug/2020:13:16:13 -0500] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 13.2.2 (refs/pipelines/172580057; go1.13.8; linux/amd64)" - <my_browser_ip> - - [01/Aug/2020:13:16:13 -0500] "GET /-/grafana/api/dashboards/home HTTP/2.0" 200 637 "https://<gitlab_domain>/-/grafana/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36" 2.09
==> /var/log/gitlab/nginx/gitlab_error.log <== 2020/08/01 13:16:13 [error] 6405#0: *115120 connect() failed (111: Connection refused) while connecting to upstream, client: <my_browser_ip>, server: <gitlab_domain>, request: "GET /-/grafana/api/search?limit=1 HTTP/2.0", upstream: "http://[::1]:3001/api/search?limit=1", host: "<gitlab_domain>", referrer: "https://<gitlab_domain>/-/grafana/?orgId=1"
==> /var/log/gitlab/nginx/gitlab_access.log <== <my_browser_ip> - - [01/Aug/2020:13:16:13 -0500] "GET /-/grafana/api/plugins?embedded=0&core=0 HTTP/2.0" 200 2 "https://<gitlab_domain>/-/grafana/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36" - <my_browser_ip> - - [01/Aug/2020:13:16:13 -0500] "GET /-/grafana/api/search?limit=1 HTTP/2.0" 200 195 "https://<gitlab_domain>/-/grafana/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36" - <my_browser_ip> - - [01/Aug/2020:13:16:13 -0500] "GET /-/grafana/api/search?limit=30&starred=true HTTP/2.0" 200 2 "https://<gitlab_domain>/-/grafana/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36" - <my_browser_ip> - - [01/Aug/2020:13:16:13 -0500] "GET /-/grafana/api/search?dashboardIds=7&limit=30 HTTP/2.0" 200 212 "https://<gitlab_domain>/-/grafana/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36" 1.64
Details of package version
gitlab-ce/buster,now 13.2.2-ce.0
Environment details
- Operating System:
Debian 9 Stretch
Running onLinux <hostname> 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux
- Bare Metal Dedicated Server in OVH Datacenter
- Installation had originally been running for 5+ years. I recently moved the installation over to a new machine where I installed gitlab-ce from scratch. During that, I also copied the latest configuration file template and modified the appropriate values with the ones I had already configured.
- Is there any other software running on the machine:
- MariaDB
- Redis (I changed the gitlab-omnibus redis port)
- Source Engine dedicated servers (Docker)
- ShoutCast relays
- Percona PMM (Docker)
- Is this a single or multiple node installation?
- Single Node
- Resources
- CPU:
Intel(R) Core(TM) i7-7700K
- Memory total:
- CPU:
total used free shared buff/cache available
Mem: 62Gi 8.8Gi 4.1Gi 1.2Gi 49Gi 53Gi
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
puma['port'] = 8081
grafana['disable_login_form'] = false
grafana['http_port'] = 3001
nginx['redirect_http_to_https'] = true
nginx['custom_gitlab_server_config']="location ^~ /.well-known {\n alias /var/www/letsencrypt/.well-known;\n}\n"
# This machine has multiple WAN IP addresses. This is is configured to
# use one of them. Note that this address is not the primary IP for the NIC.
nginx['listen_addresses'] = ['<dedicated_external_ip>', '127.0.0.1']
registry_nginx['enable'] = true