Update to 11.7 results in 500 errors for all attempt to access with browser
Summary
After update from 11.6.x (not sure which version) to 11.7 my omnibus installation now serves nothing but 500 errors to users.
Steps to reproduce
Run sudo apt-get update
on a working 11.6.x install on Ubuntu 18.04. Attempt to visit GitLab.
(How one can reproduce the issue - this is very important)
What is the current bug behavior?
When a user attempt to visit the GitLab instance web interface the site redirects the log in prompt and the responds with a 500 error.
What is the expected correct behavior?
The site should serve our the login page and allow a user to log in and interact with the website.
Relevant logs
Relevant logs
run: alertmanager: (pid 1429) 854s; run: log: (pid 456) 48218s run: gitaly: (pid 1441) 854s; run: log: (pid 452) 48218s run: gitlab-monitor: (pid 1456) 853s; run: log: (pid 457) 48218s run: gitlab-workhorse: (pid 1464) 853s; run: log: (pid 451) 48218s run: logrotate: (pid 1475) 853s; run: log: (pid 449) 48218s run: mailroom: (pid 1578) 847s; run: log: (pid 450) 48218s run: nginx: (pid 1581) 847s; run: log: (pid 459) 48218s run: node-exporter: (pid 1584) 846s; run: log: (pid 455) 48218s run: postgres-exporter: (pid 1589) 846s; run: log: (pid 468) 48218s run: postgresql: (pid 1602) 846s; run: log: (pid 453) 48218s run: prometheus: (pid 1605) 846s; run: log: (pid 1400) 871s run: redis: (pid 1622) 846s; run: log: (pid 448) 48218s run: redis-exporter: (pid 1636) 845s; run: log: (pid 458) 48218s run: sidekiq: (pid 1647) 843s; run: log: (pid 454) 48218s run: unicorn: (pid 1659) 841s; run: log: (pid 460) 48218s
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OKAccess to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 34/1 ... yes 35/2 ... yes 34/3 ... yes 35/4 ... yes 35/5 ... yes 34/7 ... yes 35/8 ... yes 34/9 ... yes 34/10 ... yes 34/11 ... yes 34/12 ... yes 34/13 ... yes 34/14 ... yes 34/15 ... yes 34/16 ... yes 34/17 ... yes 34/18 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 5
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
==> /var/log/gitlab/nginx/gitlab_error.log <== 2019/01/23 00:32:03 [error] 15743#0: *737 connect() to unix:/var/opt/gitlab/gitlab-workhorse/socket failed (111: Connection refused) while connecting to upstream, client: 192.168.1.1, server: **SANITIZED**, request: "POST /api/v4/jobs/request HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/api/v4/jobs/request", host: "**SANITIZED**" 2019/01/23 00:32:06 [error] 15743#0: *739 connect() to unix:/var/opt/gitlab/gitlab-workhorse/socket failed (111: Connection refused) while connecting to upstream, client: 192.168.1.1, server: **SANITIZED**, request: "POST /api/v4/jobs/request HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/api/v4/jobs/request", host: "**SANITIZED**" ==> /var/log/gitlab/nginx/gitlab_access.log <== **SANITIZED** - - [23/Jan/2019:10:32:36 -0500] "GET /users/sign_in HTTP/1.1" 500 2926 "https://**SANITIZED**/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0"
**/var/opt/gitlab/rails/production.log** Started GET "/users/sign_in" for **SANITIZED** at 2019-01-23 10:32:36 -0500 Processing by SessionsController#new as HTML Completed 500 Internal Server Error in 16ms (ActiveRecord: 0.7ms) ActionView::Template::Error (nil is not a valid asset source): 13: .col-sm-7.brand-holder 14: %h1 15: = brand_title 16: = brand_image 17: - if current_appearance&.description? 18: = brand_text 19: - else app/helpers/lazy_image_tag_helper.rb:14:in `image_tag' app/helpers/appearances_helper.rb:14:in `brand_image' app/views/layouts/devise.html.haml:16:in `_app_views_layouts_devise_html_haml__518802345853219748_ 70046641194440' app/controllers/application_controller.rb:113:in `render' app/controllers/sessions_controller.rb:30:in `new' lib/gitlab/i18n.rb:55:in `with_locale' lib/gitlab/i18n.rb:61:in `with_user_locale' app/controllers/application_controller.rb:417:in `set_locale' lib/gitlab/middleware/rails_queue_duration.rb:24:in `call' lib/gitlab/metrics/rack_middleware.rb:17:in `block in call' lib/gitlab/metrics/transaction.rb:55:in `run' lib/gitlab/metrics/rack_middleware.rb:17:in `call' lib/gitlab/middleware/multipart.rb:103:in `call' lib/gitlab/request_profiler/middleware.rb:16:in `call' lib/gitlab/middleware/go.rb:20:in `call' lib/gitlab/etag_caching/middleware.rb:13:in `call' lib/gitlab/middleware/correlation_id.rb:16:in `block in call' lib/gitlab/correlation_id.rb:15:in `use_id' lib/gitlab/middleware/correlation_id.rb:15:in `call' lib/gitlab/middleware/read_only/controller.rb:40:in `call' lib/gitlab/middleware/read_only.rb:18:in `call' lib/gitlab/middleware/basic_health_check.rb:25:in `call' lib/gitlab/request_context.rb:20:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call' lib/gitlab/middleware/release_env.rb:13:in `call'
Details of package version
Provide the package version installation details
||/ Name Version Architecture Description +++-=================-=============-=============-=========================================================== ii gitlab-ce 11.7.0-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis)
Environment details
- Operating System:
- Ubuntun 18.04.1
- Installation Target
- Container: LXC -- Proxmox
- Installation Type
- Upgrade from version 11.6.x
- Is there any other software running on the machine:
- No
- Is this a single or multiple node installation?
- Single
- Resources
- CPU: Intel XEON
- Memory total: 2.5GB ECC
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url "https://**SANITIZED**/" nginx['listen_port'] = 8880 nginx['listen_https'] = false nginx['proxy_set_headers'] = { "Host" => "$http_host_with_default", "X-Real-IP" => "$remote_addr", "X-Forwarded-For" => "$proxy_add_x_forwarded_for", "X-Forwarded-Proto" => "https", "X-Forwarded-Ssl" => "on", "Upgrade" => "$http_upgrade", "Connection" => "$connection_upgrade" } nginx['http2_enabled'] = true nginx['real_ip_trusted_addresses'] = [ '**SANITIZED**' ] nginx['real_ip_header'] = 'X-Real-IP' nginx['real_ip_recursive'] = 'on' gitlab_rails['time_zone'] = **SANITIZED** gitlab_rails['gitlab_email_enabled'] = true gitlab_rails['gitlab_email_from'] = '**SANITIZED**' gitlab_rails['gitlab_email_display_name'] = '**SANITIZED**' gitlab_rails['gitlab_email_reply_to'] = '**SANITIZED**' gitlab_rails['incoming_email_enabled'] = true gitlab_rails['incoming_email_address'] = "**SANITIZED**" gitlab_rails['incoming_email_email'] = "**SANITIZED**" gitlab_rails['incoming_email_password'] = "**SANITIZED**" gitlab_rails['incoming_email_host'] = "**SANITIZED**" gitlab_rails['incoming_email_port'] = 993 gitlab_rails['incoming_email_ssl'] = true gitlab_rails['incoming_email_start_tls'] = false gitlab_rails['gitlab_shell_ssh_port'] = 8222 gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = "**SANITIZED**" gitlab_rails['smtp_port'] = 587 gitlab_rails['smtp_user_name'] = "**SANITIZED**" gitlab_rails['smtp_password'] = "**SANITIZED**" gitlab_rails['smtp_domain'] = "**SANITIZED**" gitlab_rails['smtp_authentication'] = "login" gitlab_rails['smtp_enable_starttls_auto'] = true gitlab_rails['smtp_tls'] = false gitlab_rails['smtp_openssl_verify_mode'] = 'peer' nginx['hsts_max_age'] = 0 prometheus['enable'] = true prometheus['listen_address'] = '0.0.0.0:9090'