Getting 502 after upgrade from 13.1.11 to 13.5.3

Summary

After upgrading our omnibus installation from 13.1.11 to 13.5.3 we are getting constant 502 responses.

Steps to reproduce

Modified the Dockerfile changing only the 'FROM' line:

before: FROM gitlab/gitlab-ce:13.1.11-ce.0 after: FROM gitlab/gitlab-ce:13.5.3-ce.0

Merged in the changes from the diff of the gitlab.rb (from 13.1.11 to 13.5.3) which only added new commented out lines.
Updated our deployment with the new image.

What is the current bug behavior?

Attempting to access the site returns: HTTP/1.1 502 Bad Gateway

What is the expected correct behavior?

Should be able to successfully access our site.

Relevant logs

curl -I http://localhost:80

HTTP/1.1 502 Bad Gateway Server: nginx Date: Mon, 16 Nov 2020 22:32:23 GMT Content-Type: text/html; charset=utf-8 Content-Length: 2940 Connection: keep-alive Cache-Control: no-cache, no-store, max-age=0, must-revalidate Expires: Fri, 01 Jan 1990 00:00:00 GMT Pragma: no-cache

gitlab-ctl tail unicorn

==> /var/log/gitlab/unicorn/current <== 2020-11-16_18:37:14.71545 master failed to start, check stderr log for details 2020-11-16_18:37:15.72017 failed to start a new unicorn master 2020-11-16_18:37:15.73376 starting new unicorn master

==> /var/log/gitlab/unicorn/unicorn_stderr.log <== I, [2020-11-16T18:37:16.378Z #19479] INFO -- : Refreshing Gem list

==> /var/log/gitlab/unicorn/unicorn_stdout.log <== bundler: failed to load command: unicorn (/opt/gitlab/embedded/bin/unicorn)

==> /var/log/gitlab/unicorn/unicorn_stderr.log <== PG::ConnectionBad: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.2.3/lib/pg.rb:58:in initialize' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.2.3/lib/pg.rb:58:in new' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.2.3/lib/pg.rb:58:in `connect' <repeat/loop this error>

gitlab-ctl tail gitlab-workhorse

{"correlation_id":"OVh2ZwaQmC4","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"error","time":"2020-11-16T19:47:37Z","uri":"/"} {"content_type":"text/html; charset=utf-8","correlation_id":"OVh2ZwaQmC4","duration_ms":0,"host":"mysite.awsdev.mycompany.com","level":"info","method":"GET","msg":"a ccess","proto":"HTTP/1.1","referrer":"","remote_addr":"34.223.95.117:0","remote_ip":"34.223.95.117","status":502,"system":"http","time":"2020-11-16T19:47:37Z","uri":"/","user_agent":"L ogicMonitor SiteMonitor/1.0","written_bytes":2940}

Details of package version

Provide the package version installation details

# dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                     Version                   Architecture              Description
+++-========================================-=========================-=========================-=====================================================================================
ii  gitlab-ce                                13.5.3-ce.0               amd64                     GitLab Community Edition (including NGINX, Postgres, Redis)
un  gitlab-ee                                                                        (no description available)

Environment details

Operating System: Amazon Linux 2
Installation Target:
- VM: Docker FROM gitlab/gitlab-ce:13.5.3-ce.0
Installation Type, remove incorrect values:
- Upgrade from version 13.1.11
Is there any other software running on the machine: no
Is this a single or multiple node installation? Yes, Amazon ECS Cluster
Resources
- CPU: m5.xlarge EC2 instance having 4 vCPU
- Memory total: m5.xlarge EC2 instance having 16GB memory

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

gitlab_rails['secret_key_base'] = ENV["OTP_KEY_BASE"]
gitlab_rails['otp_key_base'] = ENV["DB_KEY_BASE"]
gitlab_rails['db_key_base'] = ENV["SECRET_KEY_BASE"]
external_url ENV["EXTERNAL_URL"]
gitlab_rails['gitlab_ssh_host'] = ENV["GITLAB_SSH_HOST"]
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = ENV["SMTP_ADDRESS"]
gitlab_rails['smtp_port'] = ENV["SMTP_PORT"]
gitlab_rails['smtp_authentication'] = false
gitlab_rails['gitlab_email_from'] = ENV["GITLAB_EMAIL_FROM"]
gitlab_rails['gitlab_email_display_name'] = ENV["GITLAB_EMAIL_DISPLAY_NAME"]
gitlab_rails['webhook_timeout'] = 120
gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '::1/128']
gitlab_rails['artifacts_enabled'] = true
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_background_upload'] = true
gitlab_rails['artifacts_object_store_remote_directory'] = "#{ENV['S3_BUCKET_NAME']}/artifacts"
gitlab_rails['artifacts_object_store_connection'] = {
  'provider' => 'AWS',
  'use_iam_profile' => true
}
if ENV["NON_SSO"] == "true"
  #No SSO Settings will be Configured
else
  ### OmniAuth Settings
  ###! Docs: https://docs.gitlab.com/ee/integration/omniauth.html
   gitlab_rails['omniauth_enabled'] = true
   gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
   gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'saml'
   gitlab_rails['omniauth_block_auto_created_users'] = false
   gitlab_rails['omniauth_auto_link_saml_user'] = true
   gitlab_rails['omniauth_providers'] = [
      {
        name: 'saml',
        args: {
                 assertion_consumer_service_url: ENV["ASSERTION_CONSUMER_SERVICE_URL"],
                 idp_cert_fingerprint: ENV["IDP_CERT_FINGERPRINT"],
                 idp_sso_target_url: ENV["IDP_SSO_TARGET_URL"],
                 issuer: ENV["ISSUER"],
                 name_identifier_format: ENV["NAME_IDENTIFIER_FORMAT"]
               },
        label: 'MYCOMPANY SSO' # optional label for SAML login button, defaults to "Saml"
      }
    ]
end
gitlab_rails['manage_backup_path'] = false
gitlab_rails['backup_path'] = "/backups/backuptemp"
git_data_dirs({ "default" => { "path" => ENV["GIT_DATA_DIRS"], 'gitaly_address' => 'unix:/tmp/gitaly.socket' } })
gitlab_rails['shared_path'] = ENV["SHARED_PATH"]
gitlab_rails['gitlab_shell_ssh_port'] = 7999
gitlab_rails['uploads_directory'] = ENV["UPLOADS_DIRECTORY"]
if ENV["AUTO_MIGRATE"] == "true"
  gitlab_rails['auto_migrate'] = true
else
  gitlab_rails['auto_migrate'] = false
end
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "utf8"
gitlab_rails['db_database'] = ENV["DB_DATABASE"]
gitlab_rails['db_username'] = ENV["DB_USERNAME"]
gitlab_rails['db_password'] = ENV["DB_PASSWORD"]
gitlab_rails['db_host'] = ENV["DB_HOST"]
gitlab_rails['db_port'] = "5432"
gitlab_rails['redis_host'] = ENV["REDIS_HOST"]
gitlab_rails['redis_port'] = "6379"
gitlab_rails['redis_socket'] = "/tmp/redis.socket"
registry['enable'] = false
gitlab_workhorse['listen_addr'] = "/tmp/workhorse.socket"
user['home'] = ENV["USER_HOME"]
unicorn['enable'] = true
unicorn['worker_timeout'] = 300
unicorn['socket'] = '/tmp/gitlab.socket'
puma['enable'] = false
gitlab_shell['auth_file'] = ENV["AUTH_FILE"]
postgresql['enable'] = false
redis['enable'] = false
nginx['listen_port'] = "80"
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
"X-Forwarded-Proto" => ENV["XFORWARDEDPROTO"],
"X-Forwarded-Ssl" => ENV["XFORWARDEDSSL"]
 }
logging['svlogd_num'] = 7 # keep 7 rotated log files
logging['svlogd_timeout'] = 24 * 60 * 60 # rotate after 24 hours
logging['logrotate_frequency'] = "daily" # rotate logs daily
logging['logrotate_size'] = nil # do not rotate by size by default
logging['logrotate_rotate'] = 7 # keep 7 rotated logs
logging['logrotate_dateformat'] = nil # use date extensions for rotated files rather than numbers e.g. a value of "-%Y-%m-%d" would give rotated files like production.log-2016-03-09.gz
logrotate['enable'] = true
omnibus_gitconfig['system'] = {
  "pack" => ["threads = 1"],
  "receive" => ["fsckObjects = true", "advertisePushOptions = true"],
  'receive "fsck"' => ["badEmail = warn", "missingEmail = warn", "missingCommitter = warn"],
  "repack" => ["writeBitmaps = true"],
  "transfer" => ["hideRefs=^refs/tmp/", "hideRefs=^refs/keep-around/"],
}
gitlab_ci['builds_directory'] = ENV["BUILDS_DIRECTORY"]
prometheus['enable'] = true
prometheus['listen_address'] = 'localhost:9090'
node_exporter['enable'] = true
redis_exporter['enable'] = true
gitlab_exporter['enable'] = true
prometheus_monitoring['enable'] = true
gitaly['socket_path'] = "/tmp/gitaly.socket"

Edited Nov 16, 2020 by Dale Trower