nginx: 504 Gateway Time-out after upgrade to 15.1: worker_processes is set to zero

Hello,

We had a strange issues with two (of two) omnibus-gitlab instances after upgrade from 15.0 to 15.1.

Gitlab was not responding with a 504 Gateway Time-out.

After hours of debugging and comparison with a fresh install, in nginx.conf worker_processes was set to zero.

The config look like this:

external_url 'https://git.XXXX.ch'


gitlab_rails['gitlab_email_from'] = 'XX+gitlab@XX.ch'
gitlab_rails['gitlab_email_display_name'] = 'Gitlab XXXX'
gitlab_rails['gitlab_email_reply_to'] = 'XXX+gitlab@Xxx.ch'

gitlab_rails['gitlab_default_can_create_group'] = true


gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '::1/128']

gitlab_rails['incoming_email_enabled'] = false

gitlab_rails['incoming_email_address'] = ""

gitlab_rails['incoming_email_email'] = ""

gitlab_rails['incoming_email_password'] = ""

gitlab_rails['incoming_email_host'] = ""
gitlab_rails['incoming_email_port'] = 993
gitlab_rails['incoming_email_ssl'] = true
gitlab_rails['incoming_email_start_tls'] = false

gitlab_rails['incoming_email_mailbox_name'] = "INBOX"

gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"

gitlab_rails['backup_keep_time'] = 0

gitlab_rails['gitlab_shell_ssh_port'] = 10522

gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "unicode"
gitlab_rails['db_database'] = "gitlab"
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['db_password'] = "XXXXX"
gitlab_rails['db_host'] = "10.7.0.1"
gitlab_rails['db_port'] = 5432

gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "127.0.0.1"
gitlab_rails['smtp_port'] = 25

gitlab_rails['registry_enabled'] = false

registry['enable'] = false

postgresql['enable'] = false

nginx['redirect_http_to_https'] = false
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on",
}

prometheus['enable'] = false
alertmanager['enable'] = false
node_exporter['enable'] = false
redis_exporter['enable'] = false
postgres_exporter['enable'] = false
gitlab_exporter['enable'] = false

letsencrypt['enable'] = false

gitlab_rails['kerberos_enabled'] = false
gitlab_rails['packages_enabled'] = false

gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
gitlab_rails['omniauth_block_auto_created_users'] = false
gitlab_rails['omniauth_auto_link_saml_user'] = true

gitlab_rails['omniauth_providers'] = [{
    name: 'saml',
    

    args: {
        assertion_consumer_service_url: 'https://git.XXXX.ch/users/auth/saml/callback',
        idp_cert_fingerprint: '<01:FA:DC:2F:FB:0D:EC:E9:DE:59:F4:D2:E8:5B:6E:1B:72:D9:16:88'>,
        idp_sso_target_url: 'https://sso.XXXX.ch/services/services.services.saml2.SAML2/sls',
        issuer: 'https://git.XXX.ch',
        name_identifier_format: 'urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress',
        
        
        upstream_two_factor_authn_contexts: 'urn:oasis:names:tc:SAML:2.0:ac:classes:unspecified'
        
    },
    label: 'XXXXX SSO'
}]

grafana['enable'] = false

We tried with a almost-empty config, no changes. The finial config we tried was:

external_url 'http://git.XXXX.ch'

gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "unicode"
gitlab_rails['db_database'] = "gitlab"
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['db_password'] = "XXXXX"
gitlab_rails['db_host'] = "10.7.0.1"
gitlab_rails['db_port'] = 5432


postgresql['enable'] = false

We fixed the issue by explicitly setting nginx['worker_processes'] = 4 in gitlab.rb. If we remove it and reconfigure gitlab, he set worker_process back to 0.

The line generating the default doesn't seem to have changed in years: https://gitlab.com/gitlab-org/omnibus-gitlab/-/blame/master/files/gitlab-cookbooks/gitlab/attributes/default.rb#L674

Details of package version

15.1.0-ce.0

Environment details

  • Operating System: Debian 10
  • Installation Target: LXC VM (Proxmox)
  • Installation Type:
    • Upgrade from version 15.0.3
  • single node installation

Workaround

Set nginx['worker_processes'] = 4, or other appropriate value in gitlab.rb

Edited by Jason Plum