nginx: 504 Gateway Time-out after upgrade to 15.1: worker_processes is set to zero
Hello,
We had a strange issues with two (of two) omnibus-gitlab instances after upgrade from 15.0 to 15.1.
Gitlab was not responding with a 504 Gateway Time-out.
After hours of debugging and comparison with a fresh install, in nginx.conf
worker_processes was set to zero.
The config look like this:
external_url 'https://git.XXXX.ch'
gitlab_rails['gitlab_email_from'] = 'XX+gitlab@XX.ch'
gitlab_rails['gitlab_email_display_name'] = 'Gitlab XXXX'
gitlab_rails['gitlab_email_reply_to'] = 'XXX+gitlab@Xxx.ch'
gitlab_rails['gitlab_default_can_create_group'] = true
gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '::1/128']
gitlab_rails['incoming_email_enabled'] = false
gitlab_rails['incoming_email_address'] = ""
gitlab_rails['incoming_email_email'] = ""
gitlab_rails['incoming_email_password'] = ""
gitlab_rails['incoming_email_host'] = ""
gitlab_rails['incoming_email_port'] = 993
gitlab_rails['incoming_email_ssl'] = true
gitlab_rails['incoming_email_start_tls'] = false
gitlab_rails['incoming_email_mailbox_name'] = "INBOX"
gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"
gitlab_rails['backup_keep_time'] = 0
gitlab_rails['gitlab_shell_ssh_port'] = 10522
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "unicode"
gitlab_rails['db_database'] = "gitlab"
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['db_password'] = "XXXXX"
gitlab_rails['db_host'] = "10.7.0.1"
gitlab_rails['db_port'] = 5432
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "127.0.0.1"
gitlab_rails['smtp_port'] = 25
gitlab_rails['registry_enabled'] = false
registry['enable'] = false
postgresql['enable'] = false
nginx['redirect_http_to_https'] = false
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on",
}
prometheus['enable'] = false
alertmanager['enable'] = false
node_exporter['enable'] = false
redis_exporter['enable'] = false
postgres_exporter['enable'] = false
gitlab_exporter['enable'] = false
letsencrypt['enable'] = false
gitlab_rails['kerberos_enabled'] = false
gitlab_rails['packages_enabled'] = false
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
gitlab_rails['omniauth_block_auto_created_users'] = false
gitlab_rails['omniauth_auto_link_saml_user'] = true
gitlab_rails['omniauth_providers'] = [{
name: 'saml',
args: {
assertion_consumer_service_url: 'https://git.XXXX.ch/users/auth/saml/callback',
idp_cert_fingerprint: '<01:FA:DC:2F:FB:0D:EC:E9:DE:59:F4:D2:E8:5B:6E:1B:72:D9:16:88'>,
idp_sso_target_url: 'https://sso.XXXX.ch/services/services.services.saml2.SAML2/sls',
issuer: 'https://git.XXX.ch',
name_identifier_format: 'urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress',
upstream_two_factor_authn_contexts: 'urn:oasis:names:tc:SAML:2.0:ac:classes:unspecified'
},
label: 'XXXXX SSO'
}]
grafana['enable'] = false
We tried with a almost-empty config, no changes. The finial config we tried was:
external_url 'http://git.XXXX.ch'
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "unicode"
gitlab_rails['db_database'] = "gitlab"
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['db_password'] = "XXXXX"
gitlab_rails['db_host'] = "10.7.0.1"
gitlab_rails['db_port'] = 5432
postgresql['enable'] = false
We fixed the issue by explicitly setting nginx['worker_processes'] = 4
in gitlab.rb. If we remove it and reconfigure gitlab, he set worker_process back to 0.
The line generating the default doesn't seem to have changed in years: https://gitlab.com/gitlab-org/omnibus-gitlab/-/blame/master/files/gitlab-cookbooks/gitlab/attributes/default.rb#L674
Details of package version
15.1.0-ce.0
Environment details
- Operating System: Debian 10
- Installation Target: LXC VM (Proxmox)
- Installation Type:
- Upgrade from version 15.0.3
- single node installation
Workaround
Set nginx['worker_processes'] = 4
, or other appropriate value in gitlab.rb