alertmanager fails to start after upgrade to 15.4.2
Summary
After upgrading to 15.4.2 CE (omnibus) and restarting the server, alertmanager fails to start & can't be restarted. This has happened on two individual servers running separate gitlab instances. I have provided details for both servers, but the behavior, error messages, and logs are the same between the two.
Steps to reproduce
#sudo yum upgrade gitlab-ce-15.4.2-ce.0.el7
#sudo shutdown -h now
- Power on the server & login
#sudo gitlab-ctl status
What is the current bug behavior?
The command # sudo gitlab-ctl status
reports:
down: alertmanager: 0s, normally up, want up; run: log: (pid 2484) 2259s
What is the expected correct behavior?
The alertmanager service starts
Relevant logs
Relevant logs
2022-10-19_16:36:02.95126 level=error ts=2022-10-19T16:36:02.951Z caller=main.go:250 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2022-10-19_16:36:03.96567 level=info ts=2022-10-19T16:36:03.964Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=master, revision=)" 2022-10-19_16:36:03.96568 level=info ts=2022-10-19T16:36:03.964Z caller=main.go:226 build_context="(go=go1.17.13, user=GitLab-Omnibus, date=)" 2022-10-19_16:36:03.96569 level=warn ts=2022-10-19T16:36:03.965Z caller=cluster.go:177 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2022-10-19_16:36:03.96798 level=error ts=2022-10-19T16:36:03.967Z caller=main.go:250 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2022-10-19_16:36:04.98317 level=info ts=2022-10-19T16:36:04.982Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=master, revision=)" 2022-10-19_16:36:04.98318 level=info ts=2022-10-19T16:36:04.982Z caller=main.go:226 build_context="(go=go1.17.13, user=GitLab-Omnibus, date=)" 2022-10-19_16:36:04.98318 level=warn ts=2022-10-19T16:36:04.982Z caller=cluster.go:177 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2022-10-19_16:36:04.98546 level=error ts=2022-10-19T16:36:04.985Z caller=main.go:250 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2022-10-19_16:36:06.00117 level=info ts=2022-10-19T16:36:06.000Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=master, revision=)" 2022-10-19_16:36:06.00119 level=info ts=2022-10-19T16:36:06.000Z caller=main.go:226 build_context="(go=go1.17.13, user=GitLab-Omnibus, date=)" 2022-10-19_16:36:06.00145 level=warn ts=2022-10-19T16:36:06.001Z caller=cluster.go:177 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2022-10-19_16:36:06.00421 level=error ts=2022-10-19T16:36:06.004Z caller=main.go:250 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2022-10-19_16:36:07.01917 level=info ts=2022-10-19T16:36:07.018Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=master, revision=)" 2022-10-19_16:36:07.01918 level=info ts=2022-10-19T16:36:07.018Z caller=main.go:226 build_context="(go=go1.17.13, user=GitLab-Omnibus, date=)" 2022-10-19_16:36:07.01924 level=warn ts=2022-10-19T16:36:07.019Z caller=cluster.go:177 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2022-10-19_16:36:07.02163 level=error ts=2022-10-19T16:36:07.021Z caller=main.go:250 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
Details of package version
Provide the package version installation details
rpm -qa | grep 'gitlab'
gitlab-ce-15.4.2-ce.0.el7.x86_64
Environment details
- Operating System:
- Server A: CentOS Linux release 7.3.1611 (Core)
- Server B: CentOS Linux release 7.9.2009 (Core)
- Installation Target, remove incorrect values:
- VM: VMWare
- Installation Type, remove incorrect values:
- Upgrade from version 15.0.2 on both servers
- Is there any other software running on the machine: No.
- Is this a single or multiple node installation? Single, 2 separate instances
- Resources
- CPU: Server A - 2, Server B - 4
- Memory total: 8 GB on both
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
Server A
external_url 'https://gitlab.mydomain.com/' gitlab_rails['gitlab_email_subject_suffix'] = '' nginx['enable'] = true nginx['redirect_http_to_https'] = true nginx['redirect_http_to_https_port'] = 80 nginx['ssl_certificate'] = "/etc/pki/tls/certs/star.mydomain.com.cert.bundled" nginx['ssl_certificate_key'] = "/etc/pki/tls/private/star.mydomain.com.key" nginx['ssl_ciphers'] = "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256" nginx['ssl_prefer_server_ciphers'] = "on" nginx['ssl_protocols'] = "TLSv1.1 TLSv1.2" nginx['ssl_session_timeout'] = "5m"
Server B
external_url 'https://gitlab.subdomain.mydomain.com/' gitlab_rails['gitlab_email_subject_suffix'] = '' gitlab_rails['ldap_enabled'] = true gitlab_rails['prevent_ldap_sign_in'] = false gitlab_rails['ldap_servers'] = { 'main' => { 'label' => 'LDAP', 'host' => 'duoauthproxy.mydomain.com', 'port' => 389, 'uid' => 'uid', 'encryption' => 'start_tls', 'verify_certificates' => true, 'bind_dn' => '', 'password' => '', 'verify_certificates' => true, 'tls_options' => { 'ca_file' => '/etc/duoauthproxy/ca.cert', 'ssl_version' => '', 'ciphers' => '', 'cert' => '', 'key' => '' }, 'timeout' => 10, 'active_directory' => false, 'allow_username_or_email_login' => false, 'block_auto_created_users' => false, 'base' => 'ou=People,dc=mydomain,dc=com', 'user_filter' => 'host=subdomain', 'attributes' => { 'username' => ['uid', 'userid', ], 'email' => ['mail', 'email', ], 'name' => 'cn', 'first_name' => 'givenName', 'last_name' => 'sn' }, 'lowercase_usernames' => false, 'group_base' => '', 'admin_group' => '', 'external_groups' => [], 'sync_ssh_keys' => false } } nginx['enable'] = true nginx['redirect_http_to_https'] = true nginx['redirect_http_to_https_port'] = 80 nginx['ssl_certificate'] = "/etc/pki/tls/certs/kub-admin1.mydomain.com.cert.bundled" nginx['ssl_certificate_key'] = "/etc/pki/tls/private/kub-admin1.mydomain.com.key" nginx['ssl_prefer_server_ciphers'] = "on" nginx['ssl_protocols'] = "TLSv1.1 TLSv1.2" nginx['ssl_session_timeout'] = "5m"