CentOS 8 : prometheus port conflict

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

prometheus fails to start on CentOS 8 with the default port configure (9090) because the port is already in use, as mentionned in the prometheus log file :

2022-01-06_10:26:33.25154 level=error ts=2022-01-06T10:26:33.244Z caller=main.go:608 msg="Unable to start web listener" err="listen tcp 127.0.0.1:9090: bind: address already in use"

In fact etc/services shows that this port is allocated to websm service:

websm           9090/tcp                # WebSM
websm           9090/udp                # WebSM

Steps to reproduce

Use the default configuration for prometheus listen port on a CentOS 8 machine (with Omnibus configuration if relevant): prometheus will fail to start as shown by gitlab-ctl status.

What is the expected correct behavior?

prometheus starting with the default configuration. I work around the problem by using port 19090 (defined in prometheus['listen_address']).

Relevant logs and/or screenshots

From prometheus log file :

2022-01-06_10:26:33.25154 level=error ts=2022-01-06T10:26:33.244Z caller=main.go:608 msg="Unable to start web listener" err="listen tcp 127.0.0.1:9090: bind: address already in use"

Results of GitLab environment info

Expand for output related to GitLab environment info 

Output of `gitlab-rake gitlab:check SANITIZE=true`:  

```
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.22.1 ? ... OK (13.22.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
        User output sanitized. Found 100 users of 100 limit.

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
3/2 ... yes
4/3 ... yes
5/4 ... yes
6/5 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git version >= 2.33.0 ? ... yes (2.33.1)
Git user has default SSH configuration? ... yes
Active users: ... 5
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
```

Possible fixes

Change default port for prometheus to an unallocated one (I used 19090but any other non allocated value in /etc/services could work).

Edited by 🤖 GitLab Bot 🤖