sshd bind address configuration not applying correctly
I've spent several hours trying to get my head around an gitlab-sshd issue.
I'm struggling with this config item:
gitlab_sshd['listen_address'] = '103.165.20.11:22'
Seems simple enough... but what I'm finding is that gitlab-sshd seems to respect the port from that listen address, but apparently the behaviour with respect to the address is weird.
The system in question has multiple IP addresses; let's say 103.165.20.10 and 103.165.20.11.
The OS itself has openssh-server listening on one of those addresses; the other one is for gitlab.
I have done these experiments:
Experiment 1:
OS openssh-server listening on 103.165.20.10:22
gitlab-sshd listen_address = 103.165.20.11:2222
WORKING: no conflict, and gitlab-sshd seems to be listening on 103.165.20.11:2222 and not on 103.165.20.10:2222, which is what I expect. It seems it bound to the address given and not to any.
Experiment 2:
OS openssh-server listening on 103.165.20.10:22 (did confirm listening only on specified address)
gitlab-sshd listen_address = 103.165.20.11:22
FAIL: there should be no conflict here; they are both trying to bind to different IP addresses, but sshd fails to start.
logs/sshd/current contains:
2025-04-02_08:30:50.63202 Bind to port 22 on 0.0.0.0 failed: Address already in use.
2025-04-02_08:30:50.63213 Server listening on :: port 22.
Note, it says 0.0.0.0 there, which is not the address I specified, and would seem to hint at the problem...
Experiment 3:
This is a weird experiment, I was testing a definite conflict
OS openssh-server listening on 0.0.0.0:22 (listen on all addresses)
gitlab-sshd listen_address = 103.165.20.11:22
FAIL: there is an obvious conflict, but what's surprising about this experiment, is that gitlab is spamming "can't bind 0.0.0.0:22, already in use!" (or something similar) to the log stream constantly, several times per second.
What surprises me here, is that it's actively spamming this message to the log output, but in the last experiment, it wasn't.... in the last experiment, there SHOULDN'T have been a conflict, but the server didn't start, and I did see that one error in the sshd log file which mentioned 0.0.0.0.
The behaviour shown in my experiments are all quite odd but one things for sure; if I use a free port, gitlab-sshd works normally, and it DOES seem to bind only to the address I specify correctly.
I wonder if there is an inconsistency somewhere in the software; maybe there's a pre-check in early startup, checking if the port is available or something which has INADDR_ANY instead of using the address specified in the config? Perhaps if that check succeeds, then the server is created to listen on the address given; because when the server is running, it DOES seem to respect the address specified, but the failures seem to log 0.0.0.0 in various ways to the log stream and log files...
Maybe someone needs to audit the gitlab-sshd startup path and check for any rogue uses of INADDR_ANY or 0.0.0.0 where the address from the config should have been used?