Argument checking for pprofListenAddr
There was a recent error that caused a configuration issue where an invalid argument combination was passed to workhorse:
gitlab-workhorse -logFile stdout -logFormat json -pprofListenAddr \
-apiLimit 5 -apiQueueDuration 60s -apiQueueLimit 200 -listenAddr 0.0.0.0:8181 \
-documentRoot /srv/gitlab/public -secretPath /etc/gitlab/gitlab-workhorse/secret \
-config /srv/gitlab/config/workhorse-config.toml -prometheusListenAddr 0.0.0.0:9229
Note that after -pprofListenAddr
there is no value, this will result in the following error:
{"error":"listen tcp: address -apiLimit: missing port in address","level":"error","msg":"Failed to start pprof listener","time":"2020-10-16T14:53:24Z"}
but workhorse starts anyway, except the listen address changes to localhost
:
tcp 0 0 127.0.0.1:8181 0.0.0.0:* LISTEN -
In the case of incident gitlab-com/gl-infra/production#2836 (closed), this still resulted in a healthy workhorse pod, but because the service is no longer accessible from the outside, it caused a GKE cluster to be dropped from one of our backends.
The reason why this option was passed this way, is because we default in omnibus to passing an empty string to this option for some reason https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/86537099d80cb7f6ca9ac2f24573b50d6a2e9072/files/gitlab-cookbooks/gitlab/templates/default/sv-gitlab-workhorse-run.erb#L27. The error was not specifying an empty string -pprofListenAddr ''
in the K8s config (or specifying an actual address).
I think the right thing for us to do here is to fail hard if we get an argument parse error like this.