@@ -1121,14 +1121,25 @@ Long term service and support (LTSS) for SUSE Linux Enterprise Server (SLES) 12
Announced in 14.7
{: .note}
In GitLab 15.0, you can no longer serve Sidekiq metrics and health checks over a single address and port.
:arrow_up_small: Re-written 2022-10-26
To improve stability, availability, and prevent data loss in edge cases, GitLab now serves
[Sidekiq metrics and health checks from two separate servers](https://gitlab.com/groups/gitlab-org/-/epics/6409).
Do you perform any sort of monitoring of Sidekiq? [Reding the impact statement](https://gitlab.com/gitlab-org/gitlab/-/issues/347509#affected-topology), there are three endpoints which are now split across two different ports:
When you use Omnibus or Helm charts, if GitLab is configured for both servers to bind to the same address,
a configuration error occurs.
To prevent this error, choose different ports for the metrics and health check servers:
> /metrics are now served by a dedicated server process, not an individual worker.
> /readiness and /liveness continue to be served by an in-worker HTTP server.
From 15.0, the defaults (`# sidekiq['health_checks_listen_port'] = 8092`) indicate that the additional port was added for `/readiness` and `/liveness`, with the original port being the [default for Prometheus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/15.4.0+ee.0/files/gitlab-config-template/gitlab.rb.template#L1068):
```
##! Specifies where Prometheus metrics endpoints should be made available for Sidekiq processes.
# sidekiq['metrics_enabled'] = true
# sidekiq['listen_address'] = "localhost"
# sidekiq['listen_port'] = 8082
```
Looking back, 8082 [was the original default port definition](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/14.8.6+ee.0/files/gitlab-config-template/gitlab.rb.template#L1056) for all this monitoring capability.
Read more:
-[Configure Sidekiq health checks](https://docs.gitlab.com/ee/administration/sidekiq.html#configure-health-checks)
-[Configure the Sidekiq metrics server](https://docs.gitlab.com/ee/administration/sidekiq.html#configure-the-sidekiq-metrics-server)
...
...
@@ -1139,11 +1150,10 @@ If you installed GitLab from source, verify manually that both servers are confi
-[Issue contains details on how Omnibus and Charts/K8S deployments are affected](https://gitlab.com/gitlab-org/gitlab/-/issues/347509#affected-topology).
- Omnibus: If you monitored sidekiq using URIs `/readiness` or `/liveness` then the port will change
-[relevant ports are 8082 / 8092](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/15.0.5+ee.0/files/gitlab-config-template/gitlab.rb.template#L1055)
- I think `sidekiq['health_checks_listen_port']` is the setting that needs to change
- Charts deployments unaffected unless default settings for `/readiness` or `/liveness` are being over-ridden.
-`/readiness` and `/liveness` will be on port 8092 by default from 15.0. This can be changed with `sidekiq['health_checks_listen_port'`
- If you poll those endpoints, switch to the new port, and ensure any mapping in docker or firewall ports are changed.
- Prometheus will continue to be able to poll `/health` on 8082. If you want to change that port, set `sidekiq['listen_port'] = 8082`
- Don't set them to the same port, as they're now handled by two different processes.
