feat: Support multiprocess metrics and `WEB_CONCURRENCY` > 1
What does this merge request do and why?
When WEB_CONCURRENCY
is set to a value > 1, uvicorn spawns multiple
processes to support more Web requests. Previously the app spawned a
separate HTTP metrics server for every worker, which causes port bind
failures once one HTTP server runs.
To avoid this, start the HTTP server outside of the app. We also set up the structured logging earlier to accommodate this.
Multiprocess metrics can be activated by setting
PROMETHEUS_MULTIPROC_DIR
to a writeable directory. This directory
MUST be cleaned before the application starts.
Relates to #437 (closed)
How to set up and validate locally
- Run
mkdir /tmp/metrics
- Run
WEB_CONCURRENCY=3 PROMETHEUS_MULTIPROC_DIR=/tmp/metrics poetry run ai_gateway
- Make some requests with the AI Gateway.
- Run
curl http://localhost:8082/metrics
. Make sure all metrics appear to be there.
Repeat the experiment without WEB_CONCURRENCY
and PROMETHEUS_MULTIPROC_DIR
.
Edited by Stan Hu