Skip to content

feat: Support multiprocess metrics and `WEB_CONCURRENCY` > 1

Stan Hu requested to merge sh-multiprocess-metrics into main

What does this merge request do and why?

When WEB_CONCURRENCY is set to a value > 1, uvicorn spawns multiple processes to support more Web requests. Previously the app spawned a separate HTTP metrics server for every worker, which causes port bind failures once one HTTP server runs.

To avoid this, start the HTTP server outside of the app. We also set up the structured logging earlier to accommodate this.

Multiprocess metrics can be activated by setting PROMETHEUS_MULTIPROC_DIR to a writeable directory. This directory MUST be cleaned before the application starts.

Relates to #437 (closed)

How to set up and validate locally

  1. Run mkdir /tmp/metrics
  2. Run WEB_CONCURRENCY=3 PROMETHEUS_MULTIPROC_DIR=/tmp/metrics poetry run ai_gateway
  3. Make some requests with the AI Gateway.
  4. Run curl http://localhost:8082/metrics. Make sure all metrics appear to be there.

Repeat the experiment without WEB_CONCURRENCY and PROMETHEUS_MULTIPROC_DIR.

Edited by Stan Hu

Merge request reports