Observability: Custom Metrics
What
In addition to #72 (closed), Runway service can opt-in to custom metrics by providing list of scrape targets. When configured, a sidecar container will be deployed alongside ingress container with Prometheus running in Agent Mode configured to remote write to Thanos.
Why
Right now, Model Gateway implements custom instrumentation for tracking inference counts, prompt language counts, prompt length counts, etc. Existing custom dashboard will stop reporting unless we continue to scrape custom /metrics
endpoint.
Edited by Chance Feick