Observability: Custom Metrics

What

In addition to #72 (closed), Runway service can opt-in to custom metrics by providing list of scrape targets. When configured, a sidecar container will be deployed alongside ingress container with Prometheus running in Agent Mode configured to remote write to Thanos.

Why

Right now, Model Gateway implements custom instrumentation for tracking inference counts, prompt language counts, prompt length counts, etc. Existing custom dashboard will stop reporting unless we continue to scrape custom /metrics endpoint.