[deployed] Add Local Observability Stack with Prometheus and Grafana
Problem to Solve
The project currently lacks a local observability setup for monitoring application metrics. This makes it difficult for developers to inspect the health and performance of services like the HTTP server during development. Without local metric visualization, it is also challenging to develop and validate new metrics instrumentation and dashboards before they are deployed to production.
Proposed Solution
This change introduces a local observability stack using Docker Compose, including Prometheus, Grafana, and Grafana Mimir.
Key components of the solution:
-
Docker-based Observability Stack based on the production stack:
- Prometheus: Configured to scrape metrics from the http-server-deployed service.
- Grafana: Pre-configured with a Prometheus data source and a default "GKG Overview" dashboard for immediate visualization.
- Mimir: Included for long-term metrics storage, mirroring our production setup.
- New mise tasks (observability:up, observability:down, etc.) are added for easy management of the stack.
-
Metrics Instrumentation in http-server-deployed:
- The service is instrumented using the prometheus crate to expose standard metrics via a /metrics endpoint.
- Initial metrics include HTTP request counts, request latency, and a service health gauge as baseline.
- The server now defaults to binding to a TCP socket (0.0.0.0:8080) to be easily reachable from the Dockerized Prometheus instance.