## Problem A bench-built Go service exposes `/-/metrics` with no request metrics on it. There are no request counters and no latency histograms for HTTP traffic, so the service reports rate, errors, and duration for nothing it actually serves. The endpoint is real and the Go runtime collectors are there, which makes the gap easy to miss: scrape it and you get `go_*` and `process_*` series, just nothing about requests. ## Why it looks like it works The Go chassis template passes a metrics registry into LabKit's HTTP server ([`cli/fixtures/go-chassis/chassis/cmd/main.go.template`](https://gitlab.com/gitlab-org/lab-bench/bench/-/blob/3205a6946ca01ceeba7124188f8e18c52c6ce578/cli/fixtures/go-chassis/chassis/cmd/main.go.template)): ```go srv := httpserver.NewWithConfig(&httpserver.Config{ Addr: ":" + port, ProbeAddr: ":" + probePort, Handler: mux, Logger: a.Logger(), Tracer: a.Tracer(), Metrics: a.Metrics(), // ... }) ``` `Metrics: a.Metrics()` does two things: it mounts `/-/metrics` on the probe port, and it registers the default `go_` and `process_` collectors. It does not instrument requests. Confirmed upstream at labkit [`df307e03`](https://gitlab.com/gitlab-org/labkit/-/tree/df307e038980743f2e50a83cf718b1de2b50f352): - [`v2/httpserver/middleware.go`](https://gitlab.com/gitlab-org/labkit/-/blob/df307e038980743f2e50a83cf718b1de2b50f352/v2/httpserver/middleware.go) contains no metric, counter, or histogram references. The package README documents per-request behavior as logging only: "Emits a structured `slog` line per request with `method`, `path`, `status`, and `duration_ms`." - [`v2/metrics`](https://gitlab.com/gitlab-org/labkit/-/tree/df307e038980743f2e50a83cf718b1de2b50f352/v2/metrics) is an isolated Prometheus registry plus a `BuildName` helper and label constants. Its own README example has the caller define a `CounterVec` and increment it. Neither bench generator emits metrics either: no metric references anywhere in [`cli/src/build/go.rs`](https://gitlab.com/gitlab-org/lab-bench/bench/-/blob/3205a6946ca01ceeba7124188f8e18c52c6ce578/cli/src/build/go.rs). ## What a service has to do today The [go-service-template exemplar](https://gitlab.com/gitlab-org/quality/go-service-template), a deployed Theseus service, hand-builds all of it in [`internal/transport/http.go`](https://gitlab.com/gitlab-org/quality/go-service-template/-/blob/dfc86e911fd2d64e007a2fe95a0406f8484bdf1f/internal/transport/http.go): ```go requestsTotal := prometheus.NewCounterVec(prometheus.CounterOpts{ Name: m.BuildName("http", "requests_total"), // ... }, []string{metrics.LabelEndpointID, "code", "method"}) requestDuration := prometheus.NewHistogramVec(prometheus.HistogramOpts{ Name: m.BuildName("http", "request_duration_seconds"), Buckets: metrics.DurationBuckets, }, []string{metrics.LabelEndpointID, "code", "method"}) m.MustRegister(requestsTotal, requestDuration) endpointLabel := promhttp.WithLabelFromCtx(metrics.LabelEndpointID, routePatternFromCtx) handler = routePatternMiddleware(handler) handler = promhttp.InstrumentHandlerCounter(requestsTotal, handler, endpointLabel) handler = promhttp.InstrumentHandlerDuration(requestDuration, handler, endpointLabel) ``` Note `routePatternMiddleware`. It exists so the endpoint label carries the matched `ServeMux` pattern (`GET /greetings/{id}`) rather than the raw URL path. Without it the label is per-URL and cardinality is unbounded. Any chassis implementation has to solve the same problem, and bench is better placed to do it than each service is, because the route patterns come from `bench.yaml`. ## Why this is a good fit for the chassis Route patterns are already declared in the spec. The generator knows every `HTTPEndpoint.route` and its methods at build time, so it can wire instrumentation with the correct low-cardinality label without the runtime context dance the exemplar needs. ## Scope Emit RED metrics for HTTP traffic from the generated Go entrypoint, with no developer action. Decide the metric names and label set once, here, so the gRPC and NATS transports have something concrete to match. Worth settling as part of this issue: - Metric naming. Three schemes are in use today: | Source | Names | |---|---| | go-service-template | `gitlab_http_requests_total`, `gitlab_http_request_duration_seconds` (via `BuildName`) | | rust-service-template | `http_requests_total`, `http_request_duration_seconds` | | labkit-rs `HttpMetricsLayer` (bench's Rust chassis) | OTel `http.server.request.duration`, exported as `http_server_request_duration_seconds` | The Go names should match what the Rust chassis already emits, or both should move together. This is a LabKit-level choice more than a bench one, so it's worth confirming with LabKit before picking. - Label set. The exemplar uses endpoint, code, and method. `metrics.LabelEndpointID` is a LabKit constant, so the label name is already standardized upstream. labkit-rs uses the OTel names instead: `http.route`, `http.request.method`, `http.response.status_code`. - Whether the route-pattern label is derived at build time from `bench.yaml` (bench's advantage) or at runtime from the router (the exemplar's approach). Rust is already covered. labkit-rs ships [`HttpMetricsLayer`](https://gitlab.com/gitlab-org/rust/labkit-rs/-/blob/v0.6.0/labkit/src/http/metrics_layer.rs), and the Rust chassis has mounted it since [`f75e5d9b`](https://gitlab.com/gitlab-org/lab-bench/bench/-/commit/f75e5d9b) ([`main.rs.template`](https://gitlab.com/gitlab-org/lab-bench/bench/-/blob/1ae9616a/cli/fixtures/rust-chassis/chassis/src/main.rs.template)). It labels by `http.route`, the matched route pattern. An earlier version of this issue said labkit-rs had no HTTP metrics layer; that was wrong. ## Acceptance criteria - [ ] A built example serves traffic and `/-/metrics` reports a request counter and a latency histogram for it - [ ] The endpoint label uses the declared route pattern, so two requests to `/greetings/1` and `/greetings/2` share one series - [ ] Metric names and labels are documented, and the Go and Rust chassis emit the same ones - [ ] The Go generator emits the instrumentation with no developer action - [ ] A test asserts the counter increments and the histogram observes on a request through the generated entrypoint - [ ] The epic's definition-of-done entries for gRPC and NATS metrics reference this issue's decisions rather than a nonexistent HTTP baseline ## Note on the epic's prior claim The epic body previously listed HTTP RED metrics among capabilities earlier milestones had landed, and both the gRPC ([bench#65](https://gitlab.com/gitlab-org/lab-bench/bench/-/work_items/65)) and NATS ([bench#66](https://gitlab.com/gitlab-org/lab-bench/bench/-/work_items/66)) issues asked for metrics "matching HTTP." Those references have been corrected; this issue establishes the baseline they were assuming. Part of [DM-4: Phase 1 Inbound](https://gitlab.com/groups/gitlab-org/lab-bench/-/epics/4).