Expose Prometheus metrics (!23) · Merge requests · GitLab.org / Cloud Native / charts / OpenBao

What does this MR do?

Expose Prometheus metrics

Expose telemetry configuration.
Service metrics without unauthenticated via another port.
Support custom annotations of services to apply scrape config.

Relates https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1754

Changelog: added

Background

This is a preparation for https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1754+. Once we expose the metrics, we can configure the Prometheus bundled with GitLab chart to either scrape metrics from active bao nodes only, or from both active and inactive nodes.

In any case, we will need to make the metrics available for both set of nodes, since we don't know which node will be active/inactive at orchestration time.

Test Plan

Attempt to query metrics through the Ingress without authentication

$ curl http://openbao.example.com/v1/sys/metrics?format=prometheus
{"errors":["permission denied"]}

Exec into a Pod and query the metrics endpoint without authentication

$ curl --silent http://openbao.example.com:8209/v1/sys/metrics?format=prometheus | head -n 5
# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.8613e-05
go_gc_duration_seconds{quantile="0.25"} 6.7677e-05
go_gc_duration_seconds{quantile="0.5"} 0.000199111

Author's Checklist

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

Ensure a release milestone is set.
MR title and description are up to date, accurate, and descriptive.
MR targeting the appropriate branch.
MR has a green pipeline on GitLab.com.
When ready for review, MR is labeled workflowready for review per the MR workflow.

Expected

Test plan indicating conditions for success has been posted and passes.
~~Documentation is created or updated.~~
Tests are added.