Design secure way to export limited data from production database to monitoring tools like Grafana and Slack

We have an urgent need to export some limited production database information to Grafana and Slack. Effectively we already do this via Prometheus but that only works for metrics, not any richer data. The three cases that have come up recently are:

  1. Query strings for a given queryid hash.
  2. Explain plan for a given query
  3. Repmgr status on all database machines

It would be nice to have a little microservice that exported this data via a simple REST interface that we could then access from our Grafana dashboards, Slack, or other channels.

But for a first iteration I set up a very limited grafana user on Postgres and granted it execute on a single function lookup_query() which looks up the query text of a queryid hash. The password is in 1password. I added a data source in Grafana to connect to Postgres and run this one query.

You can view the result at: https://performance.gitlab.net/dashboard/db/postgres-single-query-drill-down?orgId=1&var-environment=prd&var-queryid=2441158036&var-fqdn=All&from=now-7d&to=now

You can look at any of the top resource consuming queries here -- click on a queryid to view the history of that query: https://performance.gitlab.net/dashboard/db/postgresql-top-queries-wip?from=now-24h&to=now&var-environment=prd&orgId=1

I think it would probably be better to run a microservice on the database hosts or on a different host that has a collection of canned queries and serves up the results in json. However I was mistaken earlier when I believed Grafana could use such a data source. Afaics it doesn't have a data source type for such a microservice.

Effectively that's what I've set up though. Postgres is providing the user authentication and by only allowing Grafana to execute this function that's the "canned query". We can add a collection of functions that are all we allow Grafana to access.

Some details:

Grafana is configured to not use SSL because for some reason we don't seem to have SSL set up. It may be using md5 auth but that's still much less satisfactory than having SSL due to connection hijacking attacks. It would be vastly preferable to have a private key rather than a simple password.