Fix broken Grafana dashboards

Context

We have been alerted on Slack that some Gitlay dashboards have stopped functioning correctly, including Gitaly Feature Latency Detail.

Preliminary investigation shows that when loading the dashboard, the following error appears:

image

When the graph query is manually changed to omit the job parameter like so:

1000 * avg(gitaly:grpc_server_handling_seconds:avg5m{job="$job", grpc_method="$method"}) by (fqdn)

to

1000 * avg(gitaly:grpc_server_handling_seconds:avg5m{grpc_method="$method"}) by (fqdn)

the graph begins to display data:

image

An additional warning regarding deprecated plugins is displayed too:

image

which seems to have come into effect recently.

Proposal

  • Fix the dashboards in-place. The problems seem to occur only in dashboards with the unmanaged label.
  • Consider converting unmanaged dashboards to managed ones by defining them in Jsonnet files in the Runbooks repo.