Skip to content

Read-only metrics for repository specific primaries

Sami Hiltunen requested to merge smh-per-repository-metrics into master

Adds support for collecting read-only repository metrics taking per repository primaries in to account. The new functionality is behind a boolean flag in order to keep metric collection working for virtual storage primaries as well. In-memory implementations have been removed as there is no support for per repository primaries in the local elector.

Repository is in read-only mode when the primary is out of date. With that in mind, we need to vary the query slightly to account for the differently scoped primaries.

The old and the new queries compare the repositories and the storage_repositories tables to identify primaries which are out of date. Performance wise they should be the same although dropping the in-memory implementation allows us to query the primaries directly in the metric query. That reduces the DB round trips to get the primaries of each virtual storage. Previously there was one DB call for each virtual storage to identify the primary and then the metric query. Now we do it in a single query.

When sqlElector is used, the primary is virtual storage wide, not repository specific. The primary is stored in the shard_primaries table's node_name column, where we get it from when the collector is used with virtual storage scoped primaries.

When PerRepositoryElector is used, each repository has their own primary. The primary is stored in the repositories table's primary column.

The metric is displayed on the dashboards. It's useful for the administrators so they can see if any repositories have been switched in to read-only mode following a failover. Alternative is to run dataloss after every failover. The dashboard can be found here: https://dashboards.gitlab.net/d/8EAXC-AWz/praefect?orgId=1&refresh=30s

Closes #3198 (closed)

Edited by Sami Hiltunen

Merge request reports