Skip to content

counter: Add repository counter middleware

Will Chandler requested to merge wc/repo-count-metric into master

Gitaly currently does not expose a way to determine the number of repositories present on a storage. In a cluster an admin could query the Praefect database, but this is undocumented and is not available for non-cluster nodes.

Introduce a RepositoryCounter to manage the new gitaly_total_repositories_count gauge. For our purposes object pools are considered to be separate repositories and no effort is made to associate them with their children.

The metric will include the storage name and the first relevant directory of the repository path as prefix. The latter is included to allow admins to understand which types of repositories are present. For a standalone gitaly node prefix would be the first directory up from the storage, e.g. @hashed, @pools, or @snippets. For a Gitaly Cluster node it would be the child directory of @cluster, e.g. repositories or pools.

To get an estimate of the existing repositories on disk, we add a startup task to walk all configured storages and update the gitaly_repository_count metric with the number of repositories found. This is run in a goroutine to prevent walking a very large storage from blocking Gitaly's startup.

We avoid reporting this metric to collectors until after the walker has finished to avoid having the value jump suddently. However, the walker will still race with new repository creations received while it is running and the metric will not be fully deterministic. For example:

  • Repo @hashed/ff/ff/ffff... is created immediately after Gitaly starts up and the metric is incremented.
  • The storage walker reaches @hashed/ff/ff/ffff... and increments the metric a second time.

We could drop events receiving during the walk, but it's equally likely that the walker has already finished the new repo's directory before the event arrives and we would then undercount. Walking should finish relatively quickly, so this will probably not be too much of an issue in practice.

The walker uses ValidateRepository to confirm if a directory is a git repository, so in addition to the work done by filepath.WalkDir we are running five stat(2) calls per repository:

  • The storage's root directory
  • The directory being checked
  • Its objects directory
  • Its refs directory
  • Its HEAD file
  • Its packed-refs file

A non-repository directory would perform three stats, storage, repository, and a failed check for objects.

Edited by Will Chandler

Merge request reports