counter: Add repository counter middleware
Gitaly currently does not expose a way to determine the number of repositories present on a storage. In a cluster an admin could query the Praefect database, but this is undocumented and is not available for non-cluster nodes.
Introduce a RepositoryCounter
to manage the new
gitaly_total_repositories_count
gauge. For our purposes object pools
are considered to be separate repositories and no effort is made to
associate them with their children.
The metric will include the storage name and the first relevant
directory of the repository path as prefix
. The latter is included to
allow admins to understand which types of repositories are present. For
a standalone gitaly node prefix
would be the first directory up
from the storage, e.g. @hashed
, @pools
, or @snippets
. For a Gitaly
Cluster node it would be the child directory of @cluster
, e.g.
repositories
or pools
.
To get an estimate of the existing repositories on disk, we add a
startup task to walk all configured storages and update the
gitaly_repository_count
metric with the number of repositories
found. This is run in a goroutine to prevent walking a very large
storage from blocking Gitaly's startup.
We avoid reporting this metric to collectors until after the walker has finished to avoid having the value jump suddently. However, the walker will still race with new repository creations received while it is running and the metric will not be fully deterministic. For example:
- Repo
@hashed/ff/ff/ffff...
is created immediately after Gitaly starts up and the metric is incremented. - The storage walker reaches
@hashed/ff/ff/ffff...
and increments the metric a second time.
We could drop events receiving during the walk, but it's equally likely that the walker has already finished the new repo's directory before the event arrives and we would then undercount. Walking should finish relatively quickly, so this will probably not be too much of an issue in practice.
The walker uses ValidateRepository
to confirm if a directory is a git
repository, so in addition to the work done by filepath.WalkDir
we are
running five stat(2) calls per repository:
- The storage's root directory
- The directory being checked
- Its objects directory
- Its refs directory
- Its HEAD file
- Its packed-refs file
A non-repository directory would perform three stat
s, storage,
repository, and a failed check for objects.