Release Manager dashboard: Mirror Status
Overview
For every project in gitlab-org/security
, it is assumed that there is a Canonical project in gitlab-org
which has a push mirror to the Security repository, which itself has a push mirror to Build (dev.gitlab.org).
We want to show something akin to /chatops run mirror status
:
In this output:
- Gitaly has mirrored from Canonical to Security to Build without error.
- GitLab's Security repository has diverged on 7 branches, so Canonical cannot mirror them to Security. Security has successfully mirrored to Build.
- GitLab FOSS's Security repository has diverged on 4 branches, so Canonical cannot mirror them to Security. Security has successfully mirrored to Build.
- gitlab-pages has mirrored from Canonical to Security to Build without error.
- gitlab-shell has mirrored from Canonical to Security to Build without error.
Data gathering
The current plan is to run a scheduled CI job that gathers the mirror status and pushes to the Prometheus pushgateway. For good measure, we could also push the latest status when the ChatOps command is used, since we're already gathering the requisite data.
The data is fetched from the gitlab.com and dev.gitlab.org RemoteMirror APIs.
Prometheus format
- One entry per project (
path
, e.g.,gitlab-org/gitlab
), per mirrortype
(e.g.,security
orbuild
) - Gauge, value of
1
= success,0
= error
Example of one time series:
# HELP delivery_mirror_info GitLab Project Mirror Status
# TYPE delivery_mirror_info gauge
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/build/CNG",status="ok",type="build"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/build/CNG",status="ok",type="security"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitaly",status="ok",type="build"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitaly",status="ok",type="security"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitlab",status="ok",type="build"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitlab",status="ok",type="security"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitlab-foss",status="ok",type="build"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitlab-foss",status="ok",type="security"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitlab-pages",status="Push error: a6106921fc9960846731b268c3a71cf966ef42e9",type="security"} 0
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitlab-pages",status="ok",type="build"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitlab-shell",status="Push error: 75217544827cab4a32bc9adff54e7ce64c12c2c5",type="security"} 0
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitlab-shell",status="ok",type="build"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitlab-workhorse",status="ok",type="build"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/gitlab-workhorse",status="ok",type="security"} 1
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/omnibus-gitlab",status="Push error: a556879111ec9080b71922abb9ef7ce4ae8b62ba",type="build"} 0
delivery_mirror_info{instance="",job="mirror-status",path="gitlab-org/omnibus-gitlab",status="ok",type="security"} 1
A possible alternative format would be one entry per project, and it has security_status
and build_status
labels.
Grafana/GitLab Dashboard format
We want one panel to show an Overview of the status. Something like this:
For any mirror that has an error (status
label. That label will contain the error string returned by the mirror status (Some refs have diverged
in the ChatOps example above) -- it will almost never be the same between two projects, and it will regularly change. It is informational metadata only and should never be used in aggregate. A table visualization might be helpful for this.
Problems
- The projects are arbitrary. We can add new ones at any point and need to account for that.
- In the Overview panel example above, the order of the
path
values change on every reload. For user friendliness, they should be in a consistent order.
Nice-to-haves
- Quick links from a mirror's status to its GitLab project page.