Improve observability of project mirroring

As it is now, we are mostly blind to the state of project mirroring. One of the consequences of that is that we are not alerting on project mirroring delay.

Current state

The state of mirroring is stored in project_mirror_data table in Postgres. gitlab_exporter is surfacing this information using a number of SQL queries: https://gitlab.com/gitlab-org/gitlab-exporter/-/blob/master/lib/gitlab_exporter/database/row_count.rb#L47 . The metrics from gitlab_exporter are visualized on this dashboard: https://dashboards.gitlab.net/d/_MKRXrSmk/pull-mirrors?orgId=1&from=now-3h&to=now

Idea

One way to improve this situation is to have the sidekiq workers report the relevant metrics since they have all the information necessary to calculate them. This would also keep the instrumentation next to the implementation.

Related incidents

• multiple reports from users
• gitlab-com/gl-infra/production#7420 (closed)
• gitlab-com/gl-infra/production#7416 (closed)
• gitlab-com/gl-infra/production#7223 (closed)
• gitlab-com/gl-infra/production#7221 (closed)
• gitlab-com/gl-infra/production#6827 (closed)
• pull mirroring design discussion: #12758 (closed)

added infradev severity3 labels

added priority3 label

added Category:Source Code Management label

marked this issue as related to gitlab-com/gl-infra/production#7420 (closed)

marked this issue as related to gitlab-com/gl-infra/production#7416 (closed)

marked this issue as related to gitlab-com/gl-infra/production#7223 (closed)

marked this issue as related to gitlab-com/gl-infra/production#7221 (closed)

marked this issue as related to gitlab-com/gl-infra/production#6827 (closed)

mentioned in issue gitlab-com/gl-infra/production#7420 (closed)

cc @tlinz @oregand

@mksionek

Thank you for the ping here!

@tlinz

As this is a infradev item, I would suggest we consider scheduling this for %15.3, what are your thoughts?

What would be the weight estimate for this one @oregand / @vyaklushin ?

@tlinz I would say backend-weight2 . I believe we should already have the metric instrumentation in place. We need to identify necessary metrics and report them.

@vyaklushin as you so explicitly write "backend-weight" I wonder if other teams also have work in relation to this issue that we need to consider in the planning?

@tlinz

IMO I would estimate this at a base weight 2 - 3. I'm not sure if we need input from other teams since the sidekiq workers all the information necessary. What are your thoughts @vyaklushin ?

as you so explicitly write "backend-weight" I wonder if other teams also have work in relation to this issue

You're right, we also need to update Grafana dashboards with new metrics. I think I have no permission to modify them.

Other than that, the rest we can do ourselves.

/cc @tlinz, @oregand

@vyaklushin

AFAIK, we can touch base with @reprazent for the Grafana dashboard updates.

/cc @reprazent

O.k. that seems like something you can ask for help as you work on the issue. I will now put weight to 3.

@oregand can you confirm my assumption that this is typemaintenance ?

@tlinz

Correct! It's typemaintenance

@vyaklushin I was thinking that we should define SLIs for the actual expected time a mirror should run (project_mirror_data.next_execution_timestamp) versus when it actually starts in the RepositoryUpdateMirrorWorker. Then we could define what is an acceptable delay (1 minute?).

If we pour that into an application SLI that we add to the Sidekiq service, then we'll have actual alerts for when mirrors start too slow, or when mirrors stop executing. Meaning we'd know before we get reports from users. It would then also count towards the error budget for groupsource code.

While we're on that, we could potentially update the dashboard linked in the description. As far as I can tell it needs

1. Updated queries in gitlab-exporter based on the outcome of #216783 (closed)
2. Fix the queries for job operation rates (perhaps these can be replaced with the new SLI we're adding).

I setting this a Deliverable for %15.3 and assigning this to @vyaklushin.

mentioned in issue #12758 (closed)

added backend devopscreate groupsource code sectiondev typemaintenance labels

set weight to 3

All infradev issues need to have a proper severity label, priority label, and a milestone set. Please add those to this issue. For more details, see the handbook.

added automation:infradev-missing-labels label

mentioned in epic &6592 (closed)

changed milestone to %15.3

added Deliverable label

assigned to @vyaklushin

mentioned in issue create-stage#13025 (closed)

removed automation:infradev-missing-labels label

added workflowin dev label

mentioned in commit c41b40b3

mentioned in merge request !93018 (merged)

Update

I've discovered that we already collecting metrics that calculate the difference between:

1. scheduled and started statuses (waiting delay) - gitlab_repository_mirror_waiting_duration_seconds_bucket
2. started and finished statuses (update duration) - gitlab_repository_mirror_update_duration_seconds_bucket

This data should allow to display and alert us when waiting delay / update duration goes over the limit. The current bucket values are suboptimal. I've opened a merge request to adjust them - !93018 (merged)

mentioned in commit 2902a8cc

I've added new metrics to the Pull Mirrors dashboard.

Hi @mwasilewski-gitlab! Can you please review new metrics graphs?

as per the conversion on Zoom, this dashboard is not version controlled so there's no corresponding MR, but we discussed the changes to the dashboard

Thanks for the confirmation!

added workflowin review label and removed workflowin dev label

mentioned in commit d6e0a7de

mentioned in merge request !93290 (merged)

closed

mentioned in issue create-stage#13038 (closed)

mentioned in issue gitlab-com/gl-infra/production#7223 (closed)

added to epic &8225

mentioned in issue #22485 (closed)

mentioned in issue #387170

added mirroring label