Skip to content

Pull mirroring process is stuck for some projects

Problem

The pull mirroring process stopped working for https://gitlab.com/gitlab-community/gitlab.

image__3_

The mirroring process starts, but it never ends

Screenshot_2023-06-02_at_17.00.04

source

The last successful attempt was on 2023-05-29

Screenshot_2023-06-02_at_17.01.13

source

I see a number jobs with started state that doesn't go to 0.

Screenshot_2023-06-08_at_14.26.04

source

Investigation

I suspected a ListLFSPointers command as a reason for this problem first. Kibana link

But successful attempts from the past also have ListLFSPointers with cancelled status.

The job RepositoryUpdateMirrorWorker gets stuck. It's removed after 24 hours by StuckImportJob task. Kibana link.

In last 7 days, I estimate 1240 jobs that got stuck.

Edited by Vasilii Iakliushin