## Summary On GitLab.com, the cron job `schedule_merge_request_cleanup_refs_worker` is overridden to run **once a year** (Jan 1 at 01:01 UTC). The default in [`config/schedule.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/schedule.yml) is every minute (`* * * * *`), but the [k8s-workloads override](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/values.yaml.gotmpl#L128-131) sets: ```yaml # An override is set here due to https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3288 # Proposed improvement in https://gitlab.com/gitlab-org/gitlab/-/issues/296874 schedule_merge_request_cleanup_refs_worker: cron: '1 1 1 1 *' ``` The override was added on 2021-01-07 as an incident stopgap and has not changed since. The improvement it was waiting on ([gitlab-org/gitlab#296874](https://gitlab.com/gitlab-org/gitlab/-/issues/296874), a concurrency limit for `MergeRequestCleanupRefsWorker`) was delivered in July 2021 by [gitlab-org/gitlab!65647](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65647), but the stopgap was never removed. ## Current behavior `MergeRequestCleanupRefsWorker` is a `LimitedCapacity::Worker` (4 concurrent jobs). After each job it re-enqueues itself only while due, unstarted work remains, so the yearly kick starts 4 self-sustaining chains that drain the backlog and then die until the next January 1. Evidence from a Database Lab clone of gprd (snapshot around 2026-10-05): - The 2026-01-01 kick completed about 105M cleanups and stopped on 2026-04-02 05:46 UTC, when the backlog was momentarily empty. - Since then, nothing runs. About 53M due `merge_request_cleanup_schedules` rows are waiting, and none of the ~11M rows created since 2026-09-01 has been touched. - During 2025 the worker did not run at all (the 2024 chains had already died, and the 2025-01-01 kick is how they restarted). The next scheduled run is 2027-01-01 01:01 UTC. <details> <summary>How to verify</summary> - Grafana/Mimir: `sum(rate(sidekiq_jobs_completion_seconds_count{env="gprd", worker="MergeRequestCleanupRefsWorker"}[1h]))` shows a burst starting 2026-01-01 and zero after 2026-04-02 05:47 UTC. - Rails console: `Sidekiq::Cron::Job.find('schedule_merge_request_cleanup_refs_worker').cron` returns `'1 1 1 1 *'`, and `.last_enqueue_time` is 2026-01-01 01:01 UTC. - Kibana `pubsub-sidekiq-inf-gprd*` with `json.class: "MergeRequestCleanupRefsWorker"` has no recent hits. </details> ## Why this is a problem Merged and closed MR refs are meant to be cleaned up from Gitaly about 14 days after the MR closes ([gitlab-org/gitlab#336070](https://gitlab.com/gitlab-org/gitlab/-/issues/336070), fully rolled out in October 2023 on the assumption that the scheduler [runs every minute](https://gitlab.com/gitlab-org/gitlab/-/issues/336070#note_1132337284)). With the yearly override, refs of merged/closed MRs stay in Gitaly for up to a year, and the cleanup pipeline is dead for most of it. This does not affect `merge_request_cleanup_schedules` table size (completed rows are not deleted; retention is tracked separately), but it defeats the Gitaly ref-cleanup feature the table exists for. ## History 1. 2020-11: [production#3030](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3030) — `shard_catchall` exceeded its SLO because of `MergeRequestCleanupRefsWorker`. 1. 2021-01-07: [production#3288](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3288) — a spike of `DeleteRefs` calls on a Gitaly node. The override was [added the same day](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/commit/15962bbcd6a91561ef23b262dd79d5a6935f8235) as a stopgap. 1. 2021-07-14: [gitlab-org/gitlab!65647](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65647) added the 4-job capacity limit, closing [gitlab-org/gitlab#296874](https://gitlab.com/gitlab-org/gitlab/-/issues/296874) — the improvement the override comment points to. 1. 2023-10-26: the `merge_request_refs_cleanup` flag was fully enabled and [gitlab-org/gitlab#336070](https://gitlab.com/gitlab-org/gitlab/-/issues/336070) closed. The override is not mentioned anywhere in the rollout. ## Proposal Remove (or substantially relax) the override in k8s-workloads so the scheduler runs continuously, after agreeing a drain plan with Gitaly and infra: - The ~53M-row backlog would take roughly 1.5 to 2 months to drain at the observed Jan–Mar 2026 throughput (30–38M completions per month at 4 concurrent jobs). - The capacity limit that caused the 2021 incidents is long since in place, and [gitlab-org/gitlab#335958](https://gitlab.com/gitlab-org/gitlab/-/issues/335958) proposes making the job concurrency configurable if finer control is wanted for the drain. - Even an hourly schedule would prevent the worker chains from staying dead for months; every minute (the default) simply restarts chains promptly once the backlog is drained. The change itself is one line in [`values.yaml.gotmpl`](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/values.yaml.gotmpl#L128-131); this issue tracks agreeing the plan and coordinating the drain. ## Related - Capacity planning tracker: [gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com#3200](https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/work_items/3200) (growth anomaly on `merge_request_cleanup_schedules`, where this was found) - [gitlab-org/gitlab#603579](https://gitlab.com/gitlab-org/gitlab/-/issues/603579) — retry mechanism for failed cleanup schedules - [gitlab-org/gitlab#627775](https://gitlab.com/gitlab-org/gitlab/-/issues/627775) — `LimitedCapacity::Worker.perform_with_capacity` enqueues `max_running_jobs` regardless of `remaining_work_count`