[FF] retry_failed_keep_around_ref_writes -- retry keep-around ref writes that fail

Summary

Roll out the fix currently behind the retry_failed_keep_around_ref_writes feature flag, introduced in !248909 (merged).

Gitlab::Git::KeepAround tracked the Gitaly error and moved on when write_ref failed, so MergeRequests::KeepAroundRefsWorker reported success and its retry: 20 never engaged, leaving the merge commit unprotected from git gc. The flag turns a failed write into a reported failure that the worker retries, and swaps the commit_by / kept_around? guard ordering so an unreachable Gitaly is recorded rather than silently skipped.

  • DRI: @marc_shaw
  • Team Slack channel: #<slack-channel-of-dri-team>

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

The flag is an actor-based project flag, and it changes behaviour in two places: MergeRequests::KeepAroundRefsWorker (which starts raising) and the inline Repository#keep_around callers Ci::Pipeline, DiffPositionableNote and DraftNotes::PublishService (which get the swapped guard ordering). The inline callers run in a web request — Ci::Pipeline#keep_around_commits is an after_commit on: :create — so a raise escaping KeepAround#execute would abort pipeline creation.

Blast radius is every merge request save and every pipeline creation for an enabled project.

Watch while the flag is on, before removing it:

  • MergeRequests::KeepAroundRefsWorker failure and retry rate — a small non-zero rate is the flag working; a sustained one means the retries are not converging.
  • Jobs landing in the Sidekiq dead set after 20 attempts. This is the permanent-failure case (corrupt ref file, repository genuinely gone) and should stay near zero. There is no sidekiq_retries_exhausted handler, and KeepAroundRefsError inherits Gitlab::SidekiqMiddleware::RetryError, so these are deliberately not in Sentry — the dead set is the signal.
  • Divergence between gitlab_keeparound_refs_requested_total and gitlab_keeparound_refs_created_total. The guard swap must not change either counter's meaning.
  • No new exceptions out of Ci::Pipeline creation or draft-note publishing. Gitlab::Git::Repository::NoRepository and Gitlab::Git::CommandError are rescued inside KeepAround#execute; Gitlab::Git::ResourceExhaustedError (Gitaly overload, circuit breaker open) is not, and escapes today on both flag states — confirm it stays at its current baseline rather than rising.
  • Contention on the keep-around write lease (An identical job holds the keep-around write lease in the worker logs). Expected to be rare; a rise means retries are overlapping with freshly enqueued jobs more than anticipated.

Rollout

Run all production /chatops in #production and cross-post the results to the team channel. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set retry_failed_keep_around_ref_writes 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set retry_failed_keep_around_ref_writes true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set retry_failed_keep_around_ref_writes <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss retry_failed_keep_around_ref_writes true
/chatops gitlab run feature set --group=gitlab-org,gitlab-com retry_failed_keep_around_ref_writes true
/chatops gitlab run feature set --user=marc_shaw retry_failed_keep_around_ref_writes true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Removing the flag also removes the check_ref_first branch in Gitlab::Git::KeepAround#execute, leaving the ref check unconditionally first.

/chatops gitlab run release check <merge-request-url> <milestone>
/chatops gitlab run feature delete retry_failed_keep_around_ref_writes --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set retry_failed_keep_around_ref_writes false                                         # production
/chatops gitlab run feature set retry_failed_keep_around_ref_writes false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete retry_failed_keep_around_ref_writes --dev --pre --staging --staging-ref --production  # remove entirely