Group deletion can get stuck indefinitely when a descendant namespace is in a transient state (e.g. transfer_in_progress)

Summary

Groups::DestroyService#unsafe_execute recursively destroys child groups synchronously. If a descendant namespace destroy raises an exception while that namespace is in a transient state that has no valid transition out via reschedule_deletion (e.g. transfer_in_progress), the group deletion can get stuck in a "scheduled for deletion" loop indefinitely, with no clear error surfaced to the user or support.

Root cause

In app/services/groups/destroy_service.rb:

def unsafe_execute
  ...
  group.children.each do |group|
    # This needs to be synchronous since the namespace gets destroyed below
    DestroyService.new(group, current_user).unsafe_execute
  end
  ...
rescue Exception => e
  ...
  reschedule_deletion
  ...
  raise e
end

def reschedule_deletion
  Group.transaction do
    group.reschedule_deletion!(transition_user: current_user)
  end
end

The state machine in app/models/concerns/namespaces/stateful.rb only defines a reschedule_deletion transition from deletion_in_progress. There is no transition from transfer_in_progress.

Sequence of events when a descendant is stuck in transfer_in_progress:

  1. The descendant's unsafe_execute raises an exception (destroy fails while namespace is in transfer_in_progress).
  2. Its own rescue block calls reschedule_deletion, but the descendant's state is transfer_in_progress, so reschedule_deletion! itself raises a new StateMachines::InvalidTransition. Because this happens inside the rescue clause, it replaces the original exception (Ruby exception chaining) and the raise e line is never reached.
  3. This new exception propagates up to the parent's unsafe_execute (since the child destroy call is synchronous and un-rescued at that call site).
  4. The parent group is still in deletion_in_progress, so its own reschedule_deletion! succeeds, silently rescheduling deletion for a future date.
  5. The cycle repeats on every scheduled deletion attempt until whatever clears the descendant's transfer_in_progress state (e.g. StuckTransfersCancelCronWorker) runs, after which a retry succeeds.

Impact

  • Customers see "This group and all its data will be permanently deleted on <date>" indefinitely, with the deletion never actually completing until manually retried after the stuck state clears.
  • The real error is masked/lost (exception chaining swaps it for the InvalidTransition from the rescue handler itself).
  • This has now affected the same customer twice, and is the same outward symptom (though a different underlying trigger) as #2388 (closed) (ResourceStateEvent deletion timeout in GroupDestroyWorker), suggesting the rescue/reschedule pattern in GroupDestroyWorker/Groups::DestroyService is fragile to any exception raised deep in the descendant tree.

Proposed fix

Add a mechanism to bypass/skip transient namespace state-machine checks when hard-deleting descendants as part of a group deletion — the state of a descendant should not block deletion of the underlying records when the whole tree is being destroyed anyway. Concretely:

  • Either widen the reschedule_deletion transition to accept transfer_in_progress (and other transient states) as a valid from state, or
  • Have Groups::DestroyService clear/ignore transient namespace states on descendants before/while destroying them, instead of relying on the state machine's reschedule_deletion! succeeding on every namespace touched during the failed destroy.

Additionally, consider not swallowing/replacing the original exception in the rescue block so on-call/support has the real error to act on.

Related

  • Request For Help: gitlab-com/request-for-help#5099
  • Related prior RFH: #2388 (closed) (closed, different root cause, same symptom)