Sub-Batching Mechanism for Registry BBM

Summary

Add sub-batching support to the Registry's Batched Background Migration (BBM) framework so that each job processes its assigned row range through multiple smaller queries rather than a single large one.

Identified as a must-have during the database review of !2778 (comment 3107060883) by @krasio.

Relates to &19732 Relates to #1708

Problem

The BBM framework currently executes one query per job covering the entire batch. With 100K-row batches on the BIGINT migration (1.35B rows across 64 partitions), individual UPDATEs are not efficient enough and if not effectively calibrated could overwhelm the database.

Shrinking the batch size is a poor workaround: halving to 50K did not provide a more reasonable performance optimization while doubling framework overhead (lock acquisitions, job lookups, status updates). Sub-batching decouples query size from batch management overhead.

Sub-batching was explicitly deferred in the original BBM framework (!1614 (merged)) but should now needed as migration workloads grow.

Expected Outcome

  • Jobs split their row range into configurable sub-batches, keeping individual queries well within statement timeouts
  • Reduced row lock duration on high-traffic partitioned tables
  • Backward-compatible with existing work functions and migration definitions
  • Aligned with the GitLab Rails batched migration framework, which uses sub-batching as a core mechanism

References

  • !2778 - Database review where this was flagged
  • !1614 (merged) - Original BBM framework (sub-batching deferred)
  • &19732 - BIGINT media types column swap epic
Edited by SAhmed