Improve long-term scalability of CancelRedundantPipelinesService
## Summary In https://gitlab.com/gitlab-org/gitlab/-/issues/414833, we made found a better index to query and shortened the cancelable pipeline window from 7 to 3 days, due to the query performance being bad enough to actually degrade GitLab SaaS. In addition to the relatively narrow fixes, there was discussion of longer-term scalability solutions: https://gitlab.com/gitlab-org/gitlab/-/issues/414833#note_1485475470 Those should be further investigated so we can continue to provide this feature with as-good or better performance as we scale. ### Improvements - Decrease the duration of database queries from `CancelRedundantPipelineService` ### Risks - We could accidentally make performance worse - We could accidentally make the feature less reliable ### Involved components - CancelRedundantPipelineService ## Proposals - https://gitlab.com/gitlab-org/gitlab/-/issues/414833#note_1485475470 - Use the current query, but execute it multiple times for different time windows (timestamp-based batching strategy) - **DONE:** Get rid of the hierarchy CTE and restrict pipelines query to cancellable statuses https://gitlab.com/gitlab-org/gitlab/-/issues/463623 - **CLOSED (won't do)**: Perform all cancellations synchronously from a single job to prevent flooding Sidekiq and database connections https://gitlab.com/gitlab-org/gitlab/-/merge_requests/173049+
epic