Avoid slow queries in redundant pipelines service

What does this MR do and why?

Avoid slow queries in redundant pipelines service by iterating over the pipelines using an efficient index and avoiding recursive queries in the batching query.

The change is behind the use_offset_pagination_for_canceling_redundant_pipelines feature flag.

Note: this is a short term fix so we could re-enable the sidekiq job. Ideally, we should add a better index (which we cannot do in the short term due to the table size) that supports efficient batching.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.