Make Ci::DestroyOldPipelinesWorker better churn over the pipelines

What does this MR do and why?

This code refactors the system that automatically deletes old CI/CD pipelines to improve performance and reliability. The main changes include:

New Queue Management System: Replaced direct Redis operations with a dedicated ProjectsCleanupQueue class that better manages which projects need their old pipelines cleaned up. This provides cleaner code organization and more robust queue handling.

Optimized Pipeline Deletion: Added a new "skip_cancel" option that allows the system to delete old pipelines more efficiently by skipping unnecessary cancellation steps and side effects (like email notifications) when cleaning up very old pipelines. This is only used for automated cleanup, not user-initiated deletions.

Feature Flag Protection: The improvements are controlled by a feature flag (ci_improved_destroy_old_pipelines_worker) so they can be safely rolled out and rolled back if needed.

Better Resource Management: The new system can re-queue projects that have more pipelines to delete, ensuring large cleanup jobs don't get stuck, and increases the batch size limit from 1,000 to 2,500 projects for better efficiency.

Comprehensive Testing: Added extensive test coverage for the new queue management system to ensure reliability.

The overall goal is to make the automatic cleanup of old CI/CD pipelines faster, more reliable, and less resource-intensive while maintaining safety through feature flags.

References

https://gitlab.com/gitlab-org/gitlab/-/issues/515449+s

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.