Handle infinite loop in CI (using Redis)

What does this MR do and why?

This MR introduces pipeline chain tracking to prevent infinite recursion when pipelines push code using CI_JOB_TOKEN. The core issue occurs when a CI job pushes code that triggers a new pipeline, which also pushes code, creating an endless loop that can exhaust CI minutes and pose infrastructure risks. This implementation limits pipeline chains to a maximum depth of 5 levels while allowing legitimate multi-stage workflows to continue functioning properly.

The solution works by tracking the "chain depth" of commits in Redis - when a CI job pushes using CI_JOB_TOKEN, the system looks up the chain depth of the previous commit, increments it by one for the new commit, and rejects the push if it exceeds the maximum limit. This approach specifically targets CI_JOB_TOKEN pushes while leaving regular user pushes unaffected, and includes special handling for scheduled pipelines which are exempt from recursion checks.

This is not a security or anti-abuse feature, as infinite loops are already possible using Personal Access Tokens (PAT). Instead, this is primarily about reducing the chance of users accidentally "shooting themselves in the foot" by misconfiguring their CI pipelines. The implementation includes comprehensive integration across the Git HTTP workflow with proper error handling and Redis-based storage, allowing common automation patterns like version bumps and code formatting to work within reasonable limits while preventing accidental infinite loops.

This is a reimplementation of https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18699 using Redis. This is done to

• Make potential future downstream migration easier
• Reduce database changes
• Remove the need of manual data cleanup

Context

We added a feature to allow CI JOB TOKEN to authorize push to code to the same repository. That feature is still under the feature flag due to concerns over potential CI misconfiguration and end

References

How to set up and validate locally

1. compile workhorse

   cd workhorse
   make

2. In rails console enable the feature flag

   Feature.enable(:allow_push_repository_for_job_token)

3. Select a project and goto Setting -> CI/CD settings -> Job token permissions

4. Check Allow Git push requests to the repository option

5. Add following .gitlab-ci.yml

stages:
 - update
update-readme:
 stage: update
 image: alpine:latest
 before_script:
 - apk add --no-cache git
 - git config --global user.email "ci@example.com"
 - git config --global user.name "GitLab CI"
 - git remote set-url origin "https://gitlab-ci-token:${CI_JOB_TOKEN}@gdk.test:3000/${CI_PROJECT_PATH}.git"
 script: |
   [ "$EXIT_LOOP_NOW" = "1" ] && exit 1 || true
   echo -e "\n## Update $(date)" >> README.md
   git add README.md
   git commit -m "Update README.md with date"
   git config --global http.sslVerify false
   git push origin HEAD:$CI_COMMIT_REF_NAME -v

1. Make sure that the pipeline doesn't go to infinite loop

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Relates to #475705 (closed)