Improve CI for external repo with configurable maximum mirroring frequency on self-hosted
Release notes
Problem to solve
GitLab CI for external repositories is currently reliant on the repository mirroring capability. In 13.0, #118753 (closed) added an application limit on pull mirroring that enforced a 5 minute delay between mirroring/updates. This hard-coded maximum mirror frequency negatively impacted CI pipelines for external repos, making this CI feature effectively un-usable for teams pushing more often than once every 5 minutes. This problem applies to both self-hosted and GitLab.com users.
Intended users
User experience goal
Allow the maximum mirror frequency to be a configurable application limit so that:
- On self-hosted instances, admins can adjust or even disable the 5 minute delay
Proposal
- Make the maximum mirror frequency a configurable application limit.
- Make the limit of "create the pipeline for the first 4 refs in a single push" a configurable application limit that can be customized on self-hosted.
NOTE: Long term solution to make CI for external repositories more performant will be addressed in #235916 (closed).
Further details
While the limitation implemented in #118753 (closed) was intended to address performance issue with excessive mirroring and to prevent problems at GitLab.com at scale, this had unintended and significant negative impact on the CI for External Repository feature.
Example: If Developer A pushed a change in minute 1, all pushes from any developer are ignored from a CI perspective until minute 6. Then if at minute 6 Dev B pushes, all subsequent changes are ignored. And given that all queued mirroring attempts are purged, any push in between minutes 1 and 6 will need to be manually run by specifying the commit in the New Pipeline page - and only after the push at minute 6 occurs.
Permissions and Security
Documentation
Availability & Testing
- Unit test changes - Yes, please have test for all new/updated implementations
- End-to-end test change - Yes, with this new E2E test scenario gitlab-org/quality/testcases#144 (closed) (WIP MR that can be completed once this feature is implemented !42289 (closed))