Idea to consider: Implement Redis queue for shared runners only
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Description
We do have 5 shared runners managers configured to pick builds without tags. These runners process around 400-500k builds each day.
Proposal
Implement Redis queuing for untagged builds running on shared runners only. This will allow us to iteratively move towards the next version of CI/CD queuing while giving us time to build the proper solution.
Data
Processing of builds without tags on 25.02.2021:
| Runner | All builds on 25.02 | Builds without tags | Percentage |
|---|---|---|---|
| 2568000 | 84534 | 80580 | 95% |
| 380987 | 85546 | 81559 | 95% |
| 380986 | 89636 | 85403 | 95% |
| 44949 | 86055 | 79745 | 92% |
| 44028 | 87863 | 81303 | 92% |
See more data in #322972 (comment 518063713)
Caveats
- Builds ordering - we can only guarantee an order per runner type (like today)
- Fair scheduling - separate priority queues possible
/cc @ayufan @darbyfrey
Edited by 🤖 GitLab Bot 🤖