Implement Beaker recipe queueing
From Ales: a Beaker user should stop themselves from submitting more recipes if:
- there's > 3000 active recipes overall in Beaker
- when you have > 100 recipes in the New+Processed states
We should implement some kind of this blocker, potentially using Ales' semaphore for inspo: https://gitlab.cee.redhat.com/azelinka/udelatory/-/blob/master/beaker-stats/beaker-semaphore
The initial idea would be to add the implementation into cki-tools and then block in the pipeline before calling upt to provision resources.
Some potential issues:
-
not having the implementation in the provisioner itself means that the moment the recipe count goes down, the condition would pass and the provisioner would potentially submit more recipes than allowed. E.g. the current new recipe count is 99 and we pass and upt goes to submit 6 jobs, going above the limit
-
having the implementation in the provisioner would waste partial reruns -- e.g. 3/6 of the jobs manage to run but because jobs from different pipelines would get submitted, we don't get to the ones in the pipeline, causing half runs to time out, potentially. This is both a waste of resources (we have to rerun that full run) and time and lower test coverage for the MR if things don't get rerun
-
race conditions with multiple pipelines running and checking the status -- we have 5 pipelines with 4 gitlab test jobs each, new job count gets to 99, the check in all 20 gitlab jobs passes and they all submit the jobs, going rapidly over the limit