Configure capo runs to use the new runner tags for tests and cleanup

What does this MR do and why?

Now that #1879 (closed) has been implemented, we have two kind of runners for CAPO.

Executors

Name	Description	Projects	Tags	Concurrency
main	Deploy	sylva-projects/sylva-core	capo-infra-rocket capo-infra-rocket::main capo-infra-rocket::deploy	4
misc	Test and cleanup	sylva-projects/sylva-core	capo-infra-rocket::misc capo-infra-rocket::cleanup capo-infra-rocket::tests	4

So far, we used the max concurrency of our runners (8 slots) to limit how many clusters could be deployed at the same time.

This had the following issues:

• cleanup jobs would accumulate and multiple would run at the same time.
• test jobs could prevent other deployment to be stuck waiting for a runner slot.
• the pipeline execution time grew linearly with the number of tests jobs, with little parallelization.

With the new implementation:

• Cleanup and tests are now run on the capo-infra-rocket::tests / capo-infra-rocket::cleanup
  • for now, we're using the same executor for both. By setting different tags now we won't have to change the CI code if we decide to separate tests and cleanup.
• Added a job to keep the runner busy while tests are running.
  • this will ensure we are not deploying more than 8 clusters at the same time.

The new tag names are not set in stone, I tried to come up with something that made sense but i'm open to suggestions 😄

Example of the execution of the keep runner busy job: https://gitlab.com/sylva-projects/sylva-core/-/jobs/11261778150

Test coverage

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	capd, capo, capm3
🚀	Bootstrap Provider	kubeadm (alias kadm), rke2, okd, ck8s
🐧	Node OS	ubuntu, suse, na, leapmicro
🛠️	Deployment Options	light-deploy, dev-sources, ha, misc, maxsurge-0, logging, no-logging
🎬	Pipeline Scenarios	Available scenario list and description

• 🎬 preview ☁️ capd 🚀 kadm 🐧 ubuntu

• 🎬 preview ☁️ capo 🚀 rke2 🐧 suse

• 🎬 preview ☁️ capm3 🚀 rke2 🐧 ubuntu

• ☁️ capd 🚀 kadm 🛠️ light-deploy 🐧 ubuntu

• ☁️ capd 🚀 rke2 🛠️ light-deploy 🐧 suse

• ☁️ capo 🚀 rke2 🐧 suse

• ☁️ capo 🚀 rke2 🐧 leapmicro

• ☁️ capo 🚀 kadm 🐧 ubuntu

• ☁️ capo 🚀 rke2 🎬 rolling-update 🛠️ ha 🐧 ubuntu

• ☁️ capo 🚀 kadm 🎬 wkld-k8s-upgrade 🐧 ubuntu

• ☁️ capo 🚀 rke2 🎬 rolling-update-no-wkld 🛠️ ha 🐧 suse

• ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.4.x 🛠️ ha 🐧 ubuntu

• ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.4.x 🛠️ ha,misc 🐧 ubuntu

• ☁️ capo 🚀 rke2 🛠️ ha,misc 🐧 ubuntu

• ☁️ capm3 🚀 rke2 🐧 suse

• ☁️ capm3 🚀 kadm 🐧 ubuntu

• ☁️ capm3 🚀 ck8s 🐧 ubuntu

• ☁️ capm3 🚀 kadm 🎬 rolling-update-no-wkld 🛠️ ha,misc 🐧 ubuntu

• ☁️ capm3 🚀 rke2 🎬 wkld-k8s-upgrade 🛠️ ha 🐧 suse

• ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 ubuntu

• ☁️ capm3 🚀 rke2 🎬 sylva-upgrade-from-1.4.x 🛠️ ha 🐧 suse

• ☁️ capm3 🚀 rke2 🛠️ misc,ha 🐧 suse

• ☁️ capm3 🚀 rke2 🎬 sylva-upgrade-from-1.4.x 🛠️ ha,misc 🐧 suse

• ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 suse

• ☁️ capm3 🚀 ck8s 🎬 rolling-update 🛠️ ha 🐧 ubuntu

• ☁️ capm3 🚀 rke2|okd 🎬 no-update 🐧 ubuntu|na

Global config for deployment pipelines

• autorun pipelines
• allow failure on pipelines
• record sylvactl events

Notes:

• Enabling autorun will make deployment pipelines to be run automatically without human interaction
• Disabling allow failure will make deployment pipelines mandatory for pipeline success.
• if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.