CI: on capo, inconsistency between #replicas on MR pipelines compared to scheduled pipelines

I was investigating an MR CI run on capo which had failed on

Kustomization/sylva-system/monitoring                                                  InProgress                  Kustomization generation is 2, but latest observed generation is -1
╰┄╴HelmRelease/sylva-system/monitoring                                                 InProgress                  retrying after 1m0s
   ╰┄╴Prometheus/cattle-monitoring-system/rancher-monitoring-prometheus                Ready                       Resource is current
      ╰┄╴StatefulSet/cattle-monitoring-system/prometheus-rancher-monitoring-prometheus InProgress                  Ready: 0/1
         ╰┄╴Pod/cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0    Failed                      Pod could not be scheduled
            ├┄╴┬┄┄[Conditions]
            ┆  ╰┄╴PodScheduled                                                         False      Unschedulable    0/4 nodes are available: 4 Insufficient memory. no new claims to deallocate, preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod.

(https://gitlab.com/sylva-projects/sylva-core/-/pipelines/2574425834)

The reason is simple: 4 nodes of 16GB, too short to fit everything.

What is surprising though is that nightly scheduled pipelines have 6 nodes, and do not have this problem!

The reason is the following:

  • nightly scheduled pipelines have "🛠️ ha", which has replicas: '{{ ternary 1 3 (eq .Values.cluster.capi_providers.infra_provider "capm3") | include "preserve-type" }}' for the MachineDeployment
  • in the default MR desc template the pipelines do not have ha and don't have this

We probably want to move this replica setting out of the "🛠️ ha" options, so that it will apply everywhere.

And we should also probably get rid of this ha options... except capd all our interesting runs are HA anyways.