CI: on capo, inconsistency between #replicas on MR pipelines compared to scheduled pipelines
I was investigating an MR CI run on capo which had failed on
Kustomization/sylva-system/monitoring InProgress Kustomization generation is 2, but latest observed generation is -1
╰┄╴HelmRelease/sylva-system/monitoring InProgress retrying after 1m0s
╰┄╴Prometheus/cattle-monitoring-system/rancher-monitoring-prometheus Ready Resource is current
╰┄╴StatefulSet/cattle-monitoring-system/prometheus-rancher-monitoring-prometheus InProgress Ready: 0/1
╰┄╴Pod/cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0 Failed Pod could not be scheduled
├┄╴┬┄┄[Conditions]
┆ ╰┄╴PodScheduled False Unschedulable 0/4 nodes are available: 4 Insufficient memory. no new claims to deallocate, preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod.(https://gitlab.com/sylva-projects/sylva-core/-/pipelines/2574425834)
The reason is simple: 4 nodes of 16GB, too short to fit everything.
What is surprising though is that nightly scheduled pipelines have 6 nodes, and do not have this problem!
The reason is the following:
- nightly scheduled pipelines have "
🛠️ ha", which hasreplicas: '{{ ternary 1 3 (eq .Values.cluster.capi_providers.infra_provider "capm3") | include "preserve-type" }}'for the MachineDeployment - in the default MR desc template the pipelines do not have
haand don't have this
We probably want to move this replica setting out of the "
And we should also probably get rid of this ha options... except capd all our interesting runs are HA anyways.