CI failures on openshift-assisted-installer unit (breaking "misc" pipelines)
We've recently seen CI failures on openshift-assisted-installer unit: We had an occurence on nightly runs for an upgrade from sylva 1.5 to 1.6: https://gitlab.com/sylva-projects/sylva-core/-/jobs/14557003743 And we had this on an MR targetting main (details below) https://gitlab.com/sylva-projects/sylva-core/-/jobs/14576340576 ``` unit Kustomization/openshift-assisted-installer did not became ready after 5m0s Details on Kustomization/openshift-assisted-installer and related resources: IDENTIFIER STATUS REASON MESSAGE Kustomization/sylva-system/openshift-assisted-installer InProgress Kustomization generation is 3, but latest observed generation is 2 ╰┄╴┬┄┄[Conditions] ├┄╴Reconciling True Progressing Running health checks for revision 0.0.0-git-a815b13b@sha256:3e33707bcfa3598ca97e22fefe9c299c014350dc990d8658fcc2ee155d279730 with a timeout of 30s ├┄╴Ready Unknown Progressing Reconciliation in progress ╰┄╴Healthy Unknown Progressing Running health checks for revision 0.0.0-git-a815b13b@sha256:3e33707bcfa3598ca97e22fefe9c299c014350dc990d8658fcc2ee155d279730 with a timeout of 30s ``` ``` k get -n multicluster-engine agentserviceconfigs agent -o yaml | yq .status.conditions - lastTransitionTime: "2026-05-28T08:26:41Z" message: AgentServiceConfig reconcile completed without error. reason: ReconcileSucceeded status: "True" type: ReconcileCompleted - lastTransitionTime: "2026-05-28T08:29:51Z" message: Deployment assisted-service is not available reason: DeploymentFailure status: "False" type: DeploymentsHealthy ``` ``` $ k -n multicluster-engine get pods NAME READY STATUS RESTARTS AGE agentinstalladmission-6cfc9df8c4-bkbsf 1/1 Running 0 50m agentinstalladmission-6cfc9df8c4-n2dwv 1/1 Running 0 50m assisted-service-86d89c78b4-5zqtq 0/2 Running 10 50m capoa-bootstrap-controller-manager-855785c75c-jzhxp 1/1 Running 0 51m capoa-controlplane-controller-manager-569b68f8c7-q9tjp 1/1 Running 0 51m cluster-manager-784b8c5f98-7rlrc 1/1 Running 0 51m cluster-manager-784b8c5f98-dqht9 1/1 Running 0 51m cluster-manager-784b8c5f98-rq6p2 1/1 Running 0 51m infrastructure-operator-86768cdf54-6dzln 1/1 Running 0 51m multicluster-engine-operator-5776d6dc7-lltfs 1/1 Running 0 54m multicluster-engine-operator-5776d6dc7-xzmj8 1/1 Running 0 55m ``` The `assisted-service` pod containers are subject to numerous restarts. This seems to possibly be due to a postgres issue (?): ``` k -n multicluster-engine logs assisted-service-86d89c78b4-5zqtq postgres === PostgreSQL Startup Check === Data directory version: 12 Container image version: 13 Version mismatch detected - enabling pg_upgrade (hardlink mode) ========== $PGDATA upgrade: 12 -> 13 ========== Can't read /etc/scl/conf/rh-postgresql12, rh-postgresql12 is probably not installed. ===> Initialize new data directory; we will migrate to that. /usr/share/container-scripts/postgresql/common.sh: line 192: /opt/rh/rh-postgresql13/root/usr/bin/initdb: No such file or directory ``` (All these details can be explored with `$sylva_core/tools/serve-crustgather-artifact.sh -g https://gitlab.com -p 42451983 -j 14576340576`)
issue