Use cached node ips on upgrade when API unavailable (!9351) · Merge requests · Sylva-projects / sylva-core

What does this MR do and why?

We've seen in a lot of failed pipelines that when the api connectivity is lost, the node logs dump is missing because the retrieval of the machines IPs is based on api queries.

This MR caches the machine names + ips when the upgrade job starts in order to use them at the end if the api connectivity is lost.

notes:

ensure-sylva-toolbox is called at the beginning of the upgrade job to have access to the binaries
this will work only for capm3 + upgrade pipelines, because in the case of fresh deployments the IPs are not known, and in the capo case they are changed during the rolling upgrade

I didn't manage to fully test this in CI since I need to reproduce that kind of failure, which doesn't always happen, but it should not break anything and we can see that at least the retrieval works ok in https://gitlab.com/sylva-projects/sylva-core/-/jobs/16835844350/viewer

📥 Installing sylva-toolbox binaries
v1.10.7: Pulling from sylva-projects/sylva-elements/container-images/sylva-toolbox
Digest: sha256:ea563b87fd9479bff815d42058f46850cc2509a41c637efd03708d07e24b6ab4
Status: Image is up to date for registry.gitlab.com/sylva-projects/sylva-elements/container-images/sylva-toolbox:v1.10.7
registry.gitlab.com/sylva-projects/sylva-elements/container-images/sylva-toolbox:v1.10.7
tar: removing leading '/' from member names
$ . $CI_PROJECT_DIR/bin/env
$ .gitlab/ci/scripts/dump_machine_logs.sh management --save-machine-ips || true
# Saving management nodes IPs to the cache
>> machine_ip=192.168.100.21
>> machine_ip=192.168.100.23
>> machine_ip=192.168.100.20
>> machine_ip=192.168.100.22
mgmt-2897128803-rke2-capm3-virt-control-plane-p6smj 192.168.100.21
mgmt-2897128803-rke2-capm3-virt-control-plane-tnwb5 192.168.100.23
mgmt-2897128803-rke2-capm3-virt-control-plane-z4xhl 192.168.100.20
mgmt-2897128803-rke2-capm3-virt-md0-7brgd-5tdts 192.168.100.22
$ ./apply.sh ${ENV_PATH}

later edit: managed to reproduce the failure in https://gitlab.com/sylva-projects/sylva-core/-/jobs/16835844772

the api connectivity was lost, but the logs were retrieved OK

cc @feleouet

Test coverage

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	`capd`, `capo`, `capm3`
🚀	Bootstrap Provider	`kubeadm` (alias `kadm`), `rke2`, `okd`, `ck8s`
🐧	Node OS	`ubuntu`, `suse`, `na`, `leapmicro`
🛠️	Deployment Options	Deployment option list and description
🎬	Pipeline Scenarios	Available scenario list and description
🟢	Enabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🔴	Disabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🏗️	Target platform	Can be used to select specific deployment environment Available platform list and description
⚡	Pipeline control	`autorun`, `manual` or `blocking`. Can be used to override global config and start a deployment pipeline the required way

Global config for deployment pipelines

autorun pipelines
allow failure on pipelines
record sylvactl events

Notes:

Enabling autorun will make deployment pipelines to be run automatically without human interaction
Disabling allow failure will make deployment pipelines mandatory for pipeline success.
if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited Sep 30, 2026 by Cristian Manda

Use cached node ips on upgrade when API unavailable

What does this MR do and why?

Related reference(s)

Test coverage

CI configuration

Global config for deployment pipelines

Merge request reports