IPA randomly fails network setup on libvirt-metal CI

While working on the integration of leapmicro CI on capm3-virt MR !6366, we notice an inconsistent behavior of the baremetalhosts which fails on random hosts and random restarts while provisioning the workload cluster.

For example on two jobs on the same commit:

in https://gitlab.com/sylva-projects/sylva-core/-/jobs/12656641140 last status shows cp-0 and cp-1 provisioned and cp-2 and md-0 failed inspection

NAMESPACE         NAME                                              STATE         CONSUMER                                                                                         ONLINE   ERROR   AGE
rke2-capm3-virt   wc-2253187610-rke2-capm3-virt-workload-cp-0       provisioned   wc-2253187610-rke2-capm3-virt-cp-46c4715ea0-wxcmh                                                true     2026-01-09T08:43:13Z
rke2-capm3-virt   wc-2253187610-rke2-capm3-virt-workload-cp-1       provisioned   wc-2253187610-rke2-capm3-virt-cp-46c4715ea0-tb5nj                                                true     2026-01-09T08:43:13Z
rke2-capm3-virt   wc-2253187610-rke2-capm3-virt-workload-cp-2       inspecting    redfish-virtualmedia://172.18.0.2:8012/redfish/v1/Systems/c0014001-b10b-f001-c0de-feeb1e54ee15   2026-01-09T08:43:13Z
rke2-capm3-virt   wc-2253187610-rke2-capm3-virt-workload-md-0       inspecting    redfish-virtualmedia://172.18.0.2:8013/redfish/v1/Systems/c0014001-b10b-f001-c0de-feeb1e54ee15   inspection error   2026-01-09T08:43:13Z

in https://gitlab.com/sylva-projects/sylva-core/-/jobs/12646218053 cp-0 and cp-2 are provisioned, cp-1 fails inspection while md-0 fails provisioning

``` NAMESPACE NAME STATE CONSUMER ONLINE ERROR AGE rke2-capm3-virt wc-2251612576-rke2-capm3-virt-workload-cp-0 provisioned wc-2251612576-rke2-capm3-virt-cp-56bf13289c-v8q6m true 2026-01-08T13:45:28Z rke2-capm3-virt wc-2251612576-rke2-capm3-virt-workload-cp-1 inspecting redfish-virtualmedia://172.18.0.2:8011/redfish/v1/Systems/c0014001-b10b-f001-c0de-feeb1e54ee15 2026-01-08T13:45:28Z rke2-capm3-virt wc-2251612576-rke2-capm3-virt-workload-cp-2 provisioned wc-2251612576-rke2-capm3-virt-cp-56bf13289c-8kp67 true 2026-01-08T13:45:28Z rke2-capm3-virt wc-2251612576-rke2-capm3-virt-workload-md-0 provisioning wc-2251612576-rke2-capm3-virt-md0-4hf2x-s7pfd true 2026-01-08T13:45:28Z ```

In the console logs of md-0 things get a bit intriguing:

kubectl -n sylva-system logs libvirt-metal-workload-md-0-0 -c console-log |grep suse-network-setup
2026-01-08 13:46:30 ::  [    9.892300] suse-network-setup.sh[449]: + PATH=/bin:/usr/bin:/sbin:/usr/sbin
2026-01-08 13:46:30 ::  [    9.893902] suse-network-setup.sh[449]: + NETWORK_DATA_FILE=/mnt/ipa/openstack/latest/network_data.json
2026-01-08 13:46:30 ::  [    9.896164] suse-network-setup.sh[449]: + '[' '!' -f /mnt/ipa/openstack/latest/network_data.json ']'
2026-01-08 13:46:30 ::  [    9.898116] suse-network-setup.sh[449]: + mkdir -p /tmp/nmc/desired /tmp/nmc/generated
2026-01-08 13:46:30 ::  [    9.901281] suse-network-setup.sh[449]: + cp /mnt/ipa/openstack/latest/network_data.json /tmp/nmc/desired/_all.yaml
2026-01-08 13:46:30 ::  [    9.903635] suse-network-setup.sh[449]: + nmc generate --config-dir /tmp/nmc/desired --output-dir /tmp/nmc/generated
2026-01-08 13:46:30 ::  [    9.905995] suse-network-setup.sh[452]: [2026-01-08T13:46:28Z INFO  nmc::generate_conf] Generating config from "/tmp/nmc/desired/_all.yaml"...
2026-01-08 13:46:30 ::  [    9.908686] suse-network-setup.sh[452]: [2026-01-08T13:46:28Z INFO  nmstate::nm::profile] Skipping activation of enp0s4 as its controller bond0 will automatically activate it
2026-01-08 13:46:30 ::  [    9.911841] suse-network-setup.sh[452]: [2026-01-08T13:46:28Z INFO  nmstate::nm::profile] Skipping activation of enp0s5 as its controller bond0 will automatically activate it
2026-01-08 13:46:30 ::  [    9.915045] suse-network-setup.sh[452]: [2026-01-08T13:46:28Z ERROR nmc] Generating config failed: Detected interfaces without connection files: enp0s4
2026-01-08 14:17:36 ::  [   10.301272] suse-network-setup.sh[447]: + PATH=/bin:/usr/bin:/sbin:/usr/sbin
2026-01-08 14:17:36 ::  [   10.303272] suse-network-setup.sh[447]: + NETWORK_DATA_FILE=/mnt/ipa/openstack/latest/network_data.json
2026-01-08 14:17:36 ::  [   10.305714] suse-network-setup.sh[447]: + '[' '!' -f /mnt/ipa/openstack/latest/network_data.json ']'
2026-01-08 14:17:36 ::  [   10.308062] suse-network-setup.sh[447]: + mkdir -p /tmp/nmc/desired /tmp/nmc/generated
2026-01-08 14:17:36 ::  [   10.310092] suse-network-setup.sh[447]: + cp /mnt/ipa/openstack/latest/network_data.json /tmp/nmc/desired/_all.yaml
2026-01-08 14:17:36 ::  [   10.312734] suse-network-setup.sh[447]: + nmc generate --config-dir /tmp/nmc/desired --output-dir /tmp/nmc/generated
2026-01-08 14:17:36 ::  [   10.315538] suse-network-setup.sh[450]: [2026-01-08T14:17:36Z INFO  nmc::generate_conf] Generating config from "/tmp/nmc/desired/_all.yaml"...
2026-01-08 14:17:36 ::  [   10.318714] suse-network-setup.sh[450]: [2026-01-08T14:17:36Z INFO  nmstate::nm::profile] Skipping activation of enp0s4 as its controller bond0 will automatically activate it
2026-01-08 14:17:36 ::  [   10.322543] suse-network-setup.sh[450]: [2026-01-08T14:17:36Z INFO  nmstate::nm::profile] Skipping activation of enp0s5 as its controller bond0 will automatically activate it
2026-01-08 14:17:36 ::  [   10.326429] suse-network-setup.sh[450]: [2026-01-08T14:17:36Z INFO  nmc] Successfully generated and stored network config
2026-01-08 14:17:36 ::  [   10.329152] suse-network-setup.sh[447]: + nmc apply --config-dir /tmp/nmc/generated
2026-01-08 14:17:36 ::  [   10.331165] suse-network-setup.sh[451]: [2026-01-08T14:17:36Z INFO  nmc::apply_conf] Applying unified config...
2026-01-08 14:17:36 ::  [   10.333407] suse-network-setup.sh[451]: [2026-01-08T14:17:36Z INFO  nmc::apply_conf] Copying file... "/tmp/nmc/generated/_all/enp0s5.nmconnection"
2026-01-08 14:17:36 ::  [   10.336177] suse-network-setup.sh[451]: [2026-01-08T14:17:36Z INFO  nmc::apply_conf] Copying file... "/tmp/nmc/generated/_all/enp0s4.nmconnection"
2026-01-08 14:17:36 ::  [   10.338871] suse-network-setup.sh[451]: [2026-01-08T14:17:36Z INFO  nmc::apply_conf] Copying file... "/tmp/nmc/generated/_all/bond0.100.nmconnection"
2026-01-08 14:17:36 ::  [   10.341609] suse-network-setup.sh[451]: [2026-01-08T14:17:36Z INFO  nmc::apply_conf] Copying file... "/tmp/nmc/generated/_all/bond0.nmconnection"
2026-01-08 14:17:36 ::  [   10.344303] suse-network-setup.sh[451]: [2026-01-08T14:17:36Z INFO  nmc] Successfully applied config
2026-01-08 14:20:27 ::  [    9.870930] suse-network-setup.sh[450]: + PATH=/bin:/usr/bin:/sbin:/usr/sbin
2026-01-08 14:20:27 ::  [    9.872682] suse-network-setup.sh[450]: + NETWORK_DATA_FILE=/mnt/ipa/openstack/latest/network_data.json
2026-01-08 14:20:27 ::  [    9.874905] suse-network-setup.sh[450]: + '[' '!' -f /mnt/ipa/openstack/latest/network_data.json ']'
2026-01-08 14:20:27 ::  [    9.877113] suse-network-setup.sh[450]: + mkdir -p /tmp/nmc/desired /tmp/nmc/generated
2026-01-08 14:20:27 ::  [    9.879022] suse-network-setup.sh[450]: + cp /mnt/ipa/openstack/latest/network_data.json /tmp/nmc/desired/_all.yaml
2026-01-08 14:20:27 ::  [    9.881511] suse-network-setup.sh[450]: + nmc generate --config-dir /tmp/nmc/desired --output-dir /tmp/nmc/generated
2026-01-08 14:20:27 ::  [    9.884032] suse-network-setup.sh[453]: [2026-01-08T14:20:26Z INFO  nmc::generate_conf] Generating config from "/tmp/nmc/desired/_all.yaml"...
2026-01-08 14:20:27 ::  [    9.887077] suse-network-setup.sh[453]: [2026-01-08T14:20:26Z INFO  nmstate::nm::profile] Skipping activation of enp0s4 as its controller bond0 will automatically activate it
2026-01-08 14:20:27 ::  [    9.890734] suse-network-setup.sh[453]: [2026-01-08T14:20:26Z INFO  nmstate::nm::profile] Skipping activation of enp0s5 as its controller bond0 will automatically activate it
2026-01-08 14:20:27 ::  [    9.894338] suse-network-setup.sh[453]: [2026-01-08T14:20:26Z ERROR nmc] Generating config failed: Detected interfaces without connection files: enp0s4
2026-01-08 14:51:21 ::  [    9.798740] suse-network-setup.sh[447]: + PATH=/bin:/usr/bin:/sbin:/usr/sbin
2026-01-08 14:51:21 ::  [    9.800329] suse-network-setup.sh[447]: + NETWORK_DATA_FILE=/mnt/ipa/openstack/latest/network_data.json
2026-01-08 14:51:21 ::  [    9.802382] suse-network-setup.sh[447]: + '[' '!' -f /mnt/ipa/openstack/latest/network_data.json ']'
2026-01-08 14:51:21 ::  [    9.804329] suse-network-setup.sh[447]: + mkdir -p /tmp/nmc/desired /tmp/nmc/generated
2026-01-08 14:51:21 ::  [    9.806838] suse-network-setup.sh[447]: + cp /mnt/ipa/openstack/latest/network_data.json /tmp/nmc/desired/_all.yaml
2026-01-08 14:51:21 ::  [    9.809050] suse-network-setup.sh[447]: + nmc generate --config-dir /tmp/nmc/desired --output-dir /tmp/nmc/generated
2026-01-08 14:51:21 ::  [    9.811372] suse-network-setup.sh[450]: [2026-01-08T14:51:20Z INFO  nmc::generate_conf] Generating config from "/tmp/nmc/desired/_all.yaml"...
2026-01-08 14:51:21 ::  [    9.814096] suse-network-setup.sh[450]: [2026-01-08T14:51:20Z INFO  nmstate::nm::profile] Skipping activation of enp0s4 as its controller bond0 will automatically activate it
2026-01-08 14:51:21 ::  [    9.817365] suse-network-setup.sh[450]: [2026-01-08T14:51:20Z INFO  nmstate::nm::profile] Skipping activation of enp0s5 as its controller bond0 will automatically activate it
2026-01-08 14:51:21 ::  [    9.820704] suse-network-setup.sh[450]: [2026-01-08T14:51:20Z ERROR nmc] Generating config failed: Detected interfaces without connection files: enp0s4
2026-01-08 15:22:27 ::  [    9.342833] suse-network-setup.sh[434]: + PATH=/bin:/usr/bin:/sbin:/usr/sbin
2026-01-08 15:22:27 ::  [    9.344607] suse-network-setup.sh[434]: + NETWORK_DATA_FILE=/mnt/ipa/openstack/latest/network_data.json
2026-01-08 15:22:27 ::  [    9.346494] suse-network-setup.sh[434]: + '[' '!' -f /mnt/ipa/openstack/latest/network_data.json ']'
2026-01-08 15:22:27 ::  [    9.348448] suse-network-setup.sh[434]: + mkdir -p /tmp/nmc/desired /tmp/nmc/generated
2026-01-08 15:22:27 ::  [    9.349009] suse-network-setup.sh[434]: + cp /mnt/ipa/openstack/latest/network_data.json /tmp/nmc/desired/_all.yaml
2026-01-08 15:22:27 ::  [    9.355743] suse-network-setup.sh[434]: + nmc generate --config-dir /tmp/nmc/desired --output-dir /tmp/nmc/generated
2026-01-08 15:22:27 ::  [    9.358159] suse-network-setup.sh[451]: [2026-01-08T15:22:26Z INFO  nmc::generate_conf] Generating config from "/tmp/nmc/desired/_all.yaml"...
2026-01-08 15:22:27 ::  [    9.360806] suse-network-setup.sh[451]: [2026-01-08T15:22:26Z INFO  nmstate::nm::profile] Skipping activation of enp0s4 as its controller bond0 will automatically activate it
2026-01-08 15:22:27 ::  [    9.364068] suse-network-setup.sh[451]: [2026-01-08T15:22:26Z INFO  nmstate::nm::profile] Skipping activation of enp0s5 as its controller bond0 will automatically activate it
2026-01-08 15:22:27 ::  [    9.367208] suse-network-setup.sh[451]: [2026-01-08T15:22:26Z ERROR nmc] Generating config failed: Detected interfaces without connection files: enp0s4

So the same bmh failed and succeeded to configure IPA networking on different restarts:

13:46:28 first inspection fail
14:17:36 retry inspection success
14:20:26 reboot for provisioning (why?!) fail
subsequent retries fail

Questions compared to usual behavior would be:

why do we see this only in this CI combination (leapmicro + capm3-virt) and only on workload cluster, or maybe it happened on others, but we didn't investigate enough?
why is the restart needed between inspection and provisioning?
why is nmc generate command randomly failing since the preprovisioningNetworkData secret does not change between restarts?