Provisioning rke2 instances on capo stops at different stages - cloud init script is not always executed completely - this was seen on version 1.27

Summary

During the deployment of RKE2 on Openstack this stop at different stages like:

• having only on CP and MD deployed, but no action after that
• having two CP nodes functional and the last one in failed state
• having all CP nodes functional, but MD in failed state

Details

After checking on the cloud-init logs we can see the bellow error:

2024-02-21 13:37:21,164 - util.py[DEBUG]: Cloud-init 23.1 received SIGTERM, exiting...
  Filename: <frozen importlib._bootstrap_external>
  Function: _path_stat
  Line number: 147
    Filename: <frozen importlib._bootstrap_external>
    Function: find_spec
    Line number: 1544
      Filename: <frozen importlib._bootstrap_external>
      Function: _get_spec
      Line number: 1411
2024-02-21 13:37:21,165 - handlers.py[DEBUG]: finish: init-network/config-write-files: FAIL: running config-write-files with frequency once-per-instance
2024-02-21 13:37:21,165 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2024-02-21 13:37:21,165 - util.py[DEBUG]: Read 12 bytes from /proc/uptime
2024-02-21 13:37:21,165 - util.py[DEBUG]: cloud-init mode 'init' took 2.487 seconds (2.49)
2024-02-21 13:37:21,165 - handlers.py[DEBUG]: finish: init-network: FAIL: searching for network datasources
2024-02-21 13:37:39,724 - util.py[DEBUG]: Cloud-init v. 23.1 running 'init-local' at Wed, 21 Feb 2024 13:37:39 +0000. Up 4.86 seconds.
2024-02-21 13:37:39,726 - main.py[DEBUG]: No kernel command line url found.

Also at the same time it was noticed that also sylva-grow part was executed:

 systemctl status sylva-growparts.service
● sylva-growparts.service - Grow partitions
     Loaded: loaded (/etc/systemd/system/sylva-growparts.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2024-02-21 13:37:47 UTC; 20h ago
   Main PID: 709 (code=exited, status=0/SUCCESS)
        CPU: 402ms

Feb 21 13:37:47 mgmt-22396459-cp-878a51831e-wfg2l sylva-growparts[1684]: The filesystem on /dev/vg/lv_etcd is now 2615296 (4k) blocks long.
Feb 21 13:37:47 mgmt-22396459-cp-878a51831e-wfg2l sylva-growparts[1686]: resize2fs 1.46.5 (30-Dec-2021)
Feb 21 13:37:47 mgmt-22396459-cp-878a51831e-wfg2l sylva-growparts[1686]: Filesystem at /dev/vg/lv_containerd is mounted on /var/lib/rancher/rke2/agent/containerd; on-line resizing requ>
Feb 21 13:37:47 mgmt-22396459-cp-878a51831e-wfg2l sylva-growparts[1686]: old_desc_blocks = 1, new_desc_blocks = 4
Feb 21 13:37:47 mgmt-22396459-cp-878a51831e-wfg2l sylva-growparts[1686]: The filesystem on /dev/vg/lv_containerd is now 7844864 (4k) blocks long.
Feb 21 13:37:47 mgmt-22396459-cp-878a51831e-wfg2l sylva-growparts[1688]: resize2fs 1.46.5 (30-Dec-2021)
Feb 21 13:37:47 mgmt-22396459-cp-878a51831e-wfg2l sylva-growparts[1688]: Filesystem at /dev/vg/lv_kubelet is mounted on /var/lib/kubelet; on-line resizing required
Feb 21 13:37:47 mgmt-22396459-cp-878a51831e-wfg2l sylva-growparts[1688]: old_desc_blocks = 1, new_desc_blocks = 3
Feb 21 13:37:47 mgmt-22396459-cp-878a51831e-wfg2l sylva-growparts[1688]: The filesystem on /dev/vg/lv_kubelet is now 5229568 (4k) blocks long.
Feb 21 13:37:47 mgmt-22396459-cp-878a51831e-wfg2l systemd[1]: Finished Grow partitions.

And atfer this stage some of write-files that are defined by cloud-init are not been created anymore.