Unable to spawn a VM on Ubuntu 24 when/tmp gets auto deleted

Summary

Recent systemd causes VMs to fail on Ubuntu compute hosts

Detailed Description

Systemd 255.4-1ubuntu2 introduced automatic cleaning of /tmp after 30 days (default). Yaook libvirtd container mounts /tmp/logs into /var/log/libvirt/qemu introduced in ths MR . The systemd update causes /tmp/logs to be deleted by the host OS after 30 days. This leads to libvirt throwing No such file or directory error and not starting a new VM.

Steps to reproduce the issue

• Do not spawn any new VM on a compute host for 30 days
• /tmp/logs will disappear but it's still mounted by libvirtd container
• Try to spawn a VM

Result

• VM in Error state
• Logs say

failed to create logfile /var/log/libvirt/qemu/instance-000002e8.log: No such file or directory

Expected Result

• VM starts and is in Active state

Resolution

• Disable the 30 days /tmp deletion

sudo cat /usr/lib/tmpfiles.d/tmp.conf 
D /tmp 1777 root root - # <--- was '30d' before

• Force upgrade the compute nodes to recreate the mountpoint

 yaookctl force-upgrade compute compute-node-name --break-even-if-unnecessary --yes-i-mean-it 

Proposal

• for libvirt logs mount different directory than /tmp
• Do not mount /tmp as it may be deleted unexpectedly and break things

Specification

• /var/log/libvirt/qemu of the pod MUST NOT be mounted to /tmp or subdirectories of the host
• var/log/libvirt/qemu of the pod MUST be mounted to a different, persistent path of the host
• the persistent path on the host SHOULD get cleaned up when deleting the nova-compute node