Kubevirt VM Live Migration failed on beegfs ReadWriteMany volumn

Software environment

  • Operating system: RHEL 8.3
  • Architecture: X86
  • kernel version: 4.18.0-348.7.1.el8_5.x86_64
  • libvirt version:6.0.0
  • Hypervisor and version: Kubevirt v0.58.0

Description of problem

kubevirt vm can not migrate on beegfs volume

Steps to reproduce

I attach a beegfs pv with the vm. The beegfs pv is createed from an readwritemany sc. After I migrate the vm by virtctl cli ,there is no error. After a few seconds, i get the kubeclt event error as below "VirtualMachineInstance migration uid 8e772457-f6e8-4e62-b465-844a6d8727a0 failed. reason:Live migration failed error encountered during MigrateToURI3 libvirt api call: virError(Code=81, Domain=10, Message='Unsafe migration: Migration without shared storage is unsafe' "

Steps to reproduce the behavior. 1.create pvc csi-beegfs-dyn-pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-beegfs-dyn-pvc
spec:
accessModes:

ReadWriteMany
resources:
requests:
storage: 100Gi
storageClassName: csi-beegfs-dyn-sc
2.attach the pv to a vm

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
name: win11
spec:
running: false
template:
metadata:
labels:
kubevirt.io/domain: win11
spec:
domain:
cpu:
cores: 4
devices:
disks:
- bootOrder: 2
cdrom:
bus: sata
name: cdromiso
- bootOrder: 1
disk:
bus: virtio
name: harddrive
- cdrom:
bus: sata
name: virtiocontainerdisk
interfaces:
- name: default
model: e1000
masquerade: {}
machine:
type: q35
resources:
requests:
memory: 16G
networks:
- name: default
pod: {}
volumes:
- name: cdromiso
persistentVolumeClaim:
claimName: iso-win11
- name: harddrive
persistentVolumeClaim:
claimName: csi-beegfs-dyn-pvc
- containerDisk:
image: kubevirt/virtio-container-disk
name: virtiocontainerdisk
~

start the vm virtctl start win11 install windows with vnc, edit and delte iso in win11 vm . migrate the windows vm. virtctl migrate win11 linux consonle report hint: VM win11 was scheduled to migrate After a few seconds, i get the kubeclt event error as below "VirtualMachineInstance migration uid 8e772457-f6e8-4e62-b465-844a6d8727a0 failed. reason:Live migration failed error encountered during MigrateToURI3 libvirt api call: virError(Code=81, Domain=10, Message='Unsafe migration: Migration without shared storage is unsafe' "

Additional information

kubevirt community give me some advice as below and suggest me to open an issue on libvirt project.

IIRC this is a libvirt issue looks like beegfs is not marked as a safe shared storage in libvirt. Kubevirt live migration feature is base on it.

src/qemu/qemu_migration.c

enum {
    VIR_FILE_SHFS_NFS = (1 << 0),
    VIR_FILE_SHFS_GFS2 = (1 << 1), /* Global File System 2 */
    VIR_FILE_SHFS_OCFS = (1 << 2), /* Oracle Cluster FS (2) */
    VIR_FILE_SHFS_AFS = (1 << 3), /* Andrew File System */
    VIR_FILE_SHFS_SMB = (1 << 4), /* Server message block - windows shares */
    VIR_FILE_SHFS_CIFS = (1 << 5), /* Common Internet File System - windows shares */
    VIR_FILE_SHFS_CEPH = (1 << 6),
    VIR_FILE_SHFS_GPFS = (1 << 7), /* General Parallel File System/IBM Spectrum Scale */
    VIR_FILE_SHFS_QB = (1 << 8), /* Quobyte shared filesystem */
    VIR_FILE_SHFS_ACFS = (1 << 9), /* Oracle ASM Cluster File System */
    VIR_FILE_SHFS_GLUSTERFS = (1 << 10), /* gluster's FUSE-based client */
};
Edited by Peter Krempa