Migration

Implement live migration, using the vhost-user DEVICE_STATE feature.

(OUTDATED: Marked as a draft, because the necessary changes have not yet been merged into the vhost-crates (https://github.com/rust-vmm/vm-memory/pull/264, https://github.com/rust-vmm/vhost/pull/203, https://github.com/rust-vmm/vhost/pull/206), and so the first commit instead references those pull requests.)

To support migration, we need to extract serializable state from our central PassthroughFs object, transfer it to the destination via DEVICE_STATE, and re-apply it there to an existing PassthroughFs object. The main difficulty is to transfer our inode store and handle map to the destination:

• For every inode, we need to give the destination instructions on how to locate it. In this MR, we do this by finding a parent directory (i.e. another inode in the store) and filename for every inode, which will allow the destination to openat() it.
• For every handle, we need to tell the destination how to open the associated inode to get this handle, i.e. what flags to use for open().

This has some limitations, and it isn’t exactly trivial to figure out which those are. Some of them are:

1. The user must ensure that the shared directory contents of source and destination match when control switches from the source to the destination. virtiofsd will not do this, and not copy any data. This is especially important to note for what qemu calls “migration to file”, i.e. taking snapshots and wanting to restore them later. While technically possible without any issue, users must note that the filesystem state will not be part of the VM snapshot, and must be snapshotted/restored using external tools if desired.
2. When you have files open in the guest (handles from virtiofsd's perspective) and the respective inodes are deleted, the destination will not be able to open these handles, and the data will be lost. Pretty much the same applies if the inode is not deleted, but its access mode is changed such that it becomes inaccessible to virtiofsd.
3. When an external party renames an inode in the shared directory during migration, the destination may receive outdated directory/filename information, rendering it unable to find that inode, which will currently result in migration failing.
4. Similarly, when an external party deletes an inode in the shared directory during migration, the destination may still try to find the deleted inode, failing to do so, and thus making the whole migration fail.

In follow-up work, we will probably want to address some of these:

• Inodes and handles that cannot be found/opened on the destination should not have to lead to catastrophic failure, but users should be able to configure virtiofsd such that these errors are ignored and will instead lead to I/O errors in the guest when trying to access these files there.
  • This should address limitation number 4
  • Implemented in the latest version of this MR.
• In order to ensure that handles can be re-opened on the destination, even if the underlying inode has already been deleted, we may want to create a name (hardlink) before closing our handle on the source side (possible e.g. via silly rename, i.e. renaming instead of deleting when deleting is requested, or actually always keeping a hidden hardlink for any handle that we have open (because it can be deleted at any point, and linkat(2) says that it “will generally not work if the file has a link count of zero”).
  • This should address limitation number 2
• When file handles are available to use, and both source and destination operate on the same shared directory on the same filesystem, we could just use file handles to represent inodes instead of having to reconstruct usable paths.
  • This should address limitation number 3 (but is rarely usable)