vfio-mediated-device.txt 14.6 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14
.. include:: <isonum.txt>

=====================
VFIO Mediated devices
=====================

:Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved.
:Author: Neo Jia <cjia@nvidia.com>
:Author: Kirti Wankhede <kwankhede@nvidia.com>

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License version 2 as
published by the Free Software Foundation.

15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Virtual Function I/O (VFIO) Mediated devices[1]
===============================================

The number of use cases for virtualizing DMA devices that do not have built-in
SR_IOV capability is increasing. Previously, to virtualize such devices,
developers had to create their own management interfaces and APIs, and then
integrate them with user space software. To simplify integration with user space
software, we have identified common requirements and a unified management
interface for such devices.

The VFIO driver framework provides unified APIs for direct device access. It is
an IOMMU/device-agnostic framework for exposing direct device access to user
space in a secure, IOMMU-protected environment. This framework is used for
multiple devices, such as GPUs, network adapters, and compute accelerators. With
direct device access, virtual machines or user space applications have direct
access to the physical device. This framework is reused for mediated devices.

The mediated core driver provides a common interface for mediated device
management that can be used by drivers of different devices. This module
provides a generic interface to perform these operations:

* Create and destroy a mediated device
* Add a mediated device to and remove it from a mediated bus driver
* Add a mediated device to and remove it from an IOMMU group

The mediated core driver also provides an interface to register a bus driver.
For example, the mediated VFIO mdev driver is designed for mediated devices and
supports VFIO APIs. The mediated bus driver adds a mediated device to and
removes it from a VFIO group.

The following high-level block diagram shows the main components and interfaces
in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
48
devices as examples, as these devices are the first devices to use this module::
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96

     +---------------+
     |               |
     | +-----------+ |  mdev_register_driver() +--------------+
     | |           | +<------------------------+              |
     | |  mdev     | |                         |              |
     | |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user
     | |  driver   | |     probe()/remove()    |              |    APIs
     | |           | |                         +--------------+
     | +-----------+ |
     |               |
     |  MDEV CORE    |
     |   MODULE      |
     |   mdev.ko     |
     | +-----------+ |  mdev_register_device() +--------------+
     | |           | +<------------------------+              |
     | |           | |                         |  nvidia.ko   |<-> physical
     | |           | +------------------------>+              |    device
     | |           | |        callbacks        +--------------+
     | | Physical  | |
     | |  device   | |  mdev_register_device() +--------------+
     | | interface | |<------------------------+              |
     | |           | |                         |  i915.ko     |<-> physical
     | |           | +------------------------>+              |    device
     | |           | |        callbacks        +--------------+
     | |           | |
     | |           | |  mdev_register_device() +--------------+
     | |           | +<------------------------+              |
     | |           | |                         | ccw_device.ko|<-> physical
     | |           | +------------------------>+              |    device
     | |           | |        callbacks        +--------------+
     | +-----------+ |
     +---------------+


Registration Interfaces
=======================

The mediated core driver provides the following types of registration
interfaces:

* Registration interface for a mediated bus driver
* Physical device driver interface

Registration Interface for a Mediated Bus Driver
------------------------------------------------

The registration interface for a mediated bus driver provides the following
97
structure to represent a mediated device's driver::
98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115

     /*
      * struct mdev_driver [2] - Mediated device's driver
      * @name: driver name
      * @probe: called when new device created
      * @remove: called when device removed
      * @driver: device driver structure
      */
     struct mdev_driver {
	     const char *name;
	     int  (*probe)  (struct device *dev);
	     void (*remove) (struct device *dev);
	     struct device_driver    driver;
     };

A mediated bus driver for mdev should use this structure in the function calls
to register and unregister itself with the core driver:

116
* Register::
117

118
    extern int  mdev_register_driver(struct mdev_driver *drv,
119 120
				   struct module *owner);

121
* Unregister::
122

123
    extern void mdev_unregister_driver(struct mdev_driver *drv);
124 125 126 127 128 129 130 131 132

The mediated bus driver is responsible for adding mediated devices to the VFIO
group when devices are bound to the driver and removing mediated devices from
the VFIO when devices are unbound from the driver.


Physical Device Driver Interface
--------------------------------

133 134 135
The physical device driver interface provides the mdev_parent_ops[3] structure
to define the APIs to manage work in the mediated core driver that is related
to the physical device.
136

137
The structures in the mdev_parent_ops structure are as follows:
138 139 140 141 142

* dev_attr_groups: attributes of the parent device
* mdev_attr_groups: attributes of the mediated device
* supported_config: attributes to define supported configurations

143
The functions in the mdev_parent_ops structure are as follows:
144 145 146 147

* create: allocate basic resources in a driver for a mediated device
* remove: free resources in a driver when a mediated device is destroyed

148 149 150 151 152
(Note that mdev-core provides no implicit serialization of create/remove
callbacks per mdev parent device, per mdev type, or any other categorization.
Vendor drivers are expected to be fully asynchronous in this respect or
provide their own internal resource protection.)

153
The callbacks in the mdev_parent_ops structure are as follows:
154 155 156 157 158 159 160 161

* open: open callback of mediated device
* close: close callback of mediated device
* ioctl: ioctl callback of mediated device
* read : read emulation callback
* write: write emulation callback
* mmap: mmap emulation callback

162
A driver should use the mdev_parent_ops structure in the function call to
163
register itself with the mdev core driver::
164

165 166
	extern int  mdev_register_device(struct device *dev,
	                                 const struct mdev_parent_ops *ops);
167

168
However, the mdev_parent_ops structure is not required in the function call
169
that a driver should use to unregister itself with the mdev core driver::
170

171
	extern void mdev_unregister_device(struct device *dev);
172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193


Mediated Device Management Interface Through sysfs
==================================================

The management interface through sysfs enables user space software, such as
libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
This management interface provides flexibility to the underlying physical
device's driver to support features such as:

* Mediated device hot plug
* Multiple mediated devices in a single virtual machine
* Multiple mediated devices from different physical devices

Links in the mdev_bus Class Directory
-------------------------------------
The /sys/class/mdev_bus/ directory contains links to devices that are registered
with the mdev core driver.

Directories and files under the sysfs for Each Physical Device
--------------------------------------------------------------

194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219
::

  |- [parent physical device]
  |--- Vendor-specific-attributes [optional]
  |--- [mdev_supported_types]
  |     |--- [<type-id>]
  |     |   |--- create
  |     |   |--- name
  |     |   |--- available_instances
  |     |   |--- device_api
  |     |   |--- description
  |     |   |--- [devices]
  |     |--- [<type-id>]
  |     |   |--- create
  |     |   |--- name
  |     |   |--- available_instances
  |     |   |--- device_api
  |     |   |--- description
  |     |   |--- [devices]
  |     |--- [<type-id>]
  |          |--- create
  |          |--- name
  |          |--- available_instances
  |          |--- device_api
  |          |--- description
  |          |--- [devices]
220 221 222 223 224 225 226 227 228 229

* [mdev_supported_types]

  The list of currently supported mediated device types and their details.

  [<type-id>], device_api, and available_instances are mandatory attributes
  that should be provided by vendor driver.

* [<type-id>]

230 231
  The [<type-id>] name is created by adding the device driver string as a prefix
  to the string provided by the vendor driver. This format of this name is as
232
  follows::
233 234 235

	sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);

236
  (or using mdev_parent_dev(mdev) to arrive at the parent device outside
237
  of the core mdev code)
238

239 240 241 242 243 244 245 246 247 248 249 250 251
* device_api

  This attribute should show which device API is being created, for example,
  "vfio-pci" for a PCI device.

* available_instances

  This attribute should show the number of devices of type <type-id> that can be
  created.

* [device]

  This directory contains links to the devices of type <type-id> that have been
252
  created.
253 254 255 256 257 258 259 260 261 262 263 264 265

* name

  This attribute should show human readable name. This is optional attribute.

* description

  This attribute should show brief features/description of the type. This is
  optional attribute.

Directories and Files Under the sysfs for Each mdev Device
----------------------------------------------------------

266 267 268 269
::

  |- [parent phy device]
  |--- [$MDEV_UUID]
270 271 272 273 274
         |--- remove
         |--- mdev_type {link to its type}
         |--- vendor-specific-attributes [optional]

* remove (write only)
275

276 277 278 279
Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
fail the remove() callback if that device is active and the vendor driver
doesn't support hot unplug.

280 281
Example::

282 283
	# echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove

284
Mediated device Hot plug
285 286 287 288 289 290 291 292 293
------------------------

Mediated devices can be created and assigned at runtime. The procedure to hot
plug a mediated device is the same as the procedure to hot plug a PCI device.

Translation APIs for Mediated Devices
=====================================

The following APIs are provided for translating user pfn to host pfn in a VFIO
294
driver::
295

296 297
	extern int vfio_pin_pages(struct device *dev, unsigned long *user_pfn,
				  int npage, int prot, unsigned long *phys_pfn);
298

299 300
	extern int vfio_unpin_pages(struct device *dev, unsigned long *user_pfn,
				    int npage);
301 302 303 304 305 306 307

These functions call back into the back-end IOMMU module by using the pin_pages
and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
these callbacks are supported in the TYPE1 IOMMU module. To enable them for
other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
these two callback functions.

308 309 310 311 312 313 314 315 316 317 318 319 320
Using the Sample Code
=====================

mtty.c in samples/vfio-mdev/ directory is a sample driver program to
demonstrate how to use the mediated device framework.

The sample driver creates an mdev device that simulates a serial port over a PCI
card.

1. Build and load the mtty.ko module.

   This step creates a dummy device, /sys/devices/virtual/mtty/mtty/

321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347
   Files in this device directory in sysfs are similar to the following::

     # tree /sys/devices/virtual/mtty/mtty/
        /sys/devices/virtual/mtty/mtty/
        |-- mdev_supported_types
        |   |-- mtty-1
        |   |   |-- available_instances
        |   |   |-- create
        |   |   |-- device_api
        |   |   |-- devices
        |   |   `-- name
        |   `-- mtty-2
        |       |-- available_instances
        |       |-- create
        |       |-- device_api
        |       |-- devices
        |       `-- name
        |-- mtty_dev
        |   `-- sample_mtty_dev
        |-- power
        |   |-- autosuspend_delay_ms
        |   |-- control
        |   |-- runtime_active_time
        |   |-- runtime_status
        |   `-- runtime_suspended_time
        |-- subsystem -> ../../../../class/mtty
        `-- uevent
348 349

2. Create a mediated device by using the dummy device that you created in the
350
   previous step::
351

352
     # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" >	\
353 354
              /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create

355
3. Add parameters to qemu-kvm::
356

357 358
     -device vfio-pci,\
      sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001
359 360 361 362

4. Boot the VM.

   In the Linux guest VM, with no hardware on the host, the device appears
363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394
   as  follows::

     # lspci -s 00:05.0 -xxvv
     00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550])
             Subsystem: Device 4348:3253
             Physical Slot: 5
             Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
     Stepping- SERR- FastB2B- DisINTx-
             Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
     <TAbort- <MAbort- >SERR- <PERR- INTx-
             Interrupt: pin A routed to IRQ 10
             Region 0: I/O ports at c150 [size=8]
             Region 1: I/O ports at c158 [size=8]
             Kernel driver in use: serial
     00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00
     10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00
     20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32
     30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00

     In the Linux guest VM, dmesg output for the device is as follows:

     serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
     0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A
     0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A


5. In the Linux guest VM, check the serial ports::

     # setserial -g /dev/ttyS*
     /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
     /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10
     /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10
395

396
6. Using minicom or any terminal emulation program, open port /dev/ttyS1 or
397 398 399 400 401 402 403
   /dev/ttyS2 with hardware flow control disabled.

7. Type data on the minicom terminal or send data to the terminal emulation
   program and read the data.

   Data is loop backed from hosts mtty driver.

404
8. Destroy the mediated device that you created::
405

406
     # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove
407

408
References
409
==========
410

411 412 413 414
1. See Documentation/vfio.txt for more information on VFIO.
2. struct mdev_driver in include/linux/mdev.h
3. struct mdev_parent_ops in include/linux/mdev.h
4. struct vfio_iommu_driver_ops in include/linux/vfio.h