CXL Device Missing PCI_CAP_ID_PM (01h) in CAP List Implementation According to PCIe SPEC

Host environment

• Operating system: Rocky Linux 8.10 (Green Obsidian)
• OS/kernel version: Linux 4.18.0-553.el8_10.x86_64 #1 SMP
• Architecture: x86_64
• QEMU flavor: qemu-system-x86_64
• QEMU version: QEMU emulator version 9.0.2 (v9.0.2)
• QEMU command line:

  ./qemu-system-x86_64 -net nic -net user,hostfwd=tcp::2222-:22,hostfwd=tcp::1234-:1234 \
   -machine q35,accel=kvm,cxl=on,nvdimm=on \
   -cpu Broadwell -smp 8,sockets=2,cores=2,threads=2 \
   -m 4G -bios ~/edk2-ovmf/x64/OVMF.fd \
   -boot order=d -k /usr/share/qemu/keymaps/en-us -vga virtio \
   -drive file=~/u22_linux_6_3_240809.img,format=qcow2 \
   -drive file=~/home_disk_240809.img,format=qcow2 \
   -monitor tcp::4000,server,nowait \
   -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
   -device cxl-rp,id=rp0,port=0,bus=cxl.0,chassis=0,slot=0 \
   -object memory-backend-file,id=cxl-mem0,mem-path=/tmp/cxl-mem0,size=256M \
   -object memory-backend-file,id=lsa0,mem-path=/tmp/cxl-lsa0,size=1K \
   -device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 \
   -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=256M

Emulated/Virtualized environment

• Operating system: Ubuntu 22.04.3 LTS
• OS/kernel version: Linux qemu 6.3.0 #24 SMP PREEMPT_DYNAMIC
• Architecture: x86_64

Description of problem

• The lack of PCI_CAP_ID_PM (01h) will not cause any crash or error when running QEMU, but it is violated to the PCIe SPEC.

• When some vendors test the power management capability (e.g., Linux Runtime PM), they must manually implement this CAP list to support the D1/D2/D3_Hot d-states changes.

• We don't see any PCI_CAP_ID_PM (01h) in the CXL rootport or endpoint

  

The reason that PM Cap. (01h) must be implemented

PCI Power Management  (Linux Kernel Official Doc)

1.2. Native PCI Power Management

• The PCI Bus Power Management Interface Specification (PCI PM Spec) was introduced between the PCI 2.1 and PCI 2.2 Specifications. It defined a standard interface for performing various operations related to power management. The implementation of the PCI PM Spec is optional for conventional PCI devices, but it is mandatory for PCI Express devices. If a device supports the PCI PM Spec, it has an 8 byte power management capability field in its PCI configuration space. This field is used to describe and control the standard features related to the native PCI power management

PCI Express® Base Specification Revision 6.2 (PCI-SIG Official Doc)

5.10 SR-IOV Power Management

• The PCI Power Management Capability as described elsewhere in § Chapter 5. is required for PFs. For VFs, the PCI Power Management Capability is optional

7.5.2 PCI Power Management Capability Structure

• § Figure 7-17 illustrates the organization of the PCI Power Management Capability structure. This structure is required for all PCI Express Functions

Steps to reproduce

1. Run the qemu-system-x86 (See QEMU command line)
2. sudo lspci -xxx

Fix suggestion

May call the following API to add the PCI_CAP_ID_PM when initializing PCIe Cap.

(Code snippet)

// File: ~/qemu/hw/pci/pcie.c
int pcie_cap_init(PCIDevice *dev, uint8_t offset, uint8_t type, uint8_t port, Error **errp) {

... 

pos = pci_add_capability(dev, PCI_CAP_ID_PM, 0,
                             PCI_PM_SIZEOF, errp);
if (pos < 0) {
    return pos;
}
 
dev->exp.pm_cap = pos;
 
/*
 * Indicates that this function complies with revision 1.2 of the
 * PCI Power Management Interface Specification.
 */
pci_set_word(dev->config + pos + PCI_PM_PMC, 0x3);