Illegal instruction when I want to numactl --cpubind=0 --membind=1 to CXL Memory

Host environment

  • Operating system: Ubuntu 20.04

  • OS/kernel version: Linux 8003 5.15.0-73-generic Ubuntu SMP Mon May 15 15:18:26 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

  • Architecture: x86

  • QEMU flavor: qemu-system-x86_64

  • QEMU version: 8.0.50

  • QEMU command line:

    ./qemu-system-x86_64  -enable-kvm \
-smp 48 \
-kernel ${KERNEL_PATH} \
-cpu host \
-machine q35,accel=kvm,nvdimm=on,cxl=on\
-append "${KERNEL_CMD}" \
...
-object memory-backend-ram,id=vmem0,share=on,size=32G \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=32G

Emulated/Virtualized environment

  • Operating system: Ubuntu 22.04
  • OS/kernel version: Linux 8003 6.3.7 SMP PREEMPT_DYNAMIC Mon Jun 12 11:42:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
  • Architecture: x86

Description of problem

I ran QEMU for simulating CXL DRAM and when I tried to run numactl --cpubind=0 --membind=1 ls , I got Illegal instruction The numa node 1 was the CXL DRAM simulated by QEMU.

root@8003:~# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 node 0 size: 32090 MB node 0 free: 31325 MB node 1 cpus: node 1 size: 32768 MB node 1 free: 32768 MB node distances: node 0 1 0: 10 20 1: 20 10

When I ran on numa node 1, no failed

root@8003:~# numactl --membind=0 ls ndctl

When I ran on numa node 1(CXL DRAM),it failed.

root@8003:~# numactl --membind=1 ls [ 913.975032] traps: ls[667] trap invalid opcode ip:7fdec255d180 sp:7ffd3c507288 error:0 in ld-linux-x86-64.so.2[7fdec2546000+2a000] Illegal instruction

Steps to reproduce

  1. start the guest
  2. cxl list (we could see the simulated CXL DRAM)

root@8003:~# cxl list [ { "memdev":"mem0", "ram_size":34359738368, "serial":0, "host":"0000:0d:00.0" } ]

  1. cxl create-region -t ram -d decoder0.0 -m mem0
  2. daxctl reconfigure-device dax0.0 --mode=system-ram
  3. numactl -H

root@8003:~# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 node 0 size: 32090 MB node 0 fr ee: 31254 MB node 1 cpus: node 1 size: 32768 MB node 1 free: 32768 MB node distances: node 0 1 0: 10 20 1: 20 10

  1. numactl --membind=1 ls

root@8003:~# numactl --membind=1 ls [38441.892140] **traps: ls[861] trap invalid opcode ip:7f15db6ac180 sp:7ffc648755c8 **error:0 in ld-linux-x86-64.so.2[7f15db695000+2a000] Illegal instruction

Additional information

When I run dmesg, I found an error.

root@8003:~# dmesg|grep error [ 2.321130] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2

Since my CPU is a Xeon III, not a Xeon IV with CXL support, I'm wondering if it's because the CPU doesn't support CXL instructions, or if the Xeon III can emulate it, just because my settings don't make sense. If this is my settings problem, could you help me to deal this? Or it just caused by my Xeon III,I will update it to Xeon IV.

Edited by Yemaoxin