Problem with the Linux 6 text console when using driverctl to disable nvidia Linux driver.
This problem started when I upgraded my Arch Linux system to kernel 6.0.1. I have 2 nvidia GPUs, on the CPU bus they are 0000:08:00.0 and 0000:09:00.0. I want to disable nvidia host driver for 0000:09:00.0 so I can pass-thru to the guest. When I use driverctl for that, the text console stops working (I start X11 later with startx, I don't boot into graphics target). However, it is just the text console output: if I blindly login and type startx, I get the X11 screen for 0000:08:00.0 and I can start KVM/Qemu with direct access 0000:09:00.0. I suspect that some text console initialization has moved earlier, but I still didn't manage to tackle systemd dependencies to have driverctl loading early enough. I am attaching the part of the logs that refers to (nvidia|driverctl). Let me know if you would like any other information. Thanks!
working:
Oct 22 10:09:00 eren kernel: Linux version 5.15.74-1-lts (linux-lts@archlinux) (gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39.0) #1 SMP Sat, 15 Oct 2022 06:43:47 +0000
Oct 22 10:09:00 eren systemd[1]: Created slice Slice /system/driverctl.
Oct 22 10:09:00 eren systemd[1]: Starting Load the driverctl override for pci-0000:09:00.1...
Oct 22 10:09:01 eren kernel: nvidia: module license 'NVIDIA' taints kernel.
Oct 22 10:09:01 eren kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 511
Oct 22 10:09:01 eren kernel: nvidia 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
Oct 22 10:09:01 eren kernel: nvidia 0000:09:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
Oct 22 10:09:01 eren kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 520.56.06 Thu Oct 6 21:22:53 UTC 2022
Oct 22 10:09:01 eren systemd[1]: Starting Load the driverctl override for pci-0000:09:00.0...
Oct 22 10:09:01 eren kernel: [drm] [nvidia-drm] [GPU ID 0x00000800] Loading driver
Oct 22 10:09:01 eren kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:08:00.0 on minor 0
Oct 22 10:09:01 eren kernel: [drm] [nvidia-drm] [GPU ID 0x00000900] Loading driver
Oct 22 10:09:01 eren kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:09:00.0 on minor 1
Oct 22 10:09:01 eren kernel: nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint.
Oct 22 10:09:01 eren systemd-modules-load[435]: Inserted module 'nvidia_uvm'
Oct 22 10:09:01 eren kernel: nvidia-uvm: Loaded the UVM driver, major device number 509.
Oct 22 10:09:01 eren systemd[1]: Finished Load the driverctl override for pci-0000:09:00.0.
Oct 22 10:09:01 eren audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=driverctl@pci-0000:09:00.0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 22 10:09:01 eren systemd-udevd[488]: nvidia: Process '/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidiactl c $(grep nvidia-frontend /proc/devices | cut -d \ -f 1) 255'' failed with exit code 1.
Oct 22 10:09:01 eren systemd-udevd[488]: nvidia: Process '/usr/bin/bash -c 'for i in $(cat /proc/driver/nvidia/gpus/*/information | grep Minor | cut -d \ -f 4); do /usr/bin/mknod -Z -m 666 /dev/nvidia${i} c $(grep nvidia-frontend /proc/devices | cut -d \ -f 1) ${i}; done'' failed with exit code 1.
Oct 22 10:09:01 eren systemd[1]: Finished Load the driverctl override for pci-0000:09:00.1.
Oct 22 10:09:01 eren audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=driverctl@pci-0000:09:00.1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 22 10:09:16 eren dbus-daemon[1122]: [session uid=1000 pid=1122] Activating via systemd: service name='org.a11y.Bus' unit='at-spi-dbus-bus.service' requested by ':1.7' (uid=1000 pid=1155 comm="nvidia-settings --assign CurrentMetaMode=nvidia-au")
Oct 22 10:11:02 eren systemd[1]: driverctl@pci-0000:09:00.0.service: Deactivated successfully.
Oct 22 10:11:02 eren systemd[1]: Stopped Load the driverctl override for pci-0000:09:00.0.
Oct 22 10:11:02 eren audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=driverctl@pci-0000:09:00.0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 22 10:11:02 eren systemd[1]: driverctl@pci-0000:09:00.1.service: Deactivated successfully.
Oct 22 10:11:02 eren systemd[1]: Stopped Load the driverctl override for pci-0000:09:00.1.
Oct 22 10:11:02 eren audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=driverctl@pci-0000:09:00.1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 22 10:11:02 eren systemd[1]: Removed slice Slice /system/driverctl.
not working:
Oct 22 10:11:22 eren kernel: Linux version 6.0.2-arch1-1 (linux@archlinux) (gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39.0) #1 SMP PREEMPT_DYNAMIC Sat, 15 Oct 2022 14:00:49 +0000
Oct 22 10:11:23 eren systemd[1]: Created slice Slice /system/driverctl.
Oct 22 10:11:23 eren systemd[1]: Starting Load the driverctl override for pci-0000:09:00.1...
Oct 22 10:11:23 eren kernel: nvidia: module license 'NVIDIA' taints kernel.
Oct 22 10:11:23 eren kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 235
Oct 22 10:11:23 eren kernel: nvidia 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
Oct 22 10:11:23 eren kernel: nvidia 0000:09:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
Oct 22 10:11:23 eren systemd[1]: Finished Load the driverctl override for pci-0000:09:00.1.
Oct 22 10:11:23 eren audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=driverctl@pci-0000:09:00.1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 22 10:11:23 eren systemd-udevd[505]: nvidia: Process '/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidiactl c $(grep nvidia-frontend /proc/devices | cut -d \ -f 1) 255'' failed with exit code 1.
Oct 22 10:11:23 eren systemd-udevd[505]: nvidia: Process '/usr/bin/bash -c 'for i in $(cat /proc/driver/nvidia/gpus/*/information | grep Minor | cut -d \ -f 4); do /usr/bin/mknod -Z -m 666 /dev/nvidia${i} c $(grep nvidia-frontend /proc/devices | cut -d \ -f 1) ${i}; done'' failed with exit code 1.
Oct 22 10:11:23 eren kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
Oct 22 10:11:23 eren kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 520.56.06 Thu Oct 6 21:22:53 UTC 2022
Oct 22 10:11:23 eren kernel: [drm] [nvidia-drm] [GPU ID 0x00000800] Loading driver
Oct 22 10:11:23 eren kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:08:00.0 on minor 0
Oct 22 10:11:23 eren kernel: [drm] [nvidia-drm] [GPU ID 0x00000900] Loading driver
Oct 22 10:11:23 eren kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:09:00.0 on minor 1
Oct 22 10:11:23 eren systemd[1]: Starting Load the driverctl override for pci-0000:09:00.0...
Oct 22 10:11:23 eren systemd[1]: Finished Load the driverctl override for pci-0000:09:00.0.
Oct 22 10:11:23 eren audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=driverctl@pci-0000:09:00.0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 22 10:11:23 eren systemd-modules-load[432]: Inserted module 'nvidia_uvm'
Oct 22 10:11:23 eren kernel: nvidia-uvm: Loaded the UVM driver, major device number 511.
Oct 22 10:12:00 eren systemd[1]: driverctl@pci-0000:09:00.0.service: Deactivated successfully.
Oct 22 10:12:00 eren audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=driverctl@pci-0000:09:00.0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 22 10:12:00 eren systemd[1]: Stopped Load the driverctl override for pci-0000:09:00.0.
Oct 22 10:12:00 eren audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=driverctl@pci-0000:09:00.1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 22 10:12:00 eren systemd[1]: driverctl@pci-0000:09:00.1.service: Deactivated successfully.
Oct 22 10:12:00 eren systemd[1]: Stopped Load the driverctl override for pci-0000:09:00.1.
Oct 22 10:12:00 eren systemd[1]: Removed slice Slice /system/driverctl.