Can't install NVIDIA runtime on Flatcar
Information about my system:
$ uname -r
5.10.69-flatcar
$ docker version
Client:
Version: 19.03.15
API version: 1.40
Go version: go1.13.15
Git commit: 6ff149b
Built: Fri Jan 29 21:29:16 2021
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.15
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 6ff149b
Built: Fri Jan 29 21:29:16 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.4
GitCommit: 69a4a55aa42353c71aae40e5bb27a412d8e3149b
nvidia:
Version: 1.0.0-rc95+dev.docker-19.03
GitCommit: 7cc3ad87e3130ba1fed6c3fd7a756e88e2aac066
docker-init:
Version: 0.18.0fec3683b971d9c3ef73f284f176672c44b448662
GitCommit:
loop
, i2c_core
and ipmi_msghandler
modules are loaded.
Tried to install NVIDIA runtime for docker using instructions available on https://gitlab.com/nvidia/container-images/driver/-/tree/master/flatcar
Command I executed:
$ docker run -it --rm --privileged -v "/etc/docker:/etc/docker" \
-v "/run/nvidia:/run/nvidia" \
-v "/run/docker.sock:/run/docker.sock" \
-v "/opt/nvidia-runtime:/opt/nvidia-runtime" \
-e "RUNTIME=docker" \
-e "RUNTIME_ARGS=--socket /run/docker.sock" \
-e "DOCKER_SOCKET=/run/docker.sock" \
nvcr.io/nvidia/k8s/container-toolkit:1.7.2-ubuntu18.04 /opt/nvidia-runtime
Logs I get:
INFO[0000] Starting nvidia-toolkit
INFO[0000] Parsing arguments
INFO[0000] Verifying Flags
INFO[0000] Initializing
INFO[0000] Installing toolkit
INFO[0000] Parsing arguments: [/opt/nvidia-runtime/toolkit]
INFO[0000] Successfully parsed arguments
INFO[0000] Installing NVIDIA container toolkit to '/opt/nvidia-runtime/toolkit'
INFO[0000] Removing existing NVIDIA container toolkit installation
INFO[0000] Creating directory '/opt/nvidia-runtime/toolkit'
INFO[0000] Creating directory '/opt/nvidia-runtime/toolkit/.config/nvidia-container-runtime'
INFO[0000] Installing NVIDIA container library to '/opt/nvidia-runtime/toolkit'
INFO[0000] Finding library libnvidia-container.so.1 (root=)
INFO[0000] Checking library candidate '/usr/lib64/libnvidia-container.so.1'
INFO[0000] Skipping library candidate '/usr/lib64/libnvidia-container.so.1': error resolving link '/usr/lib64/libnvidia-container.so.1': lstat /usr/lib64: no such file or directory
INFO[0000] Checking library candidate '/usr/lib/x86_64-linux-gnu/libnvidia-container.so.1'
INFO[0000] Resolved link: '/usr/lib/x86_64-linux-gnu/libnvidia-container.so.1' => '/usr/lib/x86_64-linux-gnu/libnvidia-container.so.1.6.0'
INFO[0000] Installing '/usr/lib/x86_64-linux-gnu/libnvidia-container.so.1.6.0' to '/opt/nvidia-runtime/toolkit/libnvidia-container.so.1.6.0'
INFO[0000] Installed '/usr/lib/x86_64-linux-gnu/libnvidia-container.so.1.6.0' to '/opt/nvidia-runtime/toolkit/libnvidia-container.so.1.6.0'
INFO[0000] Creating symlink '/opt/nvidia-runtime/toolkit/libnvidia-container.so.1' -> 'libnvidia-container.so.1.6.0'
INFO[0000] Installing executable '/usr/bin/nvidia-container-runtime' to /opt/nvidia-runtime/toolkit
INFO[0000] Installing '/usr/bin/nvidia-container-runtime' to '/opt/nvidia-runtime/toolkit/nvidia-container-runtime.real'
INFO[0000] Installed '/opt/nvidia-runtime/toolkit/nvidia-container-runtime.real'
INFO[0000] Installed wrapper '/opt/nvidia-runtime/toolkit/nvidia-container-runtime'
INFO[0000] Finding library libnvidia-ml.so (root=/run/nvidia/driver)
INFO[0000] Checking library candidate '/run/nvidia/driver/usr/lib64/libnvidia-ml.so'
INFO[0000] Skipping library candidate '/run/nvidia/driver/usr/lib64/libnvidia-ml.so': error resolving link '/run/nvidia/driver/usr/lib64/libnvidia-ml.so': lstat /run/nvidia/driver: no such file or directory
INFO[0000] Checking library candidate '/run/nvidia/driver/usr/lib/x86_64-linux-gnu/libnvidia-ml.so'
INFO[0000] Skipping library candidate '/run/nvidia/driver/usr/lib/x86_64-linux-gnu/libnvidia-ml.so': error resolving link '/run/nvidia/driver/usr/lib/x86_64-linux-gnu/libnvidia-ml.so': lstat /run/nvidia/driver: no such file or directory
WARN[0000] Error finding library path for root /run/nvidia/driver: error locating NVIDIA management library: error locating library 'libnvidia-ml.so'
INFO[0000] Using library root
INFO[0000] Installing executable 'nvidia-container-runtime.experimental' to /opt/nvidia-runtime/toolkit
INFO[0000] Installing 'nvidia-container-runtime.experimental' to '/opt/nvidia-runtime/toolkit/nvidia-container-runtime.experimental'
INFO[0000] Installed '/opt/nvidia-runtime/toolkit/nvidia-container-runtime.experimental'
INFO[0000] Installed wrapper '/opt/nvidia-runtime/toolkit/nvidia-container-runtime-experimental'
INFO[0000] Installing NVIDIA container CLI from '/usr/bin/nvidia-container-cli'
INFO[0000] Installing executable '/usr/bin/nvidia-container-cli' to /opt/nvidia-runtime/toolkit
INFO[0000] Installing '/usr/bin/nvidia-container-cli' to '/opt/nvidia-runtime/toolkit/nvidia-container-cli.real'
INFO[0000] Installed '/opt/nvidia-runtime/toolkit/nvidia-container-cli.real'
INFO[0000] Installed wrapper '/opt/nvidia-runtime/toolkit/nvidia-container-cli'
INFO[0000] Installing NVIDIA container runtime hook from '/usr/bin/nvidia-container-toolkit'
INFO[0000] Installing executable '/usr/bin/nvidia-container-toolkit' to /opt/nvidia-runtime/toolkit
INFO[0000] Installing '/usr/bin/nvidia-container-toolkit' to '/opt/nvidia-runtime/toolkit/nvidia-container-toolkit.real'
INFO[0000] Installed '/opt/nvidia-runtime/toolkit/nvidia-container-toolkit.real'
INFO[0000] Installed wrapper '/opt/nvidia-runtime/toolkit/nvidia-container-toolkit'
INFO[0000] Creating symlink '/opt/nvidia-runtime/toolkit/nvidia-container-runtime-hook' -> 'nvidia-container-toolkit'
INFO[0000] Installing NVIDIA container toolkit config '/opt/nvidia-runtime/toolkit/.config/nvidia-container-runtime/config.toml'
INFO[0000] Setting up runtime
INFO[0000] Starting 'setup' for docker
INFO[0000] Parsing arguments: [/opt/nvidia-runtime/toolkit]
INFO[0000] Successfully parsed arguments
INFO[0000] Loading config: /etc/docker/daemon.json
INFO[0000] Successfully loaded config
INFO[0000] Flushing config
INFO[0000] Successfully flushed config
INFO[0000] Sending SIGHUP signal to docker
INFO[0000] Shutting Down
ERRO[0000] error running nvidia-toolkit: unable to setup runtime: error running docker command: signal: hangup