More flexible time-slicing strategy configuration

Specific use case

• Small k8s cluster for running GitLab-based CI, with two NVIDIA T400 GPUs per node.
• CI tasks are pretty lightweight; time-slicing ("nvidia.com/gpu.replicas": "4") is used to utilize the hardware more efficiently.
• Three kinds of jobs: no GPU, single GPU, dual GPU. Latter require two different devices to test GPU-GPU communication (NVSHMEM, CUDA-aware MPI).
  • The number of devices can be checked by running nvidia-smi -L in the container.

Problem

When a job has requests/limits nvidia.com/gpu: 2, typically one slice per physical device is allocated (!224), and the container gets two devices, as intended.

However, when multiple CI jobs are created, it appears that both slices might end up on a single device. In this case, the CUDA application in the container only sees a single device, and the device-to-device communication tests cannot run.

Suggested solution

It would be helpful to have a setting that makes the one-slice-per-device distribution a requirement rather than a best-effort attempt when distributing slices between devices. For our current case, a global setting would work.

Possible extensions

We can envision the case for having this configurable per-pod. While we currently only run small CI tasks on our k8s cluster (which are not performance-sensitive), we are contemplating having more performance testing. For it, the opposite behavior would be needed: being able to request all slots on a single device. That, however, is not a near-term problem for our project: likely, any performance testing will be done on a separate infrastructure.

Standalone example

With a single node (2 GPUs, 4 slices each), it looks like allocating 1 then 5 then 2 slices triggers the problem for the last container:

$ cat test_nvidia5.yml 
apiVersion: v1
kind: Pod
metadata:
  name: al42and-nvidia-test5
spec:
  containers:
    - image: registry.gitlab.com/gromacs/gromacs/ci-ubuntu-20.04-gcc-11-cuda-11.7.1
      name: al42and-nvidia-test5
      command:
        - /bin/sh
        - "-c"
        - "sleep 24h"
      resources:
        requests:
          cpu: 2
          memory: 2Gi
          nvidia.com/gpu: 2
        limits:
          cpu: 4
          memory: 4Gi
          nvidia.com/gpu: 2
  restartPolicy: Never


$ kubectl create -f test_nvidia5.yml 
pod/al42and-nvidia-test5 created

$ kubectl exec al42and-nvidia-test5 -- nvidia-smi -L
GPU 0: NVIDIA T400 4GB (UUID: GPU-<.....>)