## The restriction An action asking for a GPU on a node pool running MIG gets **exactly one slice**, whatever `resources.gpu` says. In `core/ryax/runner/domain/site/site_entities.py`: ```python else: # Has GPU request, and the GPU on this site is in any MIG mode # Only support the request of 1 GPU to enable MIG intelliscale. enough_gpu = resource_request.gpu <= 1 ``` A request for 2 is refused even on a pool with slices to spare. It is a rule, not a shortage — the message says so since ryax-tech/ryax/ryax-core!498, which is also where the surrounding GPU model changed and made this worth revisiting. It arrived in `b5c3afcc` ("VPA-Pilot with GPU MIG support", 2025-01-17), as a simplifying assumption that let IntelliScale reason about MIG at all: one execution, one slice, one profile to recommend. ## Why it might be liftable now Ryax action pods request `nvidia.com/gpu`, and the NVIDIA device plugin in `mig.strategy=single` advertises each MIG instance under that name. The worker already emits the requested count verbatim (`core/ryax/worker_k8s/domain/execution/execution_entities.py`): ```python spec["resources"]["limits"]["nvidia.com/gpu"] = str(self.gpu) ``` So on the Kubernetes side a pod asking for 2 slices would most likely just be scheduled. Nothing above it is ready for that, though. ## The question to settle first **Is there a real workload that wants this?** This should be answered before any code, because the answer may be no. MIG instances are isolated by design: they do not support peer-to-peer between each other the way two whole GPUs on NVLink do. A multi-GPU workload — anything doing NCCL collectives, model parallelism, or large-batch data parallelism — wants two *whole* cards, not two slices of one. If that holds, the cap is not a limitation so much as a statement of what MIG is for, and the right outcome is to document it and close this. The counter-case worth checking: an action running several independent inference workers in one pod, each pinned to its own slice. That is a genuine pattern and MIG suits it. Whether any Ryax user runs it is the thing to find out. ## What would have to change if we do lift it Not just the one condition — the count assumption runs through the whole GPU path: - **`has_enough_resources`** — the MIG branch, plus deciding what "enough" means: N free slices on one node, or N anywhere in the pool. - **IntelliScale** — `MIGProfile` is `(sm, memory, name)` with no count, and `recommend()` returns one profile (`intelliscale/ryax/intelliscale/domain/algorithms/simple_mig_recommender/`). Either the recommendation gains a count, or it is defined as per-slice and something else decides how many. See ryax-tech/ryax/ryax-intelliscale!33 for the shape it has now. - **`best_fit`** — `_gpu_fit_waste` measures one partition against the request. With N slices, waste is presumably N x per-slice, which changes the ranking against a whole-card pool. - **The GPU-OOM retry ladder** — `_next_gpu_memory_gb` climbs to the next larger partition. With a count, should a retry take a bigger slice, or more slices? They are not equivalent: more slices does not give a single process more addressable VRAM. - **Accounting** — `allocated_gpu / node_total_gpu` in `core/ryax/runner/domain/accounting/accounting_entities.py`, which already carries the open question verbatim: > TODO: I don't know how it behaves when we have mig enabled with on a node > with more that one GPU ? - **Docs** — `docs/docs/howto/gpu_node_pools.md` states the one-slice rule (ryax-tech/ryax/ryax-engine!81). ## Suggested first step A SPIKE, not an implementation: confirm whether a two-slice pod actually schedules and runs usefully on a MIG node, and whether any real Ryax workload wants one. If the answer is no, close this and keep the documented rule. Related: ryax-tech/ryax/roadmap#1421 (Better GPU representation in Ryax), which introduced the memory/compute model this would have to extend.