# GPU representation in Ryax This is necessary to improve UX for users: **As a user I can ask for a specific GPU brand or model, of an amount of RAM so my action work out of the box** But also to make Intelliscale recommendation less specific and thus, let the scheduler take the placement decisions. ## GPU in the Infrastructure - **Brand**: Nvidia and AMD are not providing the API and the user code might be adapted to only one of them. - **Model**: Different models provides different advantages and inconvenient and users might want a specific one (only provide the recent ones that are available in the cloud providers) - **Partitioning**: - partitions: with share of the GPU compute and memory for example - 3g.90G => 3/7 compute, 90G memory - 4g.90G or - 1g.10G with a checkbox for homogeneous Brand filter models. ## GPU resource request - **Brand**: Nvidia and AMD are not providing the API and the user code might be adapted to only one of them. - **Model**: Different models provides different advantages and inconvenient and users might want a specific one And amount of resources: - **Amount of GPU**: amount of GPU per node - **Amount of RAM**: total amount of ram available on the GPU ## Details We choose not to get the information from the cloud provider API for now because of the complexity of it. The registration is still declarative. The catalog of cards should bind, Brand, Model, Max number of partition. This can be taen from here: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/supported-gpus.html Max number of partition is useful to check that the proposed partition is valid. Also, it might be necessary for Intelliscale to normalize the compute unit (3 means nothing but 3/7 can be normalized). ## Questions - Should we propose an `Other` field for both brand and model to allow users to add custom GPU with custom max partition number, amount of RAM and compute ? This can be a good way to populate the catalog with an API. ### How to select the right Compute To be able to give compute recommendation we need to be able to compare the GPU between them. To do so, we need an ordering between cards and cards partitions. We can use a score for each GPU compute partition. For example a 4g.40Gb for a H100 that as a score of 100 means 4/7 partition of the card. This seems complicated. Without this score, a simpler UX for the user would be to ask them to select a card among the ones available. This will be a hard constraint. Then, when the recommendation kick-in it provides a partition on this particular card. ## Design ### Simple choice: Model is mandatory Because cards cannot be compared easily regarding the compute (see previous section). The resource requests MUST define a GPU model. A user can provide constraints on the action spec with the GPU model: ```yaml spec: resources: gpu: model: A100 memory_GB: 40G compute_percentage: 50 ``` Then the recommendation of Intelliscale should provide the same kind of values: `{"memory_GB": 35, "compute_percentage": 60}` which will allow to select the right partition on the model. ### Advanced: No requirement (requires a GPU compute score) With a GPU compute score we can leave the brand and the model as constraints and make the recommendation algo only give compute score and memory amount and leave the choice of the model and the brand to the scheduler if there is no constraints. ## TODO - [ ] Change GPU representation in the Runner/Worker - [X] create a catalog API of supported Brand/Model - [X] update node pool creation API: select GPU model and the partition from the catalog (ryax-core!498) - [~] domain: Add a list of GPU models present on the nodes and convert the `gpu` field to a computed property — a node pool now records its catalog brand/model and resolves them into the memory and compute one allocation gets (`NodePool.gpu_partition`, ryax-core!498). The `gpu` **count** field was deliberately left alone: it is what the worker asks Kubernetes for, and making it computed would change what actually gets requested. Reopen if that second half is still wanted. - [X] Recommendation of GPU use memory/compute instead of MIG - [X] Intelliscale must provide GPU memory/compute instead of MIG slice (ryax-intelliscale!33) - [X] Adapt resource recommendation format in all event from Intelliscale domain to PendingExecution (ryax-intelliscale!33, ryax-core!498) - [X] placement service: remove all MIG specific and use registered partition values to find available partition (ryax-core!498) - [X] Repository - [X] Make the GPU resource request format valid for the action spec (ryax-repository!286) — `gpu:` is still a count, or an object `{count, brand, model, memory: 40G, compute_fraction: 0.5}` describing ONE GPU; see the MR for the rules. `brand` is a constraint of its own (any card of a vendor), enforced by the Runner on the pool's catalog brand (ryax-core!510). - [X] Change message to studio format (ryax-core!510) — the message is built by the Action Builder in core, which reads the metadata itself, not by the Repository. - [X] Studio - [X] receive the new format (ryax-studio!495) - [X] expose it in the API (ryax-studio!495) — `resources` on `GET /modules/{id}` and the workflow action views now carry `gpu_brand`, `gpu_model`, `gpu_memory_gb`, `gpu_compute_fraction`. - [ ] UI - [ ] Register GPU using new registration API - [ ] show in action details resource request in the new format (/app/library/action/<action id>) The Runner already accepts the advanced request and reads `gpu_model`, `gpu_memory_gb` and `gpu_compute_fraction` off `WorkflowActionResources` (numbers 5-7) when they arrive; `gpu_brand` (8) comes with ryax-core!510. Studio and UI can land in any order; ryax-repository!286 must land with or after ryax-core!510, or the old Action Builder fails to build an action that uses the object form. ## Merge requests | MR | What | |---|---| | ryax-tech/ryax/ryax-core!475 | GPU catalog: brand, model, MIG profiles, `GET /gpu-models` | | ryax-tech/ryax/ryax-core!498 | Node pools by card, advanced GPU request, placement on memory/compute, first-fit / best-fit | | ryax-tech/ryax/ryax-intelliscale!33 | Recommends GB + share of a card instead of a profile name | | ryax-tech/ryax/ryax-engine!81 | Docs and chart values | | ryax-tech/ryax/ryax-core!510 | Action Builder reads the GPU request object and sends it in `ActionReady`; Runner places on the brand | | ryax-tech/ryax/ryax-repository!286 | Action spec accepts `gpu: {count, brand, model, memory, compute_fraction}` | | ryax-tech/ryax/ryax-studio!495 | Studio stores the request, sends it to the Runner, exposes it in the API | `RYAX_SCHEDULER_GPU_FIT_POLICY` (`first_fit` default, `best_fit`) chooses among the node pools that fit a request. ## Follow-ups this turned up Each is filed separately rather than growing this issue further; all three are linked as related. - ryax-tech/ryax/roadmap#1457 — **Decide whether an action can take more than one MIG slice.** A MIG pool gives one slice per action whatever the count says. May well be correct — MIG instances do not talk to each other the way two cards on NVLink do — so the issue asks the question before proposing code. - ryax-tech/ryax/roadmap#1458 — **Key recommendations by hardware, not by site.** A recommendation is only valid for the machine it was measured on, and site is the wrong key: two node pools in one site can hold different cards. Affects CPU as much as GPU. `_hardware_profile_id` is most of the key already and reaches IntelliScale unused. - ryax-tech/ryax/roadmap#1459 — **Schedule GPUs against node pool availability, not just static capacity.** Placement matches declared capacity and cannot see what is already running, so a burst of GPU executions is all sent to the same pool. A GPU cannot be over-committed the way CPU can: the extra pods sit Pending. Other things worth doing, too small to file unless they grow: - Carry the GPU request into the worker's own view if per-partition Kubernetes resources (`nvidia.com/mig-1g.10gb`) are ever wanted; today Ryax requests `nvidia.com/gpu` and relies on `mig.strategy=single`. - `node_pools.gpu_free_total` is a column with no field and no reader, and `NodePool.gpu_num_total` is never assigned. Either finish the idea behind them (see #1459) or drop them.