## The problem Placement matches a request against a node pool's **static capacity**. It has no idea what is already running there. `NodePool.has_enough_resources` compares the request to the pool's declared cpu/memory/gpu — the figures an operator typed at registration. Nothing anywhere tracks how much of that is currently in use, and `site_placement_service.schedule_one` is handed a plain `list[Site]` with no occupancy attached. It gets sharper one level up. `InternalScheduler._place` places every pending execution in a round against the *same unmodified* `sites`: ```python for connection in connections: node_pool = self.site_placement_service.schedule_one(sites, connection) ``` Nothing is decremented between iterations. Ten GPU executions in one round all "fit" the same single-GPU pool, and all ten are told to start there. ## Why it hurts GPUs specifically CPU and memory over-commit degrades: Kubernetes packs the pods, they get throttled, work still finishes. A GPU is not divisible after the fact. A pod that asks for `nvidia.com/gpu: 1` on a node whose GPUs are all taken does not run slowly — it sits `Pending` until something frees, or forever if the pool cannot scale up. From Ryax's side the execution has been "started" and simply never progresses. MIG makes it likelier, not less: a pool advertising eight slices looks eight times roomier to this code than it is to the next execution. The GPU model added in ryax-tech/ryax/ryax-core!498 makes placement much more precise about *shape* — brand, model, memory, share of a card — which makes it more obvious that the one thing it still cannot see is whether the partition it picked is free. ## Prior art, half-built The schema remembers an attempt at this: - `node_pools.gpu_free_total` is a real column (`core/ryax/runner/infrastructure/database/metadata.py`) with **no domain field and no code that reads or writes it**. - `NodePool.gpu_num_total` is documented as "the number of gpu (or MIG slices) inside the whole node pool (no matter occupied or not)" and carries its own `TODO: Check for depreciation`. Nothing assigns it either; it is only ever read, to be forwarded to IntelliScale. Worth deciding whether to finish that idea or drop the leftovers. ## Directions, roughly cheapest first 1. **Account for the round.** Deduct each allocation from a working copy of the pools as `_place` walks the list. Purely in-memory, no new source of truth, and it removes the worst case — a burst of executions all sent to one pool. 2. **Count what Ryax itself has placed.** The Runner knows which executions are STARTED/RUNNING and on which `node_pool_id`, so occupancy can be derived from the execution table rather than tracked separately. 3. **Ask the cluster.** The worker knows real allocatable-vs-allocated, and Ryax is not the only thing on these nodes — daemonsets and other workloads take capacity too. Most accurate, most plumbing, and it needs a freshness story so placement is not deciding on stale numbers. There is a pattern to copy for this rather than invent — see below. (1) and (2) are worth having regardless of (3): they fix Ryax over-committing against itself, which is the failure people will hit first. ## A pattern for (3): Karmada's cluster resource modeling <https://karmada.io/docs/userguide/scheduling/cluster-resources/> Karmada schedules workloads across member clusters, so its scheduler picks a **cluster** and the member's own kube-scheduler picks the node. That is exactly our shape: we pick a **node pool**, and Kubernetes picks the node inside it. Which means Karmada hit our problem first, including the part that makes the naive fix wrong. ### Why a single free-capacity number is not enough An aggregate "16 GPUs free in this pool" cannot answer "does this request fit", because free capacity says nothing about how it is **distributed**. Sixteen free GPUs spread one-per-node do not satisfy a request for two on one node. Spike ryax-tech/ryax/roadmap#1418 reached the same conclusion from the other direction: *"any fit-model must be per-node observed state, not aggregate (fragmentation)"*. The obvious repair — ship every node's free resources to the Runner — puts the whole cluster's node state on the scheduling path. Karmada says plainly that this is the expensive part: resource modeling *"will collect node and pod information from all clusters"* and carries *"a considerable performance burden in large-scale scenarios"*, which is why it ships a switch, `--enable-cluster-resource-modeling=false`. ### What Karmada does instead Bucket the nodes, then send **counts per bucket**. - `spec.resourceModels` defines *grades* — ordered buckets, each a set of `ranges` over `cpu`, `memory`, `storage` or `ephemeral-storage`: ```yaml - grade: 3 ranges: - name: cpu min: "4" max: "8" - name: memory min: 32Gi max: 64Gi ``` - `status.resourceSummary` then carries `allocatable`, `allocating`, `allocated` — and `allocatableModelings`, a list of `{grade, count}`: *"the number of nodes that own the resources delineated by this modeling"*. `{grade: 2, count: 10}` means ten nodes currently look like grade 2. - The scheduler estimates what fits from the histogram alone, roughly `count * min(available / requested)` summed over the grades that qualify. The histogram is the trick: it keeps the *shape* of the cluster — which is what fragmentation destroys — while staying a handful of integers per pool. ### The Ryax mapping | Karmada | Ryax | |---|---| | member cluster | node pool | | `karmada-agent` on the member | the **worker**, one per site — already scraping the K8s API | | `spec.resourceModels` | the GPU catalog we already have, plus CPU/memory ranges | | `status.resourceSummary.allocatableModelings` | a per-pool free-partition histogram on the heartbeat | | `karmada-scheduler` reading the summary | `NodePool.has_enough_resources` | Our version is **simpler than Karmada's on the dimension that matters**, because a GPU is indivisible and the partitions are already enumerated. Karmada has to invent range buckets because CPU is continuous; our grade is just the partition name. A pool's summary is then: ``` {"full": 2, "mig-3g.20gb": 5, "mig-1g.10gb": 0} # free, per node-shape ``` and `has_enough_resources` stops asking *"is the declared total bigger than the request"* and starts asking *"is there a free partition at least as large as this request"* — which is the same catalog comparison `_gpu_fit_waste` already does, against observed counts instead of a typed-in number. ### Where it lands - **Worker** computes the histogram from the K8s API — it is the only component that can see real allocation, including the daemonsets and non-Ryax pods that direction 3 is about. Cost sits on the worker, not on the scheduling path. - **Heartbeat** carries it. ryax-tech/ryax/roadmap#1446 is already adding worker status over the heartbeat, which is the natural vehicle and would make this cheap. - **Runner** keeps it beside the static capacity, never replacing it. - **Freshness**, which this issue already asks for: stamp the summary with its observation time and fall back to static capacity once it is older than some bound. A stale histogram must degrade to today's behaviour, not to a refusal. - **A switch**, as Karmada has one. On a small cluster the static check is fine and the histogram is pure cost. ### It does not replace (1) The histogram says what was free when the worker last looked. It cannot stop a single scheduling round from sending ten executions at the same free partition, because nothing decrements between iterations — that is direction (1), and it stays necessary. Karmada has the same split: the summary informs the estimate, and the scheduler still has to account for what it has just decided. ## Worth settling first - Is over-committing sometimes *right*? A pod queued behind a running one is not always wrong — it is how you keep a GPU busy. The goal is probably a bounded queue, not a hard refusal, which is a scheduling policy question rather than a data one. - How does this interact with cluster autoscaling? Refusing to place because a pool is full may be exactly what prevents a scale-up that would have served the request. - Does the external HTTP scheduler (`core/ryax/runner/infrastructure/scheduler/http_scheduler.py`) need occupancy in its payload too? Today it receives the same static picture. Related: - ryax-tech/ryax/roadmap#1421 — Better GPU representation in Ryax - ryax-tech/ryax/ryax-core!498 — the GPU shape matching this would complete