## The problem An IntelliScale recommendation is only valid for the hardware it was measured on, and nothing in the system says which hardware that was. IntelliScale keys its model by `(site_id, action_container_image)` (`intelliscale/.../simple_mig_recommender_repository_volatile.py`). The Runner then keys its store the same way and, because it has to apply a recommendation at execution-creation time — before placement has chosen anything — collapses every site's entry for the image into one (`core/ryax/runner/application/intelliscale_recommendation_store.py`, `get_merged_for_action`). **Site is the wrong key.** Two node pools in the same site can hold different cards, and the same card can appear in several sites. So the model mixes measurements from machines that are not comparable — inside IntelliScale, before the Runner's merge ever happens. ## Why it matters The two figures degrade differently: - `gpu_memory_gb` is an observed peak — a property of the workload, so it travels between machines fairly well. - `gpu_compute_fraction` is a share of a *card*. A site with A100s says `3/7`; a site with A30s says `2/4` for the same work. Neither is wrong; they answer different questions. CPU has the same problem and it is easy to miss: two nodes with the same core *count* do not deliver the same throughput, so a CPU recommendation learned on one instance type is not a recommendation for another. Today this is mostly invisible because most clusters are single-site and homogeneous. It bites on a heterogeneous cluster, where the symptom is quiet: an action over-provisioned, or a node pool wrongly judged infeasible because a memory figure learned elsewhere exceeds what its partitions offer. ## The proposal Key recommendations by **hardware**, not by site: GPU brand and model, plus the instance type standing in for the CPU model, which we do not collect. **Most of this already exists.** `_hardware_profile_id` (`core/ryax/runner/application/handlers/execution_handlers.py`) already hashes instance type, architecture, CPU millicores, memory, GPU count, `gpu_mode` and GPU total into an `hp:v1:<digest>`, and already travels to IntelliScale on `CompleteExecutionMetric` (`hardware_profile_id`, field 15) alongside `instance_type` (17), `node_pool_gpu` (22), `node_pool_gpu_mode` (23) and `node_pool_gpu_num_total` (24). IntelliScale parses **none** of them into its domain. Two gaps to close: 1. **`hp:v1` does not distinguish GPU models.** It hashes `gpu_mode`, so two pools that both report `full` collide whether they hold an A100 or an H100. Adding the brand and model from ryax-tech/ryax/ryax-core!498 means an `hp:v2`, which invalidates the existing history keyed on v1 — acceptable for a deliberate bump, but it should be a decision rather than a side effect. 2. **The Runner applies a recommendation before it knows the hardware.** Creation ends by marking the execution PENDING, which is what triggers scheduling, so the node pool is chosen afterwards. Keying by hardware only helps if the lookup happens after placement — which means the recommendation has to survive into scheduling instead of being baked into `resources_allocation` at creation. ## Why (2) is not a small change `resources_allocation` is fixed at creation today, and things lean on that: it is persisted, the OOM-retry path pre-sets it so a bump is never undone, and it is serialised into the payload for the external HTTP scheduler (`core/ryax/runner/infrastructure/scheduler/http_scheduler.py`), so finalising it later is a contract change with an out-of-process component. One shape that avoids re-ordering everything: let placement evaluate each candidate node pool against *that pool's* recommendation, so the effective request becomes per-(execution, node pool) rather than one global value, and write the chosen one onto the execution after placement — where `startV2` already writes `mig_slice`. ## Concrete first step: send the GPU compute fraction The cheapest slice of this, and the one that pays off immediately. It is smaller than it looks: **no protobuf change is needed anywhere**, because both hops already carry a schemaless Struct that is wired end to end. ``` Runner ──StartExecutionV2.metadata──▶ Worker ──ExecutionMetricsUpdated.metrics──▶ IntelliScale ``` - `StartExecutionV2` has `google.protobuf.Struct metadata = 10`, commented *"Put any metadata field this struct"*. The Runner currently sends `metadata=Struct(), # Empty for now` (`core/ryax/runner/infrastructure/messaging/handlers/execution_rmq_handler.py`), and the worker already parses it with `MessageToDict` into `execution.metadata` (`core/ryax/worker_k8s/infrastructure/rabbitmq/handlers/execution_rmq_handlers.py`). - `ExecutionMetricsUpdated.metrics` is a `Struct` built from the worker's metrics dict, so a new key reaches IntelliScale without touching a `.proto`. ### Why the Runner has to be the one to answer It is the only component holding both halves: the node pool's registered brand and model, and the catalog to resolve them against. `NodePool.gpu_partition` already yields the exact `compute_fraction`. Having the worker read DCGM's `modelName` label instead — off a sample it already filters — looks cheaper but lands wrong: DCGM reports `NVIDIA A100-SXM4-40GB` where the catalog says `A100 40GB`, so it needs a name-normalisation table that will rot. And IntelliScale cannot resolve a model name itself without a second copy of the catalog, which is worse than the assumption it replaces. ### Where to stamp it `v2_execution_handlers.startV2` already fetches the node pool (it sets `mig_slice` from it), so the value is in hand at exactly the right moment — **after** placement has chosen. That matters: with `best_fit` the pool an execution lands on may not be the one that was recommended, and it is the former we want to learn from. `StartExecutionV2Trigger` carries `node_pool_id` today and would carry the fraction alongside it. ### Cost | Where | Change | |---|---| | Runner | Put `gpu_compute_fraction` on the trigger and into the existing `metadata` Struct | | Worker | Carry it from `execution.metadata` into the metrics dict — one field on the metrics dataclass | | IntelliScale | Parse one key in `_build_execution_metric`; prefer it in `ComputeShareLadder.observed_share`, falling back to the configured denominator when absent | Roughly 40 lines plus tests, across three repos. No new message, no correlation by execution id, no ordering problem between events. ### What it buys `total_compute_slices` stops being a guess applied cluster-wide. It is currently a single setting for every site, so an A30 pool (4 slices) has its observations read as if the card split into 7 — about 43% too small — and the model learns from a distorted input. With the real fraction it becomes a fallback for workers that predate the change, and can eventually go. This does not solve the keying problem above; a recommendation is still stored per site and merged across hardware. It removes the last hardware assumption from *reading* observations, which is the prerequisite for keying them properly. ## Suggested ordering 1. The compute fraction above — small, self-contained, useful on its own. 2. Settle the key: is `hp:v2` (instance type + GPU model) the right granularity, or is `(gpu_model, instance_type)` on its own enough and more legible? 3. IntelliScale starts keying by hardware. It can do this without the Runner changing at all — it only needs to parse fields it already receives. 4. Resolve the recommendation after placement, which is the structural part. Related: - ryax-tech/ryax/roadmap#1421 — Better GPU representation in Ryax - ryax-tech/ryax/ryax-core!498 — introduced the merge this replaces, and the GPU brand/model an `hp:v2` would need - ryax-tech/ryax/ryax-intelliscale!33 — where `total_compute_slices` and `observed_share` live