Key IntelliScale recommendations by hardware, not by site

The problem

An IntelliScale recommendation is only valid for the hardware it was measured on, and nothing in the system says which hardware that was.

IntelliScale keys its model by (site_id, action_container_image) (intelliscale/.../simple_mig_recommender_repository_volatile.py). The Runner then keys its store the same way and, because it has to apply a recommendation at execution-creation time — before placement has chosen anything — collapses every site's entry for the image into one (core/ryax/runner/application/intelliscale_recommendation_store.py, get_merged_for_action).

Site is the wrong key. Two node pools in the same site can hold different cards, and the same card can appear in several sites. So the model mixes measurements from machines that are not comparable — inside IntelliScale, before the Runner's merge ever happens.

Why it matters

The two figures degrade differently:

  • gpu_memory_gb is an observed peak — a property of the workload, so it travels between machines fairly well.
  • gpu_compute_fraction is a share of a card. A site with A100s says 3/7; a site with A30s says 2/4 for the same work. Neither is wrong; they answer different questions.

CPU has the same problem and it is easy to miss: two nodes with the same core count do not deliver the same throughput, so a CPU recommendation learned on one instance type is not a recommendation for another.

Today this is mostly invisible because most clusters are single-site and homogeneous. It bites on a heterogeneous cluster, where the symptom is quiet: an action over-provisioned, or a node pool wrongly judged infeasible because a memory figure learned elsewhere exceeds what its partitions offer.

The proposal

Key recommendations by hardware, not by site: GPU brand and model, plus the instance type standing in for the CPU model, which we do not collect.

Most of this already exists. _hardware_profile_id (core/ryax/runner/application/handlers/execution_handlers.py) already hashes instance type, architecture, CPU millicores, memory, GPU count, gpu_mode and GPU total into an hp:v1:<digest>, and already travels to IntelliScale on CompleteExecutionMetric (hardware_profile_id, field 15) alongside instance_type (17), node_pool_gpu (22), node_pool_gpu_mode (23) and node_pool_gpu_num_total (24).

IntelliScale parses none of them into its domain.

Two gaps to close:

  1. hp:v1 does not distinguish GPU models. It hashes gpu_mode, so two pools that both report full collide whether they hold an A100 or an H100. Adding the brand and model from ryax-core!498 (merged) means an hp:v2, which invalidates the existing history keyed on v1 — acceptable for a deliberate bump, but it should be a decision rather than a side effect.
  2. The Runner applies a recommendation before it knows the hardware. Creation ends by marking the execution PENDING, which is what triggers scheduling, so the node pool is chosen afterwards. Keying by hardware only helps if the lookup happens after placement — which means the recommendation has to survive into scheduling instead of being baked into resources_allocation at creation.

Why (2) is not a small change

resources_allocation is fixed at creation today, and things lean on that: it is persisted, the OOM-retry path pre-sets it so a bump is never undone, and it is serialised into the payload for the external HTTP scheduler (core/ryax/runner/infrastructure/scheduler/http_scheduler.py), so finalising it later is a contract change with an out-of-process component.

One shape that avoids re-ordering everything: let placement evaluate each candidate node pool against that pool's recommendation, so the effective request becomes per-(execution, node pool) rather than one global value, and write the chosen one onto the execution after placement — where startV2 already writes mig_slice.

Concrete first step: send the GPU compute fraction

The cheapest slice of this, and the one that pays off immediately. It is smaller than it looks: no protobuf change is needed anywhere, because both hops already carry a schemaless Struct that is wired end to end.

Runner ──StartExecutionV2.metadata──▶ Worker ──ExecutionMetricsUpdated.metrics──▶ IntelliScale
  • StartExecutionV2 has google.protobuf.Struct metadata = 10, commented "Put any metadata field this struct". The Runner currently sends metadata=Struct(), # Empty for now (core/ryax/runner/infrastructure/messaging/handlers/execution_rmq_handler.py), and the worker already parses it with MessageToDict into execution.metadata (core/ryax/worker_k8s/infrastructure/rabbitmq/handlers/execution_rmq_handlers.py).
  • ExecutionMetricsUpdated.metrics is a Struct built from the worker's metrics dict, so a new key reaches IntelliScale without touching a .proto.

Why the Runner has to be the one to answer

It is the only component holding both halves: the node pool's registered brand and model, and the catalog to resolve them against. NodePool.gpu_partition already yields the exact compute_fraction.

Having the worker read DCGM's modelName label instead — off a sample it already filters — looks cheaper but lands wrong: DCGM reports NVIDIA A100-SXM4-40GB where the catalog says A100 40GB, so it needs a name-normalisation table that will rot. And IntelliScale cannot resolve a model name itself without a second copy of the catalog, which is worse than the assumption it replaces.

Where to stamp it

v2_execution_handlers.startV2 already fetches the node pool (it sets mig_slice from it), so the value is in hand at exactly the right moment — after placement has chosen. That matters: with best_fit the pool an execution lands on may not be the one that was recommended, and it is the former we want to learn from. StartExecutionV2Trigger carries node_pool_id today and would carry the fraction alongside it.

Cost

Where Change
Runner Put gpu_compute_fraction on the trigger and into the existing metadata Struct
Worker Carry it from execution.metadata into the metrics dict — one field on the metrics dataclass
IntelliScale Parse one key in _build_execution_metric; prefer it in ComputeShareLadder.observed_share, falling back to the configured denominator when absent

Roughly 40 lines plus tests, across three repos. No new message, no correlation by execution id, no ordering problem between events.

What it buys

total_compute_slices stops being a guess applied cluster-wide. It is currently a single setting for every site, so an A30 pool (4 slices) has its observations read as if the card split into 7 — about 43% too small — and the model learns from a distorted input. With the real fraction it becomes a fallback for workers that predate the change, and can eventually go.

This does not solve the keying problem above; a recommendation is still stored per site and merged across hardware. It removes the last hardware assumption from reading observations, which is the prerequisite for keying them properly.

Suggested ordering

  1. The compute fraction above — small, self-contained, useful on its own.
  2. Settle the key: is hp:v2 (instance type + GPU model) the right granularity, or is (gpu_model, instance_type) on its own enough and more legible?
  3. IntelliScale starts keying by hardware. It can do this without the Runner changing at all — it only needs to parse fields it already receives.
  4. Resolve the recommendation after placement, which is the structural part.

Related:

Edited by Michael Mercier