fix: keep CUDA nonces alive when the feeder momentarily starves

Problem (gh-19 / QUI-828)

CUDA miner attempt rate decayed ~40% over a long run — 100% GPU util at falling board power (spin-wait), full clock, cool die; a container restart instantly restored it. Onset varied 50–100 min, so it's pipeline starvation, not a hardware throttle.

Root cause: the self-feeding streaming loop (GPU/base_cuda_sampler.py::_run_streaming_loop) dropped a nonce whenever the host feeder was momentarily empty at a completion boundary. On completion it set active_slot = next_slot; if try_pop() had returned None (which happens on every new-head generation switch, shared/ising_feeder.py:441), next_slot was -1 and active_model became None — after which the loop skipped that nonce forever (if ss.active_model is None: continue) and its GPU block sat idle. Each starvation killed one block, so throughput fell block-by-block until the ctrl array reset on restart. A second consecutive starvation also overwrote the single free_slot int, leaking a physical slot.

Fix

  • Extract the host-side slot bookkeeping into a cupy-free, unit-testable GPU/slot_rotation.py (SlotState).
  • rotate_on_completion promotes NEXT -> ACTIVE if queued, otherwise goes idle without dropping the nonce; the loop revives idle nonces from the feeder once it recovers.
  • Free slots are derived from {0..N} − {active, next}, so a slot can never leak.
  • The loop terminates only when the feeder is exhausted and every nonce is drained, so a transient empty feeder no longer ends the stream either.

Tests

tests/test_slot_rotation.py (pure-Python, runs anywhere — no GPU): the legacy no-revive behaviour collapses the active-nonce count to zero under intermittent starvation (reproduces the decay), the fix holds it steady, and slots never leak across repeated starvation cycles. 8 tests, ruff clean.

The GPU integration path (_run_streaming_loop) is exercised by CI's CUDA runner; a live multi-hour run is still needed to confirm the decay is gone in production.

Closes gh-19 / QUI-828.

Merge request reports

Loading