Substrate Miner Integration

Summary

This MR rolls the v0.2 integration branch into main. It reframes Quip from a standalone P2P blockchain into a Substrate-attached miner: there is no longer a Quip-native peer-to-peer protocol, block store, sync layer, or consensus stack in this repository. The miner subscribes to an external validator over websocket, derives work from new heads (PoW) or a mempool queue, and submits results as extrinsics signed with a hybrid sr25519 + ML-DSA-44 keystore.

The change set is now 462 commits across 287 files (+56,507 / -26,359, net +30,148). The deletion sweep that opened this work still removes the entire P2P + REST + custom-telemetry codebase; the net diff turned positive as the unified streaming stack, the threads→multiprocessing rework, the Metal utilization governor, and the QPU throughput-decoupling + daily-budget work landed on top. Every phase landed via its own MR (!77 (merged) through !112 (merged)) and was reviewed in isolation; this MR is the integration step that promotes that work to main.

There is no backwards-compatibility layer with v0.1 — CLI, config schema, wire protocol, signature scheme, and genesis ownership all change. Operators upgrading need to repoint at a Substrate validator, generate a fresh hybrid keystore, and replace quip-network-node … with quip-miner cpu/gpu/qpu … in their service files.

Architecture shift

Before (v0.1, on main)	After (v0.2)
Standalone P2P node running its own chain (QUIC, peer trust, block sync, REST)	Stateless miner subscribed to a Substrate validator over websocket
Internal block.py, block_store.py, block_synchronizer.py, network_node.py, quic_*.py, node.py, node_client.py, rest_api.py, trust_store.py, peer_*.py, swim_detector.py, sync_messages.py	substrate_client.py, substrate_miner_controller.py, substrate_submitter.py, substrate_types.py, mempool_miner_controller.py, mempool_types.py, validator_pool.py, miner_core.py, work_context.py
block_signer.py (in-house ECDSA + WOTS+)	hybrid_signer.py + keystore_hybrid.py (sr25519 + ML-DSA-44, fed to substrate extrinsics)
Custom telemetry (telemetry.py, telemetry_cache.py, telemetry_sink.py, telemetry_aggregator.py, system_info.py, load_monitor.py)	Lightweight telemetry_api.py (HTTP read-only endpoint when --rest-port > 0)
quip-network-node + quip-network-simulator CLI	quip-miner group: cpu / gpu / qpu / mempool / keygen / register-solver / deregister-solver / bootstrap
genesis_block_public.json shipped in repo	Genesis owned by the validator; miner is stateless re: chain config

Net shared/ change: +11,471 / -13,121 lines across 60 files.

Mining stack — unified streaming + process model

Since !89 (merged) the miner's execution model was rebuilt. Every backend (QPU, Metal, CUDA, CPU, Modal) and both job types (PoW, mempool) now mine through one path: a stream-driver subprocess produces samplesets into a shared-memory ring, and the worker consumes the ring and runs evaluate / ratchet / stash. There is no synchronous/inline sampling path and no mode toggle. tools/lint_no_inline_sampling.py runs in CI to keep the deleted inline symbols (_sample / _sample_batch / STREAMING_PUMP / DRIVER_OWNS_FEEDER) from reappearing. Full write-up in docs/miner-architecture.md.

Layer	Responsibility	New modules
Orchestrator	chain head + submit; dispatches (job kind, feeder spec, round params) over ctl_q	shared/base_miner.py, shared/miner_worker.py
Ring	bulk (sample, energy) arrays in shared memory; metadata rides a descriptor queue	shared/shared_ring.py, shared/ring_views.py
Stream driver (subprocess)	owns feeder + sampler; one process per device	shared/stream_context.py, QPU/stream_driver.py, per-backend *_stream.py (CPU/sa_stream.py, GPU/metal_stream.py, GPU/cuda_stream.py, GPU/modal_stream.py, QPU factory in QPU/dwave_miner.py)

The sampler is the only per-backend difference; the feeder (pow / mempool, shared/ising_feeder.py) is the only per-job difference. Each backend exposes a build_persistent_context(...) factory that builds its sampler and returns a backend- and job-agnostic StreamContext. Metal init failure crashes the driver — there is no CPU fallback.

Supporting work that landed alongside the streaming stack:

• threads → multiprocessing — per-connection process isolation; the worker op-loop and stream driver are separate processes, not coroutines sharing one asyncio loop.
• QPU throughput (QPU/qpu_time_manager.py, QPU/stream_driver.py) — persistent stream driver replaces per-dispatch reconnect; decay-as-threshold + generation filtering preserve aggressive-submit; daily-budget accounting per solution.
• Metal utilization governor (GPU/macos_sensors.py, GPU/util_monitor.py, GPU/gpu_scheduler.py, GPU/metal_scheduler.py) — sensor-driven (HID / thermal / battery / displays) occupancy-budget cap with stop-aware pause; independent of the CUDA monitor.
• telemetry — shared/telemetry_process.py + shared/stats_snapshot.py serve /api/v1/status and /api/v1/stats (per-miner mode + mempool guard; qpu_access_time_us per attempt; redacted secrets).

Phase MRs (already reviewed individually)

!77 substrate foundation → !78 faucet bootstrap → !79 mine-work-item → !80 controller + miner_core → !81 deletion sweep → !82/!83 hybrid signer + faucet → !84 live verification → !85–!88 mempool stack and concurrent mode → !89 validator list + HA failover → mine-work-item refactor + on-disk solution keying → !110 QPU throughput decoupling → !111 unified streaming stack (shared ring) → !112 test-suite cleanup, plus the Metal governor series.

CLI surface

The console entry point quip-network-node (v0.1) is removed. New entry point is quip-miner:

quip-miner keygen                              # generate a hybrid keystore
quip-miner bootstrap --validator <ws>          # one-shot reachability + funding check
quip-miner register-solver --signer-key <fp>   # register on-chain as a solver
quip-miner deregister-solver --signer-key <fp>
quip-miner cpu     --validator <ws> [--mode pow|mempool|both] [--config <toml>]
quip-miner gpu     --validator <ws> [--gpu-backend local|modal] [--mode ...]
quip-miner qpu     --validator <ws> --daily-budget <usd>
quip-miner mempool --validator <ws>            # mempool-only worker

All of cpu/gpu/qpu/mempool/bootstrap/register-solver/deregister-solver accept repeated --validator URL for failover and --config path.toml for TOML configuration (CLI > TOML > defaults).

TOML schema ([miner])

Wired through today: validators, signer_key, faucet_url, rest_port, rest_host. Reserved but parsed-but-ignored: topology, [miner.cpu], [miner.gpu], [miner.qpu]. See quip.network.{cpu,gpu,qpu}.example.toml.

Migration map from the v0.1 [global] schema is in the !89 (merged) description under "Hand-off note for nodes.quip.network".

Backwards compatibility

None — this is a hard cut. Specifically:

• CLI: quip-network-node gone, no alias; --node-url removed in favor of --validator.
• Config: [global] schema rejected; the only schema is [miner].
• Wire format: there is no Quip-native P2P protocol anymore; the miner speaks substrate-interface to a validator.
• Genesis: removed from the repo. The validator owns chain config.
• Signature: ECDSA+WOTS+ keystores from v0.1 are not interoperable with the new hybrid keystore — operators run quip-miner keygen fresh.

CI / images

• :v0.2-preview Docker tag publishes from the v0.2 branch (per .shared_image_rules anchor in .gitlab-ci.yml).
• Merging this MR will make the v0.2 line publish :latest.

Verification

• All phase MRs (!77 (merged)–!112 (merged)) were individually code-reviewed and merged green.
• Phase 6 (!84 (merged)) confirmed live miner ↔️ validator end-to-end on a dev chain: head subscription, extrinsic submission, faucet flow.
• Per-phase test files were added alongside each phase; the suite under tests/ was net-reduced as ~20 obsolete P2P/consensus test files were deleted alongside the production code they covered.

Known follow-ups (not blocking this MR)

Carried over from !89 (merged) review:

• _run failover race — second concurrent caller can hit iface = None mid-reconnect; ValidatorPool.advance_rotation reduces but doesn't fully close the in-client race. Proper fix needs lock-semantics changes across _run, _raw_run, and reconnect().
• Faucet test coverage — success path and wallet-faucet-failed path lack unit coverage; a faucet integration harness is a separate workstream.
• Test edge cases — concurrent _run race, TimeoutError outside ConnectionError, shutdown-during-reconnect, empty-TOML round-trip, single-URL rotation.

Test plan (pre-merge)

• Pipeline green on v0.2 HEAD
• End-to-end smoke: quip-miner cpu --validator <ws> --signer-key <hybrid-keystore> mines and submits at least one PoW extrinsic on a dev chain
• quip-miner mempool --validator <ws> picks up a queued mempool work item and submits a result
• quip-miner cpu --mode both runs both controllers in one process without _call_lock deadlock
• Two-validator run; kill primary mid-mining → automatic failover with head subscription dropped …; failing over log line, no controller restart needed
• quip-miner cpu --config ./quip.network.cpu.example.toml happy path with TOML-only configuration