Build a CI harness that, for a merge request, measures the benchmarks its diff can reach by running the base and the head **interleaved on one machine inside a single job**, and reports the per-benchmark ratio. The value is entirely in the interleaving: a ratio measured against a co-resident control is meaningful even when the absolute numbers are not, which is the opposite of what a scheduled pipeline comparing this week's host against thirty previous ones can offer. This is the CI half of performance work. The standalone suite and the publishing of absolute numbers are the other half and are tracked separately in !2903 and #3112; they answer "how fast is Eigen", while this answers "did this merge request change anything". ### Why a cross-run comparison cannot answer it The existing scheduled pipeline compares a run against history stored on a `perf-data` branch. Two independent problems make that unable to detect a regression. Hosts vary between runs. The same job, `bench:run:x86-64:sse`, drew materially different hardware two days apart: | | 2026-09-06 | 2026-09-08 | |---|---|---| | Reported clock | 2450 MHz | 2696 MHz | | L1 data cache | 32 KiB | 48 KiB | | L2 cache | 512 KiB | 1 MiB | Blocked GEMM, TRSM and GEMV throughput is governed by the cache hierarchy, so a test across runs at the configured 5% threshold largely resolves which host the job drew. Interleaving both sides inside one job removes this term entirely. And it has never produced data to compare against: the `perf-data` branch does not exist, and `bench:analyze` was skipped or canceled in every scheduled run back to 2026-07-02. Whether to retire that pipeline is a separate decision under #3043. ### Proposed shape 1. **Select.** Map the diff to benchmark targets the same way `scripts/affected_tests.py` maps it to test targets, over the same include graph. `benchmarks/*` currently sits in that script's `IGNORED_PATTERNS`, so the graph exists and the benchmark trees are simply excluded from it today. 2. **Build both sides.** Compile the selected benchmark sources twice, against the merge base and against the head, one binary per benchmark family. 3. **Run interleaved.** Alternate the two binaries within one job, several rounds, one process per run, pinned, reporting medians with the observed spread. 4. **Report, do not gate.** Post the ratio table as an artifact or a note. A hard gate on a noisy measurement produces red pipelines that reviewers learn to ignore. 5. **Opt in by label.** The established pattern is `affected-tests` and `all-tests`; a benchmark label fits alongside them and keeps this off the default path. ### Constraints that have to be designed in, not discovered later These are measured on this project and each one has already invalidated a result. - **One benchmark family per translation unit.** Registering several families in one `.cpp` produces deltas that are pure code layout. On !2940 a combined TU read +26% and −18% on two `stableNormalize` instantiations; both collapsed to within 1% once each family got its own TU, and a `makeGivens` "1.3x regression" on an untouched path vanished the same way. - **The same effect exists one level up.** Two separately linked static libraries moved *untouched* routines by up to 28%. Where an A/B needs a library, the working construction is a shared parent library with only the changed routines compiled into the two executables, both linked against one byte-identical image. - **Carry a control.** An untouched benchmark in the same pair bounds the layout and placement noise, and a delta is only believable when it clears the control's spread. - **Ratios, not absolutes.** Nothing from a shared cloud runner should be published as an absolute number. - **Check the arguments reach the change.** A registered grid that never leaves a safe range measures nothing; this has happened here. ### Open questions - **Which runner.** Shared cloud runners give a consistent host *within* a job, which is all the interleaving needs, so they may be sufficient. A pinned self-hosted runner would additionally make numbers comparable across MRs, at the cost of scheduling. - **Runtime budget.** Selection has to be capped. A change under `Eigen/src/Core` reaches essentially every benchmark, which is exactly the case where the test selector degrades to the full suite; the harness needs a bound and a way to narrow it by hand. - **Presentation.** Artifact, JUnit report, or a note on the merge request, and how to show a spread rather than a single number. <details> <summary>Evidence for the host-variance table</summary> Google Benchmark context blocks from the two job traces, same job name, two days apart: ``` job 16330045743 (2026-09-06, scheduled) "host_name": "runner-jjqcuhrnc-project-15462818-concurrent-0", "num_cpus": 32, "mhz_per_cpu": 2450, "cpu_scaling_enabled": false, "size": 32768, <- L1 "size": 32768, "size": 524288, <- L2 job 16357140153 (2026-09-08, web) "host_name": "runner-3sk86exer-project-15462818-concurrent-0", "num_cpus": 32, "mhz_per_cpu": 2696, "cpu_scaling_enabled": false, "size": 49152, <- L1 "size": 32768, "size": 1048576, <- L2 ``` The CPU models are not quoted here because `run.benchmark.sh` captures the model name only into the combined results file, which the run-job timeouts meant was never written. Scheduled-run outcomes: ``` 2026-09-06 bench:run x3 failed job_execution_timeout (3 h each, no results artifact) 2026-08-30 bench:run failed script_failure, remaining jobs canceled 2026-08-02 bench:run x3 failed script_failure 2026-07-02 bench:run x4 failed script_failure ``` `GET /projects/15462818/repository/branches/perf-data` returns 404. </details>