Benchmarks: Add a cross-library comparison harness and shared flop accounting
Adds benchmarks/comparison/ to measure Eigen against a selectable BLAS/LAPACK library and record the conditions needed to reproduce the results. It covers twelve operations: AXPY, DOT, GEMV, GEMM, TRSM and SYRK against BLAS; POTRF, GETRF, GEQRF, SYEV, GESDD and GEEV against LAPACK. Benchmarks register float, double, std::complex<float>, and std::complex<double> cases, except that DOT is real-only. The harness supplies data for the Zen 5 benchmark page; plotting and publication live in the website repository.
Each operation uses one Google Benchmark executable. Eigen and reference implementations share a driver, shape grid, timing policy, and nominal flop count. Implementations are registered adjacently for each shape to reduce drift between measurements.
| Op | Eigen | Reference | Nominal flops (real scalars) |
|---|---|---|---|
| AXPY | y += alpha * x |
?axpy |
2n |
| DOT | x.dot(y) |
?dot (real only) |
2n |
| GEMV | y.noalias() += A * x |
?gemv |
2mn |
| GEMM | C.noalias() += A * B |
?gemm |
2mnk |
| TRSM | A.triangularView<Lower>().solveInPlace(X) |
?trsm, left, lower |
m(m+1)n |
| SYRK | C.selfadjointView<Lower>().rankUpdate(A) |
?syrk / ?herk, lower |
n(n+1)k |
| POTRF | LLT<Ref<M>> in place |
?potrf |
n^3/3 |
| GETRF | PartialPivLU<Ref<M>> in place |
?getrf |
2n^3/3 |
| GEQRF | HouseholderQR<Ref<M>> in place |
?geqrf |
2mn^2 - 2n^3/3 |
| SYEV | SelfAdjointEigenSolver, eigenvectors |
?syev / ?heev, jobz = 'V' |
9n^3 |
| GESDD | BDCSVD<M, ComputeThinU | ComputeThinV> |
?gesdd, jobz = 'S' |
14mn^2 + 8n^3 |
| GEEV | EigenSolver / ComplexEigenSolver, eigenvectors |
?geev, jobvr = 'V' |
25n^3 |
The LAPACK pairs use like-for-like algorithms: tridiagonal QR against ?syev, and divide and conquer against ?gesdd. Eigenvalue and singular-value flop counts follow the classical algorithms (Golub & Van Loan 8.3.3, 7.5.6, table 8.6.1). Both implementations are charged the same count.
Every reference result is checked against Eigen outside the timed region, once per shape. The checks cost O(n^2) where the decomposition allows it:
- GEMM and SYRK: a randomized product check.
- AXPY, DOT and GEMV: direct comparison of the results.
- TRSM:
A(Xy)againstBy. - POTRF, GETRF and GEQRF: factorization residuals applied to a random vector.
- SYEV:
A(Vx)againstV(w \circ x). - GESDD:
A(Vx)againstU(s \circ x). - GEEV: the full residual
AV - V\Lambda, because its eigenvector matrix is not unitary.
Non-finite results are rejected. Destructive operations time one operand copy in both implementations. POTRF, GETRF, GEQRF and SYEV allocate nothing inside the timed region.
vendors.cmake provides nine reference configurations: OpenBLAS, oneMKL, oneMKL with its CPUID vendor check bypassed, AOCL-BLIS, BLIS, ArmPL, NVPL, Accelerate, and netlib. The bypass is compiled into a separately named benchmark configuration and is unsupported by Intel. Library versions and LAPACK provenance are recorded; without LAPACK, the LAPACK operations build with only the Eigen implementation. Explicit library paths support installations that FindBLAS/FindLAPACK cannot locate.
run.py reads a machine profile and builds each ISA and reference configuration. It applies thread caps and optional CPU affinity, then writes per-repetition rates to results/<machine>/<commit>/. Result files record CPU and OS, compiler and flags, Eigen commit, library versions and paths, thread settings, load, and errors. Modified tracked files and excessive load require explicit overrides. The benchmark project requires C++17 and Google Benchmark; the runner requires Python 3.11 or newer.
benchmarks/bench_common.h centralizes flop formulas and the rate counter for the comparison and existing benchmarks. Its adoption corrects bench_bunchkaufman's PartialPivLU count, which previously used twice the symmetric-factorization count, and accounts for complex arithmetic in its factorization benchmarks.
Validation at head b48190d48 on master 63a4163a6: all 24 comparison CTests pass (^bench_.*_compare_(names|smoke)$). These tests run every tiny cell of the twelve binaries for every registered scalar type; DOT is real-only. The checks used a 13th Gen Intel(R) Core(TM) i7-13700HX with GCC 13.3, -O3 -mavx2 -mfma and OpenBLAS 0.3.26.
The OpenBLAS, oneMKL, bypassed oneMKL and AOCL configurations also build on an AMD RYZEN AI MAX+ 395 w/ Radeon 8060S under WSL2 with -O3 -mavx512dq -mfma. In each configuration, run.py --filter re-measured the website page's 874 Eigen cells on a tree identical to this head. The 48 runs produced no errored cell. reuse lint, codespell and clang-format-17 are clean on the branch.
The linked page measures float and double, single-threaded, at AVX-512 on that host. OpenBLAS 0.3.26, oneMKL 2025.3 (standard and bypassed) and AOCL 5.2 were measured on 2026-09-10 and 2026-09-11. Eigen was re-measured at master 63a4163a6 on 2026-09-13.
Complex measurements, the AVX2/SSE2 targets and thread scaling remain follow-up work. Profiles for Zen 4, Zen 2, Apple M4, Neoverse V2, and Cortex-X925 are included but have not been exercised with this version. Related to #3112.
Update: Rebased onto master 63a4163a6, retaining master's removal of duplicate benchmarks. This MR also incorporates the nine operations from rmlarsen1/eigen-new!12 (closed), which had been filed inside the fork and was not visible here. The diff grows from 29 files (+2.5k) to 38 files (+4.5k).