SME: thread cap at the unit count, in-place LHS, ZA packers, kernels for narrow and tiny results

Michael Teschrequested to merge
tesch1/eigen:sme-phase2-3 into master

Single-thread improvements to the SME GEMM backend from !3160 (merged), and a cap on the thread count at the number of SME units, measured on Apple M4 Pro. Against master, float GEMM gains 1.16x (geometric mean over bench_gemm, up to 2x at 8^3 and 1.9x at 32^3) and double 1.14x; the decompositions gain 1-7%. On one thread this branch is 1.12x (column-major) and 1.18x (row-major) of Accelerate's cblas_sgemm on the 24 workloads of the MpGEMM paper, and 1.02-1.13x on squares from 1000^3 up. On a 12-thread pool, float 1024^3 and 2048^3 go from 1030 and 1362 GFLOPS uncapped to 2729 and 2847 with the cap.

The design elements come from MTGEMM-A, a standalone reproduction of the MpGEMM paper (arXiv:2512.21473) built for this MR: its code is not public, so the project rebuilds the design from the paper, measures each element on the paper's workloads against Accelerate, LIBXSMM, KleidiAI and OpenBLAS, and serves as the test bed for what is ported here. Its README has the per-shape results, the ablation and the heat maps over M x N for this branch.

The multi-threaded work (one disjoint part of the result per SME unit, and a minimum task size per scalar type) is split out into !3174 (merged), which builds on this one.

Reference issue

Related to #3124 (closed) (closed; the follow-ups named in !3160 (merged)'s review: heap-buffer alignment and the SME unit count), and to #3119 (small-product dispatch), which the tiny-result kernel (item 10) addresses in part.

What does this implement/fix?

Threads and buffers:

  1. nbSmeUnits() / setNbSmeUnits() / EIGEN_SME_UNITS (Parallelizer.h). The SME units are shared by a cluster of cores: on this machine two units serve eight performance cores, so a product on the SME kernel gains nothing past 2 threads and loses to contention beyond. The count is detected once from macOS's performance-cluster topology and is 0 (no cap) everywhere else, where EIGEN_SME_UNITS or setNbSmeUnits() supply it, and parallelize_gemm caps the thread count for the scalar pairs the SME kernel serves. benchmarks/Core/bench_gemm_threads.cpp is new.
  2. scratch_malloc / scratch_new (Memory.h): the temporaries' heap fallback and gemm_blocking_space::allocateA/B allocate at EIGEN_STACK_ALIGN_BYTES, 64 in SME builds, and construct their elements.

Packing:

  1. The transposing packers (column-major RHS, row-major LHS) move four rows or four depth steps per ZA instruction (MOVA vg4), with a core-side L2 prefetch of the source on full panels; tail panels of more than 4 columns (up to 31 for float, 15 for double) of a deep block go through ZA as well (partial panels, two depth steps per store up to half a panel), where the NEON transposer ran at a fifth of the ZA rate.
  2. The column-major LHS copy moves four panels per depth step with four-vector loads.
  3. A column-major real LHS block deeper than the NEON bound is read in place instead of packed when its stride is not a multiple of 4 KB and at most 16 KB, and the block is at most 4 MB (EIGEN_SME_DIRECT_LHS_MAX_STRIDE_BYTES, EIGEN_SME_DIRECT_LHS_MAX_BLOCK_BYTES). When the RHS is a single panel each LHS element is read once, so up to half a panel wide the LHS is always read in place, and up to a full panel while depth times stride stays within EIGEN_SME_DIRECT_LHS_MAX_SPAN_BYTES (16 MB; 0 disables it). EIGEN_SME_FORCE_NEON_SMALL_BLOCKS turns the in-place read off.
  4. When all rows fit one LHS block, the packed RHS block is L2-sized (EIGEN_SME_SINGLE_PASS_RHS_BUDGET_BYTES, 4 MB), so it is not written to DRAM and read back.

Kernels:

  1. Full 2 x 2 blocks store C column by column from four-slice ZA reads, each column loaded and stored whole, and large results prefetch the next C block into L2.
  2. A RHS panel of at most svl columns stacks the four ZA tiles along M (two LHS panels against one B vector) instead of using half of the 2 x 2 grid.
  3. A block that fills one or two tiles (M or N of svl or less) spreads consecutive depth steps over the four tiles and adds the partial sums before the store: outer products into one tile wait on each other.
  4. Small results (as the kernel sees them, 2-8 rows and 2-8 columns for float, 2-4 rows for double) take a NEON outer-product kernel with the whole result in registers, on one thread, before any blocking or packing, where it beats the other paths: from depth 16, or 6 with an area of at least 24; a full result over a depth of 2048 or more keeps the SME kernel. The coeff-based product runs one dependent FMA chain per result there (4 x 4 x 512: 10.8 -> about 100 GFLOPS). EIGEN_SME_NO_NEON_SMALL_BLOCKS and EIGEN_SME_FORCE_NEON_SMALL_BLOCKS turn it off with the rest of the NEON small-block path, and it is off under EIGEN_USE_BLAS.

Floating-point status:

  1. We found that entering and leaving streaming mode sets every cumulative exception flag in FPSR (Arm DDI0616, RMHTLZ), and that the flags are set again when the OS resumes a thread in streaming mode, so after a product on the SME kernel all five flags read as raised, even for an exact result; this is in master since !3160 (merged). Each call into streaming code now saves FPSR before and restores it after, so a product leaves the caller's flags as they were. We also found that FMOPA sets no flags at all, so floating-point exceptions inside the SME kernels (an overflow, an invalid inf * 0) are not reported; TopicVectorization.dox says so. We first kept the flags saved inside the streaming region (those of its own SVE arithmetic, such as the C update), but saw them all set again in 1-3% of 256^3 products and about half of 1024^3 ones here, so we chose the deterministic save and restore. For comparison, Accelerate's cblas_sgemm left all five flags set after every call we tried, clean or not. product_sme checks every SME path (packed and in-place GEMM in both storage orders, narrow, symmetric, triangular product and solve) and scalar type with exact results.

Every new macro has an entry in PreprocessorDirectives.dox. Non-SME builds behave as before (the heap helpers for the blocking buffers are renamed and reduce to the previous allocation).

Results

Minimum of 5 repetitions, single thread, the tree's benchmarks; "approved" is adda2db4, the head of this MR at the last review:

Case master approved (adda2db4) this MR master -> MR GFLOPS master -> MR
float GEMM 8^3 52 ns 52 ns 26 ns 2.00x 20 -> 40
float GEMM 16^3 145 ns 146 ns 144 ns 1.01x 56 -> 57
float GEMM 32^3 735 ns 423 ns 386 ns 1.90x 89 -> 170
float GEMM 64^3 1457 ns 1125 ns 1337 ns 1.09x 360 -> 392
float GEMM 96^3 2477 ns 1999 ns 1883 ns 1.32x 714 -> 940
float GEMM 192^3 13.1 us 12.3 us 12.7 us 1.03x 1081 -> 1116
float GEMM 384^3 76.2 us 72.8 us 74.0 us 1.03x 1487 -> 1531
float GEMM 512^3 173.3 us 164.4 us 164.9 us 1.05x 1549 -> 1628
float GEMM 1024^3 1.55 ms 1.54 ms 1.36 ms 1.14x 1390 -> 1577
float GEMM 2048^3 12.13 ms 11.32 ms 10.70 ms 1.13x 1416 -> 1605
float GEMM 64x1024x64 12.0 us 11.8 us 12.2 us 0.98x 700 -> 687
float GEMM 1024x64x64 15.2 us 14.7 us 14.4 us 1.06x 551 -> 584
float GEMM 64x64x1024 10.4 us 8993 ns 9236 ns 1.12x 809 -> 908
double GEMM 32^3 668 ns 739 ns 689 ns 0.97x 98 -> 95
double GEMM 96^3 6755 ns 6471 ns 6186 ns 1.09x 262 -> 286
double GEMM 384^3 259.4 us 253.0 us 248.9 us 1.04x 436 -> 455
double GEMM 1024^3 5.72 ms 5.74 ms 5.04 ms 1.14x 375 -> 426
complex GEMM 384^3 267.2 us 267.4 us 267.2 us 1.00x 1695 -> 1695
complex GEMM 384^3 1.03 ms 1.06 ms 1.01 ms 1.02x 439 -> 449
LLT float n=1024 1.42 ms 1.40 ms 1.33 ms 1.07x
LLT double n=1024 2.66 ms 2.60 ms 2.48 ms 1.07x
PartialPivLU float n=1024 3.06 ms 3.06 ms 3.00 ms 1.02x
PartialPivLU double n=1024 6.14 ms 6.09 ms 5.84 ms 1.05x

Geometric means over each benchmark binary (Eigen cases):

benchmark cases master -> MR (geomean) approved -> MR (geomean) approved -> MR (worst)
bench_gemm 31 1.157 1.061 0.84
bench_gemm_double 31 1.135 1.093 1.00
bench_gemm_complex 31 1.035 1.033 0.96
bench_gemm_complex_double 31 1.045 1.053 1.00
bench_cholesky 20 1.014 1.009 0.99
bench_cholesky_double 20 1.019 1.003 0.98
bench_lu 104 1.073 1.069 0.97

The one case below 0.95x of the approved head is float 64^3 in bench_gemm (1125 -> 1337 ns): a standalone loop of the same product measures 1254 -> 1284 ns, and the gap moves with code alignment (1220 -> 1340 ns with -falign-functions=64 -falign-loops=64), so it was not pinned to a change.

Against Accelerate on one thread (VECLIB_MAXIMUM_THREADS=1), float, column-major C.noalias() += A * B, GFLOPS, from the MTGEMM-A harness (minimum of 5 trials):

n Accelerate this MR this MR / Accelerate MTGEMM-A
32 298 258 0.87x 250
64 924 801 0.87x 957
128 1421 1264 0.89x 1479
256 1682 1464 0.87x 1713
512 1772 1688 0.95x 1783
1000 1762 1789 1.02x 1766
1024 1649 1687 1.02x 1808
2048 1569 1700 1.08x 1701
3000 1523 1652 1.08x 1669
4096 1540 1740 1.13x 1689
MpGEMM paper workloads (IDs) Accelerate col this MR col ratio Accelerate row this MR row ratio
M = 64 (1-6) 935 982 1.05x 704 966 1.37x
M = 128 (7-12) 1192 1267 1.06x 1002 1256 1.25x
M = 4096 (13-18) 1473 1706 1.16x 1595 1709 1.07x
N = 256 (19-24) 1245 1530 1.23x 1422 1483 1.04x
all 24 1196 1343 1.12x 1124 1324 1.18x

Over a grid of every M and N from 4 to 4096 (powers of two) at K = 512 and 4096, this branch is below 0.9x of Accelerate on 34 of 484 cells (MTGEMM-A on 44), all at K = 512: one side of 4-16 against the other of 128-512, and small row-major products, which Eigen evaluates as C = A * B by zeroing C with core stores first; the SME unit then reads those lines slowly, while Accelerate with beta = 0 never reads C.

GEMM on the thread pool, median of 3 repetitions, bench_gemm_threads:

float n pool threads capped (GFLOPS) uncapped (GFLOPS)
1024 1 1595 (cap has no effect)
1024 2 2706 (cap has no effect)
1024 4 2725 1519
1024 8 2692 1331
1024 12 2729 1030
2048 1 1609 (cap has no effect)
2048 2 2851 (cap has no effect)
2048 4 2850 1230
2048 8 2833 1150
2048 12 2847 1362

References

Apple does not document the one-unit-per-cluster topology; the thread cap rests on the MpGEMM paper and on the measurement above.

Changes since the approved head (adda2db4): master is merged, and the vector kernels from !3161 (merged) share sme_fpsr_guard; the vector-length checks that choose the narrow and depth-split kernels are evaluated once per kernel call (inside the block loop they cost float GEMM 22% at 64^3 and 13% at 128^3 through the generated code); products keep the caller's floating-point flags (item 11); the kernels for narrow and one- or two-tile blocks and the four-slice ZA moves run only at the vector lengths whose tile shapes they assume, an LHS packed once for all column blocks is packed when a block that cannot read it in place follows one that did, and the Linux detection (CPU implementer in /proc/cpuinfo, then the cluster ids in sysfs) is gone. No Linux system runs on an SME-capable Apple chip to test it against, and following the review's rule that the cap rests on positive evidence, 0 (no cap) is the safe answer there; EIGEN_SME_UNITS and setNbSmeUnits() cover such a system. This also removes the file I/O from Parallelizer.h.

Additional information

Machine: Apple M4 Pro, 8 performance cores in two clusters of 4 (one SME unit each), macOS 27.0, SVL 512, FEAT_SME2 and SME_F64F64. Compiler: Apple clang 21.0.0, -O3 -DNDEBUG -march=armv8.6-a+sme2+sme-f64f64. Base: ab4f7316 (this MR's merge base). Each benchmark binary ran alone, after the load average fell below 4; bench_gemm zeroes C inside the timed loop.

Tests: product_sme (new subtests for small depths against large results, single-panel products read in place, every tail-panel width, and tiny results in all storage orders, with sub-blocks, strided results, Hankel views with an outer stride of 1 and exact-size buffers), product_threaded (tiny results on a pool), product_small, product_large, product_extra, product_symm, product_syrk, product_trsolve, product_trmm, product_mmtr, product_selfadjoint, product_notemporary, cholesky, lu, qr, householder, triangular, bdcsvd and eigensolver_selfadjoint pass under the SME flags; the new kernels also pass targeted sweeps under AddressSanitizer and UBSan with exact-size buffers, and a threaded run of double without FEAT_SME_F64F64. GCC 14.3 cross-builds the CI's SME targets (product_sme, product_symm, product_trsolve, product_trmm, product_syrk, product_mmtr, mixingtypes, plus product_threaded) with -march=armv9.2-a+sme2+sme-f64f64, and they pass under QEMU 10.2 at SVL 128, 256, 512 and 2048, as the CI's SME jobs run them. Changed lines are clang-format 17 clean and scripts/check_style.py --diff reports nothing.

Future work: an SME GEMV (loads into ZA reach 124 GB/s from one thread against 88 GB/s for NEON, and Accelerate's single-thread GEMV runs at about 150 GB/s); a C = A * B mode that stores the first depth block without reading C; online packing as in MpGEMM, which hides the transposing pack of a single deep panel (32 x 32 x 4096: 686 GFLOPS here, 1126 in MTGEMM-A) but changes the shared loop structure.

Appendix A: bench_gemm (float), 5 repetitions per build (min, median, cv)
Case master min master median master cv MR min MR median MR cv master -> MR
BM_EigenGemm/8/8/8 52 ns 52 ns 0.2% 26 ns 26 ns 0.5% 2.00x
BM_EigenGemm/16/16/16 145 ns 146 ns 0.3% 144 ns 144 ns 0.6% 1.01x
BM_EigenGemm/32/32/32 735 ns 737 ns 0.7% 386 ns 387 ns 0.3% 1.90x
BM_EigenGemm/64/64/64 1457 ns 1463 ns 0.4% 1337 ns 1338 ns 0.8% 1.09x
BM_EigenGemm/96/96/96 2477 ns 2584 ns 1.9% 1883 ns 1887 ns 0.2% 1.32x
BM_EigenGemm/128/128/128 5303 ns 5388 ns 1.2% 4585 ns 4588 ns 0.0% 1.16x
BM_EigenGemm/160/160/160 8519 ns 8845 ns 1.7% 7974 ns 8069 ns 1.1% 1.07x
BM_EigenGemm/192/192/192 13.1 us 13.2 us 0.7% 12.7 us 12.7 us 0.3% 1.03x
BM_EigenGemm/224/224/224 18.8 us 19.0 us 2.5% 18.1 us 18.1 us 0.6% 1.04x
BM_EigenGemm/256/256/256 27.0 us 27.0 us 1.7% 26.5 us 26.6 us 1.7% 1.02x
BM_EigenGemm/288/288/288 36.0 us 36.1 us 2.2% 35.2 us 35.3 us 0.1% 1.02x
BM_EigenGemm/320/320/320 47.2 us 48.0 us 0.9% 45.9 us 46.3 us 0.8% 1.03x
BM_EigenGemm/384/384/384 76.2 us 76.9 us 2.6% 74.0 us 74.6 us 1.1% 1.03x
BM_EigenGemm/448/448/448 119.8 us 120.5 us 0.9% 112.1 us 112.2 us 0.2% 1.07x
BM_EigenGemm/512/512/512 173.3 us 177.9 us 2.7% 164.9 us 166.1 us 0.3% 1.05x
BM_EigenGemm/768/768/768 545.2 us 546.6 us 0.4% 516.8 us 517.3 us 0.1% 1.05x
BM_EigenGemm/1024/1024/1024 1.55 ms 1.56 ms 3.0% 1.36 ms 1.36 ms 0.3% 1.14x
BM_EigenGemm/1536/1536/1536 5.05 ms 5.07 ms 0.3% 4.55 ms 4.58 ms 0.7% 1.11x
BM_EigenGemm/2048/2048/2048 12.13 ms 12.14 ms 0.1% 10.70 ms 10.73 ms 0.2% 1.13x
BM_EigenGemm/4096/4096/4096 88.64 ms 89.42 ms 0.9% 82.27 ms 82.85 ms 0.7% 1.08x
BM_EigenGemm/64/64/1024 10.4 us 10.4 us 0.2% 9236 ns 9240 ns 0.6% 1.12x
BM_EigenGemm/1024/64/64 15.2 us 15.5 us 1.4% 14.4 us 14.4 us 0.1% 1.06x
BM_EigenGemm/64/1024/64 12.0 us 12.0 us 1.7% 12.2 us 12.4 us 1.4% 0.98x
BM_EigenGemm/256/256/1024 91.3 us 91.4 us 1.7% 87.2 us 87.3 us 0.1% 1.05x
BM_EigenGemm/1024/256/256 113.4 us 113.8 us 1.3% 103.0 us 103.3 us 0.4% 1.10x
BM_EigenGemm/4096/96/96 121.5 us 125.8 us 2.9% 91.2 us 91.3 us 0.8% 1.33x
BM_EigenGemm/4096/128/128 174.0 us 174.3 us 0.2% 137.7 us 137.8 us 0.1% 1.26x
BM_EigenGemm/4096/144/144 214.0 us 214.9 us 1.5% 162.2 us 162.8 us 0.3% 1.32x
BM_EigenGemm/4096/160/160 239.2 us 241.7 us 0.7% 193.6 us 194.4 us 0.6% 1.24x
BM_EigenGemm/4096/176/176 284.4 us 285.1 us 0.1% 223.6 us 223.9 us 0.1% 1.27x
BM_EigenGemm/8192/128/128 375.1 us 380.7 us 0.9% 279.0 us 279.9 us 0.2% 1.34x
Appendix B: bench_gemm_double, 5 repetitions per build
Case master min master median master cv MR min MR median MR cv master -> MR
BM_EigenGemm/8/8/8 73 ns 73 ns 0.1% 74 ns 75 ns 0.3% 0.98x
BM_EigenGemm/16/16/16 247 ns 249 ns 0.6% 249 ns 250 ns 0.2% 0.99x
BM_EigenGemm/32/32/32 668 ns 670 ns 0.3% 689 ns 691 ns 0.2% 0.97x
BM_EigenGemm/64/64/64 2743 ns 2746 ns 0.8% 1863 ns 1871 ns 0.2% 1.47x
BM_EigenGemm/96/96/96 6755 ns 6778 ns 0.6% 6186 ns 6204 ns 0.6% 1.09x
BM_EigenGemm/128/128/128 13.7 us 13.7 us 1.2% 12.4 us 12.4 us 1.4% 1.10x
BM_EigenGemm/160/160/160 24.7 us 24.7 us 0.3% 23.0 us 23.0 us 0.7% 1.08x
BM_EigenGemm/192/192/192 38.9 us 39.0 us 0.5% 36.4 us 36.6 us 0.6% 1.07x
BM_EigenGemm/224/224/224 58.0 us 58.1 us 0.9% 54.6 us 54.7 us 1.2% 1.06x
BM_EigenGemm/256/256/256 83.7 us 83.8 us 0.9% 78.6 us 78.8 us 0.5% 1.06x
BM_EigenGemm/288/288/288 114.9 us 115.0 us 0.4% 109.3 us 109.3 us 0.4% 1.05x
BM_EigenGemm/320/320/320 154.3 us 154.5 us 0.4% 147.2 us 147.2 us 0.6% 1.05x
BM_EigenGemm/384/384/384 259.4 us 260.0 us 0.3% 248.9 us 250.6 us 0.3% 1.04x
BM_EigenGemm/448/448/448 403.8 us 404.1 us 0.3% 389.6 us 389.7 us 0.2% 1.04x
BM_EigenGemm/512/512/512 648.0 us 648.5 us 1.0% 602.8 us 605.9 us 0.5% 1.07x
BM_EigenGemm/768/768/768 2.21 ms 2.22 ms 0.9% 2.05 ms 2.05 ms 0.2% 1.08x
BM_EigenGemm/1024/1024/1024 5.72 ms 5.75 ms 0.3% 5.04 ms 5.05 ms 0.2% 1.14x
BM_EigenGemm/1536/1536/1536 18.65 ms 18.70 ms 0.2% 16.65 ms 16.72 ms 0.2% 1.12x
BM_EigenGemm/2048/2048/2048 43.27 ms 43.39 ms 0.2% 39.72 ms 39.73 ms 0.3% 1.09x
BM_EigenGemm/4096/4096/4096 334.88 ms 336.20 ms 0.3% 308.36 ms 310.02 ms 0.5% 1.09x
BM_EigenGemm/64/64/1024 28.8 us 28.8 us 1.1% 25.0 us 25.0 us 1.0% 1.15x
BM_EigenGemm/1024/64/64 37.2 us 37.2 us 1.1% 29.5 us 29.6 us 1.1% 1.26x
BM_EigenGemm/64/1024/64 29.9 us 29.9 us 1.0% 27.6 us 27.9 us 0.8% 1.08x
BM_EigenGemm/256/256/1024 318.8 us 322.1 us 0.7% 296.4 us 297.0 us 0.1% 1.08x
BM_EigenGemm/1024/256/256 347.2 us 348.0 us 0.2% 314.6 us 315.1 us 0.6% 1.10x
BM_EigenGemm/4096/96/96 281.9 us 282.9 us 0.6% 209.7 us 210.3 us 0.3% 1.34x
BM_EigenGemm/4096/128/128 467.1 us 468.8 us 0.7% 351.9 us 352.5 us 0.1% 1.33x
BM_EigenGemm/4096/144/144 587.3 us 587.9 us 0.3% 440.9 us 444.0 us 0.7% 1.33x
BM_EigenGemm/4096/160/160 731.2 us 735.2 us 0.9% 547.3 us 548.5 us 0.1% 1.34x
BM_EigenGemm/4096/176/176 904.0 us 905.9 us 0.7% 652.3 us 652.9 us 0.6% 1.39x
BM_EigenGemm/8192/128/128 1.11 ms 1.11 ms 0.5% 803.3 us 805.2 us 0.5% 1.38x
Appendix C: bench_gemm_complex, 5 repetitions per build
Case master min master median master cv MR min MR median MR cv master -> MR
BM_EigenGemm/8/8/8 99 ns 99 ns 0.2% 100 ns 100 ns 0.5% 0.99x
BM_EigenGemm/16/16/16 402 ns 403 ns 0.4% 401 ns 402 ns 0.1% 1.00x
BM_EigenGemm/32/32/32 976 ns 979 ns 0.5% 1000 ns 1004 ns 0.3% 0.98x
BM_EigenGemm/64/64/64 2205 ns 2206 ns 0.1% 2218 ns 2221 ns 0.3% 0.99x
BM_EigenGemm/96/96/96 7183 ns 7249 ns 0.9% 7119 ns 7139 ns 1.0% 1.01x
BM_EigenGemm/128/128/128 14.6 us 14.6 us 0.1% 14.5 us 14.6 us 0.8% 1.00x
BM_EigenGemm/160/160/160 26.3 us 26.4 us 0.5% 26.2 us 26.3 us 0.2% 1.00x
BM_EigenGemm/192/192/192 41.1 us 41.6 us 1.0% 40.9 us 41.0 us 1.3% 1.00x
BM_EigenGemm/224/224/224 60.9 us 60.9 us 0.0% 60.8 us 60.9 us 0.5% 1.00x
BM_EigenGemm/256/256/256 87.6 us 87.7 us 0.3% 87.6 us 87.7 us 0.1% 1.00x
BM_EigenGemm/288/288/288 119.9 us 120.9 us 0.6% 119.4 us 120.2 us 0.8% 1.00x
BM_EigenGemm/320/320/320 160.1 us 160.1 us 0.7% 159.8 us 160.0 us 0.3% 1.00x
BM_EigenGemm/384/384/384 267.2 us 267.6 us 0.3% 267.2 us 267.4 us 0.1% 1.00x
BM_EigenGemm/448/448/448 415.0 us 415.4 us 0.6% 414.2 us 414.6 us 0.4% 1.00x
BM_EigenGemm/512/512/512 663.8 us 671.1 us 0.5% 655.6 us 656.0 us 0.3% 1.01x
BM_EigenGemm/768/768/768 2.38 ms 2.38 ms 0.6% 2.14 ms 2.15 ms 0.6% 1.11x
BM_EigenGemm/1024/1024/1024 6.17 ms 6.18 ms 0.1% 5.35 ms 5.36 ms 0.3% 1.15x
BM_EigenGemm/1536/1536/1536 20.94 ms 20.96 ms 0.2% 17.63 ms 17.64 ms 0.2% 1.19x
BM_EigenGemm/2048/2048/2048 46.34 ms 46.46 ms 0.2% 40.97 ms 41.19 ms 0.6% 1.13x
BM_EigenGemm/4096/4096/4096 361.36 ms 361.64 ms 0.1% 315.21 ms 315.90 ms 0.2% 1.15x
BM_EigenGemm/64/64/1024 29.6 us 29.7 us 1.9% 29.9 us 30.4 us 1.7% 0.99x
BM_EigenGemm/1024/64/64 39.2 us 39.3 us 0.1% 39.2 us 39.2 us 0.6% 1.00x
BM_EigenGemm/64/1024/64 33.2 us 33.5 us 0.7% 33.1 us 33.1 us 0.3% 1.00x
BM_EigenGemm/256/256/1024 319.6 us 321.6 us 1.4% 323.4 us 326.3 us 0.6% 0.99x
BM_EigenGemm/1024/256/256 355.5 us 357.3 us 0.4% 355.5 us 355.8 us 0.3% 1.00x
BM_EigenGemm/4096/96/96 283.3 us 285.2 us 1.0% 272.1 us 272.3 us 0.9% 1.04x
BM_EigenGemm/4096/128/128 470.1 us 470.7 us 0.2% 451.2 us 451.7 us 0.6% 1.04x
BM_EigenGemm/4096/144/144 622.4 us 627.3 us 0.7% 605.9 us 606.2 us 0.2% 1.03x
BM_EigenGemm/4096/160/160 759.0 us 770.5 us 0.7% 710.0 us 715.0 us 0.4% 1.07x
BM_EigenGemm/4096/176/176 952.9 us 960.7 us 0.6% 887.8 us 890.6 us 0.5% 1.07x
BM_EigenGemm/8192/128/128 1.26 ms 1.27 ms 0.4% 1.10 ms 1.10 ms 0.8% 1.15x
Appendix D: bench_gemm_complex_double, 5 repetitions per build
Case master min master median master cv MR min MR median MR cv master -> MR
BM_EigenGemm/8/8/8 152 ns 152 ns 0.3% 152 ns 152 ns 0.5% 1.00x
BM_EigenGemm/16/16/16 911 ns 912 ns 0.1% 841 ns 844 ns 0.3% 1.08x
BM_EigenGemm/32/32/32 1867 ns 1872 ns 0.7% 1882 ns 1886 ns 0.5% 0.99x
BM_EigenGemm/64/64/64 7910 ns 7912 ns 0.1% 7770 ns 7776 ns 0.7% 1.02x
BM_EigenGemm/96/96/96 22.3 us 22.4 us 0.8% 22.1 us 22.1 us 0.2% 1.01x
BM_EigenGemm/128/128/128 45.8 us 45.9 us 0.8% 45.6 us 45.8 us 0.9% 1.00x
BM_EigenGemm/160/160/160 82.3 us 82.5 us 0.4% 82.1 us 82.2 us 0.4% 1.00x
BM_EigenGemm/192/192/192 136.0 us 136.1 us 0.9% 135.7 us 135.8 us 0.1% 1.00x
BM_EigenGemm/224/224/224 210.0 us 210.1 us 0.1% 209.6 us 210.8 us 0.4% 1.00x
BM_EigenGemm/256/256/256 314.6 us 315.2 us 0.2% 313.3 us 313.5 us 0.2% 1.00x
BM_EigenGemm/288/288/288 431.5 us 431.7 us 0.6% 430.4 us 430.5 us 0.6% 1.00x
BM_EigenGemm/320/320/320 587.7 us 588.1 us 0.0% 583.5 us 584.8 us 0.2% 1.01x
BM_EigenGemm/384/384/384 1.03 ms 1.03 ms 0.5% 1.01 ms 1.01 ms 0.2% 1.02x
BM_EigenGemm/448/448/448 1.71 ms 1.71 ms 0.9% 1.62 ms 1.63 ms 0.6% 1.06x
BM_EigenGemm/512/512/512 2.72 ms 2.73 ms 0.2% 2.52 ms 2.53 ms 0.5% 1.08x
BM_EigenGemm/768/768/768 9.81 ms 9.84 ms 0.4% 8.73 ms 8.74 ms 0.1% 1.12x
BM_EigenGemm/1024/1024/1024 22.62 ms 22.64 ms 0.1% 19.80 ms 19.81 ms 0.7% 1.14x
BM_EigenGemm/1536/1536/1536 74.25 ms 74.38 ms 0.2% 64.92 ms 64.96 ms 0.2% 1.14x
BM_EigenGemm/2048/2048/2048 174.76 ms 175.15 ms 0.2% 152.84 ms 153.44 ms 0.9% 1.14x
BM_EigenGemm/4096/4096/4096 1371.26 ms 1375.20 ms 0.7% 1195.46 ms 1199.11 ms 0.7% 1.15x
BM_EigenGemm/64/64/1024 89.0 us 89.1 us 0.1% 87.7 us 88.7 us 0.6% 1.01x
BM_EigenGemm/1024/64/64 101.5 us 101.6 us 0.4% 101.3 us 101.4 us 0.4% 1.00x
BM_EigenGemm/64/1024/64 98.2 us 98.4 us 0.8% 97.2 us 97.2 us 0.5% 1.01x
BM_EigenGemm/256/256/1024 1.31 ms 1.31 ms 0.5% 1.26 ms 1.27 ms 0.6% 1.04x
BM_EigenGemm/1024/256/256 1.32 ms 1.33 ms 0.4% 1.31 ms 1.31 ms 0.2% 1.01x
BM_EigenGemm/4096/96/96 964.8 us 967.3 us 0.9% 940.1 us 945.5 us 0.3% 1.03x
BM_EigenGemm/4096/128/128 1.66 ms 1.67 ms 0.3% 1.60 ms 1.60 ms 0.2% 1.04x
BM_EigenGemm/4096/144/144 2.08 ms 2.08 ms 0.3% 1.97 ms 1.97 ms 0.1% 1.06x
BM_EigenGemm/4096/160/160 2.58 ms 2.59 ms 0.7% 2.38 ms 2.38 ms 0.1% 1.08x
BM_EigenGemm/4096/176/176 3.09 ms 3.09 ms 0.4% 2.81 ms 2.81 ms 0.2% 1.10x
BM_EigenGemm/8192/128/128 3.39 ms 3.41 ms 1.0% 3.25 ms 3.26 ms 0.2% 1.04x
Appendix E: bench_cholesky (float), 5 repetitions per build
Case master min master median master cv MR min MR median MR cv master -> MR
BM_LDLT/4 77 ns 82 ns 4.5% 78 ns 82 ns 6.4% 0.99x
BM_LDLT/8 206 ns 206 ns 1.9% 202 ns 203 ns 2.1% 1.02x
BM_LDLT/16 583 ns 594 ns 2.4% 580 ns 587 ns 2.0% 1.01x
BM_LDLT/32 2024 ns 2052 ns 0.9% 2016 ns 2047 ns 1.0% 1.00x
BM_LDLT/64 9197 ns 9449 ns 1.2% 9229 ns 9465 ns 1.1% 1.00x
BM_LDLT/128 42.8 us 42.9 us 0.3% 42.8 us 43.0 us 0.3% 1.00x
BM_LDLT/256 278.3 us 280.0 us 0.6% 276.2 us 277.9 us 0.9% 1.01x
BM_LDLT/512 2.28 ms 2.29 ms 0.5% 2.29 ms 2.31 ms 0.5% 1.00x
BM_LDLT/1024 14.57 ms 14.65 ms 0.2% 14.58 ms 14.67 ms 0.3% 1.00x
BM_LDLT/1500 31.03 ms 31.05 ms 0.3% 31.01 ms 31.03 ms 0.1% 1.00x
BM_LLT/4 45 ns 46 ns 1.2% 44 ns 44 ns 0.7% 1.02x
BM_LLT/8 121 ns 121 ns 0.0% 121 ns 121 ns 0.0% 1.00x
BM_LLT/16 334 ns 337 ns 0.5% 335 ns 338 ns 0.4% 1.00x
BM_LLT/32 1521 ns 1522 ns 0.2% 1524 ns 1525 ns 0.3% 1.00x
BM_LLT/64 5362 ns 5384 ns 0.2% 5370 ns 5402 ns 0.4% 1.00x
BM_LLT/128 20.5 us 20.5 us 0.3% 20.5 us 20.5 us 0.1% 1.00x
BM_LLT/256 74.2 us 74.3 us 0.6% 72.2 us 72.2 us 1.5% 1.03x
BM_LLT/512 268.8 us 271.2 us 0.7% 252.2 us 252.8 us 0.6% 1.07x
BM_LLT/1024 1.42 ms 1.42 ms 0.1% 1.33 ms 1.34 ms 1.2% 1.07x
BM_LLT/1500 2.72 ms 2.75 ms 0.7% 2.49 ms 2.49 ms 0.7% 1.09x
Appendix F: bench_cholesky_double, 5 repetitions per build
Case master min master median master cv MR min MR median MR cv master -> MR
BM_LDLT/4 83 ns 90 ns 4.7% 84 ns 89 ns 3.4% 0.99x
BM_LDLT/8 202 ns 217 ns 3.3% 202 ns 220 ns 3.8% 1.00x
BM_LDLT/16 581 ns 606 ns 2.3% 586 ns 613 ns 2.2% 0.99x
BM_LDLT/32 2324 ns 2337 ns 0.4% 2335 ns 2354 ns 0.4% 1.00x
BM_LDLT/64 10.8 us 10.9 us 0.5% 10.8 us 10.8 us 0.1% 1.00x
BM_LDLT/128 55.0 us 55.2 us 0.3% 55.2 us 55.2 us 0.5% 1.00x
BM_LDLT/256 562.6 us 564.6 us 0.7% 561.8 us 564.5 us 0.8% 1.00x
BM_LDLT/512 3.74 ms 3.75 ms 0.4% 3.74 ms 3.75 ms 0.6% 1.00x
BM_LDLT/1024 25.72 ms 25.77 ms 0.1% 25.71 ms 25.78 ms 0.3% 1.00x
BM_LDLT/1500 53.32 ms 53.42 ms 0.2% 53.04 ms 53.11 ms 0.1% 1.01x
BM_LLT/4 49 ns 49 ns 1.7% 41 ns 41 ns 0.3% 1.20x
BM_LLT/8 125 ns 126 ns 0.3% 125 ns 125 ns 0.0% 1.00x
BM_LLT/16 326 ns 328 ns 0.4% 327 ns 329 ns 0.5% 1.00x
BM_LLT/32 1819 ns 1830 ns 0.3% 1819 ns 1822 ns 0.5% 1.00x
BM_LLT/64 7397 ns 7457 ns 0.4% 7366 ns 7396 ns 0.3% 1.00x
BM_LLT/128 32.3 us 32.5 us 0.3% 32.5 us 32.6 us 0.2% 0.99x
BM_LLT/256 115.9 us 116.0 us 0.4% 114.4 us 114.6 us 1.0% 1.01x
BM_LLT/512 494.1 us 494.6 us 1.2% 456.2 us 456.2 us 0.0% 1.08x
BM_LLT/1024 2.66 ms 2.67 ms 0.6% 2.48 ms 2.51 ms 0.9% 1.07x
BM_LLT/1500 5.56 ms 5.57 ms 0.6% 5.21 ms 5.23 ms 0.4% 1.07x
Appendix G: bench_lu, Compute, 5 repetitions per build
Case master min master median master cv MR min MR median MR cv master -> MR
PartialPivLU_Compute_float/8 128 ns 147 ns 8.4% 128 ns 147 ns 8.6% 1.00x
PartialPivLU_Compute_float/32 1822 ns 1936 ns 3.7% 1793 ns 1899 ns 3.4% 1.02x
PartialPivLU_Compute_float/64 6882 ns 7020 ns 2.3% 6864 ns 7046 ns 2.3% 1.00x
PartialPivLU_Compute_float/128 32.0 us 32.4 us 1.0% 31.5 us 32.1 us 1.2% 1.02x
PartialPivLU_Compute_float/256 120.2 us 124.9 us 1.7% 120.1 us 120.7 us 0.8% 1.00x
PartialPivLU_Compute_float/512 603.4 us 604.9 us 1.2% 576.1 us 586.6 us 1.8% 1.05x
PartialPivLU_Compute_float/1024 3.06 ms 3.10 ms 0.7% 3.00 ms 3.02 ms 0.7% 1.02x
PartialPivLU_Compute_double/8 144 ns 168 ns 11.8% 143 ns 168 ns 11.8% 1.01x
PartialPivLU_Compute_double/32 2378 ns 2435 ns 1.6% 2366 ns 2435 ns 1.8% 1.01x
PartialPivLU_Compute_double/64 10.2 us 10.3 us 0.7% 10.2 us 10.3 us 0.6% 0.99x
PartialPivLU_Compute_double/128 51.9 us 52.6 us 0.8% 51.4 us 51.8 us 0.8% 1.01x
PartialPivLU_Compute_double/256 198.7 us 202.2 us 0.8% 195.3 us 202.3 us 1.7% 1.02x
PartialPivLU_Compute_double/512 1.08 ms 1.08 ms 1.0% 1.01 ms 1.01 ms 0.3% 1.07x
PartialPivLU_Compute_double/1024 6.14 ms 6.16 ms 0.2% 5.84 ms 5.85 ms 0.2% 1.05x
FullPivLU_Compute_float/8 222 ns 244 ns 4.5% 234 ns 241 ns 1.7% 0.95x
FullPivLU_Compute_float/32 3712 ns 3744 ns 0.8% 3676 ns 3771 ns 1.4% 1.01x
FullPivLU_Compute_float/64 21.6 us 21.8 us 0.7% 21.5 us 21.9 us 1.1% 1.01x
FullPivLU_Compute_float/128 137.0 us 137.1 us 0.6% 136.1 us 137.2 us 0.7% 1.01x
FullPivLU_Compute_float/256 1.08 ms 1.08 ms 0.3% 1.07 ms 1.08 ms 0.4% 1.01x
FullPivLU_Compute_float/512 9.05 ms 9.07 ms 0.3% 9.05 ms 9.08 ms 0.3% 1.00x
FullPivLU_Compute_float/1024 69.27 ms 69.29 ms 0.0% 69.04 ms 69.32 ms 0.4% 1.00x
FullPivLU_Compute_double/8 224 ns 233 ns 5.8% 212 ns 238 ns 4.6% 1.06x
FullPivLU_Compute_double/32 5441 ns 5502 ns 0.7% 5462 ns 5512 ns 0.6% 1.00x
FullPivLU_Compute_double/64 33.8 us 33.9 us 0.3% 33.7 us 34.1 us 0.5% 1.00x
FullPivLU_Compute_double/128 239.8 us 240.5 us 0.3% 234.1 us 234.6 us 0.3% 1.02x
FullPivLU_Compute_double/256 2.28 ms 2.28 ms 0.1% 2.26 ms 2.28 ms 0.4% 1.01x
FullPivLU_Compute_double/512 17.19 ms 17.21 ms 0.2% 17.22 ms 17.27 ms 0.3% 1.00x
FullPivLU_Compute_double/1024 125.96 ms 126.15 ms 0.4% 126.96 ms 127.65 ms 0.2% 0.99x
Edited by Michael Tesch

Merge request reports