SME: NEON path for small blocks, 64-byte packed panels, generic crossovers

An SME build had no NEON GEMM kernel: below the crossover it ran the coefficient-based product and above it the ZA kernel on mostly predicated 32x32 tiles, so float 16^3 cost 2.2x a NEON build and a blocked LLT at n=32 and n=64 ran 2.0-2.3x slower than NEON. This adds a NEON path for shallow, narrow blocks inside the SME kernel and packers, aligns the packed panels to 64 bytes, and returns the SME crossovers to the generic values. On Apple M4 Pro the SME build is now within 10% of a NEON build at 8^3 and ahead from 16^3, and beats the current SME backend by 1.5-2.6x at 16^3 to 128^3 for float, 2.2-2.4x on LLT at n=32 and n=64, and 1.5-1.6x on partial-pivoting LU at n=64, with the large sizes unchanged.

Headline measurements, minimum of 5 repetitions, Apple M4 Pro single thread (all cases and their variability are in the appendices):

Case	upstream (min, ns)	NEON build (min, ns)	this MR (min, ns)	MR vs upstream	MR vs NEON
float GEMM 8^3	56	49	52	1.08x	0.93x
float GEMM 16^3	364	164	142	2.56x	1.15x
float GEMM 32^3	1,067	758	744	1.43x	1.02x
float GEMM 64^3	2,222	5,241	1,477	1.50x	3.55x
float GEMM 128^3	7,757	39,394	5,251	1.48x	7.50x
float GEMM 512^3	175,823	2,233,109	173,419	1.01x	12.88x
double GEMM 16^3	683	261	249	2.74x	1.05x
double GEMM 32^3	731	1,541	673	1.09x	2.29x
complex GEMM 16^3	891	462	404	2.21x	1.14x
complex GEMM 32^3	696	3,130	860	0.81x	3.64x
complex GEMM 32^3	2,550	6,157	1,744	1.46x	3.53x
LLT float n=32	3,742	1,586	1,492	2.51x	1.06x
LLT float n=64	12,039	5,305	5,263	2.29x	1.01x
LLT float n=128	27,267	20,508	19,958	1.37x	1.03x
LLT float n=1024	1,361,784	3,929,180	1,391,583	0.98x	2.82x
LLT float n=1500	2,579,579	10,976,766	2,657,501	0.97x	4.13x
LLT double n=32	4,268	1,974	1,828	2.33x	1.08x
LLT double n=256	138,621	177,787	115,554	1.20x	1.54x
LLT double n=1500	5,807,643	22,041,874	5,367,237	1.08x	4.11x
PartialPivLU float n=64	11,660	7,085	6,720	1.74x	1.05x
PartialPivLU double n=64	15,656	10,180	10,049	1.56x	1.01x
PartialPivLU double n=256	234,413	332,880	188,485	1.24x	1.77x
float GEMM 64x1024x64	13,866	86,557	12,094	1.15x	7.16x

Reference issue

Closes #3124 (closed).

What does this implement/fix?

Three measurements drove the design (Eigen/src/Core/arch/SME/GeneralBlockPanelKernel.h documents them next to the code):

1. Crossing modes on the packed panels is what costs. Streaming-mode stores read back by NEON stall ~100 ns plus a per-byte penalty; NEON stores read by the SME unit cost ~40 ns per KB. A NEON kernel alone gained little because the packers still ran in streaming mode, so packers and kernel now decide together.
2. Packed panels at a 16-byte offset slow the ZA kernel by 35-50%. Eigen's alloca temporaries are 16-byte aligned; in SME builds EIGEN_STACK_ALIGN_BYTES (Memory.h) makes them 64. This alone took float 32^3 from 0.75 to 0.27 us and 128^3 from 7.1 to 3.8 us in a direct measurement.
3. The per-type crossovers 78/54/60/45 sent products up to N=24 to the coefficient-based path, which is slower than a NEON GEMM.

The change:

• sme_gebp_neon consumes the SME packed panels with NEON packets: 12x8 tiles from lane-broadcast FMAs (vfmaq_laneq_*), a 3/2/1-packet row ladder plus scalar rows, k-unrolled accumulator chains for narrow tiles, complex tiles over the real view of the panels with conjugation folded into signs and interleaving vst2q stores. The panel format and blocking are unchanged.
• The four gemm_pack_* specializations gain pack_neon, a raw-pointer NEON copy or register transpose for the direct-access case (the mapper-based fallback cost 2-3x Eigen's NEON packers).
• Dispatch, fitted on this machine and traced through LLT and LU: a panel is packed with NEON when depth <= sme_neon_max_depth (24 float, 16 double and complex float, 8 complex double) and its width is at most sme_neon_max_panel (256, 128 complex); such a block runs the NEON kernel when both sides are within sme_neon_max_width (48/40/16/24), or within sme_neon_shallow_width (96/64/32/24) at depth <= 8, or one side is within sme_neon_thin_dim (32 real, 8/4 complex). The packers see one side only and the drivers pack a wide panel and run the kernel on sub-blocks, so a NEON kernel never reads a streaming-packed panel; the one crossing left is a NEON-packed shallow panel read by ZA, bounded by the 256-wide cap.
• sme_gemm_to_coeffbased_threshold and the fixed-size threshold return to the generic values; outputAreaBelowThreshold loses its integer division.
• test/product_sme.cpp gains test_neon_small_blocks: the NEON kernel on hand-packed panels over every row, column and depth tail of the tile ladder, both conjugation flags, a non-trivial alpha, panel-mode stride/offset and a general-stride result.

Costs, stated flatly: shallow square products 56-96 wide at depth 8 now take NEON and lose 25-70% against the ZA kernel (the price of the in-place decompositions, which update such blocks at offsets the ZA slice stores handle poorly); a shallow product wider than 96 with NEON-packed panels pays the per-KB read cost on ZA (128x128x16 float 1.4 -> 2.0 us). Left undone: float 8^3 stays 10% behind a NEON build (packer overhead on 8-wide panels); a thread-local "last pack mode" per side would let the kernel know how a sub-blocked panel was packed and remove the remaining crossing; the SYRK diagonal blocks still run at the 32x32 SME granularity, and SYRK packs B at the full size, so above 256 its 32x32 diagonal blocks read a streaming-packed panel with the NEON kernel; the heap-allocated packed buffers (gemm_blocking_space::allocateA/B, every threaded GEMM, and blocks above EIGEN_STACK_ALLOCATION_LIMIT) keep EIGEN_DEFAULT_ALIGN_BYTES, which on glibc leaves an mmap'd block at 16 mod 64, so the 35-50% alignment cost still applies there (follow-up).

Override switches for tuning on other SME implementations: EIGEN_SME_NEON_MAX_DEPTH, EIGEN_SME_NEON_MAX_WIDTH, EIGEN_SME_NEON_SHALLOW_DEPTH, EIGEN_SME_NEON_SHALLOW_WIDTH, EIGEN_SME_NEON_THIN_DIM, EIGEN_SME_NEON_MAX_PANEL, EIGEN_STACK_ALIGN_BYTES; EIGEN_SME_NO_NEON_SMALL_BLOCKS and EIGEN_SME_FORCE_NEON_SMALL_BLOCKS pin the path for A/B runs.

Additional information

Machine: Apple M4 Pro (sysctl -n machdep.cpu.brand_string), macOS 27.0, SVL 512, FEAT_SME2 and SME_F64F64. Compiler: Apple clang 21.0.0, -O3 -DNDEBUG -march=armv8.6-a+sme2+sme-f64f64 (on Apple silicon -march=armv9.2-a+sme2 builds but SIGILLs, as non-streaming SVE is absent). Baseline: upstream master 47a78413 with the same flags; the NEON column is this branch with -DEIGEN_ARM64_NO_SME. Benchmarks are the tree's own bench_gemm, bench_gemm_double, bench_gemm_complex, bench_gemm_complex_double, bench_cholesky, bench_cholesky_double and bench_lu, each binary run alone with --benchmark_repetitions=5 --benchmark_report_aggregates_only=true (GEMM filtered to sizes <= 512), the minimum of the five repetitions in the tables, with median and cv per case in the appendices; the raw output files are kept and can be attached on request. The host was not idle (load average 4-9 during the runs); the unchanged ZA kernel moved by up to 10% between invocations of one binary, and complex float 32^3 by 30% between binaries whose kernel code is identical, so differences of that order in the tables are layout and drift, not this change. The harness zeroes C inside the timed loop, so every product also pays the transfer of C from the core to the SME unit; a product that accumulates into C in place measures 0.31 us for float 32^3 on this branch against 0.75 us for the NEON build.

Tests: product_sme (with the new subtest), product_small, product_large, product_symm, product_syrk, product_trsolve, product_trmm, product_mmtr, cholesky, lu and mixingtypes pass under the SME flags; a 122880-case comparison against lazyProduct over shapes 1..65 x 1..70 x 1..100, four layouts, both conjugations, dual alpha, strided and row-major results passes with the NEON path forced on every block and with the default dispatch. NEON-only, EIGEN_DONT_VECTORIZE and EIGEN_ARM64_NO_SME_F64F64 builds compile. GCC: 16.1 (Homebrew, macOS, Darwin branch) compiles the NEON path and passes the same comparison with the path forced. It cannot link an SME binary on macOS, where its libgcc lacks __arm_tpidr2_save, and with -march=armv8.6-a+sme2 it rejects the pre-existing streaming svwhilelt calls without +sve, although the GCC 16 release notes list SME without SVE as supported (https://gcc.gnu.org/gcc-16/changes.html; GCC 15 refused +sme without +sve2 outright). The CI job with GCC 14 under QEMU on armv9.2-a+sme2 remains the GCC coverage, and it never exercises an SME-only target. Changed lines are clang-format 17 clean.

Update: review by @rmlarsen1 addressed in 6562a4fe (alpha packet names for codespell, the crossover specializations dropped, dst.size() in the output-area test, comments cut to the packer/kernel invariant and the panel-cap trade, the trace hook removed, the direct-pack sweep extended to depths above each NEON bound, and the new knobs documented); the description's tables follow the updated guide.

Update 2: the quiet re-run @rmlarsen1 asked for confirmed 0.93-0.96x at LLT n = 1024 and 1500 (float; upstream, branch, upstream, branch alternated). Cause: the triangular solvers pack a kc-deep panel in depth-8 slices in panel mode, so the packers saw depth 8 and NEON-packed panels that ZA then read at ~40 ns per KB. e76cb252 decides the packing mode from the panel's depth (the panel-mode stride) and, in the kernel, from strideA / strideB, which are those same depths; LLT float n = 1024 and 1500 and LLT double n = 256 to 1500 are re-measured in the tables (float 1024 now equal to upstream, 1500 within 2%, double 1024 and 1500 5-7% faster than upstream).

Appendix A: bench_gemm (float), 5 repetitions per build (min, median, cv)

Case	upstream min	upstream median	upstream cv	NEON min	NEON median	NEON cv	MR min	MR median	MR cv	MR vs upstream	MR vs NEON
BM_EigenGemm/8/8/8	56 ns	58 ns	1.1%	49 ns	49 ns	0.5%	52 ns	52 ns	0.1%	1.08x	0.93x
BM_EigenGemm/16/16/16	364 ns	365 ns	0.3%	164 ns	165 ns	0.3%	142 ns	142 ns	0.4%	2.56x	1.15x
BM_EigenGemm/32/32/32	1067 ns	1076 ns	0.5%	758 ns	759 ns	0.2%	744 ns	749 ns	0.3%	1.43x	1.02x
BM_EigenGemm/64/64/64	2222 ns	2237 ns	0.4%	5241 ns	5261 ns	0.4%	1477 ns	1482 ns	0.3%	1.50x	3.55x
BM_EigenGemm/96/96/96	4141 ns	4152 ns	0.2%	17.0 us	17.0 us	0.2%	2543 ns	2568 ns	0.7%	1.63x	6.67x
BM_EigenGemm/128/128/128	7757 ns	7782 ns	0.8%	39.4 us	39.4 us	0.1%	5251 ns	5266 ns	0.3%	1.48x	7.50x
BM_EigenGemm/160/160/160	12.8 us	12.9 us	0.4%	75.3 us	75.4 us	0.2%	8967 ns	9023 ns	0.4%	1.42x	8.40x
BM_EigenGemm/192/192/192	12.9 us	13.1 us	3.2%	123.8 us	123.9 us	0.1%	13.6 us	14.0 us	1.9%	0.95x	9.10x
BM_EigenGemm/224/224/224	18.7 us	19.1 us	1.9%	195.5 us	195.9 us	0.2%	19.7 us	20.0 us	1.6%	0.95x	9.91x
BM_EigenGemm/256/256/256	26.7 us	26.8 us	1.2%	291.9 us	293.6 us	0.4%	26.7 us	27.4 us	2.1%	1.00x	10.94x
BM_EigenGemm/288/288/288	36.1 us	36.5 us	1.2%	405.1 us	405.6 us	0.2%	35.9 us	36.7 us	1.1%	1.01x	11.30x
BM_EigenGemm/320/320/320	46.9 us	47.2 us	0.7%	553.9 us	555.4 us	1.1%	47.5 us	49.3 us	2.1%	0.99x	11.66x
BM_EigenGemm/384/384/384	76.9 us	78.1 us	1.4%	955.3 us	958.7 us	0.3%	76.3 us	77.4 us	1.6%	1.01x	12.52x
BM_EigenGemm/448/448/448	117.5 us	119.7 us	1.1%	1.52 ms	1.52 ms	0.4%	116.4 us	117.3 us	0.6%	1.01x	13.02x
BM_EigenGemm/512/512/512	175.8 us	177.0 us	0.9%	2.23 ms	2.25 ms	1.2%	173.4 us	173.9 us	0.2%	1.01x	12.88x
BM_EigenGemm/64/64/1024	10.4 us	10.5 us	1.0%	82.2 us	82.5 us	0.4%	10.7 us	10.9 us	1.4%	0.98x	7.69x
BM_EigenGemm/64/1024/64	13.9 us	14.0 us	0.4%	86.6 us	86.7 us	0.2%	12.1 us	12.3 us	0.7%	1.15x	7.16x
BM_EigenGemm/256/256/1024	91.2 us	91.9 us	0.9%	1.14 ms	1.14 ms	0.2%	92.7 us	93.2 us	0.2%	0.98x	12.28x

Appendix B: bench_gemm_double, 5 repetitions per build (min, median, cv)

Case	upstream min	upstream median	upstream cv	NEON min	NEON median	NEON cv	MR min	MR median	MR cv	MR vs upstream	MR vs NEON
BM_EigenGemm/8/8/8	110 ns	111 ns	0.9%	72 ns	72 ns	0.3%	73 ns	74 ns	1.1%	1.50x	0.98x
BM_EigenGemm/16/16/16	683 ns	687 ns	0.4%	261 ns	261 ns	0.1%	249 ns	250 ns	0.2%	2.74x	1.05x
BM_EigenGemm/32/32/32	731 ns	737 ns	0.9%	1541 ns	1550 ns	0.3%	673 ns	676 ns	0.5%	1.09x	2.29x
BM_EigenGemm/64/64/64	2919 ns	2960 ns	1.0%	10.7 us	10.7 us	0.2%	2732 ns	2736 ns	0.6%	1.07x	3.90x
BM_EigenGemm/96/96/96	6952 ns	7039 ns	1.0%	34.8 us	34.9 us	0.4%	6697 ns	6743 ns	0.6%	1.04x	5.20x
BM_EigenGemm/128/128/128	13.1 us	13.2 us	1.1%	78.2 us	78.8 us	1.2%	13.1 us	13.3 us	1.9%	1.00x	5.98x
BM_EigenGemm/160/160/160	23.4 us	23.6 us	1.9%	148.2 us	148.8 us	0.5%	23.2 us	23.7 us	1.1%	1.01x	6.39x
BM_EigenGemm/192/192/192	37.3 us	37.8 us	1.3%	247.9 us	249.8 us	0.5%	37.7 us	38.5 us	0.9%	0.99x	6.57x
BM_EigenGemm/224/224/224	56.7 us	57.0 us	2.4%	391.7 us	392.9 us	0.4%	57.1 us	57.6 us	0.7%	0.99x	6.87x
BM_EigenGemm/256/256/256	83.2 us	83.5 us	0.4%	581.5 us	582.5 us	0.2%	83.8 us	84.0 us	0.4%	0.99x	6.94x
BM_EigenGemm/288/288/288	114.2 us	114.8 us	0.8%	814.8 us	821.4 us	0.8%	116.0 us	116.5 us	0.7%	0.98x	7.02x
BM_EigenGemm/320/320/320	153.4 us	154.2 us	0.6%	1.13 ms	1.13 ms	0.6%	155.0 us	157.2 us	0.8%	0.99x	7.26x
BM_EigenGemm/384/384/384	260.9 us	263.2 us	0.7%	1.93 ms	1.95 ms	0.6%	259.8 us	262.2 us	1.1%	1.00x	7.44x
BM_EigenGemm/448/448/448	406.1 us	412.4 us	1.5%	3.04 ms	3.06 ms	0.6%	407.5 us	408.0 us	0.3%	1.00x	7.45x
BM_EigenGemm/512/512/512	680.0 us	685.2 us	0.9%	4.60 ms	4.61 ms	0.2%	664.8 us	669.9 us	0.8%	1.02x	6.92x
BM_EigenGemm/64/64/1024	27.1 us	27.9 us	2.0%	157.2 us	158.6 us	0.5%	27.0 us	27.2 us	0.5%	1.01x	5.82x
BM_EigenGemm/64/1024/64	30.7 us	31.2 us	1.6%	166.8 us	167.4 us	0.7%	28.3 us	28.7 us	0.8%	1.08x	5.89x
BM_EigenGemm/256/256/1024	331.4 us	331.8 us	0.6%	2.30 ms	2.31 ms	0.3%	323.3 us	331.9 us	1.7%	1.02x	7.12x

Appendix C: bench_gemm_complex, 5 repetitions per build (min, median, cv)

Case	upstream min	upstream median	upstream cv	NEON min	NEON median	NEON cv	MR min	MR median	MR cv	MR vs upstream	MR vs NEON
BM_EigenGemm/8/8/8	109 ns	110 ns	1.0%	96 ns	96 ns	0.2%	98 ns	99 ns	1.7%	1.11x	0.97x
BM_EigenGemm/16/16/16	891 ns	896 ns	0.8%	462 ns	464 ns	0.4%	404 ns	404 ns	0.3%	2.21x	1.14x
BM_EigenGemm/32/32/32	696 ns	706 ns	1.2%	3130 ns	3135 ns	0.2%	860 ns	868 ns	0.8%	0.81x	3.64x
BM_EigenGemm/64/64/64	2216 ns	2237 ns	0.7%	21.9 us	22.0 us	0.3%	2210 ns	2213 ns	0.3%	1.00x	9.89x
BM_EigenGemm/96/96/96	7101 ns	7174 ns	0.8%	69.9 us	70.2 us	0.6%	7042 ns	7099 ns	1.2%	1.01x	9.93x
BM_EigenGemm/128/128/128	14.2 us	14.3 us	0.5%	158.5 us	159.2 us	0.4%	13.9 us	14.0 us	0.4%	1.02x	11.38x
BM_EigenGemm/160/160/160	24.6 us	25.2 us	1.1%	303.3 us	304.5 us	0.4%	24.6 us	24.7 us	0.8%	1.00x	12.35x
BM_EigenGemm/192/192/192	39.7 us	40.2 us	1.5%	518.8 us	523.6 us	0.5%	39.9 us	40.0 us	0.9%	0.99x	13.01x
BM_EigenGemm/224/224/224	60.3 us	60.7 us	0.8%	815.0 us	817.9 us	0.4%	59.8 us	60.3 us	0.7%	1.01x	13.63x
BM_EigenGemm/256/256/256	86.6 us	87.2 us	0.5%	1.21 ms	1.22 ms	0.2%	87.0 us	87.3 us	0.2%	1.00x	13.95x
BM_EigenGemm/64/64/1024	28.2 us	28.2 us	1.3%	311.3 us	312.5 us	1.2%	28.0 us	28.2 us	0.7%	1.01x	11.12x
BM_EigenGemm/64/1024/64	31.8 us	32.2 us	0.8%	338.3 us	340.0 us	0.2%	31.7 us	31.7 us	0.5%	1.00x	10.68x
BM_EigenGemm/256/256/1024	325.4 us	330.2 us	0.9%	4.81 ms	4.83 ms	0.4%	324.3 us	327.1 us	0.8%	1.00x	14.82x

Appendix D: bench_gemm_complex_double, 5 repetitions per build (min, median, cv)

Case	upstream min	upstream median	upstream cv	NEON min	NEON median	NEON cv	MR min	MR median	MR cv	MR vs upstream	MR vs NEON
BM_EigenGemm/8/8/8	215 ns	216 ns	0.5%	152 ns	152 ns	0.2%	156 ns	157 ns	0.2%	1.38x	0.97x
BM_EigenGemm/16/16/16	1107 ns	1110 ns	1.1%	875 ns	877 ns	0.3%	910 ns	920 ns	0.7%	1.22x	0.96x
BM_EigenGemm/32/32/32	2550 ns	2568 ns	0.7%	6157 ns	6164 ns	0.2%	1744 ns	1774 ns	1.3%	1.46x	3.53x
BM_EigenGemm/64/64/64	10.4 us	10.5 us	0.5%	44.7 us	44.8 us	1.3%	7690 ns	7741 ns	0.7%	1.35x	5.81x
BM_EigenGemm/96/96/96	20.8 us	21.0 us	0.9%	141.1 us	141.5 us	0.4%	20.9 us	21.0 us	0.4%	1.00x	6.76x
BM_EigenGemm/128/128/128	44.5 us	44.6 us	1.1%	324.3 us	325.0 us	0.4%	44.2 us	45.0 us	2.7%	1.01x	7.33x
BM_EigenGemm/160/160/160	81.5 us	81.8 us	0.8%	619.1 us	619.5 us	0.1%	81.5 us	82.7 us	0.8%	1.00x	7.59x
BM_EigenGemm/192/192/192	135.5 us	135.7 us	0.1%	1.06 ms	1.06 ms	0.2%	137.2 us	137.6 us	0.6%	0.99x	7.72x
BM_EigenGemm/224/224/224	210.6 us	211.5 us	0.6%	1.67 ms	1.67 ms	0.4%	210.3 us	211.1 us	0.2%	1.00x	7.92x
BM_EigenGemm/256/256/256	315.2 us	316.2 us	0.4%	2.49 ms	2.49 ms	0.1%	320.2 us	323.1 us	0.7%	0.98x	7.77x
BM_EigenGemm/64/64/1024	88.3 us	88.5 us	0.3%	637.3 us	642.4 us	0.4%	88.8 us	88.9 us	1.3%	0.99x	7.18x
BM_EigenGemm/64/1024/64	101.2 us	101.3 us	0.6%	681.0 us	681.5 us	0.3%	97.6 us	98.2 us	0.8%	1.04x	6.97x
BM_EigenGemm/256/256/1024	1.35 ms	1.35 ms	0.4%	9.86 ms	9.86 ms	0.1%	1.34 ms	1.35 ms	2.2%	1.00x	7.34x

Appendix E: bench_cholesky (float), 5 repetitions per build (min, median, cv)

Case	upstream min	upstream median	upstream cv	NEON min	NEON median	NEON cv	MR min	MR median	MR cv	MR vs upstream	MR vs NEON
BM_LDLT/4	78 ns	83 ns	5.2%	78 ns	83 ns	5.1%	76 ns	81 ns	5.7%	1.03x	1.03x
BM_LDLT/8	209 ns	210 ns	1.8%	206 ns	211 ns	2.4%	203 ns	205 ns	2.0%	1.03x	1.01x
BM_LDLT/16	590 ns	598 ns	2.8%	594 ns	616 ns	2.5%	573 ns	583 ns	2.4%	1.03x	1.04x
BM_LDLT/32	2043 ns	2058 ns	0.8%	2046 ns	2071 ns	0.9%	1961 ns	1981 ns	1.0%	1.04x	1.04x
BM_LDLT/64	9410 ns	9661 ns	1.4%	9383 ns	9623 ns	1.3%	8753 ns	8953 ns	1.3%	1.08x	1.07x
BM_LDLT/128	43.3 us	43.5 us	1.5%	42.9 us	43.1 us	0.7%	40.7 us	41.7 us	1.8%	1.06x	1.05x
BM_LDLT/256	280.7 us	281.7 us	0.6%	281.1 us	282.7 us	0.5%	266.4 us	270.6 us	0.7%	1.05x	1.05x
BM_LDLT/512	2.30 ms	2.32 ms	0.6%	2.31 ms	2.33 ms	0.5%	2.16 ms	2.19 ms	0.7%	1.06x	1.07x
BM_LDLT/1024	14.73 ms	14.74 ms	0.8%	14.80 ms	14.95 ms	0.9%	14.01 ms	14.05 ms	0.2%	1.05x	1.06x
BM_LDLT/1500	31.33 ms	31.67 ms	0.8%	31.39 ms	32.08 ms	1.6%	30.15 ms	30.26 ms	0.6%	1.04x	1.04x
BM_LLT/4	45 ns	46 ns	2.1%	44 ns	46 ns	3.1%	45 ns	45 ns	1.9%	1.02x	1.00x
BM_LLT/8	124 ns	124 ns	0.2%	121 ns	122 ns	0.6%	120 ns	120 ns	0.3%	1.03x	1.01x
BM_LLT/16	341 ns	344 ns	0.3%	342 ns	344 ns	0.4%	331 ns	331 ns	0.3%	1.03x	1.03x
BM_LLT/32	3742 ns	3777 ns	0.7%	1586 ns	1589 ns	0.4%	1492 ns	1496 ns	0.3%	2.51x	1.06x
BM_LLT/64	12.0 us	12.1 us	0.3%	5305 ns	5405 ns	1.4%	5263 ns	5276 ns	0.6%	2.29x	1.01x
BM_LLT/128	27.3 us	27.3 us	0.4%	20.5 us	20.6 us	0.2%	20.0 us	20.0 us	0.6%	1.37x	1.03x
BM_LLT/256	76.8 us	78.2 us	1.3%	93.6 us	93.9 us	0.2%	70.5 us	70.8 us	0.6%	1.09x	1.33x
BM_LLT/512	279.3 us	281.8 us	1.2%	553.8 us	555.4 us	0.4%	263.5 us	266.4 us	0.9%	1.06x	2.10x
BM_LLT/1024	1.36 ms	1.36 ms	1.5%	3.93 ms	3.93 ms	0.2%	1.39 ms	1.40 ms	0.7%	0.98x	2.82x
BM_LLT/1500	2.58 ms	2.61 ms	1.0%	10.98 ms	10.99 ms	0.1%	2.66 ms	2.68 ms	0.9%	0.97x	4.13x

Appendix F: bench_cholesky_double, 5 repetitions per build (min, median, cv)

Case	upstream min	upstream median	upstream cv	NEON min	NEON median	NEON cv	MR min	MR median	MR cv	MR vs upstream	MR vs NEON
BM_LDLT/4	82 ns	89 ns	5.3%	84 ns	90 ns	4.4%	81 ns	88 ns	4.9%	1.01x	1.03x
BM_LDLT/8	204 ns	222 ns	4.6%	207 ns	226 ns	4.4%	201 ns	215 ns	3.6%	1.01x	1.03x
BM_LDLT/16	586 ns	611 ns	2.3%	602 ns	631 ns	2.4%	574 ns	599 ns	2.6%	1.02x	1.05x
BM_LDLT/32	2347 ns	2370 ns	0.9%	2367 ns	2383 ns	0.7%	2262 ns	2282 ns	0.6%	1.04x	1.05x
BM_LDLT/64	10.9 us	10.9 us	0.6%	10.9 us	10.9 us	0.2%	10.4 us	10.5 us	0.7%	1.04x	1.05x
BM_LDLT/128	55.0 us	55.4 us	0.7%	55.5 us	55.6 us	0.5%	52.8 us	53.0 us	1.0%	1.04x	1.05x
BM_LDLT/256	563.3 us	573.8 us	0.9%	563.7 us	568.1 us	0.9%	541.5 us	542.0 us	0.7%	1.04x	1.04x
BM_LDLT/512	3.73 ms	3.76 ms	0.6%	3.74 ms	3.76 ms	0.5%	3.56 ms	3.60 ms	1.1%	1.05x	1.05x
BM_LDLT/1024	25.90 ms	26.00 ms	0.3%	25.86 ms	25.95 ms	0.9%	25.16 ms	25.19 ms	0.7%	1.03x	1.03x
BM_LDLT/1500	53.90 ms	54.27 ms	0.5%	54.19 ms	54.30 ms	0.3%	52.04 ms	52.05 ms	0.1%	1.04x	1.04x
BM_LLT/4	41 ns	41 ns	0.8%	41 ns	41 ns	1.2%	40 ns	41 ns	0.4%	1.01x	1.01x
BM_LLT/8	124 ns	127 ns	1.2%	124 ns	125 ns	0.8%	122 ns	123 ns	0.1%	1.02x	1.02x
BM_LLT/16	326 ns	331 ns	1.1%	334 ns	337 ns	0.8%	318 ns	318 ns	0.2%	1.03x	1.05x
BM_LLT/32	4268 ns	4300 ns	1.1%	1974 ns	1979 ns	0.2%	1828 ns	1838 ns	0.6%	2.33x	1.08x
BM_LLT/64	16.5 us	16.6 us	0.6%	8035 ns	8073 ns	0.3%	7308 ns	7329 ns	0.2%	2.26x	1.10x
BM_LLT/128	43.7 us	44.2 us	0.9%	34.6 us	34.8 us	0.6%	31.7 us	31.8 us	0.5%	1.38x	1.09x
BM_LLT/256	138.6 us	141.3 us	1.4%	177.8 us	178.8 us	0.3%	115.6 us	115.6 us	0.6%	1.20x	1.54x
BM_LLT/512	516.7 us	526.9 us	1.1%	1.12 ms	1.13 ms	0.8%	499.7 us	501.8 us	0.3%	1.03x	2.23x
BM_LLT/1024	2.70 ms	2.71 ms	0.7%	7.78 ms	7.86 ms	0.7%	2.59 ms	2.60 ms	0.8%	1.04x	3.00x
BM_LLT/1500	5.81 ms	5.89 ms	1.7%	22.04 ms	22.36 ms	0.8%	5.37 ms	5.38 ms	0.9%	1.08x	4.11x

Appendix G: bench_lu, Compute, 5 repetitions per build (min, median, cv)

Case	upstream min	upstream median	upstream cv	NEON min	NEON median	NEON cv	MR min	MR median	MR cv	MR vs upstream	MR vs NEON
PartialPivLU_Compute_float/8	131 ns	152 ns	9.6%	128 ns	149 ns	10.1%	127 ns	143 ns	9.6%	1.03x	1.01x
PartialPivLU_Compute_float/32	2195 ns	2295 ns	3.2%	1868 ns	1971 ns	3.6%	1766 ns	1886 ns	3.8%	1.24x	1.06x
PartialPivLU_Compute_float/64	11.7 us	11.9 us	1.6%	7085 ns	7265 ns	2.3%	6720 ns	6912 ns	2.8%	1.74x	1.05x
PartialPivLU_Compute_float/128	33.8 us	34.0 us	1.2%	33.2 us	33.5 us	1.6%	31.4 us	31.7 us	1.3%	1.08x	1.06x
PartialPivLU_Compute_float/256	132.7 us	135.3 us	1.9%	181.9 us	184.2 us	0.9%	116.6 us	119.4 us	1.9%	1.14x	1.56x
PartialPivLU_Compute_float/512	621.2 us	628.6 us	1.3%	1.14 ms	1.14 ms	0.7%	602.6 us	616.9 us	1.8%	1.03x	1.89x
PartialPivLU_Compute_float/1024	3.07 ms	3.09 ms	1.0%	8.07 ms	8.09 ms	0.6%	3.00 ms	3.00 ms	0.2%	1.02x	2.69x
PartialPivLU_Compute_double/8	145 ns	167 ns	12.8%	174 ns	183 ns	6.1%	138 ns	164 ns	13.3%	1.05x	1.26x
PartialPivLU_Compute_double/32	3849 ns	3989 ns	1.8%	2470 ns	2496 ns	1.5%	2326 ns	2400 ns	2.1%	1.65x	1.06x
PartialPivLU_Compute_double/64	15.7 us	15.8 us	0.7%	10.2 us	10.4 us	1.1%	10.0 us	10.1 us	0.7%	1.56x	1.01x
PartialPivLU_Compute_double/128	57.0 us	58.0 us	1.3%	59.5 us	59.9 us	0.6%	50.5 us	51.1 us	1.2%	1.13x	1.18x
PartialPivLU_Compute_double/256	234.4 us	235.1 us	0.4%	332.9 us	333.4 us	1.6%	188.5 us	193.5 us	1.6%	1.24x	1.77x
PartialPivLU_Compute_double/512	1.14 ms	1.16 ms	1.1%	2.17 ms	2.18 ms	0.4%	1.05 ms	1.05 ms	0.2%	1.09x	2.07x
PartialPivLU_Compute_double/1024	6.40 ms	6.52 ms	1.2%	15.62 ms	15.72 ms	0.4%	6.02 ms	6.03 ms	0.7%	1.06x	2.59x
FullPivLU_Compute_float/8	252 ns	254 ns	4.1%	221 ns	245 ns	6.3%	218 ns	238 ns	7.7%	1.16x	1.01x
FullPivLU_Compute_float/32	3795 ns	3822 ns	1.0%	3785 ns	3835 ns	0.9%	3669 ns	3720 ns	1.1%	1.03x	1.03x
FullPivLU_Compute_float/64	22.0 us	22.2 us	1.1%	22.2 us	22.3 us	0.6%	21.5 us	21.5 us	0.6%	1.03x	1.03x
FullPivLU_Compute_float/128	136.9 us	137.1 us	0.5%	137.6 us	138.3 us	0.3%	134.3 us	135.9 us	0.8%	1.02x	1.02x
FullPivLU_Compute_float/256	1.10 ms	1.10 ms	0.2%	1.08 ms	1.09 ms	0.6%	1.06 ms	1.07 ms	0.5%	1.04x	1.02x
FullPivLU_Compute_float/512	9.22 ms	9.25 ms	0.3%	9.08 ms	9.13 ms	0.4%	8.91 ms	8.93 ms	0.2%	1.04x	1.02x
FullPivLU_Compute_float/1024	70.50 ms	70.99 ms	0.4%	70.08 ms	70.19 ms	1.6%	67.93 ms	68.22 ms	0.2%	1.04x	1.03x
FullPivLU_Compute_double/8	233 ns	241 ns	5.3%	218 ns	229 ns	4.1%	227 ns	229 ns	1.7%	1.03x	0.96x
FullPivLU_Compute_double/32	5538 ns	5623 ns	1.1%	5509 ns	5544 ns	1.2%	5281 ns	5369 ns	1.0%	1.05x	1.04x
FullPivLU_Compute_double/64	34.6 us	34.8 us	0.3%	34.4 us	34.5 us	0.3%	33.4 us	33.5 us	1.6%	1.04x	1.03x
FullPivLU_Compute_double/128	241.4 us	243.7 us	0.8%	239.6 us	240.3 us	0.3%	236.8 us	237.2 us	0.6%	1.02x	1.01x
FullPivLU_Compute_double/256	2.31 ms	2.32 ms	0.5%	2.34 ms	2.35 ms	0.3%	2.23 ms	2.24 ms	0.6%	1.04x	1.05x
FullPivLU_Compute_double/512	17.72 ms	17.73 ms	0.4%	17.68 ms	17.81 ms	0.8%	16.96 ms	16.99 ms	0.3%	1.04x	1.04x
FullPivLU_Compute_double/1024	130.28 ms	131.51 ms	0.5%	129.22 ms	130.62 ms	0.6%	124.36 ms	124.59 ms	0.4%	1.05x	1.04x