Core: fix small fixed-size kernel regressions (#3083)

Summary

#3083 (closed) reports small fixed-size kernels regressed Eigen 3.4 → 5.0.x. This MR fixes three independent SSE/AVX codegen root causes and adds a benchmark for the affected path. No public API or behaviour change.

Root cause 1 — masked-store assignment tail

SmallAssignmentPacketThreshold (!2480, 8a910e308, added for #1342 (closed)) forces small fixed-size assignments onto the scalar path. The real cost is the partial-packet tail, which a fully-unrolled LinearVectorizedTraversal emits as a packet segment — on AVX a masked store (vmaskmovps / vmaskmovpd), slow on current Intel hardware. The threshold also overshot.

Fix: in dense_assignment_loop_impl<…, LinearVectorizedTraversal, CompleteUnrolling> the size is a compile-time constant, so emit the tail as scalar stores directly (UsePacketSegment = false). The NoUnrolling loop keeps the masked path. With the masked tail gone the threshold is unnecessary, so this reverts 8a910e308, restoring the pre-!2480 / Eigen 3.4 traversal selection.

Root cause 2 — SSE predux dead-lane coupling

4fdf87bbf ("clean up intel packet reductions") routed the final 2→1 step of the SSE horizontal reductions through the packed op (predux<Packet2d> → _mm_add_pd); it was previously a scalar add (_mm_add_sd). Both give the same result, but the packed add also writes a live, unused high lane. On legacy-SSE two-operand encoding that dead value couples into the dependency graph — git bisect pinned a ~25% slowdown of #3083 (closed)'s camera kernel to exactly 4fdf87bbf.

Fix: restore the scalar final step for predux<Packet2d> / predux<Packet4f>, gated on \!EIGEN_VECTORIZE_AVX. AVX keeps the packed path — VEX three-operand encoding has no false coupling, and a scalar step would itself regress.

Root cause 3 — narrow LinearVectorized packet selection

find_best_packet<T, Size> returns the widest packet that exactly divides Size, falling through to the smallest packet otherwise. At N=9 float on AVX2 it picks Packet4f and emits 2*xmm + 1 scalar, where 1*ymm + 1 scalar would do (matching what GCC's auto-vectorizer reaches when !2480 scalarized the assignment — Florian Maurin's reviewer benchmark of this shape).

Fix: add find_largest_packet (widest packet ≤ Size, no divisibility requirement) and a selector find_assign_linear_packet that prefers the wider packet only when it strictly cuts the total op count. LinearPacketType flows through the new selector; InnerPacketType stays on find_best_packet (InnerVectorizedTraversal / SliceVectorizedTraversal still require exact divisibility). The op-count guard keeps the existing choice on ties (N=6 double on AVX-512: 3*Packet2d ties 1*Packet4d + 2 scalars → Packet2d kept, LLT/LDLT sub-vector kernels undisturbed).

Codegen, Ryzen 7800X3D:

size, target	MR2581 first commit	with 4f1d18aee
float N=9, AVX2	2*xmm + 1 scalar	1*ymm + 1 scalar
float N=17, AVX2	4*xmm + 1 scalar	2*ymm + 1 scalar
double N=11, AVX-512	5*xmm + 1 scalar	1*zmm + 1*xmm + 1 scalar
double N=6, AVX-512	3*xmm	3*xmm — unchanged
double N=4, AVX-512	1*ymm	1*ymm — unchanged

Results

Core i7-13700HX, g++ 13.3 -O3, median ns/call, double 3×3 inverse() * t (#3083 (closed) Example 2 — headline regression):

build	master	this MR
SSE2	14.3	6.5
AVX2	14.2	5.1

A third commit adds benchmarks/Core/bench_segment_tail.cpp (target bench_segment_tail); its Part C (chained kernels) shows Chained/Camera (#3083 (closed) Example 1) 22.4 → 17.9 (SSE2), 21.9 → 20.6 (AVX2); Block23 (no products) unchanged at 2.0/1.9 ns.

Florian's reviewer shape, all variants

Ryzen 7800X3D, g++ 13.3 -O3, Matrix<T, 256, 1>, storage.segment<N>(1) += storage.tail<N>(), median ns/iter, 50M iter × 4 reps. Tracks Florian's benchmark. -march=haswell matches the AVX2 packet hierarchy of his Xeon E-2176M.

N	shape	!2480	MR2581 first	this MR
float 9 AVX2	Pmax+1	3.20	2.46	2.87
float 17 AVX2	2Pmax+1	5.33	2.71	2.91
float 23 AVX2	3Pmax-1	5.87	2.67	2.68
double 5 AVX2	Pmax+1	2.65	2.80	2.65
double 7 AVX2	2Pmax-1	5.53	2.74	2.46

Tail cases at Pmax+1 / 2Pmax+1 / 3Pmax-1 are 1.8–2.2× faster than MR2581's first commit (which already cut !2480 in half via the masked-tail fix).

Trade-off: at N=17 float on AVX-512 (Pmax=16), the new selector picks Packet16f (1*ZMM + 1 scalar), but the unaligned 64-byte ZMM store at byte-offset 4 straddles a cache line and a tight loop forwards more slowly than MR2581's 4*SSE-128. On AVX2 (no ZMM) the same selector wins. The op-count metric doesn't see this — Packet16f has 2 ops, Packet4f has 5, so it's chosen.

Three-way comparison, geometry / small-matrix kernels

-march=native, 1.5 s min_time, median of 5 reps, ns/call:

case	!2480	MR2581	this MR
BM_LinePointDistance<double>	14.40	1.89	1.88
BM_QuatSetFromTwoVectors<double>	19.70	11.20	11.20
BM_LDLT_Compute<double, 6>	82.50	83.80	84.30
BM_LLT_Compute<double, 6>	66.30	73.70	71.50
BM_LLT_Compute<float, 3>	3.44	5.56	5.57
BM_Inverse<float, 2>	1.28	1.48	1.48
BM_LLT_Compute<float, 4>	(n/a)	56.8	75.1 ← trade-off

MR2581's silent wins over !2480 (LinePointDistance 7.66×, QuatSetFromTwoVectors 1.77×, chained Vector3d ops get enough ILP from SSE packets to break the !2480 scalar dependency chain) are preserved. <double, 6> decompositions stay within ±2 % of MR2581.

Trade-off at BM_LLT_Compute<float, 4> on -march=native: 1.32× regression. The 4×4 LLT's 3×3 trailing-update gets the wider Packet8f instead of Packet4f, ~32% slower in this tight loop on AVX-512. Stable; byte-identical on -march=haswell (AVX2: Packet8f is the max packet either way), so isolated to AVX-512.

Variants explored

I prototyped four alternatives to the op-count selector:

• V0 — Florian's reviewer patch (SmallAssignmentMaxCoeffReadCost = 6 + IsVectorAtCompileTime gate, restores !2480-style scalarization). Restores BM_LLT_Compute<float, 4> on its own (vector-only gate skips LLT's 2D blocks)… but loses BM_LinePointDistance<double> 7.66× silent win (Vector3d.distance() scalarizes), and still regresses BM_LLT_Compute<float, 4> 1.43× on -march=native under default inlining.
• V2 — this MR's selector + IsVectorAtCompileTime gate.
• V3 — this MR's selector + DstAlignment >= largest_alignment gate.
• V4 — both gates.

V2/V3/V4 do not measurably improve BM_LLT_Compute<float, 4> under default inlining on -march=native (all stay at ~1.33×). V3 is a no-op under default EIGEN_UNALIGNED_VECTORIZE=1. None is strictly better than the simple op-count selector, so the MR keeps the minimal form.

No dynamic-size regression

For Size == Dynamic the selector reduces to find_best_packet by construction (both best_ops and largest_ops are 0 → < false → returns best). Asm of 10 typical dynamic-size shapes (VectorXf=VectorXf, MatrixXd+=MatrixXd, MatrixXf.block(off,off,n,n)=…, VectorXd.setZero(), VectorXf *= k, …) is byte-identical to MR2581 first commit on both AVX2 and AVX-512 (modulo auto-generated debug-frame label numbers). Sanity-bench:

case	2581	this MR	ratio
BM_VecAdd/1048576 AVX2	67217 ns	67299 ns	1.001
BM_VecAdd/65536 AVX-512	3667	3720	1.014
BlockWrite_float/1024/128 AVX2	998	1000	1.002
BlockRead_double/1024/64 AVX-512	344	345	1.003
Gemv_double/1024/1024 AVX2	93491	93397	0.999

Cross-toolchain regression sweep

benchmarks/{Core,Geometry}/ (excluding gemm/symv/syr/triangular/dot/reductions which don't touch this path), 6 benches × {gcc 13.3, clang 18} × {native, haswell, sandybridge, westmere}, pinned to core 0. Persistent ≥10 % "regressions" flagged at min_time=0.3 s all involve Matrix<T, Dynamic, Dynamic> cases; at 2 s min_time they collapse to within ±2 % (consistent with the byte-identical Dynamic-size codegen). The only stable cross-toolchain regression on a fixed-size case is BM_LLT_Compute<float, 4> on gcc -march=native (above).

Regression audit — codegen diff

80-kernel SSE2/AVX2 asm diff: 52 / 80 byte-identical on SSE2, 67 / 80 byte-identical on AVX2. Every change is either a single addpd↔️addsd swap (root cause 2, identical throughput) or an assignment kernel with equal-or-fewer instructions (root cause 1 + root cause 3). No kernel gains a costly instruction.

Tests

vectorization_logic_{1,2}, packetmath, redux, commainitializer_{1,2}, basicstuff_{1..8}, linearstructure_{1..12} pass locally. Full ctest left to CI.

Follow-up

has_packet_segment is true only on the AVX backend (slow vmaskmov) and absent on AVX-512 / SVE / RVV where masking is cheap; correct placement tracked in #3086.

Refs #3083 (closed), #1342 (closed), #3086. (#3083 (closed)'s third item — dense sparseView() → SparseMatrix — is tracked separately as #3084 (closed).)