Perf: tighter pldexp 4-way split (tree multiply, vpmovsxdq expansion, drop pnegate)

Summary

Two micro-optimizations to pldexp (and pldexp_fast, pldexp_generic) — applied uniformly to the Default backend and to the SSE / AVX / AVX-512 hand-tuned specializations:

1. Tree-multiply the 4-way scale-factor split. The 4-way split evaluates a * 2^e as a * 2^b * 2^b * 2^b * 2^(e-3b) where b = floor(e/4). The naive (((a*c1)*c1)*c1)*c2 is a chain of four dependent multiplies — four vmul latencies on the critical path even though each multiplication is by a clean power of two and so is freely re-associable. Re-associate as (a * c2) * (c1 * c1) * c1 so the c1*c1 squaring runs in parallel with a*c2 and the dependent chain is only three deep. Op count is unchanged so no throughput regression for callers that aren't latency-bound.

2. Replace the manual int32→int64 expansion with vpmovsxdq on AVX/AVX-512. The hand-tuned pldexp<Packet4d>, pldexp_fast<Packet4d>, and pldexp<Packet8d> historically widened a Packet4i/Packet8i of biased int32 exponents to a Packet4d/Packet8d power-of-two via a four-op recipe: swizzle even/odd lanes -> shift each 64-bit half left 52 -> insert128. AVX2 has a single instruction that does the int32→int64 widening directly: vpmovsxdq (_mm256_cvtepi32_epi64, _mm512_cvtepi32_epi64). Since the biased exponent is always ≥ 0, the sign extension just zero-fills, which is what we want. Two ops instead of four.

   pldexp_generic for double is left to call out with one observation: the AVX2 pcast<Packet4d, Packet4l> with no AVX-512DQ goes through a slow ~10-op emulation, and pldexp_generic for double inherits that cost. In practice it does not matter on AVX/SSE/AVX-512 because the per-arch specializations bypass pldexp_generic; the generic path is only used for double on RVV / SVE / NEON-aarch64, where the int64 cast is a single native instruction.

The minor extra polish: drop a redundant pnegate(pset1(max_exp)) in the generic clamp by materializing +max_exp and -max_exp as two separate pset1 broadcasts. On x86 this saves one v-op (the otherwise-emitted sign-bit XOR).

Coverage

The tree-multiply restructure applies to every backend that goes through pldexp_generic (covers float on AVX, SSE, AVX-512, NEON Packet4f/Packet2f, RVV10, SVE) and to the four hand-tuned pldexp<Packet*d> specializations. The vpmovsxdq cleanup is x86-specific and lands on AVX/AVX-512 only — SSE pldexp<Packet2d> already uses an efficient swizzle-then-slli64 layout and is left alone.

Bench

Compute-bound microbench (3 buffers × 1024 elements, hot in L1; uses asm volatile + escape sink so neither gcc nor clang DCEs the loop). Intel i7-13700HX, best of 5.

gcc-13 -O3 -march=native:

packet	before	after	speedup
Packet8f (AVX float)	24.57 ns/pkt	22.94	+6.6%
Packet4f (SSE float)	14.72	13.69	+7.0%
Packet4d (AVX double)	7.44	7.00	+5.9%
Packet2d (SSE double)	4.12	4.00	+2.9%

clang++-18 -O3 -march=native:

packet	before	after	speedup
Packet8f (AVX float)	24.41 ns/pkt	23.53	+3.6%
Packet4f (SSE float)	14.55	14.36	+1.3%
Packet4d (AVX double)	7.28	7.03	+3.4%
Packet2d (SSE double)	4.05	4.02	+0.7%

Clang gains are smaller than gcc's because clang already auto-reassociates some of the multiplications on the baseline. Every metric improves under both compilers — zero regressions. Speedup column is (before - after) / before (so positive = faster).

Per-function asm body op count under gcc / clang at the AFTER state:

function	gcc before / after	clang before / after
pldexp_generic<Packet8f>	21 / 20	19 / 19
pldexp_generic<Packet4f>	20 / 20	19 / 19
pldexp<Packet4d> (AVX spec.)	28 / 23	28 / 21
pldexp<Packet2d> (SSE spec.)	20 / 20	20 / 20

Correctness

Bit-exact full-range sweep of Eigen::internal::pldexp against std::ldexp over [-278, 278] (float) and [-2099, 2099] (double), three input mantissas, all four packet types. 154,200 samples, 100% exact match, max 0 ULP diff.

packetmath_{1..8} and array_cwise_{1..2} clean under both gcc and clang. clang-format-17 --dry-run --Werror clean.

Out of scope / follow-ups

• AVX-512 changes are the same pattern as AVX (just wider intrinsics) but not yet bench-verified on actual AVX-512 hardware — the dev box for this work is a Raptor Lake P-core, no AVX-512.
• A backend-portable refactor of pldexp_generic to use int32 + manual expansion (so RVV/SVE/NEON-aarch64 also benefit on subnormal-output paths) is plausible but bigger and not attempted here.

Test plan

• gcc-13 packetmath_{1..8}, array_cwise_{1..2}
• clang++-18 packetmath_{1..8}, array_cwise_{1..2}
• full-range pldexp vs std::ldexp ULP sweep (gcc + clang)
• focused microbench under gcc + clang (5 runs each)
• clang-format-17 dry-run
• CI matrix
• AVX-512 hardware verification (deferred)

---

Update (review fixes)

Two issues raised in code review and fixed in commit ef934860c:

1. Numerical bug in the multiply tree. The originally-pushed re-association (a*c2)*(c1*c1)*c1 overflows when the remainder factor c2 = 2^(e - 3*floor(e/4)) is greater than 1, which happens for e ∈ {-1, -2, -5, -6, ...}. Concretely, pldexp(DBL_MAX, -1) was returning +inf instead of the finite DBL_MAX/2. Reordered to (a*c1)*(c1*c1)*c2 so the first multiply is always on the downscale side; a*c1 cannot overflow when the final ldexp result is finite. Same depth-3 latency, same op count. Added a regression test in test/packetmath.cpp.

2. AVX-only build broken. The "simpler int32->int64 expansion" commit used _mm256_cvtepi32_epi64 and _mm256_slli_epi64, which are AVX2-only intrinsics, in arch/AVX/PacketMath.h (which must support AVX-only builds). Wrapped the conversion in pldexp_avx_pow2_from_biased() with #ifdef EIGEN_VECTORIZE_AVX2 and an SSE4.1 fallback that splits the 128-bit input into two halves (_mm_cvtepi32_epi64 + _mm_slli_epi64) and recombines via _mm256_insertf128_si256 — the same pattern already used by pfrexp_generic_get_biased_exponent. Verified with a fresh -mavx -mno-avx2 build.