has_packet_segment / masked partial-packet tails enabled on the wrong backends (slow on AVX2, absent on AVX-512)
Summary
has_packet_segment<Packet> controls whether a partial-packet ("segment") load/store is used for the tail of a vectorized assignment (AssignEvaluator.h, via ploaduSegment / pstoreuSegment / assignPacketSegment). It is currently specialized to std::true_type only in the AVX backend (Eigen/src/Core/arch/AVX/PacketMath.h and arch/AVX/Complex.h), covering Packet4f/Packet8f/Packet2d/Packet4d and the integer/complex packets. Every other backend falls back to the primary std::false_type and uses a scalar tail.
That is backwards relative to how cheap masking actually is per ISA:
- AVX1 / AVX2 —
has_packet_segment = true. The segment lowers tovmaskmovps/vmaskmovpd. Masked stores in particular are slow on current Intel hardware and forward poorly to subsequent loads. This is the backend where masked tails hurt the most, yet it is the only one that opts in. (See !2581 (merged) / #3083 (closed): a fully-unrolled fixed-size assignment ran ~2x slower than master because of this; !2581 (merged) works around it by forcing a scalar tail for theCompleteUnrollingcase.) - AVX-512 —
has_packet_segmentis not specialized, so the tail is scalar. But AVX-512 has first-class mask registers (k0–k7); a writemask is essentially free on loads/stores. This is exactly where masked segments should be used, and currently are not. - ARM SVE, RISC-V V — mask-native ISAs, also not opted in.
- SSE / NEON — no cheap masked primitive; the scalar fallback is correct.
Suggested direction
- Make
has_packet_segmentreflect cheap masking: enable it for the mask-native backends (AVX-512, and evaluate SVE / RVV), whereassignPacketSegmentcan lower to a genuine masked load/store. - Reconsider the AVX1/AVX2 specialization. Options: drop it (always scalar tail there); restrict it to the load side (masked loads are far less costly than masked stores on AVX2); or keep it only for the
NoUnrolling/ dynamic-size case, where a runtime trip count can make even a mediocre masked op worthwhile. - Extend
packetmath/vectorization_logiccoverage so the segment path is exercised per-backend and the choice is validated rather than assumed.
Context
Found while fixing #3083 (closed) (!2581 (merged)). That MR resolves the immediate regression by forcing scalar tails for completely-unrolled fixed-size assignments regardless of backend; this issue tracks the broader, correct placement of masked partial-packet support.