Core: Fix SSE2 pcmp_lt<Packet2l> comparing low halves as signed
Without SSE4.2 there is no 64-bit integer compare, so pcmp_lt<Packet2l> synthesizes one from 32-bit lane compares combined lexicographically:
hi(a) < hi(b) || (hi(a) == hi(b) && lo(a) < lo(b))The high half carries the sign, but the low half carries magnitude only and must therefore be ordered as unsigned. The fallback used pcmp_lt<Packet4i> — a signed compare — for both halves, so any operand pair sharing a high half whose low halves straddle bit 31 compared backwards:
| a | b | before | correct |
|---|---|---|---|
0x0000000000000000 |
0x00000000FFFFFFFF |
false | true |
0x0000000100000000 |
0x00000001FFFFFFFF |
false | true |
0x00000000FFFFFFFF |
0x0000000000000001 |
true | false |
Biasing only the low lanes by 2^31 lets a single signed 32-bit compare serve both halves, so the fix costs two pxor and no extra compare. hi_eq is computed before the bias and is XOR-invariant, so it is unaffected.
User-visible impact
pcmp_le<Packet2l>, pmin<Packet2l>, pmax<Packet2l> and the generic integer pabsdiff are all built on this compare, so this was not confined to internals. On a plain -O2 x86-64 build (SSE2 baseline, no -march):
Eigen::Array<int64_t, 4, 1> a, b;
a << 0, 1, 0x0000000100000000ll, 5;
b << 0x00000000FFFFFFFFll, 0x00000000FFFFFFFFll, 0x00000001FFFFFFFFll, 7;
a.min(b); // returned the LARGER element in lanes 0-2pabsdiff was further off: for 0x100000000 vs 0x1FFFFFFFF it returned 0xFFFFFFFF00000001 instead of 0xFFFFFFFF.
Why the existing suite missed it
packetmath_boolean_mask_ops_notcomplex_test sets data1[i + PacketSize] to either data1[i] or Scalar(0). Equal operands are ordered correctly whatever the signedness, and for the zero case a full-range random int64_t has a zero high half with probability ~2^-32 — so its operands effectively never share a high half, and the lexicographic path is never reached.
The new packetmath_split_half_compare_test builds pairs that do share a high half, with low halves chosen around the half-width sign bit, and runs over every integer scalar whose packet has comparisons (so it also guards the equivalent decomposition in any future backend, at any width). It covers pcmp_lt, pcmp_le, pcmp_eq, pmin, pmax and pabsdiff.
The test fails at the parent commit for int64_t from SSE2 through SSE4.1, and passes from SSE4.2 up where _mm_cmpgt_epi64 takes over.
Validation
Run on a 13th Gen Intel Core i7-13700HX host.
packetmath, all 15 parts, passing at: SSE2 baseline,-msse3 -mno-ssse3,-mssse3 -mno-sse4.1,-msse4.1 -mno-sse4.2,-msse4.2,-mavx2 -mfma. The intermediate levels matter because the emulated path stays live through SSE4.1, and at SSE4.1pcmp_eqis native whilepcmp_ltis still emulated.- Confirmed the new test fails at the parent commit and passes with the fix.
- Exhaustive sweep of
pcmp_lt/pcmp_le/pcmp_eq/pmin/pmax/pabsdiffover all pairs from a boundary set (0, 1,INT64_MIN,INT64_MAX,-1,2^32 - 1,2^32,2^33 - 1): 0 mismatches after, 12 before. array_cwiseinteger-predicate subtests pass at the SSE2 baseline.- Clean under
-fsanitize=undefined,addressfor int8/int16/int32/uint32/int64/uint64. (packetmath_generic_*reports a pre-existing signed-overflow atGenericPacketMath.h:366for uint16 that reproduces on a pristinemastercheckout and is unrelated to this change.) - Builds clean at
-std=c++14with both gcc and clang. - AVX-512 configurations compile but could not be executed on this host, which has no AVX-512.
I also swept every other backend (AVX, AVX512, NEON, SVE, AltiVec, ZVector, MSA, LSX, RVV10, HVX, clang, GPU/SYCL/HIP) for the same bug class — a limb-decomposed wide compare using the wrong signedness on one half. This SSE2 fallback is the only instance in the tree; AVX's Packet4ul biases the whole 64-bit value rather than a limb and is correct, and AltiVec's split pcmp_eq<Packet2l> is signedness-invariant.
Relationship to !2817 (merged)
!2817 (merged) (Optimize NEON 64-bit integral packets) adds a 64-bit boundary test that is the first thing in the suite to reach this path, so its pipeline is currently red on every SSE2-baseline x86 job for a defect that has nothing to do with that MR. This fix unblocks it.