Core: Fix SSE2 pcmp_lt<Packet2l> comparing low halves as signed

Without SSE4.2 there is no 64-bit integer compare, so pcmp_lt<Packet2l> synthesizes one from 32-bit lane compares combined lexicographically:

hi(a) < hi(b) || (hi(a) == hi(b) && lo(a) < lo(b))

The high half carries the sign, but the low half carries magnitude only and must therefore be ordered as unsigned. The fallback used pcmp_lt<Packet4i> — a signed compare — for both halves, so any operand pair sharing a high half whose low halves straddle bit 31 compared backwards:

a b before correct
0x0000000000000000 0x00000000FFFFFFFF false true
0x0000000100000000 0x00000001FFFFFFFF false true
0x00000000FFFFFFFF 0x0000000000000001 true false

Biasing only the low lanes by 2^31 lets a single signed 32-bit compare serve both halves, so the fix costs two pxor and no extra compare. hi_eq is computed before the bias and is XOR-invariant, so it is unaffected.

User-visible impact

pcmp_le<Packet2l>, pmin<Packet2l>, pmax<Packet2l> and the generic integer pabsdiff are all built on this compare, so this was not confined to internals. On a plain -O2 x86-64 build (SSE2 baseline, no -march):

Eigen::Array<int64_t, 4, 1> a, b;
a << 0, 1, 0x0000000100000000ll, 5;
b << 0x00000000FFFFFFFFll, 0x00000000FFFFFFFFll, 0x00000001FFFFFFFFll, 7;
a.min(b);   // returned the LARGER element in lanes 0-2

pabsdiff was further off: for 0x100000000 vs 0x1FFFFFFFF it returned 0xFFFFFFFF00000001 instead of 0xFFFFFFFF.

Why the existing suite missed it

packetmath_boolean_mask_ops_notcomplex_test sets data1[i + PacketSize] to either data1[i] or Scalar(0). Equal operands are ordered correctly whatever the signedness, and for the zero case a full-range random int64_t has a zero high half with probability ~2^-32 — so its operands effectively never share a high half, and the lexicographic path is never reached.

The new packetmath_split_half_compare_test builds pairs that do share a high half, with low halves chosen around the half-width sign bit, and runs over every integer scalar whose packet has comparisons (so it also guards the equivalent decomposition in any future backend, at any width). It covers pcmp_lt, pcmp_le, pcmp_eq, pmin, pmax and pabsdiff.

The test fails at the parent commit for int64_t from SSE2 through SSE4.1, and passes from SSE4.2 up where _mm_cmpgt_epi64 takes over.

Validation

Run on a 13th Gen Intel Core i7-13700HX host.

  • packetmath, all 15 parts, passing at: SSE2 baseline, -msse3 -mno-ssse3, -mssse3 -mno-sse4.1, -msse4.1 -mno-sse4.2, -msse4.2, -mavx2 -mfma. The intermediate levels matter because the emulated path stays live through SSE4.1, and at SSE4.1 pcmp_eq is native while pcmp_lt is still emulated.
  • Confirmed the new test fails at the parent commit and passes with the fix.
  • Exhaustive sweep of pcmp_lt/pcmp_le/pcmp_eq/pmin/pmax/pabsdiff over all pairs from a boundary set (0, 1, INT64_MIN, INT64_MAX, -1, 2^32 - 1, 2^32, 2^33 - 1): 0 mismatches after, 12 before.
  • array_cwise integer-predicate subtests pass at the SSE2 baseline.
  • Clean under -fsanitize=undefined,address for int8/int16/int32/uint32/int64/uint64. (packetmath_generic_* reports a pre-existing signed-overflow at GenericPacketMath.h:366 for uint16 that reproduces on a pristine master checkout and is unrelated to this change.)
  • Builds clean at -std=c++14 with both gcc and clang.
  • AVX-512 configurations compile but could not be executed on this host, which has no AVX-512.

I also swept every other backend (AVX, AVX512, NEON, SVE, AltiVec, ZVector, MSA, LSX, RVV10, HVX, clang, GPU/SYCL/HIP) for the same bug class — a limb-decomposed wide compare using the wrong signedness on one half. This SSE2 fallback is the only instance in the tree; AVX's Packet4ul biases the whole 64-bit value rather than a limb and is correct, and AltiVec's split pcmp_eq<Packet2l> is signedness-invariant.

Relationship to !2817 (merged)

!2817 (merged) (Optimize NEON 64-bit integral packets) adds a 64-bit boundary test that is the first thing in the suite to reach this path, so its pipeline is currently red on every SSE2-baseline x86 job for a defect that has nothing to do with that MR. This fix unblocks it.

Merge request reports

Loading
Loading