Core: Vectorize isZero() and isApproxToConstant() checks

Use fused packet comparisons for float and double isZero() and isApproxToConstant(), sharing the existing visitor traversal without adding a public API. ARMv7 floats use scalar comparisons because NEON flushes subnormal operands while scalar VFP preserves them. Other scalar types retain the original scalar predicates, including C++14 custom scalars that implement only the operations needed by the requested check.

Performance

Dynamic, column-major matrices: all-zero inputs for isZero() and constant 0.5 inputs for isApproxToConstant(0.5). Allocation and initialization are excluded. Speedup = median CPU time before / after.

Intel Xeon E-2176M @ 2.70GHz, pinned to CPU 2, AVX2; GCC 13.3 and Clang 18.1.3, C++17, -O3 -DNDEBUG -march=native. Google Benchmark against origin/master (9d194e7ad4c7), with alternating before/after runs: four samples per compiler and version, ten for selected noisy cases. The CPU governor was powersave; clocks were not locked. Most candidate timings have a median absolute deviation below 10%; three GCC float isZero() cases reach 12-19%, so those ratios are approximate.

Double

Operation Dynamic size GCC speedup Clang speedup
isZero() 3 x 3 2.60x 1.81x
isZero() 4 x 4 4.08x 2.77x
isZero() 8 x 8 4.18x 3.51x
isZero() 32 x 32 4.73x 2.83x
isZero() 256 x 256 2.46x 2.30x
isZero() 1024 x 1024 1.69x 2.06x
isZero() 2048 x 2048 1.40x 1.53x
isApproxToConstant(0.5) 3 x 3 2.15x 2.19x
isApproxToConstant(0.5) 4 x 4 3.56x 2.53x
isApproxToConstant(0.5) 8 x 8 4.32x 2.37x
isApproxToConstant(0.5) 32 x 32 5.38x 2.04x
isApproxToConstant(0.5) 256 x 256 3.74x 2.30x
isApproxToConstant(0.5) 1024 x 1024 3.57x 2.07x
isApproxToConstant(0.5) 2048 x 2048 1.89x 2.21x
Float results
Operation Dynamic size GCC speedup Clang speedup
isZero() 3 x 3 2.81x 4.02x
isZero() 32 x 32 7.82x 8.28x
isZero() 256 x 256 5.24x 6.24x
isZero() 1024 x 1024 4.41x 5.42x
isZero() 2048 x 2048 2.52x 3.10x
isApproxToConstant(0.5) 3 x 3 3.13x 3.96x
isApproxToConstant(0.5) 32 x 32 8.84x 4.34x
isApproxToConstant(0.5) 256 x 256 5.74x 4.97x
isApproxToConstant(0.5) 1024 x 1024 5.87x 4.40x
isApproxToConstant(0.5) 2048 x 2048 4.31x 3.31x

Early exits do not necessarily improve: GCC's double 256 x 256 isApproxToConstant(0.5) with a failure at the first coefficient measured 1.65 ns on master and 1.84 ns here (0.90x). The corresponding Clang case improved from 1.79 ns to 1.50 ns. Passing reductions benefit more from SIMD than immediate scalar failures.

Implementation

  • Reduce comparison masks directly and preserve accumulated failures through packet and scalar tails.
  • Keep the fuzzy visitor, traits, dispatch helpers, and public check definitions together in Visitor.h. Opt into the existing LinearAccess trait, with outer/inner traversal for non-linear expressions, and preserve other visitors' initialization behavior and exit hints.
  • Keep isZero() and isApproxToConstant() free of size thresholds. Triangular checks are unchanged.
  • Select custom-scalar fallbacks through C++14-compatible SFINAE, so unused comparison operations are not instantiated.

Validation

Added tests cover fixed/dynamic sizes, packet boundaries and tails, failures at every position, both storage orders, blocks, unaligned maps, NaNs, empty inputs and custom scalars. Dispatch assertions verify the intended linear and non-linear paths. Matrix-check benchmarks cover small/large sizes and early/late failures.

  • fuzzy_matrix_checks r10 s1: GCC 13 and Clang 18 C++14 native AVX2; Clang 18 C++17 ASan/UBSan; GCC 13 C++14 without vectorization.
  • Visitor parts 11-14: GCC 13 C++14, r10 s1.
  • Earlier validation: Clang 18 C++17 baseline SSE and GCC 13 C++17 native AVX2.
  • Both custom-scalar regression cases fail to compile with the pre-correction implementation and pass with the corrected dispatch.
  • After narrowing the scope: fuzzy_matrix_checks r10 s1 passes Clang 18 C++14 native ASan/UBSan and GCC 13 C++14 without vectorization; the retained benchmark compiles and passes a Clang 18 C++17 smoke run.
  • Changed-line clang-format-17 and git diff --check pass.

Update: Following Rasmus Munk Larsen's review, retain scalar float comparisons on ARMv7, consolidate the visitor implementation, restore the shared exit hint, and limit benchmarks to the two changed predicates. Register every test through numbered CALL_SUBTEST calls and add subnormal cases covering packets, tails, fixed matrices, blocks, both storage orders, and zero/subnormal tolerances.

  • ARMv7 NEON: Clang 18, C++14, -O2 -march=armv7-a -mfpu=neon-vfpv4, QEMU Cortex-A15. The five reported queries pass on master, reproduce four wrong answers before this correction, and all pass afterwards. The full fuzzy test passes r10 s1 under QEMU.
  • Revalidated all five CMake split fuzzy targets and visitor parts 11-14 with GCC 13 C++14, r10 s1; native AVX2 with GCC 13 and Clang 18 C++14; Clang 18 C++17 ASan/UBSan; and GCC 13 C++14 without vectorization.
  • The cleaned benchmark builds and runs with both GCC 13 and Clang 18 C++17; the tables above reflect the corrected implementation.

Recommend affected-tests with all-platforms for broader backend and MSVC coverage. ARMv7 execution was emulated, not native; other non-x86 and device configurations were not validated locally.

Edited by Florian Maurin

Merge request reports

Loading
Loading