Core: Vectorize isZero() and isApproxToConstant() checks
Use fused packet comparisons for float and double isZero() and isApproxToConstant(), sharing the existing visitor traversal without adding a public API. ARMv7 floats use scalar comparisons because NEON flushes subnormal operands while scalar VFP preserves them. Other scalar types retain the original scalar predicates, including C++14 custom scalars that implement only the operations needed by the requested check.
Performance
Dynamic, column-major matrices: all-zero inputs for isZero() and constant 0.5 inputs for isApproxToConstant(0.5). Allocation and initialization are excluded. Speedup = median CPU time before / after.
Intel Xeon E-2176M @ 2.70GHz, pinned to CPU 2, AVX2; GCC 13.3 and Clang 18.1.3, C++17, -O3 -DNDEBUG -march=native. Google Benchmark against origin/master (9d194e7ad4c7), with alternating before/after runs: four samples per compiler and version, ten for selected noisy cases. The CPU governor was powersave; clocks were not locked. Most candidate timings have a median absolute deviation below 10%; three GCC float isZero() cases reach 12-19%, so those ratios are approximate.
Double
| Operation | Dynamic size | GCC speedup | Clang speedup |
|---|---|---|---|
isZero() |
3 x 3 | 2.60x | 1.81x |
isZero() |
4 x 4 | 4.08x | 2.77x |
isZero() |
8 x 8 | 4.18x | 3.51x |
isZero() |
32 x 32 | 4.73x | 2.83x |
isZero() |
256 x 256 | 2.46x | 2.30x |
isZero() |
1024 x 1024 | 1.69x | 2.06x |
isZero() |
2048 x 2048 | 1.40x | 1.53x |
isApproxToConstant(0.5) |
3 x 3 | 2.15x | 2.19x |
isApproxToConstant(0.5) |
4 x 4 | 3.56x | 2.53x |
isApproxToConstant(0.5) |
8 x 8 | 4.32x | 2.37x |
isApproxToConstant(0.5) |
32 x 32 | 5.38x | 2.04x |
isApproxToConstant(0.5) |
256 x 256 | 3.74x | 2.30x |
isApproxToConstant(0.5) |
1024 x 1024 | 3.57x | 2.07x |
isApproxToConstant(0.5) |
2048 x 2048 | 1.89x | 2.21x |
Float results
| Operation | Dynamic size | GCC speedup | Clang speedup |
|---|---|---|---|
isZero() |
3 x 3 | 2.81x | 4.02x |
isZero() |
32 x 32 | 7.82x | 8.28x |
isZero() |
256 x 256 | 5.24x | 6.24x |
isZero() |
1024 x 1024 | 4.41x | 5.42x |
isZero() |
2048 x 2048 | 2.52x | 3.10x |
isApproxToConstant(0.5) |
3 x 3 | 3.13x | 3.96x |
isApproxToConstant(0.5) |
32 x 32 | 8.84x | 4.34x |
isApproxToConstant(0.5) |
256 x 256 | 5.74x | 4.97x |
isApproxToConstant(0.5) |
1024 x 1024 | 5.87x | 4.40x |
isApproxToConstant(0.5) |
2048 x 2048 | 4.31x | 3.31x |
Early exits do not necessarily improve: GCC's double 256 x 256 isApproxToConstant(0.5) with a failure at the first coefficient measured 1.65 ns on master and 1.84 ns here (0.90x). The corresponding Clang case improved from 1.79 ns to 1.50 ns. Passing reductions benefit more from SIMD than immediate scalar failures.
Implementation
- Reduce comparison masks directly and preserve accumulated failures through packet and scalar tails.
- Keep the fuzzy visitor, traits, dispatch helpers, and public check definitions together in
Visitor.h. Opt into the existingLinearAccesstrait, with outer/inner traversal for non-linear expressions, and preserve other visitors' initialization behavior and exit hints. - Keep
isZero()andisApproxToConstant()free of size thresholds. Triangular checks are unchanged. - Select custom-scalar fallbacks through C++14-compatible SFINAE, so unused comparison operations are not instantiated.
Validation
Added tests cover fixed/dynamic sizes, packet boundaries and tails, failures at every position, both storage orders, blocks, unaligned maps, NaNs, empty inputs and custom scalars. Dispatch assertions verify the intended linear and non-linear paths. Matrix-check benchmarks cover small/large sizes and early/late failures.
fuzzy_matrix_checks r10 s1: GCC 13 and Clang 18 C++14 native AVX2; Clang 18 C++17 ASan/UBSan; GCC 13 C++14 without vectorization.- Visitor parts 11-14: GCC 13 C++14,
r10 s1. - Earlier validation: Clang 18 C++17 baseline SSE and GCC 13 C++17 native AVX2.
- Both custom-scalar regression cases fail to compile with the pre-correction implementation and pass with the corrected dispatch.
- After narrowing the scope:
fuzzy_matrix_checks r10 s1passes Clang 18 C++14 native ASan/UBSan and GCC 13 C++14 without vectorization; the retained benchmark compiles and passes a Clang 18 C++17 smoke run. - Changed-line clang-format-17 and
git diff --checkpass.
Update: Following Rasmus Munk Larsen's review, retain scalar float comparisons on ARMv7, consolidate the visitor implementation, restore the shared exit hint, and limit benchmarks to the two changed predicates. Register every test through numbered CALL_SUBTEST calls and add subnormal cases covering packets, tails, fixed matrices, blocks, both storage orders, and zero/subnormal tolerances.
- ARMv7 NEON: Clang 18, C++14,
-O2 -march=armv7-a -mfpu=neon-vfpv4, QEMU Cortex-A15. The five reported queries pass on master, reproduce four wrong answers before this correction, and all pass afterwards. The full fuzzy test passesr10 s1under QEMU. - Revalidated all five CMake split fuzzy targets and visitor parts 11-14 with GCC 13 C++14,
r10 s1; native AVX2 with GCC 13 and Clang 18 C++14; Clang 18 C++17 ASan/UBSan; and GCC 13 C++14 without vectorization. - The cleaned benchmark builds and runs with both GCC 13 and Clang 18 C++17; the tables above reflect the corrected implementation.
Recommend affected-tests with all-platforms for broader backend and MSVC coverage. ARMv7 execution was emulated, not native; other non-x86 and device configurations were not validated locally.