Loading
Core: Accelerate real vector operations with SME2
Adds SME2 kernels for float and double dot products, scaled vector addition (y += alpha*x), and column-major matrix–vector multiplication (y.noalias() += A*x). The kernels reuse Core's dispatch, cache settings, alignment helpers, and generic fallbacks.
At head 92c90383f, GEMV bounds its unscaled column sums, the kernels leave the caller's floating-point exception flags unchanged and report none of their own, and DOT keeps the IEEE 754 sign of a zero sum. The correctness fixes increased sampled DOT/AXPY times by 2–16% and GEMV times by 9–36% relative to the previous MR revision 2be092ad6. The follow-ups in the updates below make GEMV up to 2.3× faster than before them, remove a 6–12× slowdown of zero-valued DOT, and make resident DOT and AXPY 1.5–10% faster again. Earlier comparisons against the base commit are retained below and are explicitly historical.
Update (92c90383f), after @tesch1's review, measured on an Apple M4 Mac mini against 13b056920:
- An FPSR value read while streaming is unreliable. Besides SMSTART/SMSTOP, resuming a thread that was preempted in streaming mode sets the cumulative flags. Exact float DOT at
2^{20}elements left spurious flags after 11 of 2,000 calls, and AXPY after 17. DOT and AXPY now save FPSR before the streaming call and restore it afterwards, as GEMV already did, and none of those calls is flagged. The vector kernels therefore report no floating-point exceptions: ZA arithmetic raises none, and the flags of the SVE prefix, tails and reduction are discarded. Dropping the two FPSR accesses made while streaming makes resident DOT and AXPY 1.5–10% faster. The vectorization page lists the new kernels and this exception behavior. - The M4-fitted crossovers are named constants, and the
setCpuCacheSizesdocumentation is backend-neutral again. - Two things stay after measurement. Forcing SME past the L2 upper bound wins only for resident 8 MiB operands and is 2–16% slower at 16 MiB per operand on this M4 (table in the review thread). GEMV's column batch of 4 instead of 16 costs at most 0.4% on 32,768–200,000 × 512 matrices, and it keeps SME overflowing no earlier than generic GEMV.
The only case with a larger slowdown, produced double DOT at 8,192 elements, has overlapping IQRs; three reruns gave +0.5% to +9% with similar dispersion.
Apple M4: DOT and AXPY, `13b05692` versus `92c90383`
| Case | 13b056920, ns [IQR] |
92c90383f, ns [IQR] |
Time change |
|---|---|---|---|
| DOT, float, 8,192 (generic path) | 430.7 [427.6, 434.6] | 433.9 [432.0, 435.6] | +0.7% |
| DOT, float, 16,384 | 281.8 [274.9, 282.0] | 259.0 [253.8, 259.2] | -8.1% |
| DOT, float, 65,536 | 714.1 [713.9, 714.7] | 695.5 [692.6, 698.5] | -2.6% |
| DOT, double, 8,192 | 275.2 [274.8, 275.4] | 253.5 [251.4, 253.7] | -7.9% |
| DOT, double, 16,384 | 418.5 [418.0, 422.3] | 397.6 [397.0, 399.4] | -5.0% |
| DOT, double, 65,536 | 1,297.0 [1,296.7, 1,301.1] | 1,277.7 [1,275.7, 1,278.7] | -1.5% |
| DOT with producer, float, 8,192 (generic path) | 945.5 [944.8, 957.3] | 950.8 [947.2, 957.5] | +0.6% |
| DOT with producer, float, 16,384 | 2,584.9 [2,532.9, 2,644.7] | 2,596.9 [2,581.4, 2,618.6] | +0.5% |
| DOT with producer, float, 65,536 | 6,742.8 [6,737.2, 6,750.4] | 6,707.0 [6,675.2, 6,716.3] | -0.5% |
| DOT with producer, double, 8,192 | 2,535.0 [2,423.2, 2,702.6] | 2,819.4 [2,673.4, 3,005.0] | +11.2% |
| DOT with producer, double, 16,384 | 3,600.2 [3,594.7, 3,602.9] | 3,583.7 [3,577.3, 3,587.0] | -0.5% |
| DOT with producer, double, 65,536 | 12,161 [12,144, 12,170] | 12,089 [11,995, 12,207] | -0.6% |
| AXPY, float, 16,384 (generic path) | 1,025.9 [1,024.9, 1,029.7] | 1,026.6 [1,025.6, 1,029.1] | +0.1% |
| AXPY, float, 32,768 | 724.2 [723.5, 725.5] | 652.9 [652.8, 653.1] | -9.8% |
| AXPY, float, 65,536 | 1,376.5 [1,376.2, 1,376.7] | 1,304.0 [1,303.9, 1,304.0] | -5.3% |
| AXPY, double, 16,384 | 725.3 [724.3, 725.5] | 653.0 [652.8, 653.1] | -10.0% |
| AXPY, double, 32,768 | 1,374.7 [1,374.4, 1,375.3] | 1,304.2 [1,303.7, 1,304.7] | -5.1% |
| AXPY, double, 65,536 | 2,677.4 [2,676.3, 2,678.0] | 2,606.0 [2,605.4, 2,606.3] | -2.7% |
| AXPY with producer/consumer, float, 16,384 (generic path) | 3,241.3 [3,212.7, 3,262.6] | 3,230.6 [3,207.3, 3,281.6] | -0.3% |
| AXPY with producer/consumer, float, 32,768 | 6,250.8 [6,171.4, 6,318.9] | 6,259.2 [6,240.3, 6,260.8] | +0.1% |
| AXPY with producer/consumer, float, 65,536 | 11,082 [10,953, 11,089] | 10,657 [10,626, 10,726] | -3.8% |
| AXPY with producer/consumer, double, 16,384 | 6,215.9 [6,105.4, 6,272.9] | 6,069.0 [6,052.9, 6,118.4] | -2.4% |
| AXPY with producer/consumer, double, 32,768 | 10,851 [10,708, 11,069] | 10,753 [10,517, 10,799] | -0.9% |
| AXPY with producer/consumer, double, 65,536 | 21,221 [20,800, 21,635] | 21,475 [20,937, 21,496] | +1.2% |
Apple M4 (sysctl) Mac mini, 24 GiB, macOS 27.0; Apple clang 21.0.0, -std=c++14 -O3 -DNDEBUG -mcpu=apple-m4, Google Benchmark 1.9.5, this MR's benchmark sources on both sides. Each binary ran alone, alternating old/new twice, five repetitions per invocation, minimum 0.05 s; DOT CPU time, AXPY wall time, ten samples per case. The machine's CI VM was stopped; one-minute load was 2.6–3.4. GEMV's kernel and guard are unchanged.
Update (fae71a1b7, 2ea0577a2): two follow-ups, measured on Apple M4 against b1d25110e:
- GEMV's predicated remainder accumulated into ZA group 0 only, a serial FMLA chain of about 1 ns per column, and matrices with fewer than 16 streaming vectors of rows ran entirely on it. Each row block now has four predicated chunks with independent ZA groups, full or partial. GEMV performs only ZA arithmetic, so the FPSR accesses inside streaming mode (60–95 ns per call) and the
alpha == 0compare move out of the streaming call. Results are bitwise identical tob1d25110eforfloatanddouble. - A zero SME DOT result was recomputed through the generic inner product, which made exactly zero dot products, e.g. of vectors with disjoint supports, 6–12× slower than nonzero ones. ZA and the tail accumulator now start at −0, the additive identity in every rounding mode except roundTowardNegative. There
(+0) + (-0) = -0, so a zero result still takes the generic path; FPCR is read only then, outside streaming mode.
| Case | b1d25110e, ns [IQR] |
2ea0577a2, ns [IQR] |
Time change |
|---|---|---|---|
| GEMV, float, 256 × 32 | 162.1 [159.0, 162.6] | 70.1 [70.0, 70.2] | -56.8% |
| GEMV, float, 128 × 64 | 220.7 [220.7, 221.0] | 106.9 [106.8, 107.0] | -51.6% |
| GEMV, float, 1,024 × 128 | 969.0 [968.0, 969.8] | 870.1 [868.5, 870.9] | -10.2% |
| GEMV, float, 129 × 128 | 431.1 [430.6, 431.7] | 252.6 [252.4, 252.7] | -41.4% |
| GEMV, float, 257 × 128 | 455.2 [454.8, 456.3] | 386.3 [386.1, 386.6] | -15.1% |
| GEMV, float, 10,000 × 8 | 736.9 [736.5, 738.2] | 635.8 [625.8, 646.1] | -13.7% |
| GEMV, double, 128 × 32 | 157.5 [155.7, 159.3] | 70.4 [70.4, 70.5] | -55.3% |
| GEMV, double, 1,024 × 128 | 1,827.9 [1,826.1, 1,829.9] | 1,739.7 [1,738.8, 1,741.4] | -4.8% |
| GEMV, double, 257 × 32 | 256.1 [254.4, 256.5] | 167.8 [167.7, 168.9] | -34.5% |
| DOT, float, 65,536 | 721.8 [719.4, 722.4] | 730.2 [722.7, 731.2] | +1.2% |
| DOT, zero result, float, 65,536 | 8,417.3 [8,244.5, 8,468.1] | 721.5 [720.0, 723.8] | -91.4% |
| DOT, zero result, double, 65,536 | 15,088 [14,566, 15,293] | 1,318.6 [1,317.3, 1,320.3] | -91.3% |
SME-dispatched GEMV layout cases take 0.43–0.97× the previous time; cases dispatched to NEON and nonzero DOT are unchanged within their dispersion. Apple M4 (sysctl), 24 GiB, macOS 27.0; Apple clang 21.0.0, -std=c++14 -O3 -DNDEBUG -mcpu=apple-m4, Google Benchmark 1.9.5, this MR's benchmark sources on both sides. Each binary ran alone, alternating old/new twice, five repetitions per invocation, minimum 0.05 s; CPU time, ten samples per case. One-minute load was 2–5 with another user logged in. The GEMV rows use fae71a1b7's kernel, which 2ea0577a2 does not change.
Apple M4: all GEMV layout cases and DOT, `b1d25110` versus `2ea0577a`
| Case | b1d25110e, ns [IQR] |
2ea0577a2, ns [IQR] |
Time change |
|---|---|---|---|
| GEMV, float, 128 × 4 | 23.9 [23.8, 24.0] | 23.9 [23.9, 23.9] | +0.0% |
| GEMV, float, 256 × 4 | 40.2 [40.1, 40.3] | 40.0 [40.0, 40.1] | -0.5% |
| GEMV, float, 1,024 × 4 | 138.7 [138.5, 139.1] | 137.9 [137.8, 138.1] | -0.6% |
| GEMV, float, 128 × 16 | 61.5 [61.1, 61.5] | 61.9 [61.5, 62.5] | +0.7% |
| GEMV, float, 256 × 16 | 116.7 [116.5, 116.7] | 116.7 [116.7, 116.9] | +0.1% |
| GEMV, float, 1,024 × 16 | 448.1 [447.4, 448.9] | 448.5 [448.1, 449.0] | +0.1% |
| GEMV, float, 128 × 32 | 112.5 [112.1, 113.0] | 113.0 [112.8, 113.0] | +0.4% |
| GEMV, float, 256 × 32 | 162.1 [159.0, 162.6] | 70.1 [70.0, 70.2] | -56.8% |
| GEMV, float, 1,024 × 32 | 314.5 [312.7, 316.2] | 213.5 [213.4, 216.5] | -32.1% |
| GEMV, float, 128 × 64 | 220.7 [220.7, 221.0] | 106.9 [106.8, 107.0] | -51.6% |
| GEMV, float, 256 × 64 | 208.1 [207.6, 208.4] | 107.0 [106.9, 107.1] | -48.6% |
| GEMV, float, 1,024 × 64 | 491.3 [488.9, 491.6] | 390.4 [389.4, 391.1] | -20.5% |
| GEMV, float, 128 × 128 | 308.5 [308.3, 308.7] | 202.6 [202.6, 202.9] | -34.3% |
| GEMV, float, 256 × 128 | 320.3 [319.9, 323.7] | 222.1 [222.0, 222.4] | -30.6% |
| GEMV, float, 1,024 × 128 | 969.0 [968.0, 969.8] | 870.1 [868.5, 870.9] | -10.2% |
| GEMV, float, 129 × 32 | 128.7 [128.6, 129.0] | 129.3 [129.2, 129.4] | +0.5% |
| GEMV, float, 257 × 32 | 194.3 [194.1, 196.8] | 112.5 [112.4, 112.6] | -42.1% |
| GEMV, float, 129 × 128 | 431.1 [430.6, 431.7] | 252.6 [252.4, 252.7] | -41.4% |
| GEMV, float, 257 × 128 | 455.2 [454.8, 456.3] | 386.3 [386.1, 386.6] | -15.1% |
| GEMV, float, 129 × 32; offsets A +16 B, y +0 B | 128.5 [128.4, 128.7] | 128.5 [128.4, 128.6] | +0.0% |
| GEMV, float, 257 × 32; offsets A +16 B, y +0 B | 195.3 [195.1, 195.5] | 112.3 [112.2, 112.4] | -42.5% |
| GEMV, float, 129 × 128; offsets A +16 B, y +0 B | 435.8 [434.3, 437.7] | 252.7 [252.3, 253.0] | -42.0% |
| GEMV, float, 257 × 128; offsets A +16 B, y +0 B | 454.0 [452.7, 454.3] | 386.9 [386.5, 387.0] | -14.8% |
| GEMV, float, 129 × 32; offsets A +0 B, y +16 B | 128.4 [127.9, 128.9] | 129.6 [129.5, 129.7] | +0.9% |
| GEMV, float, 257 × 32; offsets A +0 B, y +16 B | 261.4 [257.4, 261.5] | 114.0 [113.7, 114.0] | -56.4% |
| GEMV, float, 129 × 128; offsets A +0 B, y +16 B | 581.0 [578.9, 583.1] | 453.7 [453.7, 454.0] | -21.9% |
| GEMV, float, 257 × 128; offsets A +0 B, y +16 B | 653.7 [647.9, 661.4] | 505.2 [505.0, 505.3] | -22.7% |
| GEMV, float, 129 × 32; offsets A +16 B, y +16 B | 127.7 [127.1, 128.4] | 128.4 [128.3, 128.6] | +0.5% |
| GEMV, float, 257 × 32; offsets A +16 B, y +16 B | 258.5 [258.2, 258.8] | 114.0 [113.9, 114.1] | -55.9% |
| GEMV, float, 129 × 128; offsets A +16 B, y +16 B | 581.3 [580.8, 581.5] | 454.4 [454.3, 454.8] | -21.8% |
| GEMV, float, 257 × 128; offsets A +16 B, y +16 B | 646.4 [644.5, 647.7] | 504.6 [504.3, 505.0] | -21.9% |
| GEMV, float, 129 × 32; padded stride | 126.6 [126.1, 127.1] | 127.0 [126.9, 127.1] | +0.3% |
| GEMV, float, 257 × 32; padded stride | 198.8 [194.8, 199.1] | 112.3 [112.3, 112.5] | -43.5% |
| GEMV, float, 129 × 128; padded stride | 439.3 [438.5, 440.8] | 251.8 [251.3, 252.2] | -42.7% |
| GEMV, float, 257 × 128; padded stride | 441.4 [441.0, 443.7] | 386.0 [385.6, 386.2] | -12.5% |
| GEMV, float, 129 × 32; offsets A +16 B, y +0 B, padded stride | 126.6 [126.4, 126.8] | 127.0 [126.9, 127.0] | +0.3% |
| GEMV, float, 257 × 32; offsets A +16 B, y +0 B, padded stride | 198.1 [197.8, 198.4] | 112.4 [112.3, 112.5] | -43.3% |
| GEMV, float, 129 × 128; offsets A +16 B, y +0 B, padded stride | 437.5 [434.0, 438.6] | 251.2 [251.2, 251.4] | -42.6% |
| GEMV, float, 257 × 128; offsets A +16 B, y +0 B, padded stride | 442.6 [439.9, 444.4] | 384.6 [384.0, 385.0] | -13.1% |
| GEMV, float, 129 × 32; offsets A +0 B, y +16 B, padded stride | 126.9 [126.9, 127.1] | 127.0 [126.9, 127.1] | +0.0% |
| GEMV, float, 257 × 32; offsets A +0 B, y +16 B, padded stride | 262.1 [260.1, 264.1] | 112.7 [112.7, 114.0] | -57.0% |
| GEMV, float, 129 × 128; offsets A +0 B, y +16 B, padded stride | 575.5 [575.0, 578.2] | 452.9 [452.6, 453.1] | -21.3% |
| GEMV, float, 257 × 128; offsets A +0 B, y +16 B, padded stride | 647.1 [645.5, 648.0] | 502.9 [502.9, 503.0] | -22.3% |
| GEMV, float, 129 × 32; offsets A +16 B, y +16 B, padded stride | 126.8 [126.4, 126.9] | 126.7 [126.7, 126.9] | -0.1% |
| GEMV, float, 257 × 32; offsets A +16 B, y +16 B, padded stride | 262.9 [260.6, 263.0] | 113.7 [113.4, 114.5] | -56.7% |
| GEMV, float, 129 × 128; offsets A +16 B, y +16 B, padded stride | 580.0 [578.0, 580.8] | 454.4 [454.3, 454.6] | -21.7% |
| GEMV, float, 257 × 128; offsets A +16 B, y +16 B, padded stride | 645.0 [644.8, 645.3] | 502.8 [502.2, 503.3] | -22.1% |
| GEMV, float, 4,096 × 4 | 529.5 [529.3, 529.9] | 528.5 [528.3, 528.7] | -0.2% |
| GEMV, float, 10,000 × 4 | 512.0 [506.1, 518.0] | 422.7 [422.5, 423.0] | -17.4% |
| GEMV, float, 4,096 × 8 | 359.3 [353.8, 365.1] | 268.3 [262.3, 269.4] | -25.3% |
| GEMV, float, 10,000 × 8 | 736.9 [736.5, 738.2] | 635.8 [625.8, 646.1] | -13.7% |
| GEMV, float, 4,096 × 16 | 551.0 [548.8, 552.8] | 451.2 [449.3, 454.8] | -18.1% |
| GEMV, float, 10,000 × 16 | 1,230.4 [1,229.2, 1,230.5] | 1,144.4 [1,143.1, 1,145.8] | -7.0% |
| GEMV, double, 128 × 4 | 39.1 [39.1, 39.1] | 39.1 [39.0, 39.1] | -0.2% |
| GEMV, double, 256 × 4 | 71.6 [71.4, 71.6] | 71.7 [71.4, 72.1] | +0.2% |
| GEMV, double, 1,024 × 4 | 284.5 [284.3, 284.7] | 267.3 [267.1, 267.4] | -6.0% |
| GEMV, double, 128 × 16 | 115.4 [115.3, 115.5] | 115.5 [115.5, 115.6] | +0.1% |
| GEMV, double, 256 × 16 | 161.7 [161.3, 162.3] | 73.6 [73.6, 73.6] | -54.5% |
| GEMV, double, 1,024 × 16 | 327.7 [327.3, 330.3] | 236.2 [236.0, 236.5] | -27.9% |
| GEMV, double, 128 × 32 | 157.5 [155.7, 159.3] | 70.4 [70.4, 70.5] | -55.3% |
| GEMV, double, 256 × 32 | 204.6 [202.4, 206.5] | 110.3 [110.2, 110.4] | -46.1% |
| GEMV, double, 1,024 × 32 | 481.9 [481.6, 482.5] | 392.9 [392.2, 393.6] | -18.5% |
| GEMV, double, 128 × 64 | 200.0 [199.7, 200.4] | 107.1 [107.1, 107.1] | -46.4% |
| GEMV, double, 256 × 64 | 291.4 [291.2, 291.7] | 193.3 [192.7, 195.0] | -33.7% |
| GEMV, double, 1,024 × 64 | 825.9 [825.3, 826.3] | 738.5 [737.7, 739.2] | -10.6% |
| GEMV, double, 128 × 128 | 317.4 [315.9, 319.9] | 219.4 [219.1, 219.7] | -30.9% |
| GEMV, double, 256 × 128 | 534.7 [533.4, 535.0] | 435.8 [435.2, 436.1] | -18.5% |
| GEMV, double, 1,024 × 128 | 1,827.9 [1,826.1, 1,829.9] | 1,739.7 [1,738.8, 1,741.4] | -4.8% |
| GEMV, double, 129 × 32 | 189.4 [189.3, 189.5] | 112.6 [112.6, 112.7] | -40.5% |
| GEMV, double, 257 × 32 | 256.1 [254.4, 256.5] | 167.8 [167.7, 168.9] | -34.5% |
| GEMV, double, 129 × 128 | 445.5 [444.7, 446.4] | 386.4 [385.5, 386.9] | -13.3% |
| GEMV, double, 257 × 128 | 725.1 [724.0, 726.4] | 658.9 [658.3, 659.4] | -9.1% |
| GEMV, double, 129 × 32; offsets A +16 B, y +0 B | 191.1 [190.9, 195.0] | 112.6 [112.5, 112.7] | -41.1% |
| GEMV, double, 257 × 32; offsets A +16 B, y +0 B | 257.5 [256.3, 257.6] | 169.8 [167.9, 170.0] | -34.0% |
| GEMV, double, 129 × 128; offsets A +16 B, y +0 B | 442.5 [442.1, 443.9] | 387.2 [387.0, 387.4] | -12.5% |
| GEMV, double, 257 × 128; offsets A +16 B, y +0 B | 723.6 [721.3, 726.4] | 652.7 [650.9, 653.7] | -9.8% |
| GEMV, double, 129 × 32; offsets A +0 B, y +16 B | 257.7 [256.7, 260.2] | 112.8 [112.7, 113.7] | -56.2% |
| GEMV, double, 257 × 32; offsets A +0 B, y +16 B | 371.7 [369.4, 372.0] | 180.4 [180.2, 180.8] | -51.5% |
| GEMV, double, 129 × 128; offsets A +0 B, y +16 B | 649.4 [649.1, 649.7] | 505.1 [504.6, 505.5] | -22.2% |
| GEMV, double, 257 × 128; offsets A +0 B, y +16 B | 1,210.3 [1,209.2, 1,211.1] | 1,066.7 [1,066.0, 1,067.6] | -11.9% |
| GEMV, double, 129 × 32; offsets A +16 B, y +16 B | 262.1 [260.8, 262.3] | 113.2 [113.1, 113.3] | -56.8% |
| GEMV, double, 257 × 32; offsets A +16 B, y +16 B | 370.2 [368.2, 371.6] | 181.6 [181.3, 181.9] | -50.9% |
| GEMV, double, 129 × 128; offsets A +16 B, y +16 B | 646.0 [644.9, 646.6] | 504.7 [504.5, 505.0] | -21.9% |
| GEMV, double, 257 × 128; offsets A +16 B, y +16 B | 1,214.3 [1,211.5, 1,215.3] | 1,068.5 [1,067.8, 1,069.7] | -12.0% |
| GEMV, double, 129 × 32; padded stride | 191.0 [189.7, 192.8] | 112.6 [112.6, 112.7] | -41.0% |
| GEMV, double, 257 × 32; padded stride | 255.3 [253.5, 255.6] | 169.1 [168.5, 169.4] | -33.8% |
| GEMV, double, 129 × 128; padded stride | 433.4 [431.9, 434.9] | 386.9 [385.7, 387.7] | -10.7% |
| GEMV, double, 257 × 128; padded stride | 699.0 [697.4, 702.2] | 636.1 [635.5, 637.8] | -9.0% |
| GEMV, double, 129 × 32; offsets A +16 B, y +0 B, padded stride | 193.0 [192.9, 193.4] | 110.3 [110.2, 110.3] | -42.9% |
| GEMV, double, 257 × 32; offsets A +16 B, y +0 B, padded stride | 251.0 [250.8, 253.7] | 170.2 [168.7, 170.5] | -32.2% |
| GEMV, double, 129 × 128; offsets A +16 B, y +0 B, padded stride | 435.4 [434.3, 439.4] | 382.8 [382.7, 383.0] | -12.1% |
| GEMV, double, 257 × 128; offsets A +16 B, y +0 B, padded stride | 701.4 [699.9, 702.5] | 638.6 [637.6, 639.4] | -8.9% |
| GEMV, double, 129 × 32; offsets A +0 B, y +16 B, padded stride | 258.1 [255.8, 259.8] | 112.7 [112.6, 112.9] | -56.3% |
| GEMV, double, 257 × 32; offsets A +0 B, y +16 B, padded stride | 370.0 [368.3, 371.4] | 181.1 [179.5, 181.4] | -51.1% |
| GEMV, double, 129 × 128; offsets A +0 B, y +16 B, padded stride | 644.0 [643.3, 644.7] | 504.1 [503.8, 505.8] | -21.7% |
| GEMV, double, 257 × 128; offsets A +0 B, y +16 B, padded stride | 1,215.9 [1,212.3, 1,216.8] | 1,067.3 [1,066.4, 1,068.1] | -12.2% |
| GEMV, double, 129 × 32; offsets A +16 B, y +16 B, padded stride | 258.6 [258.2, 258.8] | 112.3 [111.6, 112.8] | -56.6% |
| GEMV, double, 257 × 32; offsets A +16 B, y +16 B, padded stride | 368.8 [368.6, 371.6] | 179.5 [179.3, 180.9] | -51.3% |
| GEMV, double, 129 × 128; offsets A +16 B, y +16 B, padded stride | 647.6 [647.4, 650.5] | 502.7 [502.3, 502.8] | -22.4% |
| GEMV, double, 257 × 128; offsets A +16 B, y +16 B, padded stride | 1,211.7 [1,210.7, 1,215.7] | 1,066.6 [1,066.0, 1,068.5] | -12.0% |
| GEMV, double, 4,096 × 4 | 426.3 [425.9, 427.5] | 326.2 [324.1, 327.7] | -23.5% |
| GEMV, double, 10,000 × 4 | 907.7 [904.9, 908.1] | 811.9 [810.4, 812.9] | -10.6% |
| GEMV, double, 4,096 × 8 | 618.2 [602.9, 629.7] | 514.4 [494.5, 534.3] | -16.8% |
| GEMV, double, 10,000 × 8 | 1,356.3 [1,326.4, 1,387.8] | 1,261.6 [1,216.6, 1,306.9] | -7.0% |
| GEMV, double, 4,096 × 16 | 994.1 [991.6, 996.5] | 895.7 [891.0, 920.4] | -9.9% |
| GEMV, double, 10,000 × 16 | 2,339.9 [2,337.7, 2,341.4] | 2,257.4 [2,255.5, 2,258.3] | -3.5% |
| DOT, float, 16,384 | 281.8 [280.9, 282.9] | 283.1 [282.3, 286.5] | +0.5% |
| DOT, float, 65,536 | 721.8 [719.4, 722.4] | 730.2 [722.7, 731.2] | +1.2% |
| DOT, double, 16,384 | 419.6 [415.7, 429.1] | 422.4 [420.7, 429.6] | +0.7% |
| DOT, double, 65,536 | 1,306.8 [1,298.9, 1,312.2] | 1,309.9 [1,303.1, 1,317.2] | +0.2% |
| DOT, zero result, float, 4,096 (generic path) | 245.7 [212.4, 279.5] | 214.3 [213.5, 215.4] | -12.8% |
| DOT, zero result, float, 16,384 | 1,659.5 [1,605.5, 1,712.6] | 274.9 [272.6, 277.5] | -83.4% |
| DOT, zero result, float, 65,536 | 8,417.3 [8,244.5, 8,468.1] | 721.5 [720.0, 723.8] | -91.4% |
| DOT, zero result, double, 4,096 (generic path) | 446.1 [439.7, 451.2] | 434.7 [432.9, 435.7] | -2.6% |
| DOT, zero result, double, 16,384 | 5,003.0 [4,897.5, 5,043.2] | 427.6 [425.2, 428.0] | -91.5% |
| DOT, zero result, double, 65,536 | 15,088 [14,566, 15,293] | 1,318.6 [1,317.3, 1,320.3] | -91.3% |
What changes
- Contiguous real DOT and AXPY use four independent ZA groups, each holding four streaming vectors. AXPY peels the destination to a 64-byte boundary with Core's
first_aligned<64>; the prefix and tail stay within the streaming call. Allocation alignment and ordinary NEON packet operations stay unchanged. - Column-major GEMV accumulates directly from the matrix without packing. It applies
alphaafter each column batch, using the generic GEMV batch sizes (all columns below 128 columns, otherwise 16 or 4 according to the L1/stride condition), to avoid overflowing a larger unscaled sum. Each row block splits into four predicated four-vector chunks with independent ZA groups, so a partial block also keeps four FMLA chains in flight. The streaming vector length determines the number of coefficients processed. - A guard saves FPSR before each streaming call and restores it afterwards, so a call leaves the caller's flags unchanged; nothing reads FPSR while streaming. DOT seeds ZA and its tail accumulator with −0, so a zero sum keeps its IEEE 754 sign; under roundTowardNegative a zero result is recomputed by the generic inner product.
- Dispatch retains Core's fallback for unsupported scalar types, expressions, and strides. DOT specializes the shared inner-product implementation; AXPY uses
blas_traitsfor either scalar placement and preserves nested products' evaluation order. Exact AXPY aliasing works; partial overlap uses Core's traversal. Double precision requiresEIGEN_VECTORIZE_SME_F64F64, and external BLAS retains GEMV precedence. - The benchmark grids cover vectors down to one element, cache boundaries, large operands, offsets, padded matrices, and NEON producer/consumer chains. Focused
vector_smetests cover dispatch, alignment, remainders, and numerical behavior.
SME DOT is selected by .dot(). Non-conjugating transpose inner products, including (x.transpose() * y)(0, 0) and scalar conversion of x.transpose() * y, and scaled subtraction (y -= alpha * x) retain Core's generic kernels.
SME reduces instruction overhead for these low-arithmetic-intensity operations; it does not reduce the bytes transferred. Float DOT and AXPY move approximately 8 and 12 bytes per two FLOPs. Four ZA groups process 256 floats per main iteration on this M4's 512-bit streaming vector length. Generated assembly for the public expressions confirms four-vector loads/stores and ZA FMLA, including the GEMV remainder.
Dispatch thresholds
Core reports L1 = 128 KiB and L2 = 4 MiB on this M4. DOT selects SME for per-operand sizes from max(16 KiB, L1/2) through L2; AXPY uses max(4 KiB, L1) through L2. These empirical crossovers account for NEON/SME memory handoff and retain the generic path for small and very large vectors. They reuse Core's cache detection and honor later setCpuCacheSizes() overrides; nonpositive L1 or L2 disables these vector fast paths. The fixed floors are named constants (kSmeDotMinBytes, kSmeAxpyMinBytes, kSmeGemv*) fitted on this M4.
The cache suitability query is kept out of line so its initialization code does not prevent small-vector caller inlining. In the generated float benchmark loops, DOT inlines its generic fallback and AXPY inlines its initial size check; vectors below the existing size floors do not call the cache-query helper. GCC and Clang receive their respective unroll directives through a local macro, which is undefined after the kernels.
GEMV requires at least 128 rows and four columns, column-major storage, and unit vector strides. It selects SME when the matrix reaches L1, or when the matrix is at least 32 KiB and cols*sizeof(Scalar) >= 128. Padded matrix outer strides are supported. Dimension comparisons avoid overflow.
Numerical behavior and validation
The public-expression reproducer gives the following results for both float and double:
| Case | Previous head 2be092ad6 |
Current head 2ea0577a2 |
|---|---|---|
257 × 128 GEMV, entries max()/64, RHS ones, alpha = 0.25 |
+inf |
finite, approximately max()/2 |
| Exact DOT, AXPY, GEMV after clearing exceptions | spurious mask 0x9f |
mask 0 |
| DOT of 32,769 negative zeros with ones | +0 |
-0 |
New public regression tests cover float/double, main loops and masked tails, 128/129 columns, both 16- and 4-column batches, positive/negative zero, and clean/pre-existing exception status for exact, invalid and overflowing inputs, with and without SVE tails. All four vector_sme parts pass with Apple clang 21, C++14 Release, seed 1 and ten framework repetitions. All four also pass with 32-bit indices and -fsanitize=address,undefined -fsanitize-trap=undefined. Changed-line clang-format 17, the style check, and git diff --check pass. Follow-up tests add a single positive-zero product, cancellation across and within fused lanes, products that underflow to −0, all four rounding modes, and GEMV row counts up to 1,100, which reach every chunk pattern at the largest architectural SVL. At 2ea0577a2 all four parts pass again in both configurations; a mutant without the roundTowardNegative fallback fails the rounding-mode test, and one that seeds +0 without any fallback fails the signed-zero test, for both float and double. At 92c90383f all four parts pass on an Apple M4 Mac mini in C++14 Release (seed 1 with ten repetitions, and a random seed) and with 32-bit indices under AddressSanitizer with trapping or recoverable UBSan; the new exception test fails at 13b056920 for both float and double.
GCC 16.2 compiles all four parts at 2ea0577a2 with C++14, -Wall -Wextra -Wshadow -Werror=unknown-pragmas and no diagnostics (not re-run at 92c90383f); execution remains unavailable because the installed runtime lacks __arm_tpidr2_save. Recoverable UBSan inserts additional streaming transitions inside the kernels; it failed the FPSR test until 92c90383f and passes since.
The guard restores the caller's flags, and the kernels raise none of their own. ZA arithmetic suppresses floating-point exceptions architecturally, and FPSR values read while streaming are unreliable, so reporting the SVE tails' flags could not be deterministic. See Arm DDI0616 A.a, RMHTLZ and FPMulAdd_ZA/FPDotAdd_ZA. Existing SME GEMM's transition handling is outside this vector-kernel change; !3164 (merged) adds the same guard there.
Cost of the correctness fixes on Apple M4
Comparison: 2be092ad6 → b0ccd58dd, identical benchmark sources and flags. Medians in ns [inclusive IQR], ten samples per case and revision; positive time changes are slower. The FPSR guard adds register reads/writes at both sides of the streaming transitions, confirmed in generated assembly. GEMV also reads/scales/writes its destination after each bounded column batch.
| Case | Before, ns [IQR] | After, ns [IQR] | Time change |
|---|---|---|---|
| DOT, float, 65,536 | 681.0 [678.7, 683.9] | 723.7 [722.2, 724.2] | +6.3% |
| DOT, double, 65,536 | 1,272.2 [1,270.7, 1,282.8] | 1,301.1 [1,299.9, 1,304.1] | +2.3% |
| AXPY, float, 65,536 | 1,314.9 [1,312.0, 1,328.1] | 1,383.0 [1,382.5, 1,383.7] | +5.2% |
| AXPY, double, 65,536 | 2,619.9 [2,617.2, 2,621.8] | 2,691.0 [2,688.5, 2,692.6] | +2.7% |
| GEMV, float, 257 × 128 | 344.9 [339.7, 350.1] | 467.9 [461.6, 470.7] | +35.7% |
| GEMV, double, 1,024 × 128 | 1,445.9 [1,438.2, 1,473.4] | 1,842.7 [1,800.6, 1,853.5] | +27.4% |
Apple M4, 24 GiB RAM, macOS 26.7; Apple clang 21.0.0, -std=c++14 -O3 -DNDEBUG -mcpu=apple-m4, Google Benchmark 1.9.5. Each family ran alone in before/after/before/after order, five repetitions per invocation, minimum 0.05 s per repetition. DOT/GEMV use CPU time and AXPY wall time. Buffers are aligned; matrices are column-major without padding. There were no concurrent task builds or benchmarks. Affinity and clocks were not fixed; one-minute load ranged from 3.34 to 3.63. Differences comparable to their dispersion remain inconclusive.
Apple M4: all correctness-fix measurements, 10 samples per revision
| Case | Before, ns [IQR] | After, ns [IQR] | Time change |
|---|---|---|---|
| DOT, float, 16,384 | 240.2 [237.2, 243.5] | 279.1 [275.2, 281.1] | +16.2% |
| DOT, float, 65,536 | 681.0 [678.7, 683.9] | 723.7 [722.2, 724.2] | +6.3% |
| DOT, double, 16,384 | 385.1 [383.1, 388.8] | 421.4 [418.7, 423.7] | +9.4% |
| DOT, double, 65,536 | 1,272.2 [1,270.7, 1,282.8] | 1,301.1 [1,299.9, 1,304.1] | +2.3% |
| AXPY, float, 32,768 | 661.9 [658.3, 665.2] | 730.4 [728.7, 731.3] | +10.4% |
| AXPY, float, 65,536 | 1,314.9 [1,312.0, 1,328.1] | 1,383.0 [1,382.5, 1,383.7] | +5.2% |
| AXPY, double, 32,768 | 1,312.5 [1,312.0, 1,314.2] | 1,383.7 [1,382.7, 1,385.1] | +5.4% |
| AXPY, double, 65,536 | 2,619.9 [2,617.2, 2,621.8] | 2,691.0 [2,688.5, 2,692.6] | +2.7% |
| GEMV, float, 1,024 × 128 | 785.3 [781.8, 791.8] | 987.3 [978.3, 992.5] | +25.7% |
| GEMV, float, 257 × 128 | 344.9 [339.7, 350.1] | 467.9 [461.6, 470.7] | +35.7% |
| GEMV, float, 10,000 × 8 | 682.5 [662.5, 689.7] | 744.8 [743.9, 752.2] | +9.1% |
| GEMV, double, 1,024 × 128 | 1,445.9 [1,438.2, 1,473.4] | 1,842.7 [1,800.6, 1,853.5] | +27.4% |
| GEMV, double, 257 × 128 | 594.6 [593.8, 596.9] | 742.4 [732.4, 751.1] | +24.9% |
| GEMV, double, 10,000 × 8 | 1,258.5 [1,238.4, 1,277.8] | 1,381.5 [1,315.3, 1,452.7] | +9.8% |
Historical base comparison and dispatch recheck at 2be092ad
These measurements and validation notes predate b0ccd58dd; they do not measure the correctness fixes above. They retain the original base comparison, distributions, and reproduction archive.
The earlier comparison is base 82a7cb22a5 versus head 2be092ad6, with the same benchmark sources and compiler flags. Times are medians in ns [inclusive IQR], 10 samples per case per build. A base/head ratio above 1 means this MR is faster.
| Case | Base, ns [IQR] | Head, ns [IQR] | Base / head |
|---|---|---|---|
| DOT, float, 16 | 1.244 [1.238, 1.246] | 1.775 [1.766, 1.792] | 0.70× |
| DOT, float, 64 | 2.751 [2.748, 2.753] | 2.814 [2.788, 2.828] | 0.98× |
| DOT, float, 256 | 11.01 [11.01, 11.03] | 11.33 [11.26, 11.34] | 0.97× |
| DOT, float, 65,536 | 4,331 [4,327, 4,338] | 691.6 [689.6, 693.4] | 6.26× |
| DOT, double, 65,536 | 8,674 [8,667, 8,682] | 1,270 [1,263, 1,276] | 6.83× |
| AXPY, float, 16 | 1.290 [1.248, 1.341] | 1.305 [1.277, 1.389] | 0.99× |
| AXPY, float, 64 | 5.572 [5.401, 5.644] | 5.622 [5.294, 5.763] | 0.99× |
| AXPY, float, 65,536 | 5,556 [5,553, 5,560] | 1,354 [1,311, 1,408] | 4.11× |
| DOT with producer, float, 65,536; offsets +16/+16 B | 8,785 [8,779, 8,794] | 6,592 [6,538, 6,608] | 1.33× |
| AXPY, float, 1,024; offsets +16/+16 B | 72.11 [71.90, 72.41] | 80.50 [78.45, 83.09] | 0.90× |
The largest gains are for cached operands. Including a NEON producer reduces the 65,536-element float DOT gain to 1.33×. At 64 elements, float DOT and AXPY retain about 0.063 ns and 0.050 ns of overhead in their medians; the AXPY distributions overlap. Smaller DOT cases and offset-buffer AXPY retain larger relative costs, as shown above and in the complete tables below.
Some larger cases showed substantial run-to-run variation. A separate paired recheck against fa59ab01f isolates the cache-query inlining change: the largest measured slowdown among its larger-vector cases is 1.0% for 2,097,152-element float AXPY; offset-buffer float AXPY at 1,024 elements is 2.8% slower (75.84 → 77.97 ns). These are separate comparisons, not substitutions into the base/head table. Both data sets and their variability are retained below.
Measurement and validation
CPU: Apple M4, 24 GiB RAM, macOS 26.7 (25G229). Apple clang 21.0.0 (clang-2100.3.34.2), GNU C++14, -O3 -DNDEBUG -mcpu=apple-m4, Google Benchmark 1.9.5. Base: 82a7cb22a50bcd70bb7b2a7e57250ca26110bf84; head: 2be092ad6b43d494d82439744f5e820e63dd6a0e. All builds use the head's benchmark sources and disable BLAS discovery. The earlier-head timing sweep covers DOT and AXPY.
Each benchmark family runs alone, without concurrent task builds or benchmarks. The full sweep alternates base/fa59ab01f/head/base/fa59ab01f/head, five repetitions per invocation and a minimum 0.1 s per repetition. The paired recheck alternates head/fa59ab01f/head/fa59ab01f, five repetitions per invocation and a minimum 0.15 s per repetition. DOT uses CPU time; AXPY uses wall time. Affinity and clocks are not fixed; one-minute load averages span 3.0–5.1 for the full sweep and 2.5–2.8 for the recheck. No thermal warnings were reported. Differences comparable to their dispersion are inconclusive.
All vectors have dynamic size. Warm cases repeat a.dot(b) or y += Scalar(0.75) * x. Produced DOT also times a = source.array() + Scalar(0.25); mixed AXPY times the corresponding producer for y and a y.sum() consumer. Offsets are bytes from a 64-byte boundary; unspecified layouts use Eigen's default allocation. Allocation, input generation, and scalar-reference checks are outside timing.
Validation at 2be092ad6 |
Result |
|---|---|
Apple clang 21, C++14, -mcpu=apple-m4, SME enabled: vector_sme_1, vector_sme_2, cache_sizes |
All three pass, ten framework repetitions, seed 1 |
GCC 16.2, C++14, -O2 -march=armv9.2-a+sme2+sme-f64f64 -Wall -Wextra -Wshadow -Werror=unknown-pragmas |
Full vector_sme translation unit compiles without diagnostics |
Public float/double DOT, AXPY, GEMV assembly versus fa59ab01f |
All six SME kernel instruction streams are identical after normalizing labels and comments |
Changed lines: clang-format 17, repository style check, git diff --check |
Pass |
GCC runtime validation is unavailable because the installed libgcc lacks __arm_tpidr2_save. Other SME hardware/vector lengths, device builds, and external BLAS were not executed locally. Codespell and REUSE were unavailable locally, the generic clang-tidy driver did not validate SME code, and Doxygen was unavailable. No new repository source files were added by the dispatch-inlining change.
The implementation is original, informed by Arm's ACLE specification. The supplied M4 measurements motivated workload selection; no Accelerate source or binaries were inspected or timed. Thanks to @tesch1 for the alignment and NEON/SME handoff measurements in !3160 (merged), the GCC review, and the FPSR and crossover review, and @onalante-ebay for the code review.
Apple M4: DOT, earlier head versus base — 10 samples per build
| Case | Base, ns [IQR] | Head, ns [IQR] | Base / head |
|---|---|---|---|
| DOT, float, 1 | 0.4936 [0.4803, 0.4985] | 0.6342 [0.6322, 0.6385] | 0.78× |
| DOT, float, 4 | 0.7367 [0.7246, 0.7465] | 1.013 [1.009, 1.029] | 0.73× |
| DOT, float, 16 | 1.244 [1.238, 1.246] | 1.775 [1.766, 1.792] | 0.70× |
| DOT, float, 32 | 1.330 [1.328, 1.331] | 1.770 [1.763, 1.779] | 0.75× |
| DOT, float, 64 | 2.751 [2.748, 2.753] | 2.814 [2.788, 2.828] | 0.98× |
| DOT, float, 128 | 5.522 [5.518, 5.529] | 5.671 [5.650, 5.681] | 0.97× |
| DOT, float, 256 | 11.01 [11.01, 11.03] | 11.33 [11.26, 11.34] | 0.97× |
| DOT, float, 512 | 22.08 [22.06, 22.10] | 22.65 [22.47, 22.74] | 0.97× |
| DOT, float, 1,024 | 45.97 [45.88, 46.02] | 47.19 [46.83, 47.43] | 0.97× |
| DOT, float, 4,096 | 214.6 [214.0, 215.0] | 257.9 [218.9, 297.1] | 0.83× |
| DOT, float, 8,192 | 445.9 [444.8, 446.3] | 457.4 [454.4, 460.4] | 0.97× |
| DOT, float, 16,384 | 911.4 [911.1, 912.8] | 243.5 [243.0, 244.3] | 3.74× |
| DOT, float, 32,768 | 2,174 [2,171, 2,180] | 392.7 [391.6, 393.7] | 5.54× |
| DOT, float, 65,536 | 4,331 [4,327, 4,338] | 691.6 [689.6, 693.4] | 6.26× |
| DOT, float, 2,097,152 | 145,700 [144,900, 146,100] | 158,800 [154,300, 165,200] | 0.92× |
| DOT, double, 1 | 0.3739 [0.3732, 0.3752] | 0.5122 [0.5091, 0.5145] | 0.73× |
| DOT, double, 4 | 0.6232 [0.6219, 0.6236] | 1.268 [1.261, 1.276] | 0.49× |
| DOT, double, 16 | 1.372 [1.343, 1.399] | 1.776 [1.765, 1.816] | 0.77× |
| DOT, double, 32 | 2.757 [2.755, 2.761] | 2.806 [2.790, 2.820] | 0.98× |
| DOT, double, 64 | 5.544 [5.536, 5.588] | 5.622 [5.599, 5.650] | 0.99× |
| DOT, double, 128 | 11.06 [11.01, 11.08] | 11.26 [11.24, 11.27] | 0.98× |
| DOT, double, 256 | 22.11 [22.07, 22.14] | 22.57 [22.57, 22.63] | 0.98× |
| DOT, double, 512 | 45.63 [45.56, 45.71] | 46.67 [46.45, 46.94] | 0.98× |
| DOT, double, 1,024 | 99.07 [98.96, 99.25] | 100.6 [100.5, 101.0] | 0.98× |
| DOT, double, 4,096 | 446.6 [446.3, 447.1] | 450.0 [445.8, 455.3] | 0.99× |
| DOT, double, 8,192 | 911.9 [911.3, 913.7] | 236.5 [233.6, 238.4] | 3.86× |
| DOT, double, 16,384 | 2,183 [2,181, 2,184] | 386.5 [382.0, 393.0] | 5.65× |
| DOT, double, 32,768 | 4,336 [4,334, 4,339] | 679.2 [675.0, 683.0] | 6.38× |
| DOT, double, 65,536 | 8,674 [8,667, 8,682] | 1,270 [1,263, 1,276] | 6.83× |
| DOT, double, 2,097,152 | 386,300 [383,300, 393,300] | 402,400 [390,900, 410,100] | 0.96× |
| DOT, float, 64; offsets +16/+16 B | 2.755 [2.743, 2.759] | 2.899 [2.882, 2.914] | 0.95× |
| DOT, float, 1,024; offsets +16/+16 B | 46.16 [46.14, 46.24] | 46.62 [46.25, 46.96] | 0.99× |
| DOT, float, 65,536; offsets +16/+16 B | 4,356 [4,356, 4,358] | 711.7 [700.5, 714.4] | 6.12× |
| DOT, double, 64; offsets +16/+16 B | 5.484 [5.473, 5.489] | 5.677 [5.615, 5.776] | 0.97× |
| DOT, double, 1,024; offsets +16/+16 B | 99.36 [99.24, 99.40] | 105.0 [99.82, 110.0] | 0.95× |
| DOT, double, 65,536; offsets +16/+16 B | 8,690 [8,680, 8,695] | 1,345 [1,306, 1,388] | 6.46× |
| DOT with producer, float, 64; offsets +16/+16 B | 6.674 [6.518, 6.783] | 7.130 [7.086, 7.231] | 0.94× |
| DOT with producer, float, 1,024; offsets +16/+16 B | 119.5 [119.3, 119.7] | 123.3 [120.3, 126.9] | 0.97× |
| DOT with producer, float, 65,536; offsets +16/+16 B | 8,785 [8,779, 8,794] | 6,592 [6,538, 6,608] | 1.33× |
| DOT with producer, double, 64; offsets +16/+16 B | 11.91 [11.82, 12.08] | 12.86 [12.55, 13.15] | 0.93× |
| DOT with producer, double, 1,024; offsets +16/+16 B | 241.2 [239.2, 241.6] | 244.3 [240.7, 251.2] | 0.99× |
| DOT with producer, double, 65,536; offsets +16/+16 B | 17,520 [17,430, 17,620] | 12,560 [12,210, 12,580] | 1.39× |
Apple M4: AXPY, earlier head versus base — 10 samples per build
| Case | Base, ns [IQR] | Head, ns [IQR] | Base / head |
|---|---|---|---|
| AXPY, float, 1 | 0.6709 [0.6707, 0.6728] | 0.7595 [0.7485, 0.8019] | 0.88× |
| AXPY, float, 4 | 0.6686 [0.6674, 0.7105] | 0.7613 [0.7487, 0.8032] | 0.88× |
| AXPY, float, 16 | 1.290 [1.248, 1.341] | 1.305 [1.277, 1.389] | 0.99× |
| AXPY, float, 32 | 3.488 [3.483, 3.490] | 3.587 [3.556, 3.689] | 0.97× |
| AXPY, float, 64 | 5.572 [5.401, 5.644] | 5.622 [5.294, 5.763] | 0.99× |
| AXPY, float, 128 | 9.204 [9.190, 9.226] | 9.342 [9.087, 9.629] | 0.99× |
| AXPY, float, 256 | 17.21 [17.11, 17.24] | 18.47 [17.37, 20.77] | 0.93× |
| AXPY, float, 512 | 37.55 [37.52, 37.74] | 38.10 [37.52, 38.67] | 0.99× |
| AXPY, float, 1,024 | 72.03 [71.84, 72.29] | 73.99 [72.45, 75.79] | 0.97× |
| AXPY, float, 4,096 | 263.7 [263.5, 263.9] | 268.6 [264.1, 280.2] | 0.98× |
| AXPY, float, 8,192 | 519.4 [518.8, 519.9] | 526.6 [519.6, 539.4] | 0.99× |
| AXPY, float, 16,384 | 1,032 [1,031, 1,034] | 1,069 [1,036, 1,116] | 0.97× |
| AXPY, float, 32,768 | 2,796 [2,794, 2,803] | 666.8 [657.1, 706.2] | 4.19× |
| AXPY, float, 65,536 | 5,556 [5,553, 5,560] | 1,354 [1,311, 1,408] | 4.11× |
| AXPY, float, 2,097,152 | 182,600 [182,000, 183,600] | 189,800 [181,800, 200,700] | 0.96× |
| AXPY, double, 1 | 0.6710 [0.6706, 0.6713] | 0.7758 [0.7487, 0.8083] | 0.86× |
| AXPY, double, 4 | 0.7487 [0.7477, 0.7500] | 0.8987 [0.8356, 0.9748] | 0.83× |
| AXPY, double, 16 | 3.533 [3.531, 3.540] | 3.609 [3.542, 3.706] | 0.98× |
| AXPY, double, 32 | 5.508 [5.451, 5.610] | 5.574 [5.215, 5.925] | 0.99× |
| AXPY, double, 64 | 9.156 [9.040, 9.223] | 9.409 [9.108, 9.683] | 0.97× |
| AXPY, double, 128 | 17.21 [17.11, 17.30] | 18.07 [17.56, 18.65] | 0.95× |
| AXPY, double, 256 | 37.35 [37.31, 37.38] | 38.05 [37.57, 39.41] | 0.98× |
| AXPY, double, 512 | 72.12 [71.69, 72.34] | 75.72 [73.98, 78.94] | 0.95× |
| AXPY, double, 1,024 | 135.6 [135.5, 135.8] | 139.5 [136.7, 143.3] | 0.97× |
| AXPY, double, 4,096 | 519.1 [518.8, 520.0] | 533.4 [520.1, 551.9] | 0.97× |
| AXPY, double, 8,192 | 1,041 [1,032, 1,047] | 1,108 [1,035, 1,113] | 0.94× |
| AXPY, double, 16,384 | 2,805 [2,799, 2,826] | 684.9 [669.6, 703.4] | 4.10× |
| AXPY, double, 32,768 | 5,563 [5,560, 5,628] | 1,356 [1,320, 1,410] | 4.10× |
| AXPY, double, 65,536 | 11,100 [11,090, 11,100] | 2,700 [2,669, 2,768] | 4.11× |
| AXPY, double, 2,097,152 | 460,700 [432,600, 541,100] | 483,900 [434,300, 541,600] | 0.95× |
| AXPY, float, 64; offsets +16/+16 B | 5.846 [5.309, 6.029] | 7.353 [7.021, 7.748] | 0.80× |
| AXPY, float, 1,024; offsets +16/+16 B | 72.11 [71.90, 72.41] | 80.50 [78.45, 83.09] | 0.90× |
| AXPY, float, 65,536; offsets +16/+16 B | 5,595 [5,568, 5,637] | 1,407 [1,359, 1,488] | 3.98× |
| AXPY, double, 64; offsets +16/+16 B | 9.125 [9.091, 9.175] | 10.43 [10.16, 10.71] | 0.87× |
| AXPY, double, 1,024; offsets +16/+16 B | 135.7 [135.6, 136.0] | 145.1 [142.4, 148.0] | 0.94× |
| AXPY, double, 65,536; offsets +16/+16 B | 11,100 [11,100, 11,110] | 2,829 [2,707, 3,035] | 3.92× |
| AXPY with producer/consumer, float, 64; offsets +16/+16 B | 11.60 [11.45, 11.72] | 13.45 [12.72, 13.99] | 0.86× |
| AXPY with producer/consumer, float, 1,024; offsets +16/+16 B | 184.6 [183.4, 185.6] | 188.1 [184.4, 193.9] | 0.98× |
| AXPY with producer/consumer, float, 65,536; offsets +16/+16 B | 12,810 [12,780, 12,880] | 10,800 [10,750, 10,980] | 1.19× |
| AXPY with producer/consumer, double, 64; offsets +16/+16 B | 21.53 [21.03, 21.66] | 22.73 [22.06, 23.71] | 0.95× |
| AXPY with producer/consumer, double, 1,024; offsets +16/+16 B | 349.8 [349.3, 354.2] | 367.1 [356.2, 374.0] | 0.95× |
| AXPY with producer/consumer, double, 65,536; offsets +16/+16 B | 25,500 [25,470, 25,510] | 22,780 [22,060, 23,200] | 1.12× |
Apple M4: paired dispatch recheck — 10 samples per build
Comparison: fa59ab01f versus earlier head 2be092ad6. Medians in ns [inclusive IQR]; negative time changes are improvements. These 17 cases include variable measurements from the full sweep and small-vector controls.
| Case | Pre-inlining revision, ns [IQR] | Head, ns [IQR] | Time change |
|---|---|---|---|
| DOT, float, 64 | 3.267 [3.262, 3.269] | 2.763 [2.760, 2.769] | -15.4% |
| DOT, float, 256 | 17.58 [17.53, 17.64] | 11.06 [11.04, 11.08] | -37.1% |
| DOT, float, 4,096 | 217.9 [217.3, 222.1] | 215.8 [215.3, 216.0] | -1.0% |
| DOT, float, 65,536 | 692.8 [691.7, 695.3] | 682.7 [680.5, 684.1] | -1.5% |
| DOT, float, 2,097,152 | 146,800 [146,400, 147,000] | 146,700 [146,100, 147,700] | -0.1% |
| DOT, double, 65,536 | 1,268 [1,267, 1,269] | 1,270 [1,267, 1,276] | +0.2% |
| AXPY, float, 64 | 6.547 [6.542, 6.553] | 5.566 [5.122, 5.731] | -15.0% |
| AXPY, float, 256 | 17.62 [17.57, 17.72] | 17.16 [17.11, 17.19] | -2.6% |
| AXPY, float, 16,384 | 1,046 [1,037, 1,046] | 1,034 [1,032, 1,035] | -1.2% |
| AXPY, float, 65,536 | 1,313 [1,311, 1,324] | 1,316 [1,312, 1,327] | +0.3% |
| AXPY, float, 2,097,152 | 182,400 [181,900, 182,800] | 184,300 [183,300, 185,100] | +1.0% |
| AXPY, double, 8,192 | 1,036 [1,035, 1,037] | 1,035 [1,035, 1,036] | 0.0% |
| AXPY, double, 16,384 | 656.8 [655.5, 663.1] | 661.3 [656.9, 665.4] | +0.7% |
| AXPY, double, 32,768 | 1,311 [1,310, 1,312] | 1,312 [1,312, 1,314] | +0.1% |
| AXPY, double, 65,536 | 2,649 [2,623, 2,658] | 2,625 [2,622, 2,629] | -0.9% |
| AXPY, double, 2,097,152 | 432,800 [432,000, 436,600] | 434,000 [432,800, 436,500] | +0.3% |
| AXPY, float, 1,024; offsets +16/+16 B | 75.84 [75.33, 76.42] | 77.97 [77.55, 79.41] | +2.8% |
Reproduction scripts, raw observations, assembly, and validation logs: sme-small-vectors-2be092ad6.zip
Edited by Rasmus Munk Larsen