CPU (AVX2): faster Gyro-S prefill and generation - fix23 + fix37

Three commits on top of agentionai 9aa51ce (release/gyro-l), CPU backend only (ggml/src/ggml-cpu/, tests/). The target is an AVX2-only desktop CPU without AVX-512 or VNNI (measured on an Intel Core i5-8500, Coffee Lake, 6 cores, no SMT, dual-channel DDR4, the haswell variant of the portable build).

This was made with a few agents with a few days, but result is great:

#	patch	what	files
1	`0001-fix23-…patch`	fix23: prefill. Trellis experts decoded once per row and tiled; dense K-quant microtile; AVX2 `quantize_row_q8_K`	9 files, +1325 −21
2	`0002-fix23-tests-…patch`	fix23 tests: `tests/test-gyro-cpu-opt` part 1 and 2 (self-test + micro-bench)	2 files, +1230
3	`0003-fix37-…patch`	fix37: generation (one token per step). Expert dot two rows per call, attention per KV group, delta-net state in place; test part 3 + `--bench-gen`	6 files, +1720 −11

The new fast paths of fix23 (GGML_TQ_STATES, GGML_CPU_TILED_T8, GGML_CPU_FAST) and fix37 (GGML_CPU_GEN) are opt-in through environment variables and give bit-identical results whether on or off. Patch 1 also carries two paths from fix13 that are on by default and change the numbers against 9aa51ce:

the trellis decode-once: with 2 or more tokens on an expert (prefill), each row is decoded to q8_0 once and dotted as q8_0. This requantizes the weights. GGML_TQ_MIN_TOK=<large> turns it off;
the trellis L1 pair table: GGML_TQ_PAIR=0 turns it off.

So the original build answers with different, equally coherent text, while fix23 and fix37 answer identically (see Results).

##Results

Qwen3.8-Flash-Next Gyro-S TQ1_0: 512 experts, 10 used per token. Experts are TQK6/TQK7 trellis; attention, shared and dense weights are Q5_K/Q6_K/Q8_0/BF16. Intel i5-8500, model on NVMe, the three builds measured in the same session one after another, 60 s pause between runs, 2 rounds:

taskset -c 1-5 llama-completion -m Gyro-S-TQ1_0.gguf -ngl 0 -t 5 -fa on -ctk q8_0 -ctv q8_0 \ 
--no-op-offload --temp 0 -b 1024 -ub 1024 --ignore-eos --ngram-on-disk -no-cnv \ 
-c 2304 -f p1k.txt -n 1024 # 1K: 936-token prompt 
-c 4800 -f p4k.txt -n 512 # 4K: 1992-token prompt

Speeds are tokens per second; each cell is round a / round b.

test	build	prompt t/s	gen t/s	gen vs original	gen vs fix23
1K	original `9aa51ce`	4.70 / 4.73	2.79 / 2.79	—
1K	fix23 (patches 1+2)	16.20 / 12.97	3.78 / 3.52	×1.31	—
1K	fix37 (patches 1–3)	16.96 / 12.37	4.34 / 4.45	×1.58	×1.21
4K	original `9aa51ce`	4.51 / 4.54	2.55 / 2.59	—
4K	fix23	12.48 / 13.11	3.10 / 3.68	×1.32	—
4K	fix37	15.86 / 13.47	4.03 / 3.67	×1.50	×1.14

Prompt processing: ×3.1 at 1K and ×2.8–3.2 at 4K against the original, all from fix23. fix37 does not touch it. Prefill runs at 95–98 °C on this box, so it moves between 12 and 17 t/s from run to run.
Generation: ×1.3 from fix23 and another ×1.14–1.21 from fix37, ×1.5–1.6 in total. In an earlier session with the flag toggled in the same binary, fix37 measured +15% at 1K and +15.5% at 4K (1K: 3.47/3.66 → 4.10/4.09; 4K: 3.38/3.71 → 4.10/4.08).
Generated text: fix23 and fix37 produce identical text in every run. The original 9aa51ce gives a different, equally coherent answer from the first paragraph on. The difference comes only from the two fix13 defaults in patch 1: fix23 run with GGML_TQ_PAIR=0 GGML_TQ_MIN_TOK=100000 generates exactly the original's text.

Tokens per 24 hours

These follow from the averages above, as back-to-back requests on one i5-8500.

build	generation only	1K requests (936 in + 1024 out)	4K requests (1992 in + 512 out)
original	241 K	153/day, 156 K out	135 / day, 69 K out, 269 K in
fix23	315K	251/day, 257 K out	282/day, 144 K out, 561 K in
fix37	380 K	291 / day, 298 K out	321 / day, 165 K out, 640 K in

How the speedup was found

Each step was profiled on the target box, changed in one small move, checked bit for bit and measured A/B. The full step log, including every rejected step with its numbers, is kept outside the patches.

Original → fix23 (prefill). In the original profile, prefill went to re-decoding trellis expert rows for every token and to the scalar quantize_row_q8_K. Decode-once with tiles, the T8 microtile and the AVX2 quantizer took the prompt from ~4.6 to ~13–16 t/s.
fix23 → fix37 (generation). In the fix23 profile, generation went to gather in the expert dot, to attention re-reading K/V once per head (GQA 24:1), and to 3.1 MB per layer of recurrent-state copies. That gave three steps:

experts two rows per call: ×1.12–1.14 per matmul;
attention per KV group: ×2.6 per call, and no slowdown as the context grows;
delta-net state in place: about +9% of a token.

Small fixes to GET_ROWS and CONCAT add the rest. 3. Where generation stands now (fix37, 1K, 236 ms per token).

part	ms per token	limited by
dense Q5_K/Q6_K/Q8_0	~90	memory, ~32 GB/s of the 35 GB/s ceiling
experts	~75	gather
waiting at barriers	~13	thread sync

The remaining ideas measured under 1% each, or negative (see Tried and rejected).

Hardware note: an earlier 65 W power supply held this box at ~44 W and 3.1–3.3 GHz (PROCHOT). Moving to a 90 W supply and reseating the cooler gave +26–28% in every test on its own, with the same code. All numbers in this document come from the 90 W setup, and the builds being compared were always measured in the same session.

One generation step (`test-gyro-cpu-opt --bench-gen -t 5`, median of 5 runs, i5-8500)

matrix	type	shape	fix23 µs	fix37 µs
`ffn_gate/up_exps` (10 of 512 experts)	TQK6	640×2560	578.5	514.9	×1.12
`ffn_down_exps`	TQK7	2560×640	620.8	546.5	×1.14
attention, q8_0 KV, 48 q / 2 KV heads, 2048 positions	FA		2214	845	×2.62
attention, 4096 positions	FA		4405	1684	×2.62
`attn_qkv` / `ssm_out` / `hc_ffn_*` / `output`	Q5_K, Q6_K, Q8_0				×1.00 (31 GB/s, at the memory limit)

Not only Gyro-S: Laguna XS 2.1 (i5-1135G7 laptop, 4 threads)

GGML_CPU_GEN=1 also speeds up other GQA models with q8_0 KV. Laguna XS 2.1 has no trellis or delta-net layers:

Q4_K_M, MoE with 256 experts, top-8;
40 layers: 10 full-attention, 30 SWA-512;
8 KV heads, 48/64 q heads.

The gain comes from the generic one-token paths: FLASH_ATTN_EXT walking a GQA group together, and GET_ROWS/CONCAT on all threads.

Same build, cold start, 60 s pause between runs. Prompt t/s / generation t/s:

context (prompt tokens)	toggle off	`GGML_CPU_GEN=1`
1K (907), mean of 2 runs	24.4 / 9.7	24.6 / 12.5	gen ×1.28
8K (6571)	11.5 / 4.65	11.5 / 8.65	gen ×1.86
16K (13803)	5.85 / 2.84¹	6.10 / 5.97	gen ×2.10

¹ Upstream ggml-org/llama.cpp 5047748, mean of 2 runs. At 1K and 8K the fork with toggles off matches upstream within 1–2%.

The text with GGML_CPU_GEN=1 is identical to toggles off (same md5).
Prefill does not change: it is a batch path, not covered by this MR.
The 1K runs use the default weight repacking. The 8K and 16K runs use --no-repack, which also lowers RAM from 27.7 to 19.2 GB on this AVX-512 CPU. That is unrelated to this MR.

Gyro-Coder TQ2_0 (i5-8500, model on a USB disk)

Qwen3.8-Flash-Next-Gyro-Coder-TQ2_0.gguf is 75.8 GB:

256 experts, 10 used per token, all TQ2_0 trellis (16.0 GB);
a 54.4 GB PLE n-gram table, which --ngram-on-disk keeps on disk;
peak RAM 21.8 GB.

Same protocol as the Gyro-S table: 1K, original → fix23 → fix37, 2 rounds, 60 s pause. The model is read from a USB HDD. Prompt t/s / generation t/s (round a / round b):

build	prompt t/s	gen t/s	gen vs original
original	4.12 / 3.75	1.97 / 1.92	—
fix23	10.98 / 10.05	2.41 / 2.38	×1.23
fix37	11.21 / 14.49	2.72 / 2.73	×1.40

fix23 and fix37 give the same text (md5) in both rounds.
The original differs, as with Gyro-S: patch 1 turns on the fix13 trellis decode paths by default.
Generation is slower than Gyro-S, probably because the n-gram rows are read from the USB HDD for every token.

Why is it faster

Patch 1 - fix23 (prefill)

Prefill of this model was dominated by the trellis experts and the dense K-quant matmuls.

Trellis experts decoded once per row (GGML_CPU_FAST=trellis). Before, the TQ2_T/TQK6/TQK7 vec_dot decoded the whole weight row again for every token routed to the expert. With several tokens on an expert, each row is now decoded once to q8_0 and dotted against all of them in a 4×2 register tile (AVX2), with each block loaded once per tile. Tokens are passed by pointer, with no gather memcpy, and the decode uses vector divides.
SIMD state unpack (GGML_TQ_STATES=1) and the L1 pair table (fix13, GGML_TQ_PAIR, on by default). The trellis states are unpacked with pshufb, shift and mask instead of a scalar loop, and adjacent states are read from a table that fits in L1.
Dense K-quant prefill microtile (GGML_CPU_TILED_T8=1). Tokens sit in the vector lanes with int16 partial sums, so a row × 8-token group no longer ends in 8 horizontal reductions. On x86, quantize_row_q8_K is an AVX2 version that gives byte-identical output to quantize_row_q8_K_ref; until now the scalar reference run for every K-quant activation in the prefill hot path.

Patch 2 - fix23 tests

tests/test-gyro-cpu-opt.cpp, built with -DLLAMA_BUILD_TESTS=ON and without GGML_BACKEND_DL:

part 1: GGML_CPU_TILED_T8;
part 2: GGML_CPU_FAST=trellis.

Each part checks that results withthe toggle on and off are bit-identical, in child processes, and includes a micro-bench.

Patch 3 — fix37 (generation, `GGML_CPU_GEN=1`)

Profile of fix23 during generation: trellis expert dot 35% (a third of it on vpgatherdd), Q5_K/Q6_K 1-row dot 35%, libgomp barriers 12.6%, IPC 1.2. Generation used about a third of the 35.3 GB/s read bandwidth. The changes below come from that profile and from per-op cycle counts in the real graph:

MUL_MAT_ID, one token, TQ2_T/TQK6/TQK7 expert. The expert dot is bound by gathers, not memory: ~4.5 GB/s against the 35 GB/s the box can read. The token is unpacked to int16 once, in lane order with fp16 scales as float, and rows go two per call through the AVX2 pair-table kernel. That keeps two independent gather chains in flight and loads each activation vector once for both rows. The int32 lanes, scale products and FMA order are the same as in vec_dot. Result: ×1.12–1.14 per expert matmul.
FLASH_ATTN_EXT, one query row (q8_0 KV has no split-KV path). With GQA 24:1 (48 q heads on 2 KV heads), every q head re-read and re-converted the same K/V rows, so attention cost grew 30× from a short context to 4K. Now the heads of one KV group walk the cache together, one chunk of heads per thread. A q8_0 K row is dotted with 4 queries at a time, with its blocks and |k| loaded once, and V is converted to f32 once per position for the whole group. Per-head online softmax and accumulation are unchanged. Result: ×2.6, and the drop-off with context length is gone.
GATED_DELTA_NET, one token. Each 128-float row of the recurrent state goes through the decay, the dot with k, the rank-1 update and the dot with q in one pass, while it is in L1. It used to take five passes over the 128×128 state per head.
Recurrent state in place. The chain GET_ROWS(cache_s, row) → RESHAPE → GATED_DELTA_NET → VIEW(new state) → CPY(→ cache_s) is recognized once per graph, before the threads start. It applies with one token and one sequence, when every link has a single reader and none is a graph output. The op then reads the state from its cache row and writes the new state straight to the CPY destination, and the GET_ROWS and CPY nodes are skipped. That avoids copying 3.1 MB per layer × 36 layers out of the cache and back on every token, worth about 9% of a token.

The row index is read at plan time. The index is a graph input, and ggml_gallocr reuses its memory once GET_ROWS has run, so reading it later in the op is wrong. An earlier version did exactly that and crashed on the real model. The test now builds this chain through ggml_gallocr and overwrites the index after GET_ROWS, so that bug fails the test.

GET_ROWS f32 with fewer rows than threads, and CONCAT f32 of one token, are split over every thread instead of running on one. CONCAT steps its rows without two 64-bit divisions per row.

Dense Q5_K/Q6_K/Q8_0 are not touched. At 5 threads they already read ~31–32 GB/s of the 35.3 GB/s ceiling, and kernel tweaks measured ×1.00 in the graph.

Runtime toggles

variable	patch	default	effect
`GGML_TQ_PAIR`	1 (fix13)	on	L1 pair table in the trellis dot; `0` = the original gather path. Changes the numbers
`GGML_TQ_MIN_TOK`	1 (fix13)	2	trellis decode-once to q8_0 once this many tokens hit one expert; a large value turns it off. Changes the numbers (prefill)
`GGML_TQ_STATES=1`	1 (fix13)	off	SIMD trellis state unpack
`GGML_CPU_TILED_T8=1`	1	off	dense K-quant prefill microtile + AVX2 `quantize_row_q8_K`
`GGML_CPU_FAST=trellis`	1	off	trellis decode-once + tile fast path
`GGML_CPU_GEN=1`	3	off	generation fast paths above

Recommended for Gyro-S on AVX2:

export GGML_TQ_STATES=1 GGML_CPU_TILED_T8=1 GGML_CPU_FAST=trellis GGML_CPU_GEN=1

The fix37 expert kernel builds on AVX2 && F16C without AVX-512 VBMI. Builds with VBMI keep their own AVX-512 path, and other targets get a generic fallback, so GGML_CPU_GEN=1 is safe everywhere. All four portable variants (x64, sse42, haswell, icelake) build with no new warnings.

Correctness

Bit-identical toggles. The paths behind GGML_TQ_STATES, GGML_CPU_TILED_T8, GGML_CPU_FAST and GGML_CPU_GEN perform the same float operations in the same order as the code they replace. The tests check this in child processes with each toggle on and off. On the model, fix23 with every toggle unset, fix23 with all three set, and fix37 with GGML_CPU_GEN=1 generate the same text (md5 of the output).
Not bit-identical: the two fix13 defaults in patch 1, decode-once and the pair table (see the top). They change the against numbers 9aa51ce, so the generated text differs from the original's. With both turned off (GGML_TQ_PAIR=0 GGML_TQ_MIN_TOK=100000), fix23 generates exactly the original text on the 1K test: everything else in patch 1 matches 9aa51ce bit for bit.

fix23 run	md5 of the generated text
original `9aa51ce`	`661cf972`
fix23, `GGML_TQ_PAIR=0 GGML_TQ_MIN_TOK=100000`	`661cf972` (same)
fix23, `GGML_TQ_PAIR=0` only	`9d8f9ba6`
fix23, defaults (toggles unset)	`029e0ecf`
fix23 with all toggles set, and fix37 with `GGML_CPU_GEN=1`	`029e0ecf` (same as defaults)

test-gyro-cpu-opt part 3. It forks three children: GGML_CPU_GEN unset, set, and set with scalar state unpack. Their results must match bit for bit, and each child checks that its toggle is in effect. Cases:
trellis MUL_MAT_ID: TQK6/TQK7/TQ2_T, 1/2/7 tokens, 10 or 16 experts, shared or per-expert activation, Gyro-S shapes, against a double-precision reference;
MUL_MAT Q5_K/Q6_K/Q8_0, which must be unchanged by the toggle;
GET_ROWS and CONCAT;
FLASH_ATTN_EXT: 48/2, 8/1 and 40/2 heads, shared and per-head masks, 1000 positions, against a double softmax;
GATED_DELTA_NET: Gyro-S shape, KDA, 2 tokens;
the delta-net state chain allocated by ggml_gallocr, with the index overwritten after GET_ROWS, both in place (row 1 → 1) and to another row (0 → 2). A counter checks that the in-place path actually ran.

82 cases, 0 bit differences, on an i5-8500 (haswell) and an i5-1135G7.

Parts 1 and 2 (fix23) still pass.

How to get it

git clone https://gitlab.com/agentionai-group/llama.cpp
cd llama.cpp
git checkout -b gyro-fix37 9aa51ce4fdd3435a91531ffe16a17414ccc6dc6f
git am /path/to/gyro-release-fix37/0001-*.patch \ 
/path/to/gyro-release-fix37/0002-*.patch\ 
/path/to/gyro-release-fix37/0003-*.patch
git log --oneline -4 # fix37, fix23 tests, fix23, 9aa51ce

If a patch does not apply, run git am --abort, check that git rev-parse HEAD is 9aa51ce4fdd3…, and try again.

##Build

Portable build (what we run, CPU variants loaded at runtime):

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_BACKEND_DL=ON \ 
-DGGML_CPU_ALL_VARIANTS=ON -DGGML_NATIVE=OFF -DLLAMA_CURL=OFF
cmake --build build -j

Test build (static, no GGML_BACKEND_DL, which the self-test needs):

cmake -S . -B build-test -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DGGML_NATIVE=OFF \ 
-DGGML_AVX=ON -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_F16C=ON -DGGML_BMI2=ON \ 
-DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=ON
cmake --build build-test --target test-gyro-cpu-opt llama-completion -j

##Test

cd build-test/bin
./test-gyro-cpu-opt # parts 1, 2, 3; prints "part N OK", exit code 0
taskset -c 1-5 ./test-gyro-cpu-opt --bench-gen -t 5 --runs 5 --reps 10 # one generation step, fix23 vs fix37

Measure on the model

export GGML_TQ_STATES=1 GGML_CPU_TILED_T8=1 GGML_CPU_FAST=trellis
F="-ngl 0 -t 5 -fa on -ctk q8_0 -ctv q8_0 --no-op-offload --temp 0 -b 1024 -ub 1024 --ignore-eos --ngram-on-disk -no-cnv"
taskset -c 1-5 build/bin/llama-completion -m Gyro-S-TQ1_0.gguf $F -c 2304 -f p1k.txt -n 1024 # fix23
sleep 60
GGML_CPU_GEN=1 taskset -c 1-5 build/bin/llama-completion -m Gyro-S-TQ1_0.gguf $F -c 2304 -f p1k.txt -n 1024 # fix37

Compare prompt eval and eval t/s, and diff the generated text. Interleave on/off runs and pause between them: on a small desktop the CPU heats up, and the clock speed after a long prefill affects the next run.

Tried and rejected (measured on the i5-8500)

idea	result
int16 activation without row pairing	−3…5%
256 KiB state table with 64-bit gathers	+8% hot, −5…20% with cold experts
two-pass index precompute, 4 rows per call	−10…16%
tiled V accumulation in attention	+7% on Tiger Lake, −13% on the i5-8500
dense Q5_K/Q6_K/Q8_0 kernel tweaks	×1.00 in the graph at 5 threads (memory-bound)
tiny nodes on one thread without barriers	+0.9%, noise
conv state in place	unsafe: the token column dies at CONCAT
prefetching the next dense weights during the expert matmul	0% (the 9 MiB L3 is flushed by the expert stream)
parallel TOP_K / ARGSORT for the router	~1%, and the order of equal scores could change

Files

ggml/src/ggml-cpu/arch/x86/quants.c | AVX2 quantize_row_q8_K, trellis pair table (1)
ggml/src/ggml-cpu/ggml-cpu.c | trellis MUL_MAT_ID paths(1); gen expert hook, delta-net plan (3)
ggml/src/ggml-cpu/ggml-cpu-impl.h | toggles, gen declarations (1, 3)
ggml/src/ggml-cpu/llamafile/sgemm.* | single-thread q8_0 gemm entry (1)
ggml/src/ggml-cpu/ops.cpp | FA per KV group, delta net one pass, GET_ROWS/CONCAT (1, 3)
ggml/src/ggml-cpu/quants.c, quants.h | trellis decode/tile(1); gen expert dot, q8_0 multi-dot (3)
ggml/src/ggml-cpu/tiled/tiled-kernel.cpp | T8 microtile (1)
tests/test-gyro-cpu-opt.cpp, CMakeLists | parts 1, 2(2); part 3 + --bench-gen (3)

ggml-cpu : faster trellis and dense K-quant prefill on AVX2