qwen38-flash-next

qwen38-flash-next-qwen4exp (qwen38-flash-next-qwen4exp.md)

qwen38-flash-next (UD-Q2_K_XL, 73.45 GiB) serves on ${next_llama_bin} at

decode 13-21 tok/s, prefill ~236 tok/s WARM at 2.7k (~94 cold). Vision,

tool calls and coding all VERIFIED WORKING. Usable; not yet grilled. **RAM upgrade planned 2026-09-07 —

re-run the ladder after it; the predictions to check are at the bottom.**

The arch gate (solved)

Arch is qwen4exp. Our pinned ${llama_bin} (70adb1b4c, 2026-08-23) does NOT

have it — upstream PR #27742 merged 2026-08-27, four days later. Built

mainline at 465e49b9cea78a68b9c244ffb48d0ee24a82873d side-by-side as

next-llama.cpp/. Verified: qwen4exp present there, absent from llama.cpp/.

DELETION CONDITION DIFFERS from prism/ifm: this one dies when *we* rebuild

llama.cpp/ past that commit, not when something lands upstream.

The built binary's --version LIES: it reports the coolbox HEAD, because

the tarball has no .git and CMake walked up to the enclosing repo. Real SHA is

in next-llama.cpp/.coolbox-sha.

Why a 12 GB card can serve an 82 GB file

177B on paper = 125B-A6B MoE + a 51B n-gram PLE table. The PLE is not a

network: per_layer_token_embd.weight is [160, 320001536], **IQ4_NL, 26.82

GiB**, built with TENSOR_READ_LAZY — *"read rows on demand instead of loading

whole tensor; requires mmap for now"*. Measured **byte-identical in UD-IQ1_S and

UD-IQ3_XXS**, so quant choice moves only expert weights (37.11 -> 45.29 GiB).

Resident (non-PLE) by quant: IQ1_S 40.74 / IQ1_M 42.60 / Q2_K_XL 46.63 /

IQ3_XXS 49.50 GiB.

THE LESSON: lazy PLE fixes FITTING, not SPEED

I used the lazy-PLE discovery to overturn my own (correct) first read that this

box is RAM-gated. That was wrong. TENSOR_READ_LAZY only governs whether the

model loads; the 46.63 GiB of non-PLE weights are still touched every token,

and that sets tok/s.

* I GOT THE SPEED VERDICT WRONG TWICE. READ THIS BEFORE QUOTING A NUMBER *

Smoke test 2026-09-07, server-side timings, one rep each:

probe prompt_tok prefill t/s out_tok decode t/s

plain 23 10.55 85 12.89

tools 319 30.33 74 18.61

coding 38 22.87 270 20.51

vis-small 221 57.66 300 22.13

vis-large 2711 111.15 300 21.90

PREFILL SCALES 10.5x WITH PROMPT SIZE (10.6 -> 111.2 t/s from 23 -> 2711

tokens). My "prefill is 6 tok/s, unusable for agents, 2k prompt = 5 min to first

token" was measured on a 20-token prompt where FIXED OVERHEAD DOMINATES.

AND THEN A THIRD UNDER-REPORT: COLD vs WARM IS 2.5x. The 111 t/s above was

itself a FIRST-REQUEST-AFTER-LOAD number. Measured over 9 requests on 3 server

starts, a 2.7k-token prompt runs:

1st request after load ~94 t/s prefill (page cache still filling)

every request after ~236 t/s prefill

NEVER benchmark this box on the first request after a model load. The

ladder, the smoke test and the "6 tok/s prefill" panic were all cold numbers.

Warm truth: prefill ~236 t/s at 2.7k, decode 13-21 t/s (noisy at n=3 — quote the

range, not a mean).

**This is exactly [[gpt-oss-120b-cpu-moe-spill]]'s recorded warning — "short-prompt

prefill is MEANINGLESS" — and I walked into it anyway.**

DECODE also rises with load: 12.9 t/s on a trivial turn, ~22 t/s on real

work. The ladder's 6.7 was ntok/WALL-CLOCK around the whole HTTP request, which

folds in connection + prefill + load. Ladder cells are comparable RUNG-TO-RUNG

(all measured identically) and are NOT throughput figures. Always take numbers

from timings.prompt_per_second / timings.predicted_per_second.

| | Codacus (YouTube IH8XmxiwliQ) | this box |

|------------|-------------------------------|----------|

| VRAM | 12 GB | 32 GB|

| RAM | 64 GB | 31 GB |

| CPU | Ryzen 5600X | i7-8700K |

| tok/s | 24.4 | ~22 decode at load |

Comparable decode once measured properly. The gap is real but ~1.1x, not 3.6x. **VRAM does not substitute for RAM on a

spill config.** See [[gpt-oss-120b-cpu-moe-spill]] for the shape that DOES work.

ncmoe ladder (n=1 each, 2026-09-07)

ncmoe 30 ts 72,28 c262144 vis 5.4 only config holding 262144

ncmoe 26 ts 70,30 c131072 novis 6.2

ncmoe 24 ts 70,30 c131072 novis 6.7 free CUDA0 only 1090 MiB

ncmoe 24 ts 70,30 c131072 -b1024 6.4 -b 1024 COSTS speed, buys nothing

ncmoe 26 ts 70,30 c131072 vis 6.7 <- THE ENTRY

ncmoe 22 / c262144@ncmoe26 FAILED

The curve is FLAT (~8% per 2 layers) unlike gpt-oss-120b's 5.05 -> 21.14.

Reaching 24 tok/s needs ncmoe ~0, i.e. all 46.63 GiB on 32 GB of VRAM.

VISION IS FREE: ncmoe 26 *with* mmproj ties ncmoe 24 without. Do not drop

--mmproj to chase speed — it costs VRAM, not tok/s.

A NON-trap: --threads

llama.cpp's --threads default (-1) ALREADY picks 6 = the physical core count

on this box (verified: no flag -> "llama threadpool init, n_threads = 6").

6c/12t i7-8700K, siblings 0,6 1,7 2,8 3,9 4,10 5,11. I wrongly called the

absence of --threads across llama-swap.yaml a missed optimisation worth up to

+79% — it is not; llama.cpp counts PHYSICAL cores, not hardware_concurrency.

Codacus's 12 -> 6 thread finding applies only where 12 was set explicitly.

-tb 12 TESTED AND REFUTED 2026-09-07 (n=3/arm, third arm re-runs -tb 6 as

an order guard):

cold (1st req) tb6 93.92 / 94.74 tb12 81.86 -> tb12 13% WORSE

warm (reps 2-3) tb6 235.8 mean tb12 232.0 -> wash

Prefill is bandwidth-bound too, not compute-bound. Nothing to change.

Three traps this cost time on

1. --tensor-split is MANDATORY with -ncmoe. Omitting it dies with

allocating 22764.49 MiB on device 1: -ncmoe makes the first N layers

nearly weightless, and llama.cpp's default split divides by LAYER COUNT, so

CUDA0 got every light layer and CUDA1 the whole heavy tail (24 x 0.883 GiB +

KV = 22.7 GiB, matching the error exactly). Same reason gpt-oss-120b pins

78,22. The balance point is pulled DOWN from the naive 77 by the mmproj,

which loads wholly on CUDA0 and ignores the split

([[mmproj-caps-cuda0-tensor-split]]). 74,26 starves CUDA0 (KV), 68,32 starves

CUDA1 (compute pp buffers) — the two edges are different cards.

2. --load-mode none CANNOT be used, even though llama.cpp suggests it

("tensor overrides to CPU are used with mmap enabled"). No mmap means the PLE

cannot be lazy: 23 GiB experts + 26.82 GiB table = ~50 GiB of 31 GiB RAM.

Tested — hit 27 GiB resident and was killed before it thrashed the box.

3. Context and speed are directly opposed. 262144 loads ONLY at ncmoe 30,

the slowest rung. KV is cheap either way (full_attention_interval=4 -> 12

of 48 layers, 2 KV heads: q8_0 = 1.59 GiB @131072, 3.19 @262144) — it is the

WEIGHTS that do not fit. See [[kv-sizing-full-attention-layers]].

After the RAM upgrade — what to re-test

At 64 GB the whole 46.63 GiB resident set fits with the PLE page-cached

(Unsloth's own recommendation is 96 GB total). Predictions to falsify:

• ncmoe should fall to its VRAM-only floor (~22-24) and paging should stop.

The sharpest falsifier is PREFILL: it should overtake decode (normal shape).

If prefill stays below decode the bottleneck was never paging and this whole

diagnosis is wrong.

• --load-mode none becomes testable and should then be FASTER.
• UD-IQ3_XXS (85.4% top-1 vs Q2_K_XL's 82.7%) becomes affordable — 49.50 GiB

resident. Re-download; only 2.87 GiB more than what is on disk.

• MTP: PR #28243 was still OPEN at 465e49b9c. Weights exist upstream

(MTP/mtp-*-Q4_K_M.gguf, 2.59 GiB). Worth 1.35x on qwen38-awq.

• THEN grill. At ~22 tok/s the full battery is ~25-30 h (k2-horizon took ~9 h at

65 tok/s), so it is expensive but no longer absurd and would be measuring

the hardware, not the model.

REQUIRES A PATCHED CHAT TEMPLATE (found in real use, 2026-09-07)

The embedded template raises on any non-first system/developer message (line

109-110). Claude Code injects <system-reminder> blocks mid-conversation, so it

500s on turn one with CallExpression at line 110, column 28. Entry carries

--chat-template-file chat-templates/qwen38-flash-next.jinja.

The 3h45m battery never saw it — no suite sends a non-first system message.

See [[jinja-system-guard-tool-parser]] and [[grill-does-not-validate-real-use]].

### QUANT TESTED AND REJECTED: UD-IQ3_XXS IS WORSE (2026-09-08)

Second failed hypothesis for the runaways. Downloaded IQ3_XXS (76.33 GiB) and

ran round 1 on it, same ncmoe 26 / 70,30 / 131072 / q8_0 KV:

build temp score cap-hits decode

Q2_K_XL 0.2 6/8 4 ~22

Q2_K_XL 0.2 ~4/8 4 ~22

Q2_K_XL 0.6 5/8 4 ~22

IQ3_XXS 0.2 4/8 5 ~20

More bits made it MORE verbose, not less. Every task that caps at Q2_K_XL

caps at IQ3_XXS too, PLUS median_two_sorted — which used only 2469 and 11722

tokens in the two Q2_K_XL runs and now runs the full 16000. Score sits inside

the noise band; decode is ~2 tok/s slower; it costs 82 GB of disk.

IQ3_XXS REJECTED. Q2_K_XL stays. (Weights kept on

/home/giovanni/models-nvme/Qwen3.8-Flash-Next-UD-IQ3_XXS — never delete without

asking; the mmproj there is a HARD LINK to the Q2_K_XL copy, so deleting that

directory is safe for the entry but check ls -i first.)

It DOES fit at the same ncmoe (CPU-side 25.91 GiB), leaving CUDA0 1138 / CUDA1

579 MiB — tighter than Q2_K_XL's 1798 / 1463 but stable.

### SO: THE RUNAWAYS ARE ARCHITECTURAL, AND GROUNDING IS THE ONLY LEVER

Three interventions tested, two refuted:

temperature 0.2 -> 0.6 NO EFFECT 4 cap-hits either way

quant Q2_K_XL -> IQ3_XXS WORSE 4 -> 5 cap-hits

API reference (klayout) WORKS 7 runaways -> 1

Do not chase this with a sampler or a bigger quant. Give the model reference

material and it terminates. See [[klayout-api-recall-vs-composition]].

### THIS DEVALUES THE RAM UPGRADE'S QUANT PATH

The earlier table here implied 64 GB unlocks Q3_K_XL (88.3% top-1) and

Q4_K_XL (92.3%) as a quality win. Temper that. Going 82.7% -> 85.4% made the

model more verbose and no more correct, so there is no reason to expect 88.3% or

92.3% to behave differently — while costing ~half the decode rate and ~100 GB of

disk. The upgrade's real value is headroom and fewer cold-start stalls, NOT a

quality unlock. Measure one round of Q3_K_XL before committing to it.

### REAP-256 (expert-pruned, Q3_K_XL) TESTED AND REJECTED 2026-09-08

AnonimousA/Qwen3.8-Flash-Next-REAP-256-duo-GGUF, 57.69 GiB — 512 experts

pruned to 256 (expert_count=256, expert_used_count still 10). Header-probed

before download: PLE intact at 26.82 GiB IQ4_NL, experts only 25.99 GiB.

build temp score cap-hits decode ncmoe CPU-side

Q2_K_XL 0.2 6/8 4 ~22 24 22.6 GiB

Q2_K_XL 0.2 ~4/8 4 ~22 24 22.6

Q2_K_XL 0.6 5/8 4 ~22 24 22.6

IQ3_XXS 0.2 4/8 5 ~20 26 25.9

REAP-256 Q3_K_XL 0.2 3/8 6 ~22 16 8.7 <- WORST

REJECTED. Cap-hits rise monotonically away from Q2_K_XL: 4,4,4 -> 5 -> 6.

Second expert-pruned build this box has rejected (after

[[glm-4.7-flash-reap-23b]]).

AND IT WAS NOT FASTER. CPU-side fell 22.6 -> 8.7 GiB and decode did not move

(~22 t/s both) — see [[moe-decode-scales-with-active-experts]]. My ~77 t/s

projection was built on the wrong cost model and I repeated it three times before

measuring.

Vision DOES survive pruning: the unsloth mmproj works on REAP weights —

prompt_tokens 65 (text) -> 260 (small image) -> 2749 (1345x2048), answers

correct. I first called it broken because my probe printed only content and

reported VISION OK: '' on an empty string — a false pass of exactly the

[[eval-false-pass-adjacent-data]] class. **Assert on the ANSWER and on

prompt_tokens, never on the absence of an exception.**

Ladder (splits from the balanced-minus-mmproj heuristic):

ncmoe 10 / 68,32 FAILED CUDA0 short 1088 MiB

ncmoe 12 / 66,34 FAILED CUDA0 short 1152

ncmoe 14 / 65,35 FAILED CUDA0 short 865

ncmoe 16 / 64,36 LOADED free 1022 / 3797 <- picked

ncmoe 20 / 62,38 LOADED free 3948 / 3001

All three failures were CUDA0-side while CUDA1 kept 3.8 GiB free at the winning

rung — the +7-point shift is too timid, and a harder CUDA1 bias would likely buy

2-4 more GPU layers. Moot here (speed is not footprint-bound) but relevant to any

future mmproj entry.

### CAN IT GO FASTER? EVERY LEVER TESTED 2026-09-08 — ANSWER IS NO

-sm tensor IMPOSSIBLE "LLAMA_SPLIT_MODE_TENSOR not implemented

for architecture 'qwen4exp'" — refuses at

load, no crash. Same gate as nemotron_h_moe.

This was the big hope (+12.5%..+49.5% on the

four entries that switched).

MTP --spec-type BLOCKED PR #28243 still OPEN at 465e49b9c. Worth

1.35x on qwen38-awq; revisit when merged.

pruned pool (REAP) REFUTED decode tracks ACTIVE experts, not footprint

[[moe-decode-scales-with-active-experts]]

-tb 12 REFUTED 13% worse cold, wash warm (order-guarded)

--threads NO-OP llama.cpp already picks 6 (physical cores)

q4_0 KV + ncmoe 24 REAL BUT SMALL — see below

**q4_0 KV @ ncmoe 24 vs q8_0 @ ncmoe 26, n=8 per arm, INTERLEAVED (A C A C…),

one discarded warm-up per server start:**

A (q8_0, nc26) 19.68 19.95 20.01 21.12 21.37 21.49 21.57 21.61 mean 20.85

C (q4_0, nc24) 21.36 21.42 22.09 22.11 22.13 22.29 22.47 22.63 mean 22.06

Welch t=3.65 (p<0.01), C wins 57/64 pairwise, delta +1.21 t/s (+5.8%)

REAL BUT NOT TAKEN. Three reasons: (1) 1.2 tok/s changes nothing about what

the model can be used for; (2) it is CONFOUNDED — C moved ncmoe 26->24 AND KV

q8->q4, and the gain is almost certainly the two extra GPU layers, with the KV

downgrade merely paying for them; (3) it inverts [[vram-spend-priority]]

(quality over speed, prefer q8 KV) for a barely-measurable return.

Entry keeps q8_0 KV at ncmoe 26.

METHOD NOTE: at n=3 the same comparison gave A 17.59 / C 19.97 with ranges

overlapping almost completely — unreadable, and both means BELOW the true value

because the arms ran in blocks so one got a colder cache. Interleave the arms and

discard the first request per server start, or the 2.5x cold effect swamps a 6%

signal. A crude "do the ranges overlap" test also called this INCONCLUSIVE when

Welch says p<0.01 — overlap is not the right test at n=8.

THE ONLY REMAINING LEVER IS RAM. The floor is ~0.88 GiB of active experts

read per token with roughly half of it off 41 GB/s system RAM. More RAM lets

ncmoe fall, moving layers onto 300-450 GB/s VRAM. Unlike the quant path, that

moves the right variable.

GRILLED 2026-09-07 — full battery, n=1, GRILL_TEMP=0.2, 3h45m

coding R1-R4 21/23 R1 6/8 (4 cap-hits), R2-R4 15/15

round 5 5/6 complete, LOOPED 0/6, REDUNDANT 0/41

interrupt_replan PASS

office unaided 9/9 incl 3/3 chart-reading vision stages

klayout unaided 4/8 7 of 8 tasks RAN AWAY to the 16k cap

klayout +ref 7/8 1 runaway

vision 20/21 easy 6/6, med 13/13, hard 1/2

klayout-vision 5/6

textturn CLEAN 0/30 tag leaks, 0/30 fabrication

KEEP. Strong on capability, weak on restraint. 21/23 ties GLM-4.7-Flash and

kimi-distill. office 9/9 unaided is top-tier. klayout +ref 7/8 is level with

[[fable-fusion-regrilled]] (15/16), the model kept *because* it is best there.

It clears interrupt_replan outright — the task that rejected apodex 3/3 and

aquila 4/4 and that k2-horizon only half-cleared at 1/3.

THREE PREDICTIONS I MADE AND GOT WRONG — ALL UNDER-RATING IT

1. "Expect the tommy2 IQ2_M collapse: office 0/9 unaided." It scored **9/9

unaided**. [[tommy2-iq2m-grill]]'s API-recall failure is a property of THAT

model, NOT of 2-bit quantisation. Do not reason from bit depth to capability.

2. "6.7 tok/s, NOT A DRIVER." Real decode is ~22 tok/s; see the measurement

section above. Three separate cold/wall-clock errors, all pessimistic.

3. "40+ hours, not worth the grill hours." It took 3h45m and produced one

of the better score sets on the box. That estimate came from asserting

k2-horizon's grill took ~9 h without opening the log — it took 62 minutes.

Open the log before quoting a duration.

The real weakness: RUNAWAYS at temp 0.2

12 tasks hit the 16000-token cap (4 in round 1, 7 in klayout-unaided, 1 in

klayout-ref). Each costs ~12 min, which is the entire reason the battery ran

3h45m instead of ~1h. Three of the runaways still PASSED — correct code

emitted, then it kept generating. That is over-thinking, not incoherence, and it

is the [[qwen38-ridge-rejected]] signature where cap-hit COUNT leads, not score.

The API reference does not just add knowledge, it STOPS the runaways:

klayout unaided 4/8 7 runaways 88 min

klayout +ref 7/8 1 runaway 33 min

That extends [[klayout-api-recall-vs-composition]]: grounding lets the model

terminate, so a runaway is partly a symptom of missing context, not purely a

sampler defect.

### TEMPERATURE TESTED AND EXONERATED — the runaways are the MODEL

I suspected 0.2 was the wrong sampler (the GGUF declares temp 1.0, and

[[k2-horizon-arch-gate]] — the other recent qwen-lineage arch — is 0/5 on

lru_ttl at 0.2 and ships at 0.6). Wrong. Round 1 cap-hits:

0.2 run A (battery) 4 expr_eval, articulation_points, lru_ttl, wildcard_match

0.2 run B (control) 4 expr_eval, articulation_points, lru_ttl, wildcard_match

0.6 4 expr_eval, damerau_levenshtein, lru_ttl, wildcard_match

Same count at both temperatures. Score 6/8 vs 5/8 is inside the noise band.

expr_eval, lru_ttl and wildcard_match cap in ALL THREE runs — a stable

defect fingerprint, not sampler variance. **Do not raise the entry's temp to fix

this; it does not work.** The k2-horizon 0.6 precedent does NOT generalise across

the lineage.

COROLLARY: the API reference is the only lever that has moved runaways

(klayout 7 -> 1). So the mechanism is GROUNDING, not entropy — the model runs on

when it lacks the context to finish, and more randomness does not help it stop.

### n=1 SCORES ARE UNREADABLE HERE, BUT CAP-HITS ARE NOT

The 0.2 control arm re-ran round 1 at IDENTICAL settings and three of eight tasks

FLIPPED: articulation_points PASS->FAIL, lru_ttl PASS->FAIL, wildcard_match

FAIL->PASS; median_two_sorted used 4.7x more tokens (2469 vs 11722). So a score

delta under ~2 points means nothing at n=1 ([[single-run-scores-are-unreliable]]).

Cap-hit COUNT and IDENTITY reproduced exactly across both 0.2 runs — that is

the readable metric ([[qwen38-ridge-rejected]]).

### WHICH TEMP THE ENTRY ACTUALLY SERVES

--temp in the entry is only a DEFAULT: server-schema.cpp:116 binds the

request's temperature straight onto params.sampling.temp, so any client that

sends one overrides it (Claude Code, pi and kimi-code all do). **The grill's 0.2

was never what the entry serves.** The entry ships the GGUF's own

general.sampling.temp 1.0, which is UNMEASURED — today's data covers 0.2

and 0.6 only. Left at 1.0 deliberately: it is the vendor value and furthest from

the low-entropy regime, and nothing shows temperature matters here anyway.

Smoke test: what WORKS (n=1 each, 2026-09-07)

• Vision WORKS on both 365x572 and 1345x2048 (the large one tiles to 2711

prompt tokens). Descriptions were accurate and specific — correctly read an

AFA crest and a medal ribbon. The mmproj is LAZY so only a real image proves

this; /health and a text turn prove nothing.

• Tool calling WORKS: real tool_calls with finish_reason=tool_calls and

empty content — it did NOT narrate a fabricated call.

• Coding: rotate_matrix correct (transpose then reverse rows, in place).
• No tag leaks in content on any probe; reasoning_content cleanly split.

Scores are still UNMEASURED — n=1 proves capability, not quality. Expect the 2-bit API-recall failure of

[[tommy2-iq2m-grill]] (office 0/9 unaided -> 9/9 with a reference) until shown

otherwise. Related: [[k2-horizon-arch-gate]] (the other side-by-side fork),

[[vram-spend-priority]], [[decode-bandwidth-ceiling]].