qwen38

qwen38-flash-next-qwen4exp (qwen38-flash-next-qwen4exp.md)

qwen38-flash-next (UD-Q2_K_XL, 73.45 GiB) serves on ${next_llama_bin} at

decode 13-21 tok/s, prefill ~236 tok/s WARM at 2.7k (~94 cold). Vision,

tool calls and coding all VERIFIED WORKING. Usable; not yet grilled. **RAM upgrade planned 2026-09-07 —

re-run the ladder after it; the predictions to check are at the bottom.**

The arch gate (solved)

Arch is qwen4exp. Our pinned ${llama_bin} (70adb1b4c, 2026-08-23) does NOT

have it — upstream PR #27742 merged 2026-08-27, four days later. Built

mainline at 465e49b9cea78a68b9c244ffb48d0ee24a82873d side-by-side as

next-llama.cpp/. Verified: qwen4exp present there, absent from llama.cpp/.

DELETION CONDITION DIFFERS from prism/ifm: this one dies when *we* rebuild

llama.cpp/ past that commit, not when something lands upstream.

The built binary's --version LIES: it reports the coolbox HEAD, because

the tarball has no .git and CMake walked up to the enclosing repo. Real SHA is

in next-llama.cpp/.coolbox-sha.

Why a 12 GB card can serve an 82 GB file

177B on paper = 125B-A6B MoE + a 51B n-gram PLE table. The PLE is not a

network: per_layer_token_embd.weight is [160, 320001536], **IQ4_NL, 26.82

GiB**, built with TENSOR_READ_LAZY — *"read rows on demand instead of loading

whole tensor; requires mmap for now"*. Measured **byte-identical in UD-IQ1_S and

UD-IQ3_XXS**, so quant choice moves only expert weights (37.11 -> 45.29 GiB).

Resident (non-PLE) by quant: IQ1_S 40.74 / IQ1_M 42.60 / Q2_K_XL 46.63 /

IQ3_XXS 49.50 GiB.

THE LESSON: lazy PLE fixes FITTING, not SPEED

I used the lazy-PLE discovery to overturn my own (correct) first read that this

box is RAM-gated. That was wrong. TENSOR_READ_LAZY only governs whether the

model loads; the 46.63 GiB of non-PLE weights are still touched every token,

and that sets tok/s.

* I GOT THE SPEED VERDICT WRONG TWICE. READ THIS BEFORE QUOTING A NUMBER *

Smoke test 2026-09-07, server-side timings, one rep each:

probe prompt_tok prefill t/s out_tok decode t/s

plain 23 10.55 85 12.89

tools 319 30.33 74 18.61

coding 38 22.87 270 20.51

vis-small 221 57.66 300 22.13

vis-large 2711 111.15 300 21.90

PREFILL SCALES 10.5x WITH PROMPT SIZE (10.6 -> 111.2 t/s from 23 -> 2711

tokens). My "prefill is 6 tok/s, unusable for agents, 2k prompt = 5 min to first

token" was measured on a 20-token prompt where FIXED OVERHEAD DOMINATES.

AND THEN A THIRD UNDER-REPORT: COLD vs WARM IS 2.5x. The 111 t/s above was

itself a FIRST-REQUEST-AFTER-LOAD number. Measured over 9 requests on 3 server

starts, a 2.7k-token prompt runs:

1st request after load ~94 t/s prefill (page cache still filling)

every request after ~236 t/s prefill

NEVER benchmark this box on the first request after a model load. The

ladder, the smoke test and the "6 tok/s prefill" panic were all cold numbers.

Warm truth: prefill ~236 t/s at 2.7k, decode 13-21 t/s (noisy at n=3 — quote the

range, not a mean).

**This is exactly [[gpt-oss-120b-cpu-moe-spill]]'s recorded warning — "short-prompt

prefill is MEANINGLESS" — and I walked into it anyway.**

DECODE also rises with load: 12.9 t/s on a trivial turn, ~22 t/s on real

work. The ladder's 6.7 was ntok/WALL-CLOCK around the whole HTTP request, which

folds in connection + prefill + load. Ladder cells are comparable RUNG-TO-RUNG

(all measured identically) and are NOT throughput figures. Always take numbers

from timings.prompt_per_second / timings.predicted_per_second.

| | Codacus (YouTube IH8XmxiwliQ) | this box |

|------------|-------------------------------|----------|

| VRAM | 12 GB | 32 GB|

| RAM | 64 GB | 31 GB |

| CPU | Ryzen 5600X | i7-8700K |

| tok/s | 24.4 | ~22 decode at load |

Comparable decode once measured properly. The gap is real but ~1.1x, not 3.6x. **VRAM does not substitute for RAM on a

spill config.** See [[gpt-oss-120b-cpu-moe-spill]] for the shape that DOES work.

ncmoe ladder (n=1 each, 2026-09-07)

ncmoe 30 ts 72,28 c262144 vis 5.4 only config holding 262144

ncmoe 26 ts 70,30 c131072 novis 6.2

ncmoe 24 ts 70,30 c131072 novis 6.7 free CUDA0 only 1090 MiB

ncmoe 24 ts 70,30 c131072 -b1024 6.4 -b 1024 COSTS speed, buys nothing

ncmoe 26 ts 70,30 c131072 vis 6.7 <- THE ENTRY

ncmoe 22 / c262144@ncmoe26 FAILED

The curve is FLAT (~8% per 2 layers) unlike gpt-oss-120b's 5.05 -> 21.14.

Reaching 24 tok/s needs ncmoe ~0, i.e. all 46.63 GiB on 32 GB of VRAM.

VISION IS FREE: ncmoe 26 *with* mmproj ties ncmoe 24 without. Do not drop

--mmproj to chase speed — it costs VRAM, not tok/s.

A NON-trap: --threads

llama.cpp's --threads default (-1) ALREADY picks 6 = the physical core count

on this box (verified: no flag -> "llama threadpool init, n_threads = 6").

6c/12t i7-8700K, siblings 0,6 1,7 2,8 3,9 4,10 5,11. I wrongly called the

absence of --threads across llama-swap.yaml a missed optimisation worth up to

+79% — it is not; llama.cpp counts PHYSICAL cores, not hardware_concurrency.

Codacus's 12 -> 6 thread finding applies only where 12 was set explicitly.

-tb 12 TESTED AND REFUTED 2026-09-07 (n=3/arm, third arm re-runs -tb 6 as

an order guard):

cold (1st req) tb6 93.92 / 94.74 tb12 81.86 -> tb12 13% WORSE

warm (reps 2-3) tb6 235.8 mean tb12 232.0 -> wash

Prefill is bandwidth-bound too, not compute-bound. Nothing to change.

Three traps this cost time on

1. --tensor-split is MANDATORY with -ncmoe. Omitting it dies with

allocating 22764.49 MiB on device 1: -ncmoe makes the first N layers

nearly weightless, and llama.cpp's default split divides by LAYER COUNT, so

CUDA0 got every light layer and CUDA1 the whole heavy tail (24 x 0.883 GiB +

KV = 22.7 GiB, matching the error exactly). Same reason gpt-oss-120b pins

78,22. The balance point is pulled DOWN from the naive 77 by the mmproj,

which loads wholly on CUDA0 and ignores the split

([[mmproj-caps-cuda0-tensor-split]]). 74,26 starves CUDA0 (KV), 68,32 starves

CUDA1 (compute pp buffers) — the two edges are different cards.

2. --load-mode none CANNOT be used, even though llama.cpp suggests it

("tensor overrides to CPU are used with mmap enabled"). No mmap means the PLE

cannot be lazy: 23 GiB experts + 26.82 GiB table = ~50 GiB of 31 GiB RAM.

Tested — hit 27 GiB resident and was killed before it thrashed the box.

3. Context and speed are directly opposed. 262144 loads ONLY at ncmoe 30,

the slowest rung. KV is cheap either way (full_attention_interval=4 -> 12

of 48 layers, 2 KV heads: q8_0 = 1.59 GiB @131072, 3.19 @262144) — it is the

WEIGHTS that do not fit. See [[kv-sizing-full-attention-layers]].

After the RAM upgrade — what to re-test

At 64 GB the whole 46.63 GiB resident set fits with the PLE page-cached

(Unsloth's own recommendation is 96 GB total). Predictions to falsify:

• ncmoe should fall to its VRAM-only floor (~22-24) and paging should stop.

The sharpest falsifier is PREFILL: it should overtake decode (normal shape).

If prefill stays below decode the bottleneck was never paging and this whole

diagnosis is wrong.

• --load-mode none becomes testable and should then be FASTER.
• UD-IQ3_XXS (85.4% top-1 vs Q2_K_XL's 82.7%) becomes affordable — 49.50 GiB

resident. Re-download; only 2.87 GiB more than what is on disk.

• MTP: PR #28243 was still OPEN at 465e49b9c. Weights exist upstream

(MTP/mtp-*-Q4_K_M.gguf, 2.59 GiB). Worth 1.35x on qwen38-awq.

• THEN grill. At ~22 tok/s the full battery is ~25-30 h (k2-horizon took ~9 h at

65 tok/s), so it is expensive but no longer absurd and would be measuring

the hardware, not the model.

REQUIRES A PATCHED CHAT TEMPLATE (found in real use, 2026-09-07)

The embedded template raises on any non-first system/developer message (line

109-110). Claude Code injects <system-reminder> blocks mid-conversation, so it

500s on turn one with CallExpression at line 110, column 28. Entry carries

--chat-template-file chat-templates/qwen38-flash-next.jinja.

The 3h45m battery never saw it — no suite sends a non-first system message.

See [[jinja-system-guard-tool-parser]] and [[grill-does-not-validate-real-use]].

### QUANT TESTED AND REJECTED: UD-IQ3_XXS IS WORSE (2026-09-08)

Second failed hypothesis for the runaways. Downloaded IQ3_XXS (76.33 GiB) and

ran round 1 on it, same ncmoe 26 / 70,30 / 131072 / q8_0 KV:

build temp score cap-hits decode

Q2_K_XL 0.2 6/8 4 ~22

Q2_K_XL 0.2 ~4/8 4 ~22

Q2_K_XL 0.6 5/8 4 ~22

IQ3_XXS 0.2 4/8 5 ~20

More bits made it MORE verbose, not less. Every task that caps at Q2_K_XL

caps at IQ3_XXS too, PLUS median_two_sorted — which used only 2469 and 11722

tokens in the two Q2_K_XL runs and now runs the full 16000. Score sits inside

the noise band; decode is ~2 tok/s slower; it costs 82 GB of disk.

IQ3_XXS REJECTED. Q2_K_XL stays. (Weights kept on

/home/giovanni/models-nvme/Qwen3.8-Flash-Next-UD-IQ3_XXS — never delete without

asking; the mmproj there is a HARD LINK to the Q2_K_XL copy, so deleting that

directory is safe for the entry but check ls -i first.)

It DOES fit at the same ncmoe (CPU-side 25.91 GiB), leaving CUDA0 1138 / CUDA1

579 MiB — tighter than Q2_K_XL's 1798 / 1463 but stable.

### SO: THE RUNAWAYS ARE ARCHITECTURAL, AND GROUNDING IS THE ONLY LEVER

Three interventions tested, two refuted:

temperature 0.2 -> 0.6 NO EFFECT 4 cap-hits either way

quant Q2_K_XL -> IQ3_XXS WORSE 4 -> 5 cap-hits

API reference (klayout) WORKS 7 runaways -> 1

Do not chase this with a sampler or a bigger quant. Give the model reference

material and it terminates. See [[klayout-api-recall-vs-composition]].

### THIS DEVALUES THE RAM UPGRADE'S QUANT PATH

The earlier table here implied 64 GB unlocks Q3_K_XL (88.3% top-1) and

Q4_K_XL (92.3%) as a quality win. Temper that. Going 82.7% -> 85.4% made the

model more verbose and no more correct, so there is no reason to expect 88.3% or

92.3% to behave differently — while costing ~half the decode rate and ~100 GB of

disk. The upgrade's real value is headroom and fewer cold-start stalls, NOT a

quality unlock. Measure one round of Q3_K_XL before committing to it.

### REAP-256 (expert-pruned, Q3_K_XL) TESTED AND REJECTED 2026-09-08

AnonimousA/Qwen3.8-Flash-Next-REAP-256-duo-GGUF, 57.69 GiB — 512 experts

pruned to 256 (expert_count=256, expert_used_count still 10). Header-probed

before download: PLE intact at 26.82 GiB IQ4_NL, experts only 25.99 GiB.

build temp score cap-hits decode ncmoe CPU-side

Q2_K_XL 0.2 6/8 4 ~22 24 22.6 GiB

Q2_K_XL 0.2 ~4/8 4 ~22 24 22.6

Q2_K_XL 0.6 5/8 4 ~22 24 22.6

IQ3_XXS 0.2 4/8 5 ~20 26 25.9

REAP-256 Q3_K_XL 0.2 3/8 6 ~22 16 8.7 <- WORST

REJECTED. Cap-hits rise monotonically away from Q2_K_XL: 4,4,4 -> 5 -> 6.

Second expert-pruned build this box has rejected (after

[[glm-4.7-flash-reap-23b]]).

AND IT WAS NOT FASTER. CPU-side fell 22.6 -> 8.7 GiB and decode did not move

(~22 t/s both) — see [[moe-decode-scales-with-active-experts]]. My ~77 t/s

projection was built on the wrong cost model and I repeated it three times before

measuring.

Vision DOES survive pruning: the unsloth mmproj works on REAP weights —

prompt_tokens 65 (text) -> 260 (small image) -> 2749 (1345x2048), answers

correct. I first called it broken because my probe printed only content and

reported VISION OK: '' on an empty string — a false pass of exactly the

[[eval-false-pass-adjacent-data]] class. **Assert on the ANSWER and on

prompt_tokens, never on the absence of an exception.**

Ladder (splits from the balanced-minus-mmproj heuristic):

ncmoe 10 / 68,32 FAILED CUDA0 short 1088 MiB

ncmoe 12 / 66,34 FAILED CUDA0 short 1152

ncmoe 14 / 65,35 FAILED CUDA0 short 865

ncmoe 16 / 64,36 LOADED free 1022 / 3797 <- picked

ncmoe 20 / 62,38 LOADED free 3948 / 3001

All three failures were CUDA0-side while CUDA1 kept 3.8 GiB free at the winning

rung — the +7-point shift is too timid, and a harder CUDA1 bias would likely buy

2-4 more GPU layers. Moot here (speed is not footprint-bound) but relevant to any

future mmproj entry.

### CAN IT GO FASTER? EVERY LEVER TESTED 2026-09-08 — ANSWER IS NO

-sm tensor IMPOSSIBLE "LLAMA_SPLIT_MODE_TENSOR not implemented

for architecture 'qwen4exp'" — refuses at

load, no crash. Same gate as nemotron_h_moe.

This was the big hope (+12.5%..+49.5% on the

four entries that switched).

MTP --spec-type BLOCKED PR #28243 still OPEN at 465e49b9c. Worth

1.35x on qwen38-awq; revisit when merged.

pruned pool (REAP) REFUTED decode tracks ACTIVE experts, not footprint

[[moe-decode-scales-with-active-experts]]

-tb 12 REFUTED 13% worse cold, wash warm (order-guarded)

--threads NO-OP llama.cpp already picks 6 (physical cores)

q4_0 KV + ncmoe 24 REAL BUT SMALL — see below

**q4_0 KV @ ncmoe 24 vs q8_0 @ ncmoe 26, n=8 per arm, INTERLEAVED (A C A C…),

one discarded warm-up per server start:**

A (q8_0, nc26) 19.68 19.95 20.01 21.12 21.37 21.49 21.57 21.61 mean 20.85

C (q4_0, nc24) 21.36 21.42 22.09 22.11 22.13 22.29 22.47 22.63 mean 22.06

Welch t=3.65 (p<0.01), C wins 57/64 pairwise, delta +1.21 t/s (+5.8%)

REAL BUT NOT TAKEN. Three reasons: (1) 1.2 tok/s changes nothing about what

the model can be used for; (2) it is CONFOUNDED — C moved ncmoe 26->24 AND KV

q8->q4, and the gain is almost certainly the two extra GPU layers, with the KV

downgrade merely paying for them; (3) it inverts [[vram-spend-priority]]

(quality over speed, prefer q8 KV) for a barely-measurable return.

Entry keeps q8_0 KV at ncmoe 26.

METHOD NOTE: at n=3 the same comparison gave A 17.59 / C 19.97 with ranges

overlapping almost completely — unreadable, and both means BELOW the true value

because the arms ran in blocks so one got a colder cache. Interleave the arms and

discard the first request per server start, or the 2.5x cold effect swamps a 6%

signal. A crude "do the ranges overlap" test also called this INCONCLUSIVE when

Welch says p<0.01 — overlap is not the right test at n=8.

THE ONLY REMAINING LEVER IS RAM. The floor is ~0.88 GiB of active experts

read per token with roughly half of it off 41 GB/s system RAM. More RAM lets

ncmoe fall, moving layers onto 300-450 GB/s VRAM. Unlike the quant path, that

moves the right variable.

GRILLED 2026-09-07 — full battery, n=1, GRILL_TEMP=0.2, 3h45m

coding R1-R4 21/23 R1 6/8 (4 cap-hits), R2-R4 15/15

round 5 5/6 complete, LOOPED 0/6, REDUNDANT 0/41

interrupt_replan PASS

office unaided 9/9 incl 3/3 chart-reading vision stages

klayout unaided 4/8 7 of 8 tasks RAN AWAY to the 16k cap

klayout +ref 7/8 1 runaway

vision 20/21 easy 6/6, med 13/13, hard 1/2

klayout-vision 5/6

textturn CLEAN 0/30 tag leaks, 0/30 fabrication

KEEP. Strong on capability, weak on restraint. 21/23 ties GLM-4.7-Flash and

kimi-distill. office 9/9 unaided is top-tier. klayout +ref 7/8 is level with

[[fable-fusion-regrilled]] (15/16), the model kept *because* it is best there.

It clears interrupt_replan outright — the task that rejected apodex 3/3 and

aquila 4/4 and that k2-horizon only half-cleared at 1/3.

THREE PREDICTIONS I MADE AND GOT WRONG — ALL UNDER-RATING IT

1. "Expect the tommy2 IQ2_M collapse: office 0/9 unaided." It scored **9/9

unaided**. [[tommy2-iq2m-grill]]'s API-recall failure is a property of THAT

model, NOT of 2-bit quantisation. Do not reason from bit depth to capability.

2. "6.7 tok/s, NOT A DRIVER." Real decode is ~22 tok/s; see the measurement

section above. Three separate cold/wall-clock errors, all pessimistic.

3. "40+ hours, not worth the grill hours." It took 3h45m and produced one

of the better score sets on the box. That estimate came from asserting

k2-horizon's grill took ~9 h without opening the log — it took 62 minutes.

Open the log before quoting a duration.

The real weakness: RUNAWAYS at temp 0.2

12 tasks hit the 16000-token cap (4 in round 1, 7 in klayout-unaided, 1 in

klayout-ref). Each costs ~12 min, which is the entire reason the battery ran

3h45m instead of ~1h. Three of the runaways still PASSED — correct code

emitted, then it kept generating. That is over-thinking, not incoherence, and it

is the [[qwen38-ridge-rejected]] signature where cap-hit COUNT leads, not score.

The API reference does not just add knowledge, it STOPS the runaways:

klayout unaided 4/8 7 runaways 88 min

klayout +ref 7/8 1 runaway 33 min

That extends [[klayout-api-recall-vs-composition]]: grounding lets the model

terminate, so a runaway is partly a symptom of missing context, not purely a

sampler defect.

### TEMPERATURE TESTED AND EXONERATED — the runaways are the MODEL

I suspected 0.2 was the wrong sampler (the GGUF declares temp 1.0, and

[[k2-horizon-arch-gate]] — the other recent qwen-lineage arch — is 0/5 on

lru_ttl at 0.2 and ships at 0.6). Wrong. Round 1 cap-hits:

0.2 run A (battery) 4 expr_eval, articulation_points, lru_ttl, wildcard_match

0.2 run B (control) 4 expr_eval, articulation_points, lru_ttl, wildcard_match

0.6 4 expr_eval, damerau_levenshtein, lru_ttl, wildcard_match

Same count at both temperatures. Score 6/8 vs 5/8 is inside the noise band.

expr_eval, lru_ttl and wildcard_match cap in ALL THREE runs — a stable

defect fingerprint, not sampler variance. **Do not raise the entry's temp to fix

this; it does not work.** The k2-horizon 0.6 precedent does NOT generalise across

the lineage.

COROLLARY: the API reference is the only lever that has moved runaways

(klayout 7 -> 1). So the mechanism is GROUNDING, not entropy — the model runs on

when it lacks the context to finish, and more randomness does not help it stop.

### n=1 SCORES ARE UNREADABLE HERE, BUT CAP-HITS ARE NOT

The 0.2 control arm re-ran round 1 at IDENTICAL settings and three of eight tasks

FLIPPED: articulation_points PASS->FAIL, lru_ttl PASS->FAIL, wildcard_match

FAIL->PASS; median_two_sorted used 4.7x more tokens (2469 vs 11722). So a score

delta under ~2 points means nothing at n=1 ([[single-run-scores-are-unreliable]]).

Cap-hit COUNT and IDENTITY reproduced exactly across both 0.2 runs — that is

the readable metric ([[qwen38-ridge-rejected]]).

### WHICH TEMP THE ENTRY ACTUALLY SERVES

--temp in the entry is only a DEFAULT: server-schema.cpp:116 binds the

request's temperature straight onto params.sampling.temp, so any client that

sends one overrides it (Claude Code, pi and kimi-code all do). **The grill's 0.2

was never what the entry serves.** The entry ships the GGUF's own

general.sampling.temp 1.0, which is UNMEASURED — today's data covers 0.2

and 0.6 only. Left at 1.0 deliberately: it is the vendor value and furthest from

the low-entropy regime, and nothing shows temperature matters here anyway.

Smoke test: what WORKS (n=1 each, 2026-09-07)

• Vision WORKS on both 365x572 and 1345x2048 (the large one tiles to 2711

prompt tokens). Descriptions were accurate and specific — correctly read an

AFA crest and a medal ribbon. The mmproj is LAZY so only a real image proves

this; /health and a text turn prove nothing.

• Tool calling WORKS: real tool_calls with finish_reason=tool_calls and

empty content — it did NOT narrate a fabricated call.

• Coding: rotate_matrix correct (transpose then reverse rows, in place).
• No tag leaks in content on any probe; reasoning_content cleanly split.

Scores are still UNMEASURED — n=1 proves capability, not quality. Expect the 2-bit API-recall failure of

[[tommy2-iq2m-grill]] (office 0/9 unaided -> 9/9 with a reference) until shown

otherwise. Related: [[k2-horizon-arch-gate]] (the other side-by-side fork),

[[vram-spend-priority]], [[decode-bandwidth-ceiling]].

qwen38-mtp-awq-trial (qwen38-mtp-awq-trial.md)

Trialled 2026-08-24. Same Qwen3.8-27B AWQ INT4 body as [[qwen38-awq-vllm-trial]]

plus a real multi-token-prediction draft head. Served as qwen38-mtp on :9165.

Measured (identical prompts, same box, back-to-back)

| | qwen38-awq | qwen38-mtp | |

|---|---|---|---|

| decode, 2000-tok generation | 28.2 tok/s | 41.7 tok/s | 1.48x (n=1, optimistic) |

| decode, 3-run mean @192k | — | 39.9 tok/s | spread 39.4-40.7 |

| decode, 45 battery tasks | — | 38.1 tok/s mean / 39.5 median | ~1.35x — USE THIS |

| prefill, ~30k prompt | 34.9 s | 36.4 s | 0.96x (4% slower) |

| size | ~16 GB | 19 GB | |

| cold load | ~3.7 min | ~4.7 min | |

The 33-38% prefill regression from [[qwen38-atomic-install]] did NOT reproduce.

That was the main reason to be wary of MTP on this family; here the prefill cost is

~4%, comfortably paid for by a 48% decode gain. Net win for interactive/agentic use,

where decode dominates.

The config does NOT declare MTP — the weights do

config.json has no num_nextn_predict_layers / mtp_num_hidden_layers at all,

so metadata alone suggests "MTP" is just a repo-name claim. It is not: the weight

index carries 15 real mtp.* tensors (mtp.fc, a full single-layer block with

attention + MLP + both layernorms, mtp.norm, 2 pre-fc norms). Probe the WEIGHT

INDEX, not the config, to tell a real MTP build from a stub ([[agentworld-vision-stub]]

is the stub case). vLLM copes with the missing field:

getattr(config, "mtp_num_hidden_layers", 1) = 1, matching the single mtp.layers.0.

Serving requirements (vLLM 0.26.0)

• --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'.

qwen3_5_mtp is a REGISTERED method name in vllm/config/speculative.py; a wrong

string fails the load outright, so a successful load is itself evidence MTP wired up.

• --mamba-cache-mode align IS LOAD-BEARING — Qwen3_5MTP.__init__ raises

NotImplementedError on mamba_cache_mode == "all" (qwen3_5_mtp.py:218).

• Draft head is BF16 while the body is INT4; vLLM forces mtp.fc unquantized on

purpose. Mixed precision here is expected, not a packaging error.

• --max-model-len 196608 — the SAME window as qwen38-awq. It was set to 131072

first on the assumption that the BF16 draft head would starve the KV pool; that

assumption was WRONG and cost a context tier for nothing. **Verified 2026-08-24:

needle 3/3 at 182,830 real tokens** (depths 0.1/0.5/0.9, 313-358 s per depth —

cold prefills, not the ~1-2 s warm-cache false pass of

[[needle-warm-cache-invalidates-repeats]]), and the worker survived three

consecutive 183k prefills with no OOM. The head costs ~0.9 GB/card, not a tier.

The general lesson: "it loads" is not evidence a window WORKS

([[gemma4-ft-real-window-and-crashes]] loaded at 262k then OOMed under load), and

a filler prompt answering "OK" only proves ALLOCATION. A needle test proves the

context was processed AND retrievable, at the same memory pressure. Use it.

GRILLED 2026-08-24 — MTP COSTS NO QUALITY

Full 13-suite battery, 1 rep, 212 min. Envelope CLEAN (0 control-token leaks,

0 tool-envelope problems, 0 stream-only leaks across 126 turns).

| suite | qwen38-mtp | field |

|---|---|---|

| coding R1-R4 | 21/23 | IDENTICAL to plain qwen38-awq |

| R5 long-horizon | 6/6, 0 looped, 0% redundant | perfect |

| KLayout unaided | 0/8 | normal ([[klayout-api-recall-vs-composition]]) |

| KLayout +ref | 8/8 | best measured on this harness |

| klvision | 5/6 | > bonsai 4/6 |

| closed loop | 0/4 | normal |

| closed loop +ref | 4/4 | vs qwen3vl-8b 1/4, bonsai 0/4 |

| office unaided | 9/9 | ONLY model to manage this |

| office +ref | 9/9 | ref neither helps nor hurts |

| real-case gate | PASS 7/7 | |

**The headline: 21/23 coding is EXACTLY plain qwen38-awq's score, at ~1.35x the

decode speed.** Speculative decoding is meant to be lossless (the main model

verifies every draft token); this is that claim measured, not assumed. R5 6/6 with

zero looping rules out the degradation-as-looping mode that would be the first

symptom.

Office 9/9 UNAIDED is the standout and it corrects a generalisation. Every other

model here needs the API reference for office (gemma4-ft 6/9, qwen3vl-8b 3/9,

bonsai 0/9), which had suggested the suite measures API RECALL rather than

capability. qwen38-mtp shows a model can simply HAVE the recall.

Cap-hits: 4 total (1 in R1 expr_eval, 3 in unaided KLayout). Per

[[qwen38-ridge-rejected]] the cap-hit COUNT is the leading indicator of a quality

regression, so this is the number to watch on a re-grill. expr_eval also fails on

plain qwen38-awq, so it is not MTP-specific. 5 abnormal finish_reason turns (4.0%).

n=1 — [[round5-is-a-sample-not-a-measurement]]. The perfect scores (8/8, 4/4,

9/9, 7/7) are the ones most likely to move on a repeat.

Speed: quote 38 tok/s, not 41.7

Three measurements of the same model, and the number SHRANK each time the sample

got more realistic:

41.7 tok/s one synthetic 2000-tok generation (n=1)

39.9 tok/s 3 runs of the same prompt at the 192k config

38.1 tok/s MEAN over 45 graded tasks in the full battery (median 39.5,

range 27.1-43.0)

38.1 is the honest figure — it spans short tasks where per-request overhead

weighs more and prefill-heavy ones where MTP gives nothing. A single long clean

generation is the best case, not the average.

Caveat on the RATIO: qwen38-awq's 28.2 tok/s is itself a single bench run, not a

battery mean, so ~1.35x compares an average against a point measurement. The

direction is solid; do not defend the second decimal.

Runtime: 212 min, and why the estimate was wrong

I predicted ~30 min by calibrating on R4 (38.3s, 5 short tasks). Wrong by 7x. R4's

tasks are ~300 tokens; R1's are 7-10k, and KLayout+ref alone took 62 min because

its large API-reference prefills get NO MTP benefit (prefill measured 4% SLOWER).

Calibrate a battery estimate on a PREFILL-HEAVY suite, not the cheapest one —

MTP speeds decode only, so its advantage shrinks exactly where a battery spends

most of its time.

qwen38-awq-vllm-trial (qwen38-awq-vllm-trial.md)

qwen38-awq serves cyankiwi/Qwen3.8-27B-AWQ-INT4 on vLLM (TP=2, 196608 ctx,

fp8 KV, --tool-call-parser qwen3_coder + --reasoning-parser qwen3, util 0.97,

--max-num-seqs 8). Name says AWQ but quant_method is **compressed-tensors

INT4 W4A16** (group 32, asymmetric, mse observer) — vLLM auto-detects. Quant is

sound: all 48 Gated-DeltaNet (linear-attention) layers + lm_head + vision tower

+ MTP head kept BF16; only the 16 full-attention layers + MLPs are int4. Same

recipe as [[thinkingcap-awq-trial]] (cyankiwi's Qwen3.6-27B AWQ-INT4).

NOT the same weights as the llama.cpp qwen38. cyankiwi quanted the BASE

Qwen/Qwen3.8-27B (confirmed from config.json base_model_name); the llama.cpp

qwen38 serves the UNCENSORED fine-tune of the same base — see

[[qwen38-uncensored-mtp-probe]]. Same arch (qwen3_5, 64 layers / 16 full-attn),

different post-training. So this is the base model on vLLM, that is the

uncensored variant on llama.cpp.

Grill: 3-rep full battery (bench/grill-qwen38-awq-full.sh, GRILL_MAX_TOKENS=32000,

temp 0.2, ~28 tok/s, 696 min). DRIVEN THROUGH llama-swap :8090 — do NOT HUP or

swap models mid-battery (tears down the loaded model → empty response →

JSONDecodeError that fakes a failure; same trap as the 2026-08-17 qwen38 run).

| suite | rep1 | rep2 | rep3 | read |

|---|---|---|---|---|

| r1 (8 coding) | 5/8 | 6/8 | 5/8 | 2 STABLE runaways: expr_eval + wildcard_match hit 32000 cap every rep. articulation_points 2/3, median_two_sorted 2/3 = variance |

| r2 (5 algo) | 4/5 | 5/5 | 5/5 | text_justify one-off (rep1) |

| r3 (5 tool/format) | 5/5 | 5/5 | 5/5 | qwen3_coder parser clean; restraint test correctly declined a tool call |

| r4 (5 agentic loop) | 5/5 | 5/5 | 5/5 | lean: 1183-1206 tok, 9 steps, ~49s wall, 131-134 tok/step, 0 redundant |

| r5 (6 long-horizon) | 6/6 | 6/6 | 5/6 | looped=0/6 ALL 3 reps, redundant=0.0% — ZERO looping |

| klayout (unaided) | 0/8 | 0/8 | 0/8 | expected baseline — [[klayout-api-recall-vs-composition]] |

| klayout-ref (API ref) | 7/8 | 7/8 | 7/8 | stable top-tier, ties [[fable711-gptq-int4-vllm-trial]] |

| office | 9/9 | 9/9 | 9/9 | struct+pdf+vision clean |

| office-ref | 9/9 | 9/9 | 9/9 | same (already maxed) |

| realcase gate | 7/7 PASS | 7/7 PASS | 0/7 RUNAWAY | 2/3 — rep3 blew past 32k cap, emitted unterminated string literal → 0/7. [[realcase-spec-plus-apiref-multiplicative]] |

Verdict: accurate + clean-agentic, but SLOW and runaway-prone. The standout

is zero R5 looping across 3 reps — the long-horizon round where many

candidates here have looped (coder-prune's multifile_refactor looped in all 8

runs; the rejected [[qwen38-ridge-rejected]] and gemma-4-26b-a4b had genuine R5

loops). KLayout+ref 7/8 ties the best, office a clean sweep, real-case gate

capable (7/7 twice). Decode is ~28 tok/s — dense-27B class, same as

fable-711-gptq and thinkingcap, NOT a speed model (vs the MoE 30B-A3B entries

at ~90-140 tok/s). The weakness is the runaway tendency: 2 stable cap-hits

in r1 (expr_eval, wildcard_match, every rep) plus the realcase gate 1/3 of the

time. Cap-hit COUNT is the leading indicator — see [[qwen38-ridge-rejected]] —

and 2/repeat is moderate (thinkingcap had 0; ridge was rejected at 3/repeat).

Reasoning note: this IS a thinking model. It emits a think block that the

qwen3 parser splits into the reasoning field — but vLLM 0.26.0 exposes that

field under the key reasoning, NOT reasoning_content. Checking

reasoning_content (the OpenAIClients convention / older vLLM key) falsely

shows empty and led to a wrong "no think tags" call on the first load-test.

content holds the final formatted answer; reasoning holds the internal

working. The parser works. (A thinking model also fits the runaway tendency

above — thinking models burn budget, see [[budget-sensitivity-is-a-model-property]].)

Role: the concurrent vLLM counterpart to the llama.cpp qwen38. That entry

has 262k context + MTP (~30 tok/s) but is single-stream; this one trades 32k

less context (196k vs 262k) and MTP for 8-way concurrency + real tool/reasoning

parsers. Keep both — different runtimes, different fine-tunes (base vs

uncensored), different workloads. Overlaps heavily with fable-711-gptq (both

vLLM dense 27B, ~28 tok/s, vision, KLayout 7/8); qwen38-awq's edge is the cleaner

R5 (0 looping) and the 2/3 real-case gate.

qwen38-ridge-rejected (qwen38-ridge-rejected.md)

Qwen3.8-27B-Ridge 3.7bpw — REJECTED, and the reason is runaways

empero-ai/Qwen3.8-27B-Ridge-GGUF, 11.73 GiB + 0.87 BF16 mmproj. Installed

alongside the served qwen38 2026-08-18, grilled, removed same day, weights

DELETED (+12.6 GiB). Quantization of Qwen's official checkpoint (rev

1d4bf0f2) — not an abliteration.

What "Ridge" is, and it is a real idea

Architecture-aware mixed quant for this hybrid arch: 64 layers =

16 x (3 x GatedDeltaNet -> FFN + 1 x GatedAttn -> FFN). The card's claim is that

generic IQ2_XS / UD-IQ2 do not treat the GDN state (ssm_alpha / ssm_beta)

or the GDN mixers as first-class, and Ridge keeps them higher-precision.

Credible — SSM state params are few but disproportionately

quantization-sensitive, like norms. MTP head retained (blk.64/nextn;

verified 866 tensors, block_count 65). The card is honest: *"Not lossless.

+9% wiki-style PPL vs our BF16 convert"* — contrast [[gemma4-mtp-drafter-blocked]]'s

AEON neighbour, whose card argues KL drift away as a feature.

The config was BETTER than the incumbent's — that was the appeal

| | served qwen38 | Ridge |

|---|---|---|

| weights + projector | 19.05 GiB | 12.60 GiB |

| KV depth @262144 | q4_0 (policy exception) | q8_0 |

| decode | ~30 tok/s | 44.9 tok/s (+50%) |

| free VRAM after image | 1332 / 1121 MiB | 1140 / 3095 MiB |

| split | 52,48, uniquely determined | 50,50, first rung, no tuning |

The smaller weights pay for q8_0 KV at the full 262144, so it needed none of

the "prefer KV q8" policy exception qwen38 makes. Vision, tool call and the

non-first-system template all verified. This is the config we wanted.

THE MEASUREMENT THAT KILLED IT

Round 1 + realcase, GRILL_MAX_TOKENS=32000, temp 0.2, through llama-swap —

settings identical to qwen38's 2026-08-17 battery, so directly comparable.

| | qwen38 (3 reps) | Ridge (2 reps) |

|---|---|---|

| round 1 | 7/8, 7/8, 7/8 = 21/24 | 5/8, 6/8 = 11/16 |

| spread | ZERO | differs on wildcard_match |

| cap-hits / repeat | 2, 1, 1 | 3, 3 |

| realcase | 7/7, 7/7, 0/7 runaway | 7/7, 7/7 |

The +9% PPL does not show up as wrong answers — it shows up as RUNAWAYS.

Ridge fails expr_eval AND articulation_points in both repeats, plus

wildcard_match in one, every time by burning the full 32,000 tokens. qwen38

fails only expr_eval. A quant that cannot stop generating is worse than the

score alone suggests: those tasks cost ~850 s each.

Realcase is a wash, not the win it first looked like. Both models hit the cap

there; Ridge avoided a runaway in 2 attempts where qwen38 had one in 3, which

n=2 cannot establish. (Beware: the jsonl aggregate reports realcase differently

from the per-repeat logs — read logs/realcase-rep*.log, which show 7/7 gates,

not the jsonl's row counts. Misreading that initially made Ridge look far ahead.)

Confound — this does NOT prove "3.7bpw is too low"

Ridge differs from qwen38 in two ways: quantization (3.7 vs ~5.5 bpw) AND

abliteration (Qwen original vs Heretic). Exactly the flaw that sank the

[[qwen38-atomic-install]] A/B. All that is established is that *this build* is

worse at round 1. A clean bpw test would need Ridge against a Ridge-recipe quant

of the same weights at a higher bpw, which does not exist.

Transferable

• A faster model that runaways more is a net loss, even at +50% tok/s: the

runaway tasks each burn the full budget, so wall-clock per suite barely

improves while the score drops.

• Cap-hit COUNT is the leading indicator, more diagnostic than pass/fail —

it separates "got it wrong" from "could not stop", and only the latter

correlates with quant aggressiveness here.

• Speed measured in the grill (37-45 tok/s) matched the probe's 44.9, so the

single-prompt probe number was honest for once.

qwen38-atomic-install (qwen38-atomic-install.md)

qwen38-atomic — AtomicChat Qwen3.8-27B AD-Q5_K_M (REMOVED)

> REMOVED 2026-08-18, same day it was installed. Weights DELETED (+20 GB).

> A matched sweep found it 33-38% slower at prefill than the served

> qwen38 on identical prompts (940 vs 704 tok/s at 17k; 799 vs 580 at 69k),

> with decode a wash at short context and *worse* at depth (13.7 vs 16.3 at

> 197k). The abliteration A/B it was meant to enable was also confounded: the

> two builds differ in quant recipe as well as abliteration, so no score

> difference could be attributed to either. Everything below still stands as

> measured — keep it for the two transferable lessons.

Installed 2026-08-18. Q5_K_M 18.84 GiB + mmproj-Qwen3.8-27B-F16.gguf 0.86 GiB

on /mnt/ssk500/gguf/, port 9148, -c 262144, q4_0 KV, --spec-type draft-mtp,

--tensor-split 52,48. Wired to all six harnesses.

Deliberately the SAME serving config as [[qwen38-uncensored-mtp-probe]] —

same base (Qwen/Qwen3.8-27B), same context, KV depth, split and MTP flags. The

difference is only the weights: qwen38 is Heretic-abliterated, this is a

plain imatrix quant of Qwen's original weights. Identical config is what makes

them comparable; change one knob and that is lost.

MTP IS PRESENT AND THE REPO NEVER SAYS SO

No mtp / draft / speculative in any filename, tag, or README, and no

separate draft file — unlike JonathanColetti's repo, which advertises all of it.

But the GGUF header says otherwise: block_count 65, 866 tensors, and a 65th

block of

blk.64.nextn.eh_proj.weight blk.64.nextn.enorm.weight

blk.64.nextn.hnorm.weight blk.64.nextn.shared_head_norm.weight

which is Qwen's MTP head under its nextn name. Measured draft acceptance

0.72 / 0.72 / 0.95 (mean draft len 2.43-2.90) at 29.8 tok/s — the same ~2x

over the ~16 tok/s a Qwen3.8 dense 27B manages without it.

Probe the GGUF, not the filenames. A filename grep calls this build BOTH

text-only (the projector is mmproj-* here but was *-vision-f16 on the other

repo) AND MTP-less, and is wrong twice. Read block_count, then range-fetch past

the KV block — the 151k-token vocab pushes tensor names several MB in, so a

first-8KB read shows none of them.

--tensor-split 52,48 IS UNIQUELY DETERMINED — AND 54,46 IS A TRAP

This file is 0.66 GiB larger than the abliterated twin's, which spends nearly all

the remaining headroom. Measured at 262144, with an image encode as the gate:

| split | result |

|---|---|

| 52,48 | loads, encodes, 922 / 713 MiB free — SERVED |

| 54,46 | LOADS FINE, then SEGFAULTS on the first image (564/617 free) |

| 50,50 | will not load — OOM device 1 (1226 MiB) |

| 56,44 | will not load — OOM device 0 (884 MiB) |

54,46 is the dangerous one. It passes every load-time check and dies on the

first vision request, because the ViT compute buffer is allocated lazily and

llama.cpp GGML_ASSERT-aborts the *process* — killing every in-flight request,

not just the one that triggered it. This is the third time that lazy buffer has

bitten (it ended the 2026-08-16 battery and forced qwen38 off 262144 at q8_0 KV).

Never accept a split, context or KV change on "it loads".

There is no tuning room here: one step either way and it either will not load or

will crash on an image. Headroom is thin — 713 MiB on the 4060 Ti against

qwen38's 1121, and 455 MiB is where a ViT encode has aborted before. The

1345x2048 test photo fits; a materially larger one may not. **If an image ever

aborts this entry, drop -c to 229376 (frees ~1.2 GiB) — do NOT touch the

split.**

Verified at install

Text, vision, tool call (enum correctly mapped to celsius), and a non-first

system message returning BANANA (same patched template as qwen38 — same base

and tokenizer, so the stock template's raise_exception landmine applies here

too). Vision is noticeably more detailed than qwen38 on the same photo: it named

the AFA crest and the Le Coq Sportif logo, and separately identified the subject

as resembling Maradona. Cold load 91 s from ssk500.

Quality metadata: a KLD table, not the usual useless PPL

The repo publishes KL-divergence against the source weights, which actually

discriminates:

AD-Q6_K 0.00107 98.67% AD-Q4_K 0.01126 95.59%

AD-Q5_K_M 0.00419 97.34% AD-IQ2_XXS 0.25663 79.44%

Monotonic and meaningful — unlike JonathanColetti's wikitext-2 PPL table, whose

entire spread sat inside its own ±0.25 error bar and ranked Q8_0 *below* Q5_K_M.

Prefer KLD when a repo offers it. Q6_K (23.29 GiB) does not fit here at any

useful context, so Q5_K_M is the ceiling on this box.

UNGRILLED — do not treat the two as interchangeable

qwen38 has a 10-hour 3-repeat battery behind it (22/23 coding with **zero

spread**, 54/54 office, 19/24 KLayout+ref, needle 3/3 at 234k). This entry has

none. The interesting experiment it enables is a controlled abliteration A/B on

identical serving config — but per [[genesis-hermes-v5]], cross-build grill

comparisons are only sound when the config matches, which is exactly why it does.

qwen38-uncensored-mtp-probe (qwen38-uncensored-mtp-probe.md)

Qwen3.8-27B-Uncensored (Heretic abliteration) — SERVED as qwen38

JonathanColetti/Qwen3.8-27B-Uncensored-GGUF, **Q5_K_M 18.19 GiB + the vision

projector 0.86 GiB**, downloaded to /mnt/ssk500/gguf/ 2026-08-17 and probed

with bench/trial-qwen38u-install.sh. **It REPLACED the unsloth build in the

qwen38 llama-swap entry the same day** — served at -c 262144, q4_0 KV,

--spec-type draft-mtp, --tensor-split 52,48, port 9146. Same base as the

superseded [[qwen38-27b-install]]

(Qwen/Qwen3.8-27B), arch qwen35, block_count 65, head_count_kv 4, so the

same 34.0 KiB/tok q8_0 KV math applies.

Refusal directions removed with Heretic

(co-minimises refusal count against KL divergence from the base; no fine-tuning).

Abliteration quality was NOT measured here and neither was coding quality.

The projector is not called mmproj

It ships as Qwen3.8-27B-Uncensored-vision-f16.gguf. A triage grep for

mmproj reports NONE and wrongly concludes the build is text-only — it is not,

it is byte-comparable to unsloth's mmproj-F16 (0.86 GiB) and works via

--mmproj. Generalise the check in [[model-triage-checklist]] caveat 7 to

mmproj|vision|proj, not the literal string.

MTP doubles throughput (the 64k cost below was later ELIMINATED — see q4_0 KV)

Every quant ships in MTP and noMTP flavours (plus a standalone

draft-Q8_0.gguf). Measured on an identical prompt, --spec-draft-n-max 2:

| arm | context | tok/s |

|---|---|---|

| plain | 196608 | 16.0 |

| MTP --spec-type draft-mtp | 131072 | 32.1 |

2.0x. Draft acceptance 0.81 (143/176) then 0.69 (69/100), mean draft

length 2.62 / 2.38 — comfortably above the 50% floor at which the card says to

fall back to the noMTP quant, and in the same band as fable-fusion's 78.4%.

The card warns acceptance *may* fall because the draft head was trained against

the unmodified model; at 0.69-0.81 it evidently did not fall far.

This matters because the one real complaint about the installed qwen38 is that

it is a DENSE 27B at ~16 tok/s, the slowest model on the box. The plain arm here

measured 16.0 tok/s — a wash with qwen38, so abliteration aside, *MTP is

the entire reason to prefer this build.*

--tensor-split CANNOT buy the VRAM the MTP draft needs

MTP at 196608 OOM'd needing ~1.2 GiB on device 1. fable-fusion's entry records

that its MTP draft context allocates entirely on CUDA1 and that 58,42 exists

to compensate, so the obvious move was to bias weights off CUDA1. **It does not

work here** — it just relocates the failure:

196608 48,52 -> OOM device 1 (970 MiB + 260 MiB)

196608 56,44 -> OOM device 0 (884 MiB)

196608 60,40 -> OOM device 0 (1657 MiB, worse)

131072 48,52 -> LOADS, vision OK, 32.1 tok/s

The draft context is a real ~1.2 GiB that has to come from somewhere; the split

only chooses which card goes short. Context is the lever, not the split.

Don't burn probe cycles re-deriving this — go straight to reducing -c.

VRAM at the working config after an image encode: 14505/16311 (5060 Ti) and

13307/16380 (4060 Ti) — ~1.8 and ~3.0 GiB free, roomier than the installed

qwen38 at 196608.

Plain arm, verified at -c 196608

Loads in 92 s; vision correct and more detailed than qwen38 on the same photo

(it named the AFA crest); tool call well-formed with the enum mapped to

celsius; and the non-first system message returned BANANA — which

confirms the patched template *and* that the model obeys a mid-conversation

system turn (qwen38's equivalent smoke test hit its token cap before

answering, so this is the cleaner confirmation of the two).

Free VRAM after the image, plain @196608: 1882 / 1627 MiB — about 300 MiB more

than qwen38, consistent with the 0.65 GiB smaller file.

q4_0 KV is the RIGHT lever, not a smaller quant — 262144 + MTP + vision

Measured 2026-08-17. The question was how to buy back the 64k that MTP costs.

Two candidates: drop the weights Q5->Q4 (−2.53 GiB) or drop KV q8_0->q4_0

(34.0 -> 18.0 KiB/tok, −4.0 GiB at 262144). **KV wins — it saves more and

costs no weight precision**, and it needs no download since the Q5_K_M file is

already on disk. Same KV depth fable-fusion and qwen3next-thinking run.

**Q5_K_M + --spec-type draft-mtp + q4_0 KV loads the FULL 262144, encodes an

image, and holds ~30-34 tok/s** — roughly 2x the 16.0 tok/s plain arm, at the

full native window. So the earlier "MTP costs you 64k of context" trade-off is

NOT inherent; it was an artifact of insisting on q8_0 KV.

Split matters here and 48,52 is the wrong one — free VRAM after the image:

| split | 5060 Ti free | 4060 Ti free | total free | tok/s |

|---|---|---|---|---|

| 48,52 | 532 MiB | 1921 MiB | 2453 | 33.5 |

| 52,48 | 1332 MiB | 1121 MiB | 2453 | 29.9 |

| 54,46 | 960 MiB | 587 MiB | 1547 | 31.1 |

52,48 is the pick: same total free as 48,52 but balanced instead of leaving

532 MiB on one card — 532 MiB is within touching distance of the 455 MiB that

hard-aborted qwen38 at 262144. Note 54,46 consumes ~900 MiB MORE in total than

52,48, not less; buffer allocation is not monotonic in the split, so read the

measurement rather than extrapolating.

Do NOT read the tok/s column as a split ranking. Each figure is n=1 and

throughput here is a function of draft acceptance, which is stochastic:

acceptance ranged 0.57-0.84 across these runs (mean draft len 2.15-2.69). The

~11% spread is inside that noise — see [[single-run-scores-are-unreliable]].

One reporting caveat: the 52,48 vision call returned an EMPTY content string. It

was not a failure (no error, server alive afterwards, and the encode plainly

worked) — the 120-token budget went to reasoning before any content was emitted,

the same artifact as qwen38's smoke tests. The 54,46 run returned real text.

RESOLVED — the 262144 window IS real at q4_0 KV: needle 3/3 at 234,318

Measured 2026-08-17 through llama-swap (bench/needle-qwen38-20260817/):

control 31,596 real tok depths 0.1/0.5/0.9 -> 3/3 PASS (~37 s each)

far end 234,318 real tok depths 0.1/0.5/0.9 -> 3/3 PASS (527/530/619 s)

This REFUTES the lineage generalisation for this build.

[[qwen36-27b-deep-retrieval-fails]] recorded that fable-fusion / vision-coder /

bonsai all needle 3/3 at 32k and 0/3 at 222k, and called it architectural for

the Qwen3.6 27B family. Qwen3.8-27B-Uncensored retrieves cleanly at 234k — and it

does so on the CHEAPER q4_0 KV, which was the depth most likely to hurt. So q4_0

KV is vindicated here specifically, and the deliberate exception to the

"prefer KV q8" policy is paid for.

Caveat on scope: this is one code at three depths, not a recall curve — it shows

the window is addressable, not that reasoning over 234k of context is reliable.

Note also the cost: ~9-10 minutes of prefill for a far-end query (pp=234318),

so the far end is usable but never interactive.

A HARNESS TRAP, and it was self-inflicted. The first far-end run scored 2/3:

depth 0.9 came back as a JSONDecodeError on an empty HTTP response, which reads

exactly like a model failure. Cause was kill -HUP on llama-swap, sent to reload

an unrelated config change while a 9-minute request was in flight.

[[llama-swap-setup]] documents HUP as the no-sudo config reload with "same PID,

service stays active" — that is true of llama-swap itself but it **does NOT

preserve the loaded model process**, which is torn down. Never HUP while a long

probe is running, and treat an empty-response JSON error mid-suite as

infrastructure until proven otherwise.

The open decision

Installing means choosing between two configs of the same model:

plain @ 196608, 16 tok/s vs MTP @ 131072, 32 tok/s. Both fit; nothing

measured here says which is worth more, because **no coding quality was measured

on this build at all** — and qwen38 itself still has only one grill round

behind it, so an unproven abliteration would be compared against an unproven

base.

qwen38-27b-install (qwen38-27b-install.md)

Qwen3.8-27B — installed as qwen38

> SUPERSEDED 2026-08-17, same day. The qwen38 entry now serves

> JonathanColetti's Heretic-abliterated Q5_K_M with its MTP head at 262144 /

> q4_0 KV / split 52,48 — see [[qwen38-uncensored-mtp-probe]]. The unsloth

> UD-Q5_K_XL GGUF and its mmproj were DELETED from ssk500 (+20 GB), so the

> config below is no longer reachable without a ~19.7 GB re-download. Everything

> in this file about the LAZY ViT BUFFER, the template landmine and the grill

> still applies — the replacement shares the base model and the arch.

Installed 2026-08-17 from unsloth/Qwen3.8-27B-GGUF, **UD-Q5_K_XL 18.83 GiB +

mmproj-BF16 0.87 GiB**, on /mnt/ssk500/gguf/ (deliberate pick, not a default).

llama-swap entry qwen38 / alias qwen3.8-27b-192k, port 9146, -c 196608,

q8_0 K and V, --reasoning-format deepseek, --tensor-split 48,52. Cold load

86 s. Verified end-to-end THROUGH llama-swap on 8090 (process PPID = the

llama-swap MainPID, not a hand-started squatter): text, tool call, a non-first

system message and an image in the same request, ~1.3 GiB free on CUDA0 after

the encode.

It is a DENSE 27B, not an MoE — no num_local_experts in config.json. That

is the headline operational difference from everything else in the 27B class

here: measured ~16 tok/s against fable-fusion's 32, because a dense model

reads every weight every token. A task that runs to a 16k budget therefore costs

~17 minutes of wall clock. Arch is qwen35 in the GGUF (model_type qwen3_5),

the same hybrid family as fable-fusion: 64 layers, full_attention_interval 4,

so only 16 layers carry a context-scaling KV cache (n_kv_head 4,

head_dim 256) = 34.0 KiB/tok at q8_0, which is what makes a 192k window

affordable on a dense 27B at all.

"It loads" is NOT the test for a vision entry — the ViT buffer is lazy

The finding worth keeping. -c 262144 loads clean, reports healthy, and answers

text correctly with ~455 MiB free on CUDA0. The first image request then

kills the process outright:

``

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 315.16 MiB on device 0:

cudaMalloc failed: out of memory

ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 330473472

GGML_ASSERT(...) failed <- clip_encode, via mtmd_batch_encode

`

The ViT compute buffer is allocated lazily, on the first image, so a

vision-capable entry sized against load-time VRAM is a live crash rather than a

tight fit. And llama.cpp GGML_ASSERT-aborts the process instead of

returning a 500, so every in-flight request dies with it and the symptom

downstream is Connection refused, with no error attributable to the request

that caused it.

This is what ended the 2026-08-16 Qwen3.8 battery mid-run (see

bench/grill-qwen38-udq5kxl-20260816/): round 1 completed 3 tasks, then every

remaining suite logged REQ-ERR ... Connection refused and the run was recorded

as "server died", cause unknown. It was the vision stage of the office suite

reaching a server that had no room for the encoder.

So the acceptance gate for any --mmproj entry is encode an image, at the

largest image you have. 196608 was accepted only after

pictures/01rotulo-01-superAuDmsabo.jpg (1345x2048 — the same photo

fable-fusion is verified against) encoded successfully, leaving ~1.8 GiB free on

the 5060 Ti and ~1.0 GiB on the 4060 Ti. bench/trial-qwen38-install.sh

implements this gate: a context is accepted only if it loads AND survives the

image AND the server is still answering /health afterwards.

Also seen at load: graph_reserve: failed to allocate compute buffers followed

by retrying without pipeline parallelism. That warning is a **downgrade, not a

failure** — at 262144 it was the last warning before the crash, so treat it as a

sizing alarm rather than noise.

--tensor-split 48,52 — and the port-squat that faked a null result

Measured at 196608 with the image encode included (nvidia-smi idx0 = 5060 Ti =

llama.cpp CUDA1; idx1 = 4060 Ti = CUDA0 — the inversion in

[[gpu-device-ordering]]):

| split | 5060 Ti free | 4060 Ti (CUDA0) free |

|---|---|---|

| 50,50 | 1844 MiB | 1057 MiB |

| 49,51 | identical to 50,50 | identical |

| 48,52 | 1572 MiB | 1327 MiB |

| 42,58 | 700 MiB | 3431 MiB |

A is CUDA0 = the 4060 Ti, so lowering A relieves it. 48,52 is served:

CUDA0 is where the ViT buffer lands, so headroom there is exactly what prevents

the abort above. 42,58 overshoots and starves the other card to 700 MiB. Note

49,51 is a no-op — the split is per-layer, so a 1-point change need not move

a boundary.

The first version of this measurement was WRONG and it is worth knowing how.

It concluded "all splits identical to within 2 MiB, tensor-split is not a lever

here" — because a leftover trial server still held port 9146. Every new server

under test exited immediately on

srv start: couldn't bind HTTP server socket, hostname: 127.0.0.1, port: 9146

while the probe cheerfully read the squatter and reported plausible numbers.

Identical-to-2-MiB results across different configs is the tell: that is not a

null result, it is the same process answering twice. The same squatter also

served the entire grill run — llama-swap proxies to 127.0.0.1:9146 and does not

care who is listening — so the grill never exercised llama-swap's own startup of

the entry. Same family as [[llama-swap-setup]]'s hand-started-llama-swap trap,

one layer down. **Free the port and confirm the server under test is the one

answering before believing any VRAM comparison.** bench/trial-qwen38-install.sh

deliberately leaves its server up on 9146 at the end, so kill it before anything

else binds that port.

Quant choice

UD-Q5_K_XL over Q6_K deliberately. Plain **Q6_K (21.31 GiB) OOM'd at 262144 at

every -ub from 512 down to 64** (probed 2026-08-16) — -ub bottoms out as a

lever because the wall is context-dominated, not batch-dominated. The 2.48 GiB

saved buys the vision tower plus a usable compute buffer, worth more than Q5→Q6.

Template landmine (the usual one)

The stock Qwen3.8 template raises 'System message must be at the beginning.' —

the same trap as fable-fusion / vision-coder, surfacing as a 400 for any agent

client that sends a mid-conversation system turn. Patched copy at

chat-templates/qwen3.8-27b-nonfirst-system.jinja (that one raise_exception

replaced by an ordinary <|im_start|>system render). See

[[jinja-system-guard-tool-parser]].

Separately, the template defaults reasoning_effort to xhigh and injects a

"think carefully, validate assumptions" preamble. It accepts medium and low

via --chat-template-kwargs '{"reasoning_effort":"medium"}'. Left at the default

— capping reasoning blind is what got reverted on [[qwen3next-thinking-verdict]]

— but it is the first knob to reach for on output length: the 2026-08-16 trial

burned 25,842 tokens / 1679 s on a single round-1 task (expr_eval) at a 32k

budget.

The 2026-08-16 trial scripts mis-report — do not trust their load.log

bench/trial-qwen38-udq5kxl.sh and bench/grill-qwen38-udq5kxl.sh both

pgrep -f "llama-server.*Qwen3.8-27B-Q6_K" — a copy-paste from the Q6_K script.

Against the UD-Q5_K_XL binary that pattern never matches, so:

stop_server never killed anything, every -ub probe was declared FAILED while

the server was in fact coming up, and **four servers were left racing for the

same VRAM**. Its load.log says FATAL: did not load ctx=262144 at any probed

-ub while battery.log on the same run says server confirmed up ... -ub 256.

Both files are from the same script; the FATAL is the false one. Related:

[[pgrep-f-self-match-deadlock]]. bench/trial-qwen38-install.sh tracks the

launched PID directly instead.

Verified at the installed config (2026-08-17)

Plain generation; tool call (well-formed tool_calls, enum mapped to

celsius); non-first system message returns 200; vision correct on the 2048px

photo (identified the jersey, the striped pattern and the sponsor logo).

Grill round 1, single run — 7/8, zero runtime errors

bench/grill-qwen38-20260817/r1.log, run through llama-swap at the installed

config, GRILL_MAX_TOKENS=16000, temp 0.2. 72 minutes for 8 tasks.

PASS median_two_sorted 687.6s 10832 tok PASS kth_smallest_bst 38.7s 595

FAIL expr_eval 1023.4s 16000 tok PASS wildcard_match 109.8s 1740

PASS damerau_levenshtein 724.8s 11413 tok PASS search_rotated 30.9s 479

PASS articulation_points 1023.3s 16000 tok PASS lru_ttl 618.1s 9751

The run is clean as an infrastructure test: 8/8 tasks got a response, no

REQ-ERR, no 400 from the template, no tool-parser 500, NRestarts 0, and the

server was still healthy afterwards — against the 2026-08-16 battery which died

after 3 tasks.

The one FAIL is a budget artifact, not a defect. expr_eval stopped at

*exactly* 16000 out_tok with SyntaxError: unterminated string literal, and the

dumped file ends mid-sentence inside the model's REASONING

( 2 -3 4 ** 5: elements [(+,2), ) — it was still thinking at the cap

and never emitted a code block at all. Do not read this as an expr_eval miss:

the same task consumed 25,842 tokens at a 32k budget on 2026-08-16 and failed

there too, so expr_eval needs a separate measurement, not a bigger number.

Two other tasks also hit the cap and articulation_points` passed anyway.

Consistent 15.4-15.9 tok/s across every task, no fade — which is the dense

27B cost, and it is what makes this model expensive to grill: a capped task is

~17 min, so a full 3-repeat battery is a multi-hour commitment, unlike the MoE

entries. Score quality is UNMEASURED — one round, one repeat, and per

[[single-run-scores-are-unreliable]] and [[round5-is-a-sample-not-a-measurement]]

nothing here should be treated as a verdict.

qwen38-27b-hybrid-linear-attention (qwen38-27b-hybrid-linear-attention.md)

Triggered by Giovanni asking to check unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL, which

led to discovering the base model (Qwen's own org) and trying it across THREE

serving paths in one night. Dense 27.32B params, architectures:

Qwen3_5ForConditionalGeneration, GGUF/vLLM arch string qwen35/Qwen3_5.

Hybrid design: 64 real layers + 1 unused MTP head, full_attention_interval=4

so only 16 of 64 layers carry real attention KV (rest are linear_attention,

an SSM/RWKV-style recurrent mechanism) — the same genre of trick as

nemotron-lightning's Mamba hybrid, meant to make long context cheap in VRAM.

rope_theta=10,000,000 (genuinely large, unlike nemotron's suspicious

unscaled 10,000 — this one looks like real long-context design, not a

converter artifact). Vision-capable (mmproj present), reasoning model (own

model card recommends up to 262,144 reasoning + 131,072 final-response

tokens for agentic tasks, within an advertised 1M window — NOTE: this specific

27B GGUF/config only ever declared 262,144, never verified whether 1M applies

to a different sibling size or was aspirational marketing copy).

VERDICT: REJECTED for this box on ALL THREE paths tried

Nothing here says the model is bad — see "why release this at all" below. It

is a genuine mismatch with a 2x consumer-GPU (4060 Ti + 5060 Ti, PCIe not

NVLink) box running single-stream (--parallel 1).

### Path 1 — vLLM, cyankiwi/Qwen3.8-27B-AWQ-INT4 (19.57 GiB)

Repo name says AWQ; quantization_config.quant_method actually says

compressed-tensors (INT4, group_size 32, asymmetric, mse observer) — a

naming/reality mismatch, though vLLM auto-detects and loads it fine either

way. linear_attn layers, vision tower, and MTP head are excluded from

quantization (kept at original precision).

262144 does not fit — confirmed TWICE, once on possibly-contended VRAM

and once on a verified-clean 18 MiB/18 MiB GPU state, both times with the

IDENTICAL vLLM-reported number: needs 4.09 GiB KV cache, only 3.18-3.19 GiB

available, vLLM's own estimated ceiling is 203,840 tokens. Not contention,

not a fluke — a hard, reproducible constraint of this quant+model+hardware.

131072 DOES load (confirmed, served real health-check traffic for an

extended period) — but Giovanni explicitly did not want to settle for that;

see the two GGUF attempts below, both aimed at holding the FULL 262144 he

asked for via a smaller quant of the same model instead of a smaller context.

Gotcha found and fixed while probing this: vLLM's /health returns an EMPTY

200 body, so curl | grep -q '.' (require non-empty output) is a permanent

false negative — it silently loops the full timeout window then reports

FAILED even on a genuinely healthy server. Killed a working 131072 load this

way before the bug was caught. Fixed in bench/trial-qwen38-awq.sh: check

the HTTP status code only, never body content, for any future /health poll

against vLLM.

### Path 2 — llama.cpp, plain Q6_K GGUF (21.31 GiB, unsloth repo)

Verified REAL K-quant content via a GGUF header read (131 Q6_K tensors, 9.82

GiB) — unlike unsloth's misleading nemotron UD- build earlier the same

night, this dense model's clean hidden dims (5120, 17408, both /256 exact)

mean the K-quant path never needed a legacy fallback.

262144 does not fit at ANY -ub. Probed 512/256/128/64, all four OOM'd

on the COMPUTE BUFFER (not weights, not KV) — same "context-dominated, not

ub-dominated" wall documented for nemotron's 1M load

([[kv-sizing-full-attention-layers]]): 512 needed ~2.17 GiB, 256 ~1.6/1.19

GiB (split CUDA0/CUDA1), 128 ~1.31/1.11 GiB, 64 ~1.17/1.07 GiB — steadily

shrinking but with diminishing returns, never closing the gap. Already used

a BALANCED 50,50 tensor-split from the start (unlike nemotron's original

48,52 mistake), so rebalancing wasn't an available lever this time — the

model is simply ~2.5 GiB too big for 262144 on this box, full stop.

### Path 3 — llama.cpp, UD-Q5_K_XL GGUF (18.83 GiB, unsloth repo)

262144 DOES load, but only at -ub 256 (512 failed identically to Q6_K's

pattern) — and the load event itself exposed a probe-script bug worth

recording separately: llama-server's /health returns 503 while genuinely

still loading (not a crash), and this specific load took long enough

(~70s+) that the original 1000s probe window SHOULD have covered it, but a

stale port-bind on a later probe step confused the picture. The lesson:

when a -ub step "fails," check whether the PORT is actually held by a

still-loading earlier attempt before concluding OOM. trial-qwen38-udq5kxl.sh

still has the shorter probe windows; a future rerun should widen the

per-attempt patience or watch the specific PID rather than trusting a

generic port-bind failure as proof of OOM.

**Decode throughput measured 15.4-16.0 tok/s — flat across three separate

completions** (2612, 25842, and 10033 output tokens, all showing the same

~16 t/s). 5-6x slower than every existing model on this box

(gemma-awq ~93, nemotron-lightning ~90-95, every MoE model 85-95+). This is

architectural, not a fluke: dense 27B (full parameter activation every

token, unlike the MoE models that dominate this box's fast options) at a

forced -ub 256 (small batch, itself forced by the compute-buffer ceiling

above) over PCIe TP=2 (not NVLink) is memory-bandwidth-bound in the worst

possible way for this hardware.

**The server died partway through R1, consistent with an OOM-kill, not a

graceful failure.** median_two_sorted PASS (164.3s), expr_eval FAIL

(1678.8s, 25,842 tok, NOT a cap-hit — genuine logic miss on 1++2 not

raising ValueError, consistent with expr_eval being the field-universal

hard task per [[klayout-api-recall-vs-composition]]'s sibling finding),

damerau_levenshtein PASS (637.1s). Then articulation_points onward:

every subsequent suite (R2-R5, klayout, klayout-ref, office, office-ref,

realcase) returned Connection refused — the server itself was gone, not

producing bad output. The server's own log ends CLEANLY mid-generation

(task 4, decoding normally at 15.97 t/s, n_decoded=2747) with NO error

message at all — no CUDA OOM message (which llama.cpp always prints before

exiting, confirmed by every failed load attempt above showing one), no

signal trace. That pattern — clean truncation, zero logged error — is the

signature of an external SIGKILL, most likely the Linux OOM killer (system

RAM, not VRAM) given how tight this load already was. UNCONFIRMED: no

dmesg/journalctl -k access in-session to find a Killed process

... llama-server line; Giovanni could confirm with sudo dmesg | tail -50

if it matters later. Practical upshot either way: this config is loadable

but NOT STABLE for sustained serving at 262144 on this box — it survived

two long generations then died on the third, unprompted.

Why release a model like this at all — it is not a bad model, it is a mismatch

• Built for different hardware. TP over PCIe (this box) pays a real

cross-GPU sync tax every layer that NVLink or a single big datacenter GPU

doesn't. Most of tonight's slowness would not exist on proper multi-GPU

interconnect or a single H100/H200-class card.

• Dense vs MoE is a deliberate tradeoff, not an oversight. Every

parameter fires every token — no routing/expert-imbalance noise, generally

considered more predictable for hard reasoning. MoE (what is fast on this

box) trades some of that for speed.

• The architecture's real selling point is memory, not speed. The

hybrid linear-attention design is what let 262144 context fit in 32 GiB of

consumer VRAM AT ALL for a dense 27B — a normal full-attention transformer

this size likely couldn't. That is a genuine engineering win that shows up

as "smaller memory footprint," not "fast on a laptop-class GPU pair."

• The model card itself telegraphs a different intended use. Recommending

up to 262,144 tokens of pure reasoning before answering is a model built to

think for a very long time on hard problems, evaluated on accuracy/

benchmark leaderboards, not interactive latency — the same design

philosophy as OpenAI's o-series or DeepSeek-R1.

What would be needed to reconsider

Not a config change on THIS box — the compute-buffer ceiling and the

memory-bandwidth-bound decode are both hardware-shaped, not tuning-shaped.

Would need either datacenter-class GPUs (NVLink and/or much higher per-GPU

bandwidth) or a use case where 16 tok/s and occasional crashes are

acceptable (unattended batch/offline, no interactivity expected) — neither

matches how models get used on this box today.