qwen38-mtp

qwen38-mtp-awq-trial (qwen38-mtp-awq-trial.md)

Trialled 2026-08-24. Same Qwen3.8-27B AWQ INT4 body as [[qwen38-awq-vllm-trial]]

plus a real multi-token-prediction draft head. Served as qwen38-mtp on :9165.

Measured (identical prompts, same box, back-to-back)

| | qwen38-awq | qwen38-mtp | |

|---|---|---|---|

| decode, 2000-tok generation | 28.2 tok/s | 41.7 tok/s | 1.48x (n=1, optimistic) |

| decode, 3-run mean @192k | — | 39.9 tok/s | spread 39.4-40.7 |

| decode, 45 battery tasks | — | 38.1 tok/s mean / 39.5 median | ~1.35x — USE THIS |

| prefill, ~30k prompt | 34.9 s | 36.4 s | 0.96x (4% slower) |

| size | ~16 GB | 19 GB | |

| cold load | ~3.7 min | ~4.7 min | |

The 33-38% prefill regression from [[qwen38-atomic-install]] did NOT reproduce.

That was the main reason to be wary of MTP on this family; here the prefill cost is

~4%, comfortably paid for by a 48% decode gain. Net win for interactive/agentic use,

where decode dominates.

The config does NOT declare MTP — the weights do

config.json has no num_nextn_predict_layers / mtp_num_hidden_layers at all,

so metadata alone suggests "MTP" is just a repo-name claim. It is not: the weight

index carries 15 real mtp.* tensors (mtp.fc, a full single-layer block with

attention + MLP + both layernorms, mtp.norm, 2 pre-fc norms). Probe the WEIGHT

INDEX, not the config, to tell a real MTP build from a stub ([[agentworld-vision-stub]]

is the stub case). vLLM copes with the missing field:

getattr(config, "mtp_num_hidden_layers", 1) = 1, matching the single mtp.layers.0.

Serving requirements (vLLM 0.26.0)

• --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'.

qwen3_5_mtp is a REGISTERED method name in vllm/config/speculative.py; a wrong

string fails the load outright, so a successful load is itself evidence MTP wired up.

• --mamba-cache-mode align IS LOAD-BEARING — Qwen3_5MTP.__init__ raises

NotImplementedError on mamba_cache_mode == "all" (qwen3_5_mtp.py:218).

• Draft head is BF16 while the body is INT4; vLLM forces mtp.fc unquantized on

purpose. Mixed precision here is expected, not a packaging error.

• --max-model-len 196608 — the SAME window as qwen38-awq. It was set to 131072

first on the assumption that the BF16 draft head would starve the KV pool; that

assumption was WRONG and cost a context tier for nothing. **Verified 2026-08-24:

needle 3/3 at 182,830 real tokens** (depths 0.1/0.5/0.9, 313-358 s per depth —

cold prefills, not the ~1-2 s warm-cache false pass of

[[needle-warm-cache-invalidates-repeats]]), and the worker survived three

consecutive 183k prefills with no OOM. The head costs ~0.9 GB/card, not a tier.

The general lesson: "it loads" is not evidence a window WORKS

([[gemma4-ft-real-window-and-crashes]] loaded at 262k then OOMed under load), and

a filler prompt answering "OK" only proves ALLOCATION. A needle test proves the

context was processed AND retrievable, at the same memory pressure. Use it.

GRILLED 2026-08-24 — MTP COSTS NO QUALITY

Full 13-suite battery, 1 rep, 212 min. Envelope CLEAN (0 control-token leaks,

0 tool-envelope problems, 0 stream-only leaks across 126 turns).

| suite | qwen38-mtp | field |

|---|---|---|

| coding R1-R4 | 21/23 | IDENTICAL to plain qwen38-awq |

| R5 long-horizon | 6/6, 0 looped, 0% redundant | perfect |

| KLayout unaided | 0/8 | normal ([[klayout-api-recall-vs-composition]]) |

| KLayout +ref | 8/8 | best measured on this harness |

| klvision | 5/6 | > bonsai 4/6 |

| closed loop | 0/4 | normal |

| closed loop +ref | 4/4 | vs qwen3vl-8b 1/4, bonsai 0/4 |

| office unaided | 9/9 | ONLY model to manage this |

| office +ref | 9/9 | ref neither helps nor hurts |

| real-case gate | PASS 7/7 | |

**The headline: 21/23 coding is EXACTLY plain qwen38-awq's score, at ~1.35x the

decode speed.** Speculative decoding is meant to be lossless (the main model

verifies every draft token); this is that claim measured, not assumed. R5 6/6 with

zero looping rules out the degradation-as-looping mode that would be the first

symptom.

Office 9/9 UNAIDED is the standout and it corrects a generalisation. Every other

model here needs the API reference for office (gemma4-ft 6/9, qwen3vl-8b 3/9,

bonsai 0/9), which had suggested the suite measures API RECALL rather than

capability. qwen38-mtp shows a model can simply HAVE the recall.

Cap-hits: 4 total (1 in R1 expr_eval, 3 in unaided KLayout). Per

[[qwen38-ridge-rejected]] the cap-hit COUNT is the leading indicator of a quality

regression, so this is the number to watch on a re-grill. expr_eval also fails on

plain qwen38-awq, so it is not MTP-specific. 5 abnormal finish_reason turns (4.0%).

n=1 — [[round5-is-a-sample-not-a-measurement]]. The perfect scores (8/8, 4/4,

9/9, 7/7) are the ones most likely to move on a repeat.

Speed: quote 38 tok/s, not 41.7

Three measurements of the same model, and the number SHRANK each time the sample

got more realistic:

41.7 tok/s one synthetic 2000-tok generation (n=1)

39.9 tok/s 3 runs of the same prompt at the 192k config

38.1 tok/s MEAN over 45 graded tasks in the full battery (median 39.5,

range 27.1-43.0)

38.1 is the honest figure — it spans short tasks where per-request overhead

weighs more and prefill-heavy ones where MTP gives nothing. A single long clean

generation is the best case, not the average.

Caveat on the RATIO: qwen38-awq's 28.2 tok/s is itself a single bench run, not a

battery mean, so ~1.35x compares an average against a point measurement. The

direction is solid; do not defend the second decimal.

Runtime: 212 min, and why the estimate was wrong

I predicted ~30 min by calibrating on R4 (38.3s, 5 short tasks). Wrong by 7x. R4's

tasks are ~300 tokens; R1's are 7-10k, and KLayout+ref alone took 62 min because

its large API-reference prefills get NO MTP benefit (prefill measured 4% SLOWER).

Calibrate a battery estimate on a PREFILL-HEAVY suite, not the cheapest one —

MTP speeds decode only, so its advantage shrinks exactly where a battery spends

most of its time.