Kept / trial active in llama-swap.yaml
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| No results-*.log found for this id/alias. | ||||
Trialled 2026-08-09, vllm/trial-qwen36-35b-20260809.sh +
grill-qwen36-35b-20260809.sh, 87 min, 24 suite runs, 0 failed. Full 12-suite
vision set, same protocol and jsonl as [[regrill-20260808-four-model-trust]].
QuantTrio/Qwen3.6-35B-A3B-AWQ, 24 GB, arch Qwen3_5MoeForConditionalGeneration.
| suite | combined | field position |
|---|---|---|
| coding R1-R4 | 41/46 | 5th (tied qwen3-coder, qwen3-vision) |
| R5 agentic | 12/12 | tied best |
| klayout from memory | 2/16 | 2nd (fable-fusion 5/16) |
| klayout + API ref | 15/16 | TIED BEST with fable-fusion |
| klvision | 9/12 | 3rd |
| closed loop, NO ref | 4/8 | 2nd non-zero EVER (fable-fusion 6/8) |
| closed loop + ref | 6/8 | 3rd (fable-fusion, qwen3-vl-thinking 8/8) |
| office (non-vision) | 10/12 | 2nd (glm-flash-awq 12/12) |
| office + ref | 12/12 | tied best |
| cap-hits | 9 | 2nd WORST (qwen3-vl-thinking 23) |
| throughput | 88 tok/s | |
**Vision survives INT4 and it is the second model to reach the closed loop
unaided.** modules_to_not_convert keeps visual, linear_attn, self_attn,
shared_expert, mlp.gate, layer 0 and mtp in full precision — only routed
expert MLPs are 4-bit. The loop pass was VERIFIED by re-executing the artifact,
not trusted from the grader: loop_array gives 12 polygons, area 48,000,000
DBU^2, bbox 17.0 x 12.0 um — identical to the hand-checked reference in
[[closed-loop-works-with-api-ref]]. Its one loop failure is a 16k RUNAWAY on
loop_ring, not a wrong answer.
IT DOES NOT REPLACE qwen3-coder. Coding is a tie (41/46 each, R5 12/12
each), but the cost is not close:
qwen3-coder median 323 coding tok max 1171 0 cap-hits 141 tok/s
qwen36-35b median 2824 coding tok max 16000 9 cap-hits 88 tok/s
8.7x the tokens at 62% of the speed. It is a REASONING model; qwen3-coder is an
Instruct model with no think block. Same trade [[qwen3-coder-regrilled]] found
against glm-flash-awq, and the same conclusion.
IT DOES REPLACE qwen3-vision AND qwen3-vl-thinking — both, with one entry.
| | qwen36-35b | qwen3-vl-thinking | qwen3-vision |
|---|---|---|---|
| coding R1-R4 | 41/46 | 35/46 | 41/46 |
| R5 | 12/12 | 7/12 | 10/12 |
| klayout + ref | 15/16 | 12/16 | 12/16 |
| loop no ref | 4/8 | 0/8 | 0/8 |
| loop + ref | 6/8 | 8/8 | 4/8 |
| office (nv) | 10/12 | 2/12 | 0/12 |
| cap-hits | 9 | 23 | 2 |
It loses only loop+ref to qwen3-vl-thinking and raw speed to qwen3-vision.
Against qwen3-vl-thinking specifically it is +6 coding, +5 R5, +3 KLayout and
less than half the cap-hits.
The 9 cap-hits are NOT starvation — 32k diagnostic run 2026-08-10, 63 min
(vllm/diag32k-qwen36-35b.sh; R1/R2 x2 repeats, the four capped domain suites
x1). Verdict across 50 task pairs:
STARVED 0 the budget was NEVER binding on any task
FLAKY 2 capped once, later PASSED using far FEWER than 16k
RUNAWAY 3 burn the whole budget at 16k AND 32k
REGRESSED 3 PASSED at 16k, FAILED at 32k
unchanged 39
More budget makes it WORSE. dbu_rect passed at 1984 tok and blew the full
32000 at the larger budget; loop_array:code passed at 2727 then failed at 3283.
Coding was completely budget-insensitive (R1 6/8 6/8 and R2 4/5 5/5 at BOTH
budgets, 8 runs), but the domain suites regressed. 16k is the correct cap and
the cap-hits are not an argument for more room.
Runaways only occur WITHOUT the API reference. klayout-from-memory went
1/8 -> 0/8 with a 16x token blow-up; klayout+ref was 7/8 at both budgets with
every task inside 1377-3819 tok. The reference removes the runaway behaviour as
well as adding ~13 points — another entry for
[[klayout-api-recall-vs-composition]].
Serving notes (all three cost a restart to find).
--tool-call-parser qwen3_coder, not hermes — it emits Qwen3-Coder XML (<tool_call><function=name><parameter=k>); hermes threw and returned
tool_calls: null with the raw XML in content.
--reasoning-parser qwen3 — the chat template emits <think> by default.10 of 40 layers are full-attention (rest are linear/DeltaNet) with 2 KV heads,
so KV is cheap and the WEIGHTS bind — 12 GB/GPU, the tightest run here.
NOT PROMOTED pending your call; no llama-swap entry yet, weights on the
NVMe. Promoting it means replacing two vLLM entries with one and accepting a
loop+ref regression of 8/8 -> 6/8.
SERVED 2026-08-11 as qwen36-35b on port 9138 — but this is NOT the
promotion described above. Nothing was retired: qwen3-vision and
qwen3-vl-thinking both stay, so the loop+ref 8/8 -> 6/8 trade was never
taken. It is simply reachable now instead of needing a hand-launched trial
script. Weights are on ssk500, not the NVMe — moved by the 2026-08-11
re-tiering ([[nvme-working-set]]), which costs ~+45s on a cold load and is the
right side of the trade at 71 s/task.
"qwen36-35b":
# RECONNECTED 2026-08-11. Trialled 2026-08-09 and left unserved pending a
# promotion call; this entry makes it reachable again WITHOUT retiring
# anything. QuantTrio/Qwen3.6-35B-A3B-AWQ, 24 GB, arch
# Qwen3_5MoeForConditionalGeneration. See auto-memory qwen36-35b-awq-trial.
#
# WHAT IT IS FOR: it is the KLayout+reference co-leader (15/16, tied with
# fable-fusion) and only the SECOND model ever non-zero on the UNAIDED
# closed loop (4/8; fable-fusion 6/8). Coding 41/46 with R5 12/12 and
# office+ref 12/12. On paper it replaces qwen3-vision AND qwen3-vl-thinking
# with one entry — losing only loop+ref (6/8 vs vl-thinking's 8/8) — but
# nothing is retired here, so pick it deliberately.
#
# DO NOT DRIVE GENERAL CODING WITH IT. It ties qwen3-coder's 41/46 at 8.7x
# the tokens (median 2824 vs 323) and 62% of the speed, with 9 cap-hits
# against 0. 71 s/task — a reasoning model where qwen3-coder is Instruct.
#
# 16k IS THE CORRECT OUTPUT BUDGET and the 9 cap-hits are NOT starvation:
# the 32k diagnostic (vllm/diag32k-qwen36-35b.sh, 50 task pairs) found 0
# starved, 3 runaways at BOTH budgets and 3 tasks that REGRESSED from pass
# to fail with more room. More budget makes it worse, not better.
#
# Runaways only happen WITHOUT an API reference — klayout-from-memory blew
# up 16x on tokens while klayout+ref stayed inside 1377-3819 tok at both
# budgets. Load the klayout-classes skill before pointing it at KLayout.
#
# THREE SERVING FACTS, EACH COST A RESTART TO FIND:
# * --tool-call-parser qwen3_coder, NOT hermes. It emits Qwen3-Coder XML
# (<tool_call><function=name><parameter=k>); hermes threw and returned
# tool_calls: null with the raw XML left in content.
# * --reasoning-parser qwen3 — the chat template emits <think> by default.
# * --gpu-memory-utilization 0.92, not the 0.90 used by its neighbours.
# THE WEIGHTS BIND, not the KV: 11.21 GiB/GPU of weights + 1.47 peak
# activation + 0.53 CUDAGraph. Only 10 of 40 layers are full-attention
# (rest are linear/DeltaNet) with 2 KV heads and head_dim 256, so at fp8
# KV costs 5.00 KiB/token PER GPU under TP=2 (2*10*2*256*1 / 2) and
# ~0.62 GiB covers 131072. Dropping util to 0.90 takes ~0.3 GiB out of
# the margin.
#
# CONTEXT 131072 -> 180224 on 2026-08-24. The "~1.0 GiB for KV" figure
# above is CORRECT — vLLM itself reports 0.97 GiB available KV. A 200000
# attempt was made first and REFUSED at startup:
# ValueError: max seq len (200000) needs 0.99 GiB KV cache, larger than
# available KV cache memory (0.97 GiB). estimated max model length 197024.
# DO NOT SIZE THIS FROM IDLE nvidia-smi. Loaded and idle at 131072 the
# cards showed 13.53/13.48 GiB used against a 14.65/14.72 GiB reservation,
# i.e. ~1.13 GiB of apparent slack — but that slack is NOT KV headroom: it
# is activation peak, CUDA graphs and the 16384-token vision encoder
# budget. Reading it as free KV over-predicted the ceiling by ~1.8x.
# KV cost itself is reliable: 5.00 KiB/token/GPU at fp8 under TP=2
# (2*10 full-attn*2 kv-heads*256 head-dim*1 byte / 2), i.e. 0.62 GiB at
# 131072 and a predicted 0.95 at 200000 vs vLLM's actual 0.99 — trust the
# per-token cost, take the BUDGET from vLLM's own error, never from free VRAM.
# 180224 -> ~0.86 GiB of the 0.97 available, leaving ~0.11 GiB. Chosen over
# 197024 (vLLM's bare ceiling) because that ceiling is computed WITHOUT a
# real image in flight, and this is a VISION model whose image buffers
# allocate dynamically ON TOP — exactly how qwen25vl-72b died the same day
# (loaded at ~0.9 GiB spare, then hard-aborted on its FIRST image).
# VERIFIED 2026-08-24 at 180224: vLLM allocated a 196,423-token KV pool
# (concurrency 1.09x); needle 3/3 PASS at 166,012 real prompt tokens,
# depths 0.25/0.5/0.9, ~75 s prefill each. Those passes are GENUINE, not
# warm-cache free rides — this engine runs enable_prefix_caching=False
# (checked in the startup config), which is what
# [[needle-warm-cache-invalidates-repeats]] warns about. A real image was
# then served in 2.8 s with the server still healthy afterwards, so the
# vision buffers coexist with the bigger KV pool. The window is REAL, not
# nominal — the [[qwen36-27b-deep-retrieval-fails]] 0/3@222k cliff is NOT
# reproduced by this 35B at 166k.
# CAVEAT — the window may be NOMINAL: [[qwen36-27b-deep-retrieval-fails]]
# has the 27B sibling at 3/3 @32k but 0/3 @222k. 200000 sits near that
# band, so needle-test at depth before trusting it, and RESTART between
# runs — a warm prefix cache returns in 1-2 s and PASSES WITHOUT READING
# ([[needle-warm-cache-invalidates-repeats]]). Fall back to ~163840 if the
# needle fails: 163k that works beats 200k that lies.
cmd: |
/usr/bin/env
HF_HOME=/home/giovanni/.cache/huggingface
CUDA_DEVICE_ORDER=PCI_BUS_ID
PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
${vllm_bin} serve QuantTrio/Qwen3.6-35B-A3B-AWQ
--served-model-name qwen36-35b
--tensor-parallel-size 2
--host 127.0.0.1 --port 9138
--max-model-len 180224
--max-num-seqs 8
--gpu-memory-utilization 0.92
--kv-cache-dtype fp8
--override-generation-config '{"temperature":0.6}'
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
proxy: http://127.0.0.1:9138