Kept / trial active in llama-swap.yaml · aliases: qwen3.8-27b-awq-196k
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| No results-*.log found for this id/alias. | ||||
qwen38-awq serves cyankiwi/Qwen3.8-27B-AWQ-INT4 on vLLM (TP=2, 196608 ctx,
fp8 KV, --tool-call-parser qwen3_coder + --reasoning-parser qwen3, util 0.97,
--max-num-seqs 8). Name says AWQ but quant_method is **compressed-tensors
INT4 W4A16** (group 32, asymmetric, mse observer) — vLLM auto-detects. Quant is
sound: all 48 Gated-DeltaNet (linear-attention) layers + lm_head + vision tower
+ MTP head kept BF16; only the 16 full-attention layers + MLPs are int4. Same
recipe as [[thinkingcap-awq-trial]] (cyankiwi's Qwen3.6-27B AWQ-INT4).
NOT the same weights as the llama.cpp qwen38. cyankiwi quanted the BASE
Qwen/Qwen3.8-27B (confirmed from config.json base_model_name); the llama.cpp
qwen38 serves the UNCENSORED fine-tune of the same base — see
[[qwen38-uncensored-mtp-probe]]. Same arch (qwen3_5, 64 layers / 16 full-attn),
different post-training. So this is the base model on vLLM, that is the
uncensored variant on llama.cpp.
Grill: 3-rep full battery (bench/grill-qwen38-awq-full.sh, GRILL_MAX_TOKENS=32000,
temp 0.2, ~28 tok/s, 696 min). DRIVEN THROUGH llama-swap :8090 — do NOT HUP or
swap models mid-battery (tears down the loaded model → empty response →
JSONDecodeError that fakes a failure; same trap as the 2026-08-17 qwen38 run).
| suite | rep1 | rep2 | rep3 | read |
|---|---|---|---|---|
| r1 (8 coding) | 5/8 | 6/8 | 5/8 | 2 STABLE runaways: expr_eval + wildcard_match hit 32000 cap every rep. articulation_points 2/3, median_two_sorted 2/3 = variance |
| r2 (5 algo) | 4/5 | 5/5 | 5/5 | text_justify one-off (rep1) |
| r3 (5 tool/format) | 5/5 | 5/5 | 5/5 | qwen3_coder parser clean; restraint test correctly declined a tool call |
| r4 (5 agentic loop) | 5/5 | 5/5 | 5/5 | lean: 1183-1206 tok, 9 steps, ~49s wall, 131-134 tok/step, 0 redundant |
| r5 (6 long-horizon) | 6/6 | 6/6 | 5/6 | looped=0/6 ALL 3 reps, redundant=0.0% — ZERO looping |
| klayout (unaided) | 0/8 | 0/8 | 0/8 | expected baseline — [[klayout-api-recall-vs-composition]] |
| klayout-ref (API ref) | 7/8 | 7/8 | 7/8 | stable top-tier, ties [[fable711-gptq-int4-vllm-trial]] |
| office | 9/9 | 9/9 | 9/9 | struct+pdf+vision clean |
| office-ref | 9/9 | 9/9 | 9/9 | same (already maxed) |
| realcase gate | 7/7 PASS | 7/7 PASS | 0/7 RUNAWAY | 2/3 — rep3 blew past 32k cap, emitted unterminated string literal → 0/7. [[realcase-spec-plus-apiref-multiplicative]] |
Verdict: accurate + clean-agentic, but SLOW and runaway-prone. The standout
is zero R5 looping across 3 reps — the long-horizon round where many
candidates here have looped (coder-prune's multifile_refactor looped in all 8
runs; the rejected [[qwen38-ridge-rejected]] and gemma-4-26b-a4b had genuine R5
loops). KLayout+ref 7/8 ties the best, office a clean sweep, real-case gate
capable (7/7 twice). Decode is ~28 tok/s — dense-27B class, same as
fable-711-gptq and thinkingcap, NOT a speed model (vs the MoE 30B-A3B entries
at ~90-140 tok/s). The weakness is the runaway tendency: 2 stable cap-hits
in r1 (expr_eval, wildcard_match, every rep) plus the realcase gate 1/3 of the
time. Cap-hit COUNT is the leading indicator — see [[qwen38-ridge-rejected]] —
and 2/repeat is moderate (thinkingcap had 0; ridge was rejected at 3/repeat).
Reasoning note: this IS a thinking model. It emits a think block that the
qwen3 parser splits into the reasoning field — but vLLM 0.26.0 exposes that
field under the key reasoning, NOT reasoning_content. Checking
reasoning_content (the OpenAIClients convention / older vLLM key) falsely
shows empty and led to a wrong "no think tags" call on the first load-test.
content holds the final formatted answer; reasoning holds the internal
working. The parser works. (A thinking model also fits the runaway tendency
above — thinking models burn budget, see [[budget-sensitivity-is-a-model-property]].)
Role: the concurrent vLLM counterpart to the llama.cpp qwen38. That entry
has 262k context + MTP (~30 tok/s) but is single-stream; this one trades 32k
less context (196k vs 262k) and MTP for 8-way concurrency + real tool/reasoning
parsers. Keep both — different runtimes, different fine-tunes (base vs
uncensored), different workloads. Overlaps heavily with fable-711-gptq (both
vLLM dense 27B, ~28 tok/s, vision, KLayout 7/8); qwen38-awq's edge is the cleaner
R5 (0 looping) and the 2/3 real-case gate.
"qwen38-awq":
aliases: [qwen3.8-27b-awq-196k]
# vLLM COUNTERPART TO `qwen38`. NOT the same weights: cyankiwi's quant is
# of the BASE Qwen/Qwen3.8-27B (base_model_name in config.json), whereas
# the llama.cpp `qwen38` serves the UNCENSORED fine-tune of the same base.
# So this is the base model on vLLM, that is the uncensored variant on
# llama.cpp -- same arch, different post-training. The gap this fills:
# Qwen3.8-27B had NO vLLM entry (only llama.cpp), so this adds 16-way
# concurrency, fp8 KV and real tool/reasoning parsers. GRILLED 2026-08-19/20
# (3 reps): accurate + ZERO R5 looping but SLOW (~28 tok/s) with 2 stable
# runaways (expr_eval, wildcard_match) and a 1/3 realcase runaway -- see
# auto-memory/qwen38-awq-vllm-trial.md.
#
# REPO NAME IS MISLEADING: quant_method is compressed-tensors (INT4,
# group_size 32, asymmetric, mse observer) -- W4A16, not classic AWQ.
# vLLM auto-detects from config.json; no --quantization flag needed.
# QUANT IS SOUND, verified 2026-08-19 from config.json: all 48 Gated
# DeltaNet (linear-attention) layers are IGNORED and kept BF16, plus
# lm_head, the vision tower and the MTP head; only the 16 full-attention
# layers + their MLPs are int4. Same recipe as `thinkingcap`
# (cyankiwi's Qwen3.6-27B AWQ-INT4, the box's best model) -- NOT the
# broken avyukth build that RTN'd the DeltaNet layers.
# ARCH qwen3_5 (Qwen3_5ForConditionalGeneration): 64 layers, 16
# full-attention / 48 linear-attention, 4 kv_heads, head_dim 256,
# 262144 native. Vision + MTP present; MTP is NOT enabled on vLLM here
# (the head is kept BF16 but unused, like nemotron-lightning's blk.52).
# CONTEXT 196608, NOT 262144. The 2026-08-16 standalone trial measured at
# util 0.97 that 262144 wants 4.09 GiB KV against 3.18 GiB free -- the
# weights are 19.57 GiB, ~0.76 GiB/card heavier than bonsai's 18, so
# vLLM's estimated max is 203840 (vs bonsai's 252448). 196608 sits ~3.5%
# under that. 262k is unreachable on 16 GB cards for this heavier 27B;
# if you ever want it, a smaller quant is the only lever, not util
# (already 0.97). Step --max-model-len down (131072) before touching the
# quant if a big image OOMs the vision encoder -- this is a vision model
# and 0.97 leaves thin headroom (see the bonsai-awq note on the same risk).
# PARSERS: --tool-call-parser qwen3_coder -- the chat template emits the
# XML function/parameter dialect (verified from chat_template.jinja). The
# 2026-08-16 trial used `hermes` and was NEVER grilled, so the error was
# never caught; qwen3_coder matches every other qwen3_5 chat entry here
# (qwen36-35b, bonsai-awq, fable-711-gptq). --reasoning-parser qwen3
# splits the think block into reasoning_content.
# SAMPLER: temp 0.6 = the Qwen3.8 thinking preset. Shipped
# generation_config.json says 1.0; the card's thinking mode is 0.6,
# matching the `qwen38` llama.cpp entry. top_p/top_k come from the shipped
# config (0.95/20) and are not restated (box convention).
cmd: |
/usr/bin/env
HF_HOME=/home/giovanni/.cache/huggingface
CUDA_DEVICE_ORDER=PCI_BUS_ID
PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
${vllm_bin} serve cyankiwi/Qwen3.8-27B-AWQ-INT4
--served-model-name qwen38-awq
--tensor-parallel-size 2
--host 127.0.0.1 --port 9141
--max-model-len 196608
--max-num-seqs 8
--gpu-memory-utilization 0.97
--kv-cache-dtype fp8
--override-generation-config '{"temperature":0.6}'
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
proxy: http://127.0.0.1:9141