Active active in llama-swap.yaml
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| No results-*.log found for this id/alias. | ||||
prism-ml/Ternary-Bonsai-27B-AWQ-4bit, 18 GB on ssk500, arch
Qwen3_5ForConditionalGeneration (64 layers, full_attention_interval 4 = 16
full-attention layers — same shape as fable-711-gptq and thinkingcap). Served
since 2026-08-11 as bonsai-awq, llama-swap port 9139.
The point of the trial: the box already runs this model's TERNARY GGUF as
bonsai at 1.71bpw on the PrismML llama.cpp fork ([[bonsai-prism-fork]]). The
AWQ is the same weights at 4-bit on stock vLLM, so it looked like a controlled
test of what extreme quantization costs.
| suite | combined | field position |
|---|---|---|
| coding R1-R4 | 41/46 | 5th, tied qwen3-coder / qwen3-vision / qwen36-35b |
| R5 agentic | 12/12 | tied best |
| klayout from memory | 0/16 | tied worst |
| klayout + API ref | 11/16 | last |
| klvision | 10/12 | tied best |
| closed loop, both arms | 0/8 and 0/8 | only model 0 on BOTH |
| office (non-vision) | 6/12 | mid |
| office + ref | 12/12 | tied best |
| spread / cap-hits | 5 / 4 | |
| throughput | 30 tok/s, 167.6 s/task | SLOWEST ON THE BOX |
The 0/8 loop is GENUINE, not the harness artifact. All four loop+ref
tasks were re-run on 2026-08-10 under the fixed prompt that cost five of six
models score ([[loop-prompt-imports-gap]]) — it still failed, on hallucinated
API and DBU errors, including NameError: name 'db' is not defined. Every
other model in that re-run gained.
**It does NOT answer the ternary-vs-4-bit question, and that is the useful
finding.** The ternary sibling's numbers (19/23 coding, 21/21 vision) come from
the 2026-07-19 harness — a different suite, different denominators
([[office-loop-denominator-fix]]) and a different engine. Comparing 41/46 to
19/23 across that gap is exactly the unsound cross-build baseline
[[genesis-hermes-v5]] was rejected for. Settling it needs the ternary re-run on
the current instrument; until then both entries stay so the comparison remains
*runnable*, which is the whole reason bonsai-awq is served at all.
Nothing here is its best column — 41/46 is a five-way tie, its two
tied-best suites (klvision, office+ref) are shared with faster models, and it
is last on klayout+ref. At 168 s/task it is ~25x the driver. Serve it for the
quant comparison, not for work.
Serving is the ordinary Qwen3_5 recipe: TP=2, fp8 KV, --tool-call-parser
qwen3_coder, --reasoning-parser qwen3.
This is now the box's 2nd-longest PROVEN window, behind north-mini (3/3 @
440k) and ahead of gemma-awq (3/3 @ 248k... which is longer — see below) —
and it is the one genuinely good column this model has.
against 3.94 GiB available at utilization 0.97, reporting "estimated maximum
model length is 252448". 245760 is the practical ceiling, ~2.7% under that.
concurrent long request, by design.
control 3/3 at 33,225. Answers land in content.
tokens, 646s) — so the window is reachable the way clients actually reach it,
not only via the backend port. This also narrowed the "llama-swap 400s on
~220k prompts" gotcha in [[qwen36-27b-deep-retrieval-fails]] to llama.cpp
entries only.
--gpu-memory-utilization 0.97, which is real risk on aVISION model: thin headroom is what OOMs a ViT encode on a large image (the
fable-fusion entry documents exactly that). Step the window back down before
sacrificing anything else if that ever bites.
The needle result REFUTES a documented claim —
[[qwen36-27b-deep-retrieval-fails]] recorded the ternary bonsai at 0/3 @222k
and generalized to the whole qwen35 27B lineage. The AWQ build of the SAME model
passes at 224k. Quant AND engine both changed, so the cause is not isolated;
that memory now carries the narrowed version.
Note the ordering caveat: gemma-awq is verified at 247,592 and remains the
long-context pick — it is 6x faster and scores higher on everything. bonsai-awq
being 2nd-longest is not a reason to reach for it at 168 s/task.
"bonsai-awq":
# RECONNECTED 2026-08-11, trialled 2026-08-10.
# prism-ml/Ternary-Bonsai-27B-AWQ-4bit, 18 GB — the smallest competitive
# footprint on the box.
#
# NOT A DUPLICATE OF THE `bonsai` ENTRY ABOVE. Same weights, different
# quantization and a different ENGINE: `bonsai` is the author's ternary
# 1.71bpw GGUF and runs ONLY on ${prism_llama_bin} (the PrismML fork);
# this is their standard AWQ 4-bit release on stock vLLM. Keeping both is
# the point — the pair is the box's controlled quant experiment, and the
# answer was that 4-bit did NOT repair what ternary cost (see below).
#
# WHAT THE 4-BIT BOUGHT: coding 41/46, mid-field, and vision 10/12. What it
# did NOT buy is KLayout — 0/16 from memory, 11/16 with the API reference,
# and 0/8 on BOTH closed-loop arms. That 0/8 is GENUINE, not the missing-
# imports harness gap that understated everyone else's loop scores: it was
# re-run under the fixed prompt on 2026-08-10 and still failed, on
# hallucinated API and DBU errors (auto-memory loop-prompt-imports-gap).
#
# IT IS THE SLOWEST MODEL ON THE BOX: 168 s/task at 30 tok/s, ~25x the
# driver. Nothing here is its best column, so this entry exists to keep the
# ternary-vs-4-bit comparison runnable, not to be picked for work.
#
# Same Qwen3_5ForConditionalGeneration hybrid as fable-711-gptq and
# thinkingcap: 64 layers at full_attention_interval 4 = 16 full-attention
# layers, which is what makes KV expensive here.
#
# 245760, NOT the native 262144 — RAISED from 131072 on 2026-08-11.
# 262144 DOES NOT FIT and the engine says so outright: it wants 4.09 GiB of
# KV against 3.94 GiB available and refuses to start, reporting "estimated
# maximum model length is 252448". That is already at utilization 0.97;
# no setting on this box reaches 262144 with fp8 KV. 245760 sits ~2.7%
# under the engine's own estimate rather than on the edge, and loads with
# 251,903 tokens of KV (1.02x concurrency at the full window).
#
# THE 0.97 UTILIZATION IS THE PRICE OF THE WINDOW, and it is not free: this
# is a VISION model, and thin headroom is exactly what OOMs a ViT encode on
# a large image — the fable-fusion entry documents that failure mode at the
# same margin. If a big image ever kills the encoder here, step
# --max-model-len back down (196608 / 131072) before touching anything
# else; the window is worth less than the vision tower.
#
# RETRIEVAL VERIFIED, not merely configured: needle 3/3 at 224,293 REAL
# tokens (depths 0.1/0.5/0.9, ~420s per probe) with a 3/3 control at
# 33,225. This MATTERS because the TERNARY GGUF of this same model — the
# `bonsai` entry above — scores 0/3 at 222k, and
# qwen36-27b-deep-retrieval-fails generalized that to the whole qwen35 27B
# lineage. THAT GENERALIZATION DOES NOT HOLD FOR THIS BUILD. Two things
# differ from the failing run (AWQ 4-bit vs Q2_0 ternary, and vLLM+fp8 KV
# vs llama.cpp+q8_0 KV), so the CAUSE is not isolated — only the outcome.
# Tool/reasoning parsers as for every Qwen3_5 entry: qwen3_coder + qwen3.
cmd: |
/usr/bin/env
HF_HOME=/home/giovanni/.cache/huggingface
CUDA_DEVICE_ORDER=PCI_BUS_ID
PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
${vllm_bin} serve prism-ml/Ternary-Bonsai-27B-AWQ-4bit
--served-model-name bonsai-awq
--tensor-parallel-size 2
--host 127.0.0.1 --port 9139
--max-model-len 245760
--max-num-seqs 8
--gpu-memory-utilization 0.97
--kv-cache-dtype fp8
--override-generation-config '{"temperature":0.7}'
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
proxy: http://127.0.0.1:9139