bonsai

bonsai-vs-qwen3vl8b-same-harness (bonsai-vs-qwen3vl8b-same-harness.md)

Both grilled 2026-08-23 on the CURRENT harness — the first apples-to-apples

comparison. Prior numbers were NOT comparable: ternary bonsai was scored on the

2026-07-19 suite and bonsai-awq on the 2026-08-10 one, and denominators changed

between both and today ([[office-loop-denominator-fix]]).

| suite | bonsai (27B ternary Q2_0) | qwen3vl-8b (8B Q4_K_M) |

|---|---|---|

| coding R1-R4 | 21/23 | 21/23 |

| R5 long-horizon | 5/6 | 4/6 |

| R5 looping | 0/6 | 0/6 |

| KLayout unaided | 0/8 | 1/8 |

| KLayout +ref | 3/8 | 3/8 |

| klvision | 4/6 | 5/6 |

| closed loop | 0/4 | 0/4 |

| closed loop +ref | 0/4 | 1/4 |

| office unaided | 0/9 | 3/9 |

| office +ref | 9/9 | 9/9 |

| real-case gate | PASS 7/7 | FAIL 0/7 |

| envelope | CLEAN (3 abnormal finish, 2.6%) | CLEAN (2 abnormal, 1.7%) |

| runtime, 1 rep | 153 min | 39 min |

| VRAM | 15.5 GB | 11.9 GB |

| tok/step (R4) | 160 | 50 |

bonsai's vision reputation does NOT survive re-measurement

[[bonsai-prism-fork]] records "VISION 21/21 PERFECT". On today's suite it scores

4/6 — BELOW the 8B's 5/6. The old figure came from a 21-task vision suite that

no longer exists in that form (klvision is 6 tasks now), so "21/21 vs 5/6" was

never a like-for-like claim. An informal colour probe was ALSO run and is RETRACTED: it looked like bonsai

erred ("Orange") where qwen3vl-8b said "Pink", but the "ground truth" came from

crude RGB thresholds that misfiled warm reds as pink. Proper HSV hue binning on

pictures/DADa.png gives red 44.9% / orange 36.6% / pink 8.6% of chromatic pixels

— so "Orange" was defensible and "Pink" was the weaker answer. gemma-awq also

answers "Orange". THE KLVISION 4/6 vs 5/6 RESULT IS UNAFFECTED (graded suite,

not this probe), but do not cite the colour probe as evidence about either model.

Do not reach for bonsai on the strength of the 21/21 number.

The one decisive gap: real-case

qwen3vl-8b fails at stage 1 (SPEC_PARSE), which gates all 7 downstream checks, so

0/7 is ONE early failure, not seven. The cause is a deterministic bug, not

variance: it wrote row[0] on a pandas Series (label lookup -> KeyError: 0)

where row.iloc[0] was needed. It will recur. bonsai passes 7/7.

What is IDENTICAL, and why that matters

Same coding score, same KLayout+ref, same 0/4 closed loop, same 0/9 -> 9/9 office

jump. A 27B at 1.71bpw and an 8B at Q4 landing on the same numbers says these

suites measure API RECALL, not capacity ([[klayout-api-recall-vs-composition]]).

The closed-loop 0 now reproduces across TWO quantizations (ternary + AWQ) and two

harness generations ([[bonsai-awq-trial]] scored 0/8 on both arms), so it is

structural to the family, not a build artifact.

Verdict

• bonsai for work that must actually RUN end-to-end (real-case 7/7). Costs

4x the wall-clock and 3.2x the tokens per step.

• qwen3vl-8b as the cheap co-resident vision model on the 4060 Ti — better

vision, 3.6 GB smaller, 3.2x leaner output. NOT a substitute on real tasks.

n=1 for both — [[round5-is-a-sample-not-a-measurement]]. The real-case failure

is a deterministic bug rather than variance, so a re-run is unlikely to rescue it,

but the closer suites (klvision 4/6 vs 5/6) could move.

[[bonsai-prism-fork]] | [[bonsai-awq-trial]] | [[klayout-api-recall-vs-composition]]

Lesson: a self-written "ground truth" is a measurement too

The colour probe above was scored against a check *I wrote in the same session*,

and it was wrong — crude RGB thresholds filed warm reds as "pink", so the model

that answered correctly was marked wrong. pictures/DADa.png is a PowerPoint

screenshot whose main slide is a MATTERHORN SUNSET; orange/red is obviously

dominant to a human eye, which is what all three models said. The verifier was

the only thing that disagreed.

Verify the checker before believing a result that contradicts several models at

once — model consensus against a hand-rolled oracle is evidence about the ORACLE.

Related in kind to [[eval-false-pass-adjacent-data]], where the harness scored the

wrong artifact: both are the grader failing, not the model.

For descriptive vision, DADa.png is a good probe precisely BECAUSE it is a busy

screenshot: gemma-awq described the ribbon state, the 6-slide pane, the selected

slide's red border, "Slide 6 of 6", and the slide title correctly — all checkable.

bonsai-prism-fork (bonsai-prism-fork.md)

Bonsai-27B on the PrismML fork (2026-07-19)

prism-ml/Ternary-Bonsai-27B (Q2_0 g128, 7.17GB — true 1.71 bits/weight

ternary incl. embeddings/LM head; Qwen3.6-27B hybrid-attention backbone ~75%

linear, real vision tower — a qwen3vl_merger CLIP projector, 461M params).

Runs ONLY on the PrismML fork build at

~/coolbox/prism-llama.cpp (github PrismML-Eng/llama.cpp branch "prism",

built like the main llama.cpp: CUDA 13, -DCMAKE_CUDA_COMPILER=

/usr/local/cuda/bin/nvcc REQUIRED — nvcc isn't on PATH — arch "89;120").

Wired into the MAIN llama-swap.yaml via a second macro prism_llama_bin

(no parallel swap instance; a prism.coolboxapp.com subdomain was considered

and dropped — everything serves through llama.coolboxapp.com).

Gotchas (all cost time once):

• -hf repo:Q2_0 tag-matches the repo's PQ2_0 file (substring hit) — a

future format nothing supports — and fails to load. Pin -hff

Ternary-Bonsai-27B-Q2_0.gguf. (Q2_g64 = mainline format, fork rejects it.)

• Embedded chat template has the non-first-system raise_exception guard

([[jinja-system-guard-tool-parser]]) → tools requests 400. Patched copy:

chat-templates/bonsai-256k.jinja, verified tool_calls end-to-end.

• pkill -f "port NNNN" kills the invoking shell too (pattern matches its

own cmdline) — use pkill -x llama-server.

Measured: full native 262k ctx at q8/q8 KV = ~10.1/10.8GB per GPU

(~5.5GB free each; linear attention → cheap KV). ~35-37 t/s decode (dense

27B). Grill (temp 0.7): 19/23 coding (r1 6/8: fails expr_eval+lru_ttl;

r2 3/5: fails weighted_interval_scheduling+text_justify, same pair as

glm-flash; r3 5/5; r4 5/5 agentic, 172 tok/step) + VISION 21/21 PERFECT

(only model besides vision-coder to sweep; the only one to pass both hard

tasks that flunked kimi-vl 0/2).

bonsai-duo entry (port 9201, --parallel 2): two slots × 131k ctx, same

VRAM; verified 2 concurrent 600-tok generations in 19.2s wall = half the

sequential 38.4s (~64 t/s aggregate). Use for two-agent workloads.

mmproj / vision (2026-07-21): the repo ships TWO vision projectors —

Ternary-Bonsai-27B-mmproj-BF16.gguf (461M params, full BF16, 0.86GB) and a

-Q8_0 variant (0.58GB). An earlier llama-swap comment called the tower

"HQQ-4bit" — WRONG; GGUF metadata confirms genuine BF16 (general.file_type

32, clip.projector_type=qwen3vl_merger). llama-swap now PINS the BF16 file

via an explicit --mmproj <abs HF-cache snapshot path> on both bonsai and

bonsai-duo (quality-first policy). Why absolute path: --mmproj takes a

literal FILE only — no repo-relative filename selector — and the -hf

auto-pick is quant-distance based (find_best_sibling), NOT guaranteed to

choose BF16 when a repo has multiple mmproj. Path is tied to revision abbae72.

**DSpark spec-decode — WORKS but only at LOW ctx; NOT wired to 262k bonsai

(load-tested 2026-07-21):** history: first wrongly called REJECTED (trusted stale

in-repo docs), then wired to bonsai/bonsai-duo, then REVERTED after load-testing

exposed a hard 262k incompatibility. Facts:

• Trained drafter EXISTS in the SAME repo prism-ml/Ternary-Bonsai-27B-gguf:

Ternary-Bonsai-27B-dspark-Q4_1.gguf (1.95GB, DEFAULT) + -dspark-bf16.gguf.

DOWNLOADED to cache at rev abbae72 (repo main == abbae72, so it dropped into the

same snapshot; refs/main unchanged). GGUF: arch dspark, block_size=4 (key

dspark.dspark.block_size), taps [1,16,31,46,61], markov_rank 256, conf head.

• The installed llama-server (fork 9fcaed7) DOES serve it — server-context.cpp

reads the drafter's target_layers/block_size and engages llama_set_capture_layers.

The speculative.cpp:44 comment + docs/dspark-scope.md ("deferred") are STALE.

• HARD BLOCKER at 262k: dspark "full-context staging" sets the DRAFT ctx

n_batch = full context. At c=262144 → tries to alloc ~547GB → CUDA OOM → the

draft ctx fails → params.draft.ctx_dft==nullptr → dspark SILENTLY disables

("common_speculative_init: no implementations specified"). So dspark is

fundamentally incompatible with the 262k window on 16GB cards.

• At c=16384 it WORKS (needs -fit off — the fork's default -fit on auto-fit

hangs at "fitting params to device memory" with a draft model): "target tap

capture engaged on 5 layers", ~49.5% draft accept, 37.8→54.6 t/s = 1.44x

decode (beats the README's H100 1.34x). Output coherent (bonsai is a thinking

model → text lands in reasoning_content, content empty till </think>).

• DECISION: kept bonsai + bonsai-duo at full 262k, dspark OFF (quality/long-ctx

first, per box policy). ADDED a separate bonsai-fast entry (alias

ternary-bonsai-27b-dspark-16k, port 9202, models.json contextWindow 16384) that

IS the dspark variant: -md <Q4_1> --spec-type draft-dspark --spec-draft-n-max 4

-ngld 99 -fit off at -c 16384. --spec-draft-n-max MUST == block_size(4) or the

server errors. Use bonsai/bonsai-duo for long context, bonsai-fast for fast

short-context. Renamed from bonsai-spec → bonsai-fast at user request.

SYNCED to all four consumers (2026-07-21): models.json, statusline-command.sh

(alias + short name, ordered before *bonsai* so it isn't shadowed), the Klayout

plain copy, and OpenCode ~/.config/opencode/opencode.json (attachment:true).

NOT yet load-tested through the live systemd service (it was down during this

work); verified via standalone + throwaway llama-swap only. Needs a

sudo systemctl restart llama-server to go live.

How to apply: best vision-per-GB on the box and a competent agentic

coder; gpt-oss-20b still rules pure coding ([[gpt-oss-20b]]), vision-coder

rules vision+coding combined ([[vision-coder-heavy-grill]]). See also

[[kimi-vl]], [[kimi-distill]].

2026-08-23 — the 21/21 VISION FIGURE IS SUPERSEDED. Re-grilled on the

current harness it scores 4/6 on klvision, BELOW qwen3vl-8b (an 8B) at 5/6.

The old number came from a 21-task suite that no longer exists, so it was never

like-for-like. Real-case 7/7 is where bonsai still genuinely wins.

See [[bonsai-vs-qwen3vl8b-same-harness]].

bonsai-awq-trial (bonsai-awq-trial.md)

prism-ml/Ternary-Bonsai-27B-AWQ-4bit, 18 GB on ssk500, arch

Qwen3_5ForConditionalGeneration (64 layers, full_attention_interval 4 = 16

full-attention layers — same shape as fable-711-gptq and thinkingcap). Served

since 2026-08-11 as bonsai-awq, llama-swap port 9139.

The point of the trial: the box already runs this model's TERNARY GGUF as

bonsai at 1.71bpw on the PrismML llama.cpp fork ([[bonsai-prism-fork]]). The

AWQ is the same weights at 4-bit on stock vLLM, so it looked like a controlled

test of what extreme quantization costs.

| suite | combined | field position |

|---|---|---|

| coding R1-R4 | 41/46 | 5th, tied qwen3-coder / qwen3-vision / qwen36-35b |

| R5 agentic | 12/12 | tied best |

| klayout from memory | 0/16 | tied worst |

| klayout + API ref | 11/16 | last |

| klvision | 10/12 | tied best |

| closed loop, both arms | 0/8 and 0/8 | only model 0 on BOTH |

| office (non-vision) | 6/12 | mid |

| office + ref | 12/12 | tied best |

| spread / cap-hits | 5 / 4 | |

| throughput | 30 tok/s, 167.6 s/task | SLOWEST ON THE BOX |

The 0/8 loop is GENUINE, not the harness artifact. All four loop+ref

tasks were re-run on 2026-08-10 under the fixed prompt that cost five of six

models score ([[loop-prompt-imports-gap]]) — it still failed, on hallucinated

API and DBU errors, including NameError: name 'db' is not defined. Every

other model in that re-run gained.

**It does NOT answer the ternary-vs-4-bit question, and that is the useful

finding.** The ternary sibling's numbers (19/23 coding, 21/21 vision) come from

the 2026-07-19 harness — a different suite, different denominators

([[office-loop-denominator-fix]]) and a different engine. Comparing 41/46 to

19/23 across that gap is exactly the unsound cross-build baseline

[[genesis-hermes-v5]] was rejected for. Settling it needs the ternary re-run on

the current instrument; until then both entries stay so the comparison remains

*runnable*, which is the whole reason bonsai-awq is served at all.

Nothing here is its best column — 41/46 is a five-way tie, its two

tied-best suites (klvision, office+ref) are shared with faster models, and it

is last on klayout+ref. At 168 s/task it is ~25x the driver. Serve it for the

quant comparison, not for work.

Serving is the ordinary Qwen3_5 recipe: TP=2, fp8 KV, --tool-call-parser

qwen3_coder, --reasoning-parser qwen3.

2026-08-11: raised to 245760 and needle-VERIFIED at 224k

This is now the box's 2nd-longest PROVEN window, behind north-mini (3/3 @

440k) and ahead of gemma-awq (3/3 @ 248k... which is longer — see below) —

and it is the one genuinely good column this model has.

• 262144 is IMPOSSIBLE here. The engine refuses: 4.09 GiB of KV needed

against 3.94 GiB available at utilization 0.97, reporting "estimated maximum

model length is 252448". 245760 is the practical ceiling, ~2.7% under that.

• Loads with 251,903 tokens of KV, 1.02x concurrency — no room for a second

concurrent long request, by design.

• needle 3/3 at 224,293 real tokens, depths 0.1/0.5/0.9, ~420s per probe;

control 3/3 at 33,225. Answers land in content.

• Re-verified THROUGH the llama-swap proxy (:8090, depth 0.5, same 224,293

tokens, 646s) — so the window is reachable the way clients actually reach it,

not only via the backend port. This also narrowed the "llama-swap 400s on

~220k prompts" gotcha in [[qwen36-27b-deep-retrieval-fails]] to llama.cpp

entries only.

• The window costs --gpu-memory-utilization 0.97, which is real risk on a

VISION model: thin headroom is what OOMs a ViT encode on a large image (the

fable-fusion entry documents exactly that). Step the window back down before

sacrificing anything else if that ever bites.

The needle result REFUTES a documented claim —

[[qwen36-27b-deep-retrieval-fails]] recorded the ternary bonsai at 0/3 @222k

and generalized to the whole qwen35 27B lineage. The AWQ build of the SAME model

passes at 224k. Quant AND engine both changed, so the cause is not isolated;

that memory now carries the narrowed version.

Note the ordering caveat: gemma-awq is verified at 247,592 and remains the

long-context pick — it is 6x faster and scores higher on everything. bonsai-awq

being 2nd-longest is not a reason to reach for it at 168 s/task.

bonsai-fast-dspark-removed (bonsai-fast-dspark-removed.md)

Update 2026-07-27, later same day: bonsai-fast itself was subsequently

removed from llama-swap.yaml per waht_to_remove.md/[[2026-07-27-model-cleanup]]

— unrelated to this fix (disk-neutral, since it shared bonsai's GGUF blob).

The removal rationale cited was actually this fix's *pre-fix* 1/23 score,

already stale by the time it was acted on. The dspark diagnosis below remains

the load-bearing part of this memory: it applies to any future dspark-enabled

entry, not just bonsai-fast specifically.

bonsai-fast (llama-swap.yaml, ternary-bonsai-27b-dspark-16k) scored 1/23 on

the grill (see [[grill-round5-agentic-loops]]). Root-caused via direct testing

(bypassing llama-swap, isolating variables one at a time) to the **dspark

speculative-decode drafter** (Ternary-Bonsai-27B-dspark-Q4_1.gguf), not the

base model, template, or context size:

• A trivial prompt (add(a,b)) produced valid code in reasoning_content in

under a second — the weights and template are fine.

• Grill-style (code-generation) prompts truncated to just the model's first

reasoning sentence ~40-50% of the time, with an **identical signature every

time** (13 completion tokens, same 69-char reasoning fragment) — too

consistent to be sampling noise, and far too fast (0.9-1.3s) to be the

16384-token context cap (dspark's full-context draft staging is why this

entry is capped at 16k in the first place — see the cmd comment).

• Isolated dspark as the variable directly: 5/5 clean runs with the drafter

removed vs 2/3 early-stops with it present, everything else byte-identical.

• Tried --spec-draft-p-min 0.00→0.6 (the accept-probability floor), hoping to

filter a spurious low-confidence draft token (e.g. a premature EOS) — did

not fix it (still 4/8 early stops). The identical-signature repeat points

at a stateful bug in dspark's "target tap capture" mechanism (residual

activations leaking across requests on the long-lived slot,

n_slots=1 --parallel 1), not a tunable sampling threshold — not fixable

from the serving side with flags currently exposed.

Fix (2026-07-27): removed -md, --spec-type draft-dspark,

--spec-draft-n-max 4, -ngld 99, -fit off from the bonsai-fast entry in

llama-swap.yaml; restarted llama-server.service to pick it up. Verified 6/6

clean requests post-fix (was ~50% failure), then re-grilled: 21/23, R5 5/6, 0

loops, 36.4 t/s decode — matching vanilla bonsai's 36.3 t/s almost exactly,

confirming it's the same weights minus the (broken) speedup. dspark's own

measured benefit (2026-07-21 note, now removed from the yaml) was only 1.44x

(37.8→54.6 t/s) and came with its own gotchas already (spec-draft-n-max must

equal the drafter's block_size; -fit off required or the fork's auto-fit

hangs; ctx_shift auto-disables under capture) — not worth a ~50% request

failure rate.

Why: discovered via the bench grill sweep; user then asked to fix the

serving config specifically (not just document the bug).

How to apply: if dspark is ever reconsidered for bonsai-fast (e.g. a

prism-fork update that fixes the tap-capture statefulness), re-verify with the

same repeated-identical-request stress test (5-8x, watch for a consistent

early-stop signature) before trusting a "faster t/s" measurement alone — a

single successful load-test run does not catch a ~50% intermittent failure

rate. Do not re-add dspark to this entry without that verification.