Active active in llama-swap.yaml · aliases: qwen3.6-27b-fable-711-256k
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| 20260907-094706 | 12/20 | — | — | results-fable-fusion-kv-q8-rep3-20260907-094706.log |
| 20260907-090727 | 25/40 | — | — | results-fable-fusion-kv-q8-20260907-090727.log |
| 20260812-081242 | 57/71 | 122.7 | — | results-fable-fusion-ff3-20260812-081242.log |
| 20260812-072957 | 63/71 | 142.2 | — | results-fable-fusion-ff2-20260812-072957.log |
| 20260812-065044 | 60/71 | 135.0 | — | results-fable-fusion-ff1-20260812-065044.log |
| 20260727-130406 | 22/23 | 138.8 | — | results-fable-fusion-20260727-130406.log |
Run 2026-08-09, vllm/grill-fable-fusion-20260809.sh, 73 min, 12 suites x 2
repeats, 0 failed. Same protocol and jsonl as
[[regrill-20260808-four-model-trust]], driven through llama-swap on :8090.
NOT the same model as fable-711-gptq. This is the llama.cpp Q4_K_M +
mmproj on the NVMe (262k ctx, MTP speculative decode, 35s load); that one is the
vLLM GPTQ-INT4 on ssk500. Same base weights, different engine and quant. Their
scores are NOT interchangeable — see the loop rows below.
| suite | combined | field position |
|---|---|---|
| coding R1-R5 | 55/58 | 2nd (fable-711-gptq 56/58) |
| klayout from memory | 5/16 | BEST (others 0-2/16) |
| klayout + API ref | 15/16 | BEST (fable-711-gptq 14/16) |
| klvision ground truth | 10/12 | tied best |
| closed loop, NO ref | 6/8 | ONLY non-zero — everything else 0/8 |
| closed loop + ref | 8/8 | tied best with qwen3-vl-thinking |
| office | 15/18 (10/12 non-vision) | 2nd, behind glm-flash-awq 12/12 |
| office + ref | 18/18 | tied best |
| cap-hits | 0 across all 24 runs | |
| throughput | 33 tok/s | the cost |
THE FINDING: it closes the loop unaided. [[closed-loop-never-reaches-vision]]
established that stage 1 is KLayout-from-memory, that every model scores ~0
there, and that the vision stage therefore never runs — 0/4 for
fable-711-gptq, qwen3-vision AND qwen3-vl-thinking (all 0/8 after the
2026-08-09 denominator fix, [[office-loop-denominator-fix]]). fable-fusion
scores 3/4 on the code stage and 3/4 on the vision stage that follows. It writes
correct KLayout geometry from memory, renders it, and reads its own render back.
That falsifies the generalisation, not the diagnosis. "The loop is gated
behind KLayout recall" remains true. "No model can reach it" was drawn from five
models, none of which was this one, and is now wrong. Recorded in
[[closed-loop-works-with-api-ref]] as well.
**Its 23/23 record was slightly optimistic and its 21/21 vision does not
transfer.** Real coding is a very steady 22/23 on R1-R4, twice, with ZERO task
churn — only R5's search_navigate flipped. expr_eval fails for every model on
the box and is not a discriminator. QUALIFIED 2026-08-10: thinkingcap passes it, 1 of 2 runs, 9109 tok / 324s — the only model that ever has. See [[thinkingcap-awq-trial]]. The old 21/21 came from grill_vision.py
(21 tasks), a different suite; on klvision it is 10/12, tied not exceptional.
Practical upshot: fable-fusion is the KLayout model. It is the only one that
does not need the klayout-classes skill loaded to be useful — though the
reference still helps (5/16 -> 15/16). The price is 33 tok/s, a quarter of
qwen3-vl-thinking's 136, because it is a DENSE 27B; MTP speculative decode buys
33 vs the GPTQ sibling's 28 but does not change the class.
Keep both fable variants. The GGUF (this one) for KLayout and unaided loop
work; [[fable711-gptq-int4-vllm-trial]] for its spread-0 coding. They are not
substitutes: the vLLM one scores 0/4 on the no-reference loop.
UPDATED 2026-08-10: under the fixed loop prompt fable-fusion is **8/8 on BOTH
arms** (was 6/8 no-ref). It remains the only model that closes the loop unaided,
but "everything else is 0" is no longer true — qwen3-vl-thinking and qwen36-35b
both score 4/8. Its real distinction now: it is the ONLY model for which the API
reference buys nothing on the loop. See [[loop-prompt-imports-gap]].
Motivation was policy, not a hunch: this is the QUALITY pick and it runs the
CHEAPEST KV on the box, against [[vram-spend-priority]] ("upgrade KV q4->q8
before chasing tok/s"). Only 16 of 65 layers are full-attention
(full_attention_interval=4), so KV is 4.00 GiB at q4_0 / 8.00 GiB at q8_0
for the full 262144 — much cheaper than a naive 65-layer count suggests
([[kv-sizing-full-attention-layers]]).
q8_0 @262144 DOES NOT FIT. OOMs on an 887.99 MiB alloc on device 0. The
naive budget (17.23 weights + 0.87 mmproj + 8.00 KV = 26.1 of ~30.3 usable) is
wrong because it omits TWO costs: the mmproj tower loads wholly on CUDA0
ignoring --tensor-split ([[mmproj-caps-cuda0-tensor-split]]), and
--spec-type draft-mtp loads a draft model. Largest window at q8_0: 196608.
Grilled n=3 at q8_0 @196608 vs this file's q4_0 @262144 baseline:
| suite | q8_0 @196k | q4_0 @262k | |
|---|---|---|---|
| klayout unaided | 8/24 (33%) | 5/16 (31%) | parity |
| klayout + REF | 19/24 (79%) | 15/16 (94%) | -15 pts |
| closed loop unaided | 10/12 (83%) | 6/7 (86%) | parity |
REVERTED. The +ref arm is the PRODUCTION path — KLayout work here always
runs with the klayout-classes skill ([[klayout-api-recall-vs-composition]]) —
and it lost in all three reps with no spread (6/8, 6/8, 7/8).
CAVEAT THAT MATTERS: this does NOT disprove q8_0 KV. The test changed KV
depth AND window together — my error. The +ref prompts carry the API reference
and are the LONGEST inputs in the suite, so the 25% context cut is the likelier
culprit than KV depth. To isolate, run q4_0 @196608; if that also lands ~79%
the window is to blame and q8_0 becomes worth revisiting if VRAM frees up.
Unaided spread was 4/8, 3/8, 1/8. Rep1's 4/8 looked like the best unaided
KLayout result ever recorded here; the mean is 2.67, i.e. baseline. Do not read
one rep of this suite — cf. [[single-run-scores-are-unreliable]].
Checked 2026-09-07, every link:
| link | status |
|---|---|
| exllamav3 supports the arch | arch_string = "Qwen3_5ForConditionalGeneration" |
| ...WITH VISION | YES — architecture/qwen3_5.py imports Qwen3VLVisionModel and reads vision_config / vision_start_token_id |
| self-quantize tooling | present: conversion/{convert_model,measure_model,calibration_data}.py |
| fp16 source | philipjohnbasile/Qwen3.6-27B-Fable-Fusion-711-bf16, 51.7 GiB, vision_config: PRESENT |
**The text-only limitation on published EXL3 quants is the QUANTIZER'S CHOICE,
not an exllamav3 limit** — same trap as [[kat-coder-exl3-trial]]. A self-built
quant that keeps vision_config and the vision tensors WILL serve images.
Do NOT use the one published build: darkbit1001/...-Uncensored-Heretic-NM-DAU-MTP-EXL3-4.15bpw-hb16
is (1) 4.15bpw, LOWER than Q4_K_M's ~4.85 — a downgrade, not an upgrade,
(2) a DavidAU-lineage modification, and that catalogue is 3-for-3 REJECTED here
([[davidau-catalog-verdicts]]), (3) vision tower ABSENT.
RECOMMENDATION: do not do it as a QUALITY play. [[gemma-exl3]] ran exactly
this experiment on the driver — 6.10bpw vs the AWQ — and came back PARITY on all
four arms at 51 vs 93 tok/s. Here it would likely be worse: fable-fusion already
gets +49.5% from -sm tensor on llama.cpp, which EXL3 forfeits unless TP
compensates. The ONE honest case for it is CAPACITY: llama.cpp serves this entry
at --parallel 1, so EXL3 tensor-parallel ([[exl3-tabbyapi-backend]], +48% with
nccl) is the only route to concurrent or higher-throughput KLayout work.
Added to llama-swap.yaml (port 9114) to test whether [[fable-fusion]] — the
box's best all-rounder (23/23 coding + 21/21 vision) — tolerates
--parallel 2 given two things that made this the first *untested* parallel
trial on the box: (1) VRAM was already thin at --parallel 1 (~1.1-1.4GB
free/GPU, per fable-fusion's load-bearing 58,42 split), and (2) it's the
first parallel trial on a model with an MTP/speculative-decode head — every
prior trial ([[parallel-agent-slots]]'s coder-x4, bonsai-duo) ran on
non-speculative models.
Result: clean. Same weights/mmproj/template/sampler/split as the solo
entry, -c 262144 --parallel 2 (131072/slot — -c is TOTAL, split not
shared). Loaded without OOM on first try, no need to step down the
196608/131072 fallback ladder that was prepared in case it didn't fit.
VRAM measured: 1514MiB (5060Ti/CUDA1) / 1274MiB (4060Ti/CUDA0) free with
the model resident — actually *more* headroom than the solo entry's
last-measured 642/984MiB free moments before the restart that swapped it in
fresh (likely per-slot KV being exactly halved outweighs any per-slot
compute-buffer overhead; the solo measurement may also have reflected some
allocator drift after 19 minutes of a live process, not a clean-load
number). Static across a 2x400-token concurrent generation — no growth.
Throughput measured (2x 400-token concurrent completions, plain prompts,
default sampler): solo baseline 32.55 t/s; concurrent per-stream 26.68 /
26.54 t/s → aggregate ~53.2 t/s = 1.63x, each stream ~1.22x slower than
solo. Consistent with this box's established memory-bandwidth-bound decode
pattern ([[decode-bandwidth-ceiling]]; coder-x4 measured 1.43x at 2-concurrent,
bonsai-duo ~1.73x) — MTP draft acceptance apparently didn't disrupt the
pattern (single-request smoke test showed draft_n_accepted 2/2).
NOT yet done — do before trusting this for real agentic work:
and two throughput-measurement completions, neither exercised realistic
agentic tool-calling).
the failure mode that retired coder-x4 (grill round 5 caught genuine
tool-call loops that the original 39/40 quality check missed). Don't
assume 2-slot fable-fusion is loop-safe without that same style of check.
llama-swap.yaml + models.json + statusline(auto-synced) — it does NOT appear in kilo.jsonc/opencode.json/kimi
config.toml, so it's only reachable by direct model id, not through
those agent-client pickers. Leave it that way until/unless it graduates
past "trial."
How to apply: if this graduates from trial to a real serving entry,
re-run at least the R4 agentic-loop round from the standard grill at 2-way
concurrency before recommending it for unsupervised agentic work.
User asked to make both -x2 entries permanent/client-visible in the same
request. Propagated through bin/gen_kilo_config.py, opencode.json,
~/.kimi/config.toml, and the local-model-router skill (flagged there as
"load-tested only, not yet grilled at concurrency", distinct from
kimi-distill-x2's fully-grilled row). The quality/loop-safety gap noted
above is UNCHANGED — only VRAM+throughput were ever measured on this entry.
Client-visible now, but genuinely less proven than kimi-distill-x2. Run the
same round4/round5-concurrent treatment kimi-distill-x2 got
([[kimi-distill-x2-trial]]) before trusting this for unsupervised agentic
work — that treatment is what would actually close this gap, not more time
elapsed.
"fable-fusion":
aliases: [qwen3.6-27b-fable-711-256k]
# STORAGE TIER: NVMe. Briefly moved to /mnt/ssk500 on 2026-08-16, then
# REVERSED THE SAME DAY once this entry was confirmed both the box-wide
# default across every coding harness AND the only model that closes the
# KLayout loop unaided -- exactly the "swaps in constantly, pays the
# slow-disk penalty over and over" profile that must stay on fast
# storage (see coder-agentic's original NVMe rationale, same logic).
# DavidAU Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP (GGUF
# 17.2GiB) — a multi-stage fine tune + merge over 7 base models on Qwen/Qwen3.6-27B,
# Heretic-decensored (ARA method: refusals 4/100 vs the base's 99/100, KL div 0.047).
# Same "qwen35" arch as the vision-coder entry, i.e. a HYBRID SSM/attention model:
# block_count 65 but full_attention_interval 4, so only ~16 layers carry a KV cache
# (n_kv_head 4, k/v_length 256) — that's why 256k is affordable on 32GB. Vision is
# active (needs an explicit mmproj, see below) and it ships an MTP head
# (nextn_predict_layers 1) → --spec-type draft-mtp self-speculative decoding.
# QUANT: MTP-Q4_K_M. The repo has both "regular" and "MTP" NEO-imatrix quants at every
# size; MTP is picked because draft-mtp already measures 0.80-0.90 acceptance on the
# sibling vision-coder. NOT a bigger quant: this file is 17.2GiB where vision-coder's
# nominally-equal Q4_K_M is 15.7GB (DavidAU keeps the output tensor at fp16 and the MTP
# tensors at Q8_0), so Q5_K_M (19.7GiB) / Q6_K (22.4GiB) would push context under the
# 128k floor we hold elsewhere. Q4_K_M also keeps the grill comparable to vision-coder.
# MMPROJ: repo ships THREE projectors (BF16/F16/F32) and `-hf` cannot choose between
# them, so it is passed by absolute snapshot path — same situation as bonsai. BF16 is
# the quality-first pick and matches the dtype Qwen trains in. Path is tied to rev
# b73a29e8; re-point it if the repo is re-pulled at a newer revision.
# TEMPLATE: the GGUF's embedded template is BYTE-IDENTICAL to the base Qwen3.6-27B one
# and carries the same landmine as vision-coder — it raise_exception's on any system
# message that is not messages[0], which llama.cpp surfaces as a 400 "Unable to
# generate parser". chat-templates/fable-711-256k.jinja is the embedded template with
# that one line replaced by an ordinary <|im_start|>system render; the result diffs
# CLEAN against the proven chat-templates/vision-coder.jinja.
# SAMPLER: card's "thinking mode, precise coding" preset (temp 0.6 / top-p 0.95 / top-k
# 20 / min-p 0). Deliberately NO --repeat-penalty: the card is explicit that rep-pen
# above 1.0 and temp above 1.0 both degrade MTP draft acceptance. vision-coder ended up
# served at 0.2 after its grill showed quality flat across 0.2-0.85; revisit this after
# fable's own grill rather than assuming the same holds.
# CONTEXT: 262144 = n_ctx_train, and it FITS — but only with the tensor-split below.
# TENSOR-SPLIT 58,42 IS LOAD-BEARING, do not "simplify" it back to 50,50: the MTP
# draft context is allocated entirely on CUDA1, so an even weight split leaves CUDA1
# 3GB heavier than CUDA0. At 50,50 the model loads at 196608 (CUDA1 15221MiB / only
# 1090MiB free, CUDA0 12183MiB / 4197MiB free) and OOMs outright at 262144 with
# "cudaMalloc failed ... device 1" -> "failed to create MTP context". Biasing weights
# onto CUDA0 evens it out: measured at 262144 CUDA1 15187MiB, CUDA0 14907MiB, ~1.1GB
# and ~1.4GB free. Headroom is genuinely thin — VERIFIED to survive the largest image
# in pictures/ (1345x2048, ~1.1GB still free after the ViT encode), but if a bigger
# image ever OOMs the encoder, step -c down 262144 -> 196608 -> 131072 (each 64k of
# KV is ~1.2GB) before touching the quant or the split.
# LOAD TEST 2026-07-23 (all at -c 262144, split 58,42):
# - loads in ~10s, generates cleanly; thinking lands in reasoning_content as expected.
# - MTP draft acceptance 69/88 = 78.4% (card says fall back to the non-MTP quant below
# 50%, so MTP is the correct pick here); decode ~32 t/s, prompt ~58 t/s.
# - non-first system message -> 200 and the model OBEYS it, confirming the template
# patch (this exact case is a 400 on the stock embedded template).
# - vision: correct subject AND accurate in-image OCR on the 1345x2048 test photo.
# - tool-calling: well-formed tool_calls with correct enum mapping (Celsius->celsius).
# GRILL 2026-07-23 (results-fable-20260723-150840.log) — BEST ALL-ROUNDER ON THIS BOX:
# CODING 23/23 PERFECT (r1 8/8, r2 5/5, r3 5/5, r4 5/5) at temp 0.2 — the FIRST model
# here to take a perfect coding grill AND a perfect vision grill in the same run.
# Passes expr_eval (the lineage-hard unary-minus-vs-power case gpt-oss-20b clears
# only ~1/3 of the time) though it burns 8337 tok / 250s on it, and text_justify,
# the column-width off-by-one its sibling vision-coder FAILS.
# VISION 21/21 PERFECT (6/6 easy, 13/13 med, 2/2 hard) — object/landmark, in-image
# OCR, scene, colour, counting.
# KNOWLEDGE 59/59 PERFECT (9/9, 24/24, 26/26) + deep-cut tier 66/66 PERFECT (hard
# 30/30, extreme 36/36) — ties vision-coder, beats gpt-oss-20b's 65/66.
# TOTAL 169/169 across all four harnesses — a clean sweep, a first on this box.
# Decode held 32-35 tok/s across every task with no fade as context filled.
# Verdict: strictly better than vision-coder on coding (23 vs 22) at equal vision and
# equal knowledge, for ~2.8GB more VRAM and a tighter split. vision-coder remains the
# cheaper/faster option when its token-efficient thinking matters; prefer fable when
# correctness matters more than latency. Also uncensored, unlike everything else here
# that scores this well.
# *** -sm tensor (was `--tensor-split 58,42`). DO NOT RESTORE --tensor-split: passing it
# with -sm tensor does not start -- llama_params_fit is unimplemented for
# SPLIT_MODE_TENSOR and the load dies on a bogus ~20 GB "cudaMalloc failed"
# that is NOT an OOM. Tensor mode splits itself, evenly, with nothing to tune.
# Layer split runs the cards SEQUENTIALLY so the faster 5060 Ti idles waiting;
# tensor runs them concurrently. auto-memory/sm-tensor-rejects-tensor-split.md
#
# GRILLED 2026-09-02 BOTH ARMS SAME-DAY (bench/grill-smt-fable-fusion-*,
# r1-r4 n=2 + R5 n=3 per arm, same port + same cmd so -sm is the ONLY variable):
# coding 22/23 23/23 tensor | 23/23 22/23 layer
# R5 6/6 6/6 5/6 tensor | 6/6 6/6 6/6 layer
# R5 LOOPS 0/6 IN EVERY REP OF BOTH ARMS <- the signal that matters
# vision 5/6 tensor | 5/6 layer (klvision, real rendered images)
# decode 44.4 vs 29.7 tok/s in-suite over 46 tasks/arm = +49.5%
# Differences are within this box's known R5/coding sampling spread.
# *** q8_0 KV WAS TRIED 2026-09-07 AND REVERTED — IT MEASURED WORSE. ***
# Motivation was sound on paper: this is the QUALITY pick (the only model
# that closes the KLayout loop unaided) and it runs the cheapest KV on the
# box, against [[vram-spend-priority]]. Only 16 of 65 layers are
# full-attention (full_attention_interval=4), so KV is 4.00 GiB at q4_0 and
# 8.00 GiB at q8_0 for the full 262144.
#
# q8_0 @262144 DOES NOT FIT — OOMs on an 887.99 MiB alloc on device 0. The
# naive budget (17.23 weights + 0.87 mmproj + 8.00 KV = 26.1 of ~30.3) omits
# that the mmproj tower loads WHOLLY on CUDA0 ignoring --tensor-split
# ([[mmproj-caps-cuda0-tensor-split]]) AND that --spec-type draft-mtp loads
# a draft model. Largest window that fits at q8_0 is 196608.
#
# GRILLED n=3 at q8_0 @196608 vs this config's recorded baseline:
# klayout unaided 8/24 33% vs 5/16 31% parity
# klayout + REF 19/24 79% vs 15/16 94% *** -15 pts ***
# closed loop 10/12 83% vs 6/7 86% parity
# The +ref arm is the PRODUCTION path (KLayout work always runs with the
# klayout-classes skill), and it lost consistently in all 3 reps. Reverted.
#
# CAVEAT: that test changed KV depth AND window together, so it does NOT
# disprove q8_0 KV — the +ref prompts carry the API reference and are the
# LONGEST in the suite, making the 25% context cut the likelier culprit.
# To isolate, run q4_0 @196608; if it also lands ~79% the window is to blame
# and q8_0 becomes worth revisiting if VRAM ever frees up.
# Unaided spread was 4/8, 3/8, 1/8 — do NOT read one rep of this suite.
cmd: |
/usr/bin/env
CUDA_DEVICE_ORDER=PCI_BUS_ID
${llama_bin}
-m /home/giovanni/models-nvme/Qwen3.6-27B-Fable-Fusion-711-Q4_K_M.gguf
--mmproj /home/giovanni/models-nvme/Qwen3.6-27B-Fable-Fusion-711-mmproj-BF16.gguf
--alias fable-fusion
--jinja --chat-template-file chat-templates/fable-711-256k.jinja
-ngl 99 -c 262144 -fa on
-sm tensor
--spec-type draft-mtp --spec-draft-n-max 2
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0
--cache-type-k q4_0 --cache-type-v q4_0
--host 127.0.0.1 --port 9113 --parallel 1
proxy: http://127.0.0.1:9113