Active active in llama-swap.yaml · aliases: kat-coder-mtp-gguf, katmtp
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| 20260909-093448 | 61/69 | 125.6 | — | results-kat-coder-mtp-gguf-r15x3-20260909-093448.log |
kat-coder-fast (aliases kat-coder-mtp-gguf, katmtp), port 9194.
gbuzhf/KAT-Coder-V2.5-Dev-MTP-GGUF UD-Q4_K_XL, 21.29 GiB, on the STOCK
${llama_bin} — qwen35moe is in our pinned build, no fork needed.
kat-coder (EXL3), different build — and it wins on everything but one axis| | GGUF Q4_K_XL | EXL3 5bpw |
|--------------------|--------------|-----------|
| 23-task n=3 | 20.33/23 | 20.33/23 |
| decode | 127 tok/s| 76 |
| cap-hits | 0 of 15 runs | — |
| context | 262144 | 258048 |
| Claude Code | works | 404 ([[tabbyapi-no-anthropic-messages-endpoint]]) |
| exllamav3 slot leak| none | 503s after ~40 req |
| interrupt_replan | 1/14 loops | 0/3 |
Coding is IDENTICAL to two decimals. MTP engages (nextn_predict_layers=1,
blk.40.nextn.*, loader logs "creating MTP draft context"). TEXT-ONLY — 0
vision tensors, so vision + klayout-vision can only score 0.
-sm tensor IS LOAD-BEARING: +18.7%, and a hand split CANNOT match it-sm tensor 127.17 t/s (119.18-130.42) vram 13927/13881
--tensor-split 50,50 107.16 (104.18-112.94) vram 13835/15615
--tensor-split 58,42 103.93 (100.46-107.40)
--tensor-split 65,35 FAILED to load
Ranges disjoint, so the gain is real. **A layer-count split is not a weight
split**: 50,50 puts MORE on the SLOWER 4060 Ti. Trying to correct that by hand
(58,42) did NOT help and 65,35 would not load. Tensor mode splits WITHIN each
tensor and balances by construction. qwen35moe is NOT arch-gated for it,
unlike qwen4exp ([[sm-tensor-rejects-tensor-split]]).
The rounds1-5 x3 grill hit 2 loops in 3 reps and I wrote that into the entry as
the model's behaviour. n=14 on the identical config gives 1/14 (~7%).
A loop is expensive when it fires (87 tool-steps / 17,007 tokens) but it is not
the norm.
Consequence: the EXL3-vs-GGUF looping difference is UNPROVEN. 0/3 vs 1/14 is
statistically indistinguishable. Do not justify keeping both entries on that
basis until someone runs n>=14 on the EXL3 build too.
baseline temp 0.6, no penalty 1/14 7%
vendor temp 1.0 + presence 1.5 1/14 7% no better
DRY temp 0.6 + 0.8/1.75/2 4/14 29% WORSE
MTP off --spec-type none 1/5 no change
The Qwen3.5-35B-A3B card explicitly says presence_penalty "reduce[s] endless
repetitions" and recommends 1.5 for general/thinking tasks (0.0 only for
"precise coding"). It does not help this model. DRY — which fixed
[[k2-horizon-arch-gate]] — BACKFIRES here. There is no sampler fix.
The open question was whether presence_penalty 1.5 might rescue
[[apodex-11-mini-trial]] (loops 3/3), [[xyz-aquila-mini-trial]] (4/4) and
[[bigbang-v1-trial]] — all the same 35B-A3B lineage, all graded at
presence_penalty 0 (the flag appears ZERO times in llama-swap.yaml). **Answer:
no.** It does not move a 7% rate, so there is no reason to expect it to move a
100% one — and those two are deterministic where this is stochastic.
aquila's think-tag leak and bigbang's office 3/9 ceiling stand regardless.
The first matrix showed vendor-sampler and DRY at 0/5 and baseline at 1/5 —
apparently a fix. Both regressed at n=14 (1/14, 4/14). If the true rate is 20%,
P(0/5) = 0.33 — a third of the time by chance. **On a stochastic failure this
box needs n>=14**; P(0 loops | 20%) = 0.044 there. Same lesson as the KV
comparison, where a blocked n=3 gave means BELOW the true value
([[single-run-scores-are-unreliable]]).
"kat-coder-fast":
aliases: [kat-coder-mtp-gguf, katmtp]
# KAT-Coder-V2.5-Dev-MTP, UD-Q4_K_XL GGUF (21.29 GiB) — the SAME base model as
# `kat-coder` above, but a different build, and THEY ARE NOT INTERCHANGEABLE.
#
# `interrupt_replan` LOOP RATE = 1/14 (~7%), measured 2026-09-09.
# *** AN EARLIER NOTE HERE SAID "LOOPS 2/3" — THAT WAS A BAD SAMPLE. *** The
# rounds1-5 x3 grill happened to hit 2 loops in 3 reps; n=14 on the same
# config gives 1/14. A loop, when it happens, is expensive (87 tool-steps /
# 17,007 tokens) but it is NOT the norm.
#
# THE EXL3-vs-GGUF LOOPING DIFFERENCE IS UNPROVEN. `kat-coder` (EXL3) is
# recorded at 0/3; this build is 1/14. Those are statistically
# indistinguishable — do NOT justify keeping both entries on that basis until
# someone runs n>=14 on the EXL3 too.
#
# FOUR CANDIDATE FIXES TESTED, ALL FAILED (bench/results-*-replan-*):
# baseline temp 0.6, no penalty 1/14 7%
# temp 1.0 + presence-penalty 1.5 1/14 7% (the Qwen3.5 card's
# general-task setting)
# temp 0.6 + DRY 0.8/1.75/2 4/14 29% WORSE than baseline
# --spec-type none (MTP off) 1/5 no change
# The Qwen3.5-35B-A3B card says presence_penalty "reduce[s] endless
# repetitions" and recommends 1.5 for general tasks — it does not help here.
# DRY is the k2-horizon fix and it BACKFIRES on this model.
#
# WHAT IT IS FOR: a fast one-shot text coder.
# 23-task 20.33/23 (n=3: 21, 21, 19) — IDENTICAL to the EXL3 build
# decode ~127 tok/s vs the EXL3's 76 (+67%)
# 0 cap-hits in 15 suite-runs
# 262144 ctx, and it SERVES /v1/messages so Claude Code can use it —
# the EXL3 entry cannot ([[tabbyapi-no-anthropic-messages-endpoint]])
# no exllamav3 slot leak (that backend 503s permanently after ~40 requests)
#
# -sm tensor IS LOAD-BEARING, measured n=5/4 interleaved:
# -sm tensor 127.17 tok/s (119.18-130.42) vram 13927/13881 <- THIS
# --tensor-split 50,50 107.16 (104.18-112.94) vram 13835/15615
# --tensor-split 58,42 103.93 (100.46-107.40)
# --tensor-split 65,35 FAILED to load
# Ranges are disjoint, so +18.7% is real. Note 50,50 loads the SLOWER 4060 Ti
# more heavily (a layer-count split is not a weight split); tensor mode splits
# WITHIN each tensor and balances by construction. **-sm tensor REJECTS
# --tensor-split** — passing both aborts with a bogus 20006 MiB OOM
# ([[sm-tensor-rejects-tensor-split]]). Unlike qwen4exp, qwen35moe is not
# arch-gated for tensor mode.
#
# Runs on the STOCK ${llama_bin} — qwen35moe is in our pinned build, no fork.
# MTP is real and engages: `nextn_predict_layers=1`, blk.40.nextn.* tensors,
# loader logs "creating MTP draft context". TEXT-ONLY (0 vision tensors), so
# the vision + klayout-vision suites can only score 0 — do not run them.
# temp 0.6 = the Qwen3.5 family default and what both builds were graded at;
# these scores are NOT comparable to the ~450 bench lines recorded at 0.2.
cmd: |
/usr/bin/env
CUDA_DEVICE_ORDER=PCI_BUS_ID
${llama_bin}
-m /home/giovanni/models-nvme/Kwaipilot_KAT-Coder-V2.5-Dev-MTP-UD-Q4_K_XL.gguf
--alias kat-coder-fast
--jinja
-ngl 99 -c 262144 -fa on
-sm tensor
--spec-type draft-mtp --spec-draft-n-max 2
--cache-type-k q8_0 --cache-type-v q8_0
-b 2048 -ub 512
--temp 0.6 --top-p 0.95 --top-k 20
--host 127.0.0.1 --port 9194 --parallel 1
proxy: http://127.0.0.1:9194