kat-coder

kat-coder-mtp-gguf-trial (kat-coder-mtp-gguf-trial.md)

kat-coder-fast (aliases kat-coder-mtp-gguf, katmtp), port 9194.

gbuzhf/KAT-Coder-V2.5-Dev-MTP-GGUF UD-Q4_K_XL, 21.29 GiB, on the STOCK

${llama_bin} — qwen35moe is in our pinned build, no fork needed.

Same base as kat-coder (EXL3), different build — and it wins on everything but one axis

| | GGUF Q4_K_XL | EXL3 5bpw |

|--------------------|--------------|-----------|

| 23-task n=3 | 20.33/23 | 20.33/23 |

| decode | 127 tok/s| 76 |

| cap-hits | 0 of 15 runs | — |

| context | 262144 | 258048 |

| Claude Code | works | 404 ([[tabbyapi-no-anthropic-messages-endpoint]]) |

| exllamav3 slot leak| none | 503s after ~40 req |

| interrupt_replan | 1/14 loops | 0/3 |

Coding is IDENTICAL to two decimals. MTP engages (nextn_predict_layers=1,

blk.40.nextn.*, loader logs "creating MTP draft context"). TEXT-ONLY — 0

vision tensors, so vision + klayout-vision can only score 0.

-sm tensor IS LOAD-BEARING: +18.7%, and a hand split CANNOT match it

-sm tensor 127.17 t/s (119.18-130.42) vram 13927/13881

--tensor-split 50,50 107.16 (104.18-112.94) vram 13835/15615

--tensor-split 58,42 103.93 (100.46-107.40)

--tensor-split 65,35 FAILED to load

Ranges disjoint, so the gain is real. **A layer-count split is not a weight

split**: 50,50 puts MORE on the SLOWER 4060 Ti. Trying to correct that by hand

(58,42) did NOT help and 65,35 would not load. Tensor mode splits WITHIN each

tensor and balances by construction. qwen35moe is NOT arch-gated for it,

unlike qwen4exp ([[sm-tensor-rejects-tensor-split]]).

* THE "LOOPS 2/3" FIGURE WAS A BAD SAMPLE — IT IS 1/14 *

The rounds1-5 x3 grill hit 2 loops in 3 reps and I wrote that into the entry as

the model's behaviour. n=14 on the identical config gives 1/14 (~7%).

A loop is expensive when it fires (87 tool-steps / 17,007 tokens) but it is not

the norm.

Consequence: the EXL3-vs-GGUF looping difference is UNPROVEN. 0/3 vs 1/14 is

statistically indistinguishable. Do not justify keeping both entries on that

basis until someone runs n>=14 on the EXL3 build too.

Four candidate fixes, ALL FAILED

baseline temp 0.6, no penalty 1/14 7%

vendor temp 1.0 + presence 1.5 1/14 7% no better

DRY temp 0.6 + 0.8/1.75/2 4/14 29% WORSE

MTP off --spec-type none 1/5 no change

The Qwen3.5-35B-A3B card explicitly says presence_penalty "reduce[s] endless

repetitions" and recommends 1.5 for general/thinking tasks (0.0 only for

"precise coding"). It does not help this model. DRY — which fixed

[[k2-horizon-arch-gate]] — BACKFIRES here. There is no sampler fix.

SO: DO NOT RE-TEST THE LOOP-REJECTED MODELS ON A SAMPLER THEORY

The open question was whether presence_penalty 1.5 might rescue

[[apodex-11-mini-trial]] (loops 3/3), [[xyz-aquila-mini-trial]] (4/4) and

[[bigbang-v1-trial]] — all the same 35B-A3B lineage, all graded at

presence_penalty 0 (the flag appears ZERO times in llama-swap.yaml). **Answer:

no.** It does not move a 7% rate, so there is no reason to expect it to move a

100% one — and those two are deterministic where this is stochastic.

aquila's think-tag leak and bigbang's office 3/9 ceiling stand regardless.

METHOD: n=5 WAS ACTIVELY MISLEADING HERE

The first matrix showed vendor-sampler and DRY at 0/5 and baseline at 1/5 —

apparently a fix. Both regressed at n=14 (1/14, 4/14). If the true rate is 20%,

P(0/5) = 0.33 — a third of the time by chance. **On a stochastic failure this

box needs n>=14**; P(0 loops | 20%) = 0.044 there. Same lesson as the KV

comparison, where a blocked n=3 gave means BELOW the true value

([[single-run-scores-are-unreliable]]).

kat-coder-exl3-trial (kat-coder-exl3-trial.md)

KAT-Coder-V2.5-Dev-MTP (EXL3 5bpw) — TRIAL, kept not promoted

P4pps3n/KAT-Coder-V2.5-Dev-MTP-exl3-5bpw-hq, 23 GB on /mnt/ssk500/exl3,

entry kat-coder on port 9191. Base is Kwaipilot KAT-Coder-V2.5-Dev, a

coder fine-tune of Qwen3.5-35B-A3B — config is structurally identical to

turboderp/Qwen3.5-35B-A3B-exl3 (40 layers, 256 experts, 8/tok, 262144).

* THE RESULT THAT MATTERS: IT CLEARS interrupt_replan *

That task is THE agentic discriminator here ([[grill-round5-agentic-loops]]) and

it has killed every other fine-tune of this base:

| model | interrupt_replan | outcome |

|---|---|---|

| apodex | loops 3/3 | REJECTED ([[apodex-11-mini-trial]]) |

| aquila | loops 4/4 — "deterministic nameable defect" | not promoted ([[xyz-aquila-mini-trial]]) |

| bigbang | loops | REJECTED ([[bigbang-v1-trial]]) |

| KAT-Coder | PASS 3/3 probe + PASS inside full R5 (6 steps, 0 redundant) | first of the lineage to pass |

**Probe it FIRST on any model of this lineage — grill_round5.py interrupt_replan

takes minutes and decides the question.** A full grill here is hours.

Grill, n=3 (bench/kat-coder-rep{1,2,3})

| | rep1 | rep2 | rep3 |

|---|---|---|---|

| 23-task | 20/23 | 20/23 | 21/23 |

| R5 complete | 6/6 | 5/6 | 6/6 |

| R5 looped | 0/6 | 1/6 | 0/6 |

| redundant | 0% | 7.7% | 2.5% |

Mean 20.33/23 — level with glm-flash (20.0, measured the same day), just

under qwen38-awq's recorded 21/23. The single loop was search_navigate,

NOT interrupt_replan, and did not reproduce.

Deterministic failures: expr_eval 3/3 (a known flake here — glm-flash failed

it on BOTH vLLM and SGLang the same day), json_strict 3/3 (see below),

wildcard_match 2/3.

SPEED + CONCISION are the real win

~76 tok/s warm (20.7 cold -> 73.1 -> 79.3 — the cold-first-run effect again,

[[sglang-vs-vllm-backend-ab]]) vs qwen38-exl3's 22.3. And ~26x MORE CONCISE:

~490 tokens per grill task where qwen38-exl3 burns ~13,000. A full rep takes

2.5 min vs 88 min for qwen38-exl3's round 1 alone. A3B sparsity + a real MTP

head (3113 mtp.* tensors; exllamav3 has a dedicated qwen3_5_mtp.py;

--draft-mode mtp logs "Using main model MTP component for drafting").

* GOTCHA 1: --vision false IS MANDATORY — true IS A HARD CRASH *

ValueError: Required tensor model.visual.patch_embed.proj.weight not found

The config declares Qwen3_5MoeForConditionalGeneration, which implies vision,

but the quant has 0 visual tensors — text-only. **EXL3-CANDIDATES.md lists

this repo as vision ✅ and that is WRONG.** Grep

model.safetensors.index.json for visual/vision before believing any

vision claim about a quant.

* GOTCHA 2: IT ALWAYS FENCES JSON, AND NO PROMPT FIXES IT *

Asked for bare JSON it returns ``` `json\n{...} ``. 4/4 fenced: the

grill's system prompt, temperature 0.0, NO system prompt, and an emphatic

"NO CODE FENCES, no ` characters, reply must start with {". Deterministic.

The fix is CONSTRAINED DECODING, verified:

response_format {"type":"json_schema", ...} -> BARE valid JSON

json_schema (Tabby native) -> BARE valid JSON

response_format {"type":"json_object"} -> STILL FENCED (ignored)

SCOPE IT CORRECTLY — this is a JSON-fencing habit, not disobedience. The

JSON CONTENT is always correct; tool calling passes 3/3

(tool_weather/tool_select/tool_restraint); and format_lines, the OTHER

negative-constraint format task, passes 3/3. It only bites a pipeline that

parses raw output as JSON without supplying a schema.

Verdict

TRIAL, not a driver. It does not beat the incumbents on score. Kept because

it is fast, strikingly token-efficient, and the only model of this lineage whose

agentic behaviour is clean.

Context is 258048 and the doubling is FREE (verified 2026-09-05)

Raised from 131072. VRAM at 258048 is 13988/13309 MiB — IDENTICAL to 131072,

because only 10 of 40 layers are full attention (~10 KiB/token,

[[kv-sizing-full-attention-layers]]). A real ~40k-token prompt answered

correctly in 129 s.

It is 258048, NOT 262144. Ask for 262144 and TabbyAPI rounds DOWN to a

page multiple; the startup log prints the truth (max_seq_len: 258048).

EXL3-CANDIDATES projected "262k" — trust the log, not the projection.

**WHEN RE-TESTING A TP CONTEXT CEILING, WAIT ~60 s AFTER KILLING THE PREVIOUS

SERVER.** Two attempts "failed" first, both with

RuntimeError: cudaHostRegister(0x.., 67108864) failed: invalid argument

which is the previous process's pinned 64 MB host arena not yet released — NOT a

VRAM shortage, though it reads exactly like one and was nearly recorded as

"262k does not fit". 25 s is not enough (the figure in

[[exl3-tabbyapi-backend]]); 60 s worked. A third failure came from a cleanup bug

of mine: ps -o pid,args with $1 ~ /llama-server$/ matches the PID, not the

executable, so the previous model was never killed and the test competed with

30 GB already resident.

TP setup is identical to [[exl3-tabbyapi-backend]]: nccl backend +

cuda_malloc_async: false`.

kat-coder-v2-5-dev (kat-coder-v2.5-dev.md)

kat-coder — Kwaipilot KAT-Coder-V2.5-Dev — REMOVED 2026-07-28, RE-ADDED 2026-08-02

> READ THE 2026-08-02 SECTION FIRST — it revises the verdict below, which

> came from one run each. Current serving: port 9121, --tensor-split 48,52,

> weights re-downloaded to the ssd. Everything in this blockquote and the two

> sections after it describes the ORIGINAL July trial and is kept for history.

> Removed same day as added in July, on the user's call: redundant with

> [[kimi-distill]] — same base, lower score (19/23 vs 22/23), slower (88.4 vs

> 102.5 t/s). llama-swap entry deleted, models.json + both statuslines

> resynced. The patched template is KEPT, but **NOT at the path this file

> originally gave**: chat-templates/kat-coder-256k.jinja was generalised to

> chat-templates/qwen35moe-nonfirst-system-256k.jinja in fc67a142 (the

> pocket-35b commit), since the guard is a whole-family issue. Verified

> 2026-08-02: the two files are BYTE-IDENTICAL (7759 chars = the 7764-char

> original minus the documented 5-char patch) and the family-neutral copy is

> load-bearing for two live entries. A future re-add reuses it verbatim.

> The old path is recoverable from git at 4d01af9c if ever needed.

>

> Weights WERE gone (checked 2026-08-02: nothing in the HF cache,

> /mnt/ssk500 or models-nvme) — confirming a removal really does delete them.

> RE-DOWNLOADED 2026-08-02, 19.92 GiB, now on the ssd at

> /mnt/ssk500/models/models--bartowski--Kwaipilot_KAT-Coder-V2.5-Dev-GGUF

> with the usual HF-cache symlink. NOT on the nvme working set — cold load is

> ~1m33s; promote it only if it earns a keep.

>

> Re-verified present and healthy 2026-08-07: blob is 20 GB, readable,

> refs/main holds a real commit hash and the snapshot symlink resolves — so

> it is NOT in the dangling-ref state of [[llama-cpp-offline-hf-refs]] that

> makes an offline -hf resolve to '' and kill the llama-swap child in <1s.

> **Checking this needs du -sh --apparent-size or ls -lL, not plain

> du -sh**: the HF cache entry is a SYMLINK to /mnt/ssk500, so plain du

> reports 4.0K and looks exactly like deleted weights. That misread nearly

> got the model written off a second time.

> **This file is retained for the caveat-9 A/B result below, which outlives the

> model.**

bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF:Q4_K_M, 19.92 GiB. Apache 2.0,

released 2026-07. Base Qwen3.6-35B-A3B — same base as chat, uncensored

and [[kimi-distill]]. Real sparse MoE (35B total / 3B active), arch qwen35moe,

no MTP head. Served on port 9115 at the full -c 262144 with q8_0 KV,

--tensor-split 45,55 → 1616/3702 MiB free. 88.4 t/s decode (peak 100.7).

TEXT-ONLY. The open-weight release ships language-model weights only; the

upstream config.json still carries a vision_config but the tower is not

included. Do not add --mmproj, and keep it OUT of sync-models.sh's

VISION_IDS (see [[sync-models-hook]] — a wrong id there silently mislabels

modality for every downstream client).

Verdict: good, but does NOT displace the leaders

19/23, R5 6/6, 0 loops, redundant 2.8%, tok/step 115.9. Round 5 is a clean

sweep with unusually low wall times (3.8-9.9s/task, 3-10 steps). Misses were

spread, not systematic: expr_eval (lineage-wide hard case, kimi-distill misses

it too), weighted_interval_scheduling, json_strict, chain_book.

Keep it for long-horizon agentic work where speed + 256k context matter, but

fable-fusion (22/23) and kimi-distill (22/23, faster) remain the better

default picks. It is arguably redundant with kimi-distill — same base, lower

score, slower — so it is a fair cleanup candidate if VRAM/disk pressure returns.

2026-08-02 RE-ADD + 3-REPEAT RE-TEST — the verdict below is REVISED

Re-downloaded (19.92 GiB; the weights really were gone) and re-tested properly:

metric_sweep.py kat-coder --repeats 3 ->

bench/metric-sweep-20260802-082106.jsonl. Served on port 9121 (9115 was

reassigned while it was away) at --tensor-split 48,52 (rebalanced from 45,55;

2406/2914 MiB free vs the old 2076/4131).

| n=3 | coding r1-4 | round 5 | flaky | loops |

|---|---|---|---|---|

| coder-agentic | 21.7/23 | 9.3/10 | 2 | 0.7 |

| kimi-distill | 21.0/23 | 9.0/10 | 3 | 0.0 |

| kat-coder | 19.7/23 | 9.3/10 | 5 | 0.3 |

Token economy, common basis (17 coding + 7 r5 tasks all three passed in every

repeat): coding 3844 / 9888 / 55841 tokens for coder-agentic / kat-coder /

kimi-distill.

What held: 19.7/23 reproduces the July 19/23, and the SAME tasks fail.

json_strict and chain_book failed 3/3 — systematic, so the July note that

"misses were spread, not systematic" was WRONG. expr_eval and

weighted_interval_scheduling are flaky.

What changed:

1. The gap to kimi-distill is 1.3 points, not 3 — both July numbers were

single runs at opposite extremes.

2. Round 5 goes to kat-coder (9.3 vs 9.0), tying coder-agentic. The agentic

RL DID transfer; it just doesn't show on short-form coding.

3. It is 5.6x leaner than kimi-distill on coding tokens, so "redundant with

kimi-distill — lower score, slower" is wrong on the economy axis.

**REVISED VERDICT: still drop it, but it is redundant with [[coder-agentic-default-driver]],

NOT with kimi-distill.** coder-agentic dominates it on all four metrics —

higher coding, equal round 5, fewer flaky tasks, 2.6x leaner. kat-coder's 5

flaky tasks are the most of the three, making it the least dependable.

**Spin-off worth chasing: kimi-distill burns 14.5x coder-agentic's tokens for a

LOWER score** (55841 vs 3844, 21.0 vs 21.7). That is a stronger argument against

kimi-distill than anything here is against kat-coder.

The original finding, now superseded: "agentic RL lost to distillation"

This is the cleanest post-training A/B run on this box. kat-coder and

kimi-distill share base, quant class, context and sampler — graded at the same

settings on purpose — so the only real variable is the post-train:

execution-feedback agentic RL over 100k+ verifiable repo environments vs K2.6

reasoning distillation.

• Round 5: tie (6/6, 0 loops each).
• 23-task suite: distillation wins, 22/23 vs 19/23.

The 3-point gap exceeds the ±1-2 single-run flake band, so probably real — but

it is one run each; re-measure before treating it as settled. Its headline

SWE-bench Verified 69.4 did not translate into a lead here: vendor agentic

benchmarks and this harness measure different things.

Sampler deviation, deliberate: the card recommends temp 1.0 / top-p 0.95 /

top-k 20 / presence_penalty 1.5. The entry uses kimi-distill's profile

instead (temp 0.6 / top-p 0.95 / top-k 20 / min-p 0). Reason: the grill sends

temperature and max_tokens but NOT presence_penalty, so a server-level 1.5

would apply through the whole run when no other entry in the sweep has one — and

penalising repeated tokens hits code (indentation, keywords) hardest. Trying the

vendor profile for real serving is still an open follow-up.

Template guard — HAD it, patched, and the smoke test does NOT catch it

Confirmed live (not assumed): the embedded template carries

raise_exception('System message must be at the beginning.') at line 85, the

guard from [[jinja-system-guard-tool-parser]]. **The trap is that it passes a

naive check** — plain chat returns 200, and even a SIMPLE single-turn request

with a tools array returns 200. It only fires on a **multi-turn tool

conversation** (system+user+assistant+system+user + tools) → HTTP 400

"Unable to generate parser for this template". That shape is grill rounds 4/5

and real Claude Code usage.

Fix: chat-templates/kat-coder-256k.jinja renders a non-first system message as

its own ChatML turn instead of raising (one-line diff, -5 chars). The FIRST

system message is emitted before the loop at template line 62, so it is

unaffected. --chat-template-file in the entry is load-bearing.

Extraction shortcut worth reusing: pull the template from the running

server's /props endpoint (chat_template key) instead of parsing the GGUF —

llama-server has already decoded it, which sidesteps the u64-vs-u32 string-length

gotcha documented in [[jinja-system-guard-tool-parser]] entirely.

How to apply

When adding any new finetune, test the guard with a multi-turn system+tools

request, not a single-turn one — a 200 on the simple case proves nothing. And

when two models share a base, hold sampler/context/budget identical so the

comparison isolates the post-train; see [[genesis-hermes-v5]] for why a

cross-build baseline is unsound.