k2-horizon

k2-horizon-arch-gate (k2-horizon-arch-gate.md)

K2-Horizon (IFM/MBZUAI) — SOLVED by a side-by-side fork build

Triaged 2026-09-04/05 as unservable. RESOLVED 2026-09-06 — it runs.

* WHAT THIS FILE GOT WRONG (three claims, all load-bearing) *

1. "NOT servable by ANY backend." False as written. It was true only of the

four OPTIMIZED serving backends. IFM/K2-Horizon-MoVA-36B-A4B is tagged

custom_code on HF and ships modeling_k2_horizon.py (1,115 lines) +

configuration_k2_horizon.py wired through auto_map — so stock transformers

with trust_remote_code=True ran it the whole time. **That is how IFM ran it

before release.** Checking four backend registries is not the same as asking how

the authors ran it; the reference implementation ships WITH the weights.

2. "DO NOT build the vendor fork — it would shadow the mainline binary."

False, and this repo already disproved it: prism_llama_bin has pointed at a

second llama.cpp fork build (PrismML, for ternary Bonsai) for months. A fork in

its own tree behind its own macro shadows nothing. The reasoning was imported

from [[nanbeige-4.2-3b]] without checking whether the local precedent applied.

3. "MoVA is a NOVEL attention mechanism, so the port needs real kernel work."

Overstated. K2HorizonMoVAAttention.forward routes only the V projection

through 64 value-experts (4 active), then calls the ordinary

ALL_ATTENTION_FUNCTIONS path with a standard past_key_values.update().

Attention itself is plain GQA on a standard KV cache — which is why the GQA KV

arithmetic below DOES hold, and why the fork port is only 5 commits.

The build (the actual fix)

ifm_llama_bin: ifm-llama.cpp/build/bin/llama-server # llama-swap.yaml macro

Source: MBZUAI-IFM/llama.cpp branch model/K2Horizon — 5 commits / 17 files

on top of upstream 4e97ac86e, i.e. a clean delta on a real upstream point and

NEWER than our mainline build (2026-08-23).

Fetch it as a codeload TARBALL, not git clone. Two clones died with

RPC failed; curl 92 / curl 18 because a concurrent large download saturated

the link and git's pack negotiation cannot resume. The 36 MB tarball worked

first try. Build flags copied verbatim from llama.cpp/build/CMakeCache.txt

(89;120, CUDA/FA/NCCL/GRAPHS, compression=size) so the binaries differ ONLY in

arch support — and pin -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc, or

configure fails with "No CMAKE_CUDA_COMPILER could be found" in a detached shell

that lacks CUDA on PATH.

Measured 2026-09-06 (first load)

VRAM 14,893 MiB + 15,533 MiB (~30.4 GiB — very tight, but loads)

ctx 131072 at q4_0 K and V

decode 62.97 and 66.14 tok/s over two generations

correctness reverse_words + LRU-with-TTL both answered correctly

finish stop both times

Faster than granite (23) and gemma-exl3 (51); below kat-coder (76). A THINKING

model: a 400-token budget produced EMPTY content because the whole budget went

to reasoning — the reasoning/content split is clean, but budget accordingly

(1,696 reasoning chars for a one-line answer).

Two benign loader warnings, both expected:

• common_fit_params: ... n_gpu_layers already set by user to 99, abort — a

WARNING, not the fatal --fit off case ([[granite-4.2-30b]]); load continues.

• special_eos_id is not in special_eog_ids — comes from IFM's own tokenizer

metadata, not our quant. Cosmetic: finish_reason: stop works.

GRILLED n=3, 2026-09-06 — KEEP AS TRIAL, DO NOT PROMOTE

Rounds 1-5 x3 at temp 0.6 + DRY (see the sampler section below — the bench

standard 0.2 is FATAL to this model, so **these numbers are NOT comparable to

other models' 0.2 scores**). Vision suites skipped: no mmproj.

bench/results-k2-horizon-r15x3-20260906-182502.log.

| | rep1 | rep2 | rep3 | |

|---|---|---|---|---|

| 23-task | 20/23 | 20/23 | 19/23 | mean 19.67 |

| R5 complete | 6/6 | 6/6 | 5/6 | |

| R5 looped | 0/6 | 0/6 | 1/6 | 1 of 18 |

| redundant | 2.7% | 0.0% | 10.3% | |

19.67 sits just under kat-coder (20.33) and glm-flash (20.0).

### The verdict rests on ONE task: interrupt_replan loops 1 in 3

rep1 PASS 6 steps 1467 tok 1 redundant

rep2 PASS 5 steps 1412 tok 0 redundant

rep3 FAIL 8 steps 748 tok 3 redundant stop=loop

That is THE agentic discriminator ([[grill-round5-agentic-loops]]): apodex

looped it 3/3 (REJECTED), aquila 4/4 (not promoted), and kat-coder clearing it

0/3 was the whole reason that entry exists. k2-horizon at 1/3 is **better than

the rejected models and short of the one that cleared it**. For a model whose

entire pitch is agentic work, a third of runs looping on interrupt-and-replan is

the wrong place to be unreliable.

I called it "cleared" after rep2 and was wrong. Two passes is not a result on

this round — [[round5-is-a-sample-not-a-measurement]] exists precisely for this,

and rep3 produced the loop. Never report an R5 task as cleared before n=3.

### Rounds 3 and 4 are PERFECT: 30/30 across all reps

Tool-calling, format adherence, tool_restraint (correctly NOT calling a tool)

and the multi-turn agentic loop never failed once in 15 attempts each, at

1-2 s and 47-99 tokens per task. **A thinking model that spends ~1700 reasoning

chars on a trivial coding question emits a 47-token tool call** — it does not

burn reasoning on tool invocations. Every coding loss is in rounds 1-2

(algorithmic synthesis). The shape is genuinely agentic-not-coding.

### expr_eval fails 3/3 and is sampler-proof

316 s / 322 s / 155 s (16000, 16000, 8585 tok). It ran away at EVERY sampler

tested — 0.2, 0.2+DRY, 1.0, 0.6+DRY, 1.0+DRY. Same task granite runs away on and

laguna-s scores 0/20 on at 3-bit; the most-failed task on this box.

### textturn n=30 (90 turns): NO TAG LEAK, but 3% FABRICATION

turns with TAG LEAK : 0 (0%)

turns FABRICATING run : 3 (3%)

This is a real exception to [[textturn-leak-comparison]], which recorded 3/3

llama.cpp+jinja setups leaking vs 3/3 vLLM clean and blamed the serving layer.

k2-horizon is llama.cpp + jinja + a thinking model and is clean at n=90 — the

serving-layer split is a strong tendency, NOT a law.

The 3% fabrication (claiming to have run something it did not) is the worse

defect of the two for agentic duty: a tag leak is cosmetic and the 8091 sanitizer

strips it, a fabricated tool result is a wrong answer that looks right.

HARNESS BUG FOUND: grill_textturn.py computes

elif tot_leak or tot_fab: verdict = "LEAKY" but then prints only the tag-leak

explanation ("Claude Code renders the tag inline... fix --reasoning-format").

A pure-fabrication run is therefore labelled LEAKY and sent chasing a nonexistent

tag leak. Same family as [[eval-false-pass-adjacent-data]] — read the COUNTS, not

the verdict string.

### VERDICT: KEEP as a selectable trial, NOT promoted

Excellent tool discipline, clean on the leak check that has caught other models,

competitive 19.67, ~65 tok/s. But interrupt_replan 1/3 plus 3% fabrication

disqualify it from agentic driver duty, which is the only role its strengths

argue for.

* -sm tensor IS ARCHITECTURALLY IMPOSSIBLE FOR THIS MODEL *

Tested 2026-09-06 at four contexts. The failure MODE changes with context, and

the high-context OOMs MASK the real blocker:

| ctx | failure |

|---|---|

| 131072 | OOM — allocating 3456.00 MiB on device 0: cudaMalloc failed |

| 98304 | OOM — 511.09 MiB on device 0 |

| 65536 | GGML_ASSERT(split_states_equal(src_ss[0], src_ss[2])) |

| 32768 | same assert (ggml/src/ggml-backend-meta.cpp:756) |

Once VRAM stops binding, an internal split-state assertion fires — almost

certainly the routed value-expert path, since MoVA produces V through experts

before the attention call. No context or batch value moves it. Same class as

nemotron_h_moe in [[sm-tensor-rejects-tensor-split]]; add k2-horizon to that

file's list of archs that CANNOT use tensor mode.

METHOD NOTE worth more than the result: I nearly stopped at "OOM, needs less

context" and would have recorded a VRAM ceiling that does not exist. Dropping

context until memory stopped binding is what exposed the real cause. **When a

feature OOMs, reduce until it stops OOMing before concluding the cause is

memory.**

No Xid 79 and no hard crash across four attempts — clean aborts, VRAM released

every time. The [[xid79-gpu-fell-off-bus-sm-tensor]] risk did not materialise

here (that remains a single unreproduced occurrence).

Layer split: the 5060 Ti CANNOT be given its bandwidth share

Decode in --split-mode layer is a SEQUENTIAL pipeline (CUDA0's layers, then

CUDA1's), so the faster card idles waiting. Bandwidth says the time-optimal split

is ~61/39 (GDDR7 ~448 GB/s vs GDDR6 ~288 GB/s). It does not fit: 29 of 48

layers on the 5060 Ti is ~16.2 GiB against its 16311 MiB total.

Measured, n=3 each, same prompt:

| split | layers fast/slow | tok/s | 5060 Ti free | 4060 Ti free |

|---|---|---|---|---|

| 50,50 | 24 / 24 | 64.69 | 1374 MiB | 831 MiB |

| 52,48 | 25 / 23 | 64.88 | 802 MiB | 1405 MiB |

+0.3% = noise, exactly as [[mmproj-caps-cuda0-tensor-split]] warns

("rebalancing bought 1.3%"). One layer is 2% of the work; at a 1.55x speed ratio

that cannot be worth more than ~1%.

52,48 IS KEPT ANYWAY, for RELIABILITY not speed — it moves the ~800 MiB

squeeze onto the 5060 Ti and leaves the 4060 Ti 1405 MiB, and

[[nvrm-xid31-mmu-fault-4060ti]] tracks the 4060 Ti and presents as a fake OOM.

The gap is SUB-LAYER, so -ts cannot even it out. One layer is ~572 MiB

(444 weights + 128 KV); the imbalance was 543 MiB. Any move mirrors the gap

rather than closing it — predicted 802/1403, measured 802/1405.

Non-layer tensors did NOT follow CUDA0 here, contradicting

[[gpu-card-assignment-policy]]: with 24/24 layers the 4060 Ti (CUDA1) carried

612 MiB MORE, and the output head over a 250,624-token vocab is ~360 MiB of

that. Do not assume that rule holds for every arch.

Compute buffers are STATIC, sized by -b/-ub at load, not by prompt length.

A 40,627-token prompt cost 10 MiB of extra VRAM and returned correctly at

~1350 tok/s prefill. So a tight post-load figure is stable headroom, not a

margin that erodes as context fills — worth knowing before rejecting a config

for looking tight.

The arch string is spelled BOTH ways

HF config.json model_type: k2_horizon (underscore)

GGUF general.architecture: k2-horizon (HYPHEN)

The fork registers the hyphen form. **The re-check grep must search horizon,

not either exact spelling.** Also note grep -ri horizon in SGLang's models dir

false-positives on max_future_horizon / max_past_horizon (attention windows,

unrelated) — grep k2.horizon there instead.

Upstream is still empty (verified 2026-09-06)

master 74a7c897f has zero horizon hits, and a PR search returns

total_count: 0. The only upstream artifact is the bug report ggml-org#28361

("K2-Horizon models fail to load"), open, one comment that just re-quotes the

model card. Watch that issue, not a nonexistent PR. Delete ifm-llama.cpp/,

the macro and the entry once the arch lands.

The gate that WAS (kept for the backend-registry evidence)

Both members are K2HorizonForCausalLM / model_type: k2_horizon:

| backend | supported? |

|---|---|

| llama.cpp (our build 2026-08-23) | NO |

| llama.cpp MASTER (checked upstream) | NO — no "horizon" in src/llama-arch.{cpp,h} |

| vLLM | NO |

| exllamav3 | NO |

| SGLang | NO |

This is not a stale local build — upstream master has nothing either. IFM's

own GGUF card says so: *"require a version of llama.cpp containing K2 Horizon

architecture support. PR to llama.cpp is in progress"*, with support living

only at MBZUAI-IFM/llama.cpp/tree/model/K2Horizon.

~~DO NOT build the vendor fork for a trial~~ — SUPERSEDED, see the top.

The stated reason (it would shadow the mainline binary) was wrong: prism_llama_bin

already does exactly this. [[nanbeige-4.2-3b]] may deserve the same second look.

RE-CHECK IS ONE GREP on each llama.cpp upgrade:

grep -i horizon llama.cpp/src/llama-arch.cpp — search horizon, not k2_horizon

(the GGUF spells it with a HYPHEN).

Which member is worth trying when it lands: the MoVA, not the 32B

| model | shape | KV @ q8 | note |

|---|---|---|---|

| K2-Horizon-32B | DENSE, 64 layers, hidden 5120 | 128 KiB/tok | dense 32B -> ~20 t/s class, same trap as [[granite-4.2-30b]] |

| K2-Horizon-MoVA-36B-A4B | MoE, 100 experts / 8 active, 48 layers | 96 KiB/tok | ~4B active -> driver-class speed |

Both claim 524288 context; max_position_embeddings is 524288 but at 96-128

KiB/token that is 48-64 GiB of KV — **the real window here is KV-bound, not

config-bound**, exactly as for [[granite-4.2-30b]].

"MoVA" = Mixture-of-Values attention. ~~a NOVEL attention mechanism, so the

port needs real kernel work and the GQA-derived KV arithmetic may not hold~~ —

WRONG, see correction 3 at the top. Only the V projection is routed; the

attention call and KV cache are standard, so **the KV arithmetic above IS

correct** and was confirmed on the running model.

Weights — SERVED from our own quant

/mnt/models/gguf/K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf, 20.83 GiB, quantized

2026-09-06 from IFM's own BF16 (IFM/K2-Horizon-MoVA-36B-A4B-GGUF, 69.78 GiB,

the ONLY file that repo ships) using the fork's llama-quantize — mainline's

cannot parse the tensors either. Source BF16 kept at

/mnt/wd1tb/models/gguf/ifm-bf16/.

Q4_K_M over Q3_K_M despite Q3 fitting 128k at q8 KV, because low-bit builds hide

defects here ([[tommy2-iq2m-grill]] office 0/9 at 2-bit; laguna-s expr_eval 0/20

at 3-bit). Q4_K_M + q4_0 KV = 26.8 GiB predicted; 30.4 GiB measured resident,

which is tighter than the arithmetic suggested — compute buffers, not KV.

* Checkpoint_0002500 IS IFM'S OWN NAMING, NOT A THIRD-PARTY MISTAKE. *

The parked abenzerps Q4_K_M was initially distrusted because

general.name = Checkpoint_0002500 looked like a mid-training snapshot. **IFM's

official BF16 carries the identical string**, plus identical size_label

91x3.7B and identical 798 tensors — it is an artifact of IFM's conversion. So

that quant was probably fine and the 70 GB download bought PROVENANCE, not

correctness. **Read the official file's header before calling a third-party

quant suspect.**

Download gotcha, cost ~40 min: hf download stalled dead at 31 GiB

unauthenticated (process alive, 0 B/s), and WITH a token hung in Xet negotiation

at 0 bytes *and* used a different ETag, opening a fresh blob that stranded the

32.8 GiB already fetched. Both partials carried the same sha256 in the filename,

so content was identical and only the transport differed. Fix: plain

curl -C - with an Authorization: Bearer header against the classic

/resolve/main/ URL, seeded by renaming the stranded partial into place.

Author benchmark claims have NOT transferred on this box — the card leans on

tau3-Banking, and [[gemma-4-12b-agentic-yuxinlu1]] was rejected with "the

author's tau2-telecom claim did not transfer". Grill before believing.

The reusable lesson (4th instance in one day)

The architecture registry gates before capacity does. Check the arch list

BEFORE costing out a quant or a RAM upgrade: FreeToken rejecting

Qwen3NextForCausalLM ([[freetoken-ram-ceiling]]), exllamav3 lacking

Glm4MoeLite ([[exl3-tabbyapi-backend]]), SGLang having no fast MLA kernel

([[sglang-tp2-mixed-arch-norm-kernel]]), and this.