Kept / trial active in llama-swap.yaml · aliases: north-mini-code-1.0-cohere2moe-488k
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| 20260728-085350 | 22/23 | 174.1 | — | results-north-mini-20260728-085350.log |
unsloth/North-Mini-Code-1.0-GGUF:UD-Q4_K_XL, 17.93 GiB. Arch cohere2moe
(Cohere2MoeForCausalLM) — supported in b10133 (LLM_ARCH_COHERE2MOE +
src/models/cohere2moe.cpp). Real sparse MoE: 128 experts, 8/tok. 49 layers,
32 q-heads / 4 kv-heads, head_dim 128, vocab 262144, rope_theta 50000.
Port 9115. 79-94 t/s decode, ~356-616 t/s prompt.
layer_types interleaves 1 full-attention layer per 3 sliding
(sliding_window: 4096), so only 13 of 49 layers grow with context; the
other 36 are capped at 4096 tokens. With 4 kv-heads:
``
KV/token/layer = 4 x 128 x 2 = 1024 elements
q8_0 @ 500k = 13 x 1024 x 1.0625 x 500000 = 6.59 GiB
+ 36 sliding layers, capped = 0.15 GiB -> ~6.74 GiB
`
So 17.93 GiB weights + ~6.7 GiB KV fits the 31 GiB pool with room — the **full
native window at q8_0**, not the usual q4 compromise. First time on this box that
maximum context and quality-first KV were not in tension (cf.
[[vram-spend-priority]]).
NEEDLE 3/3 PASS at 440,181 tokens (depths 0.1/0.5/0.9, answer in content,
~710s each) — bench/needle-north-mini-20260728.log. ~1.7x the previous best
verified recall here (262144 on [[kimi-distill]]).
> READ THE 2026-07-28 UPDATE AT THE BOTTOM BEFORE QUOTING ANY NUMBER HERE.
> This section was written from a single run and overstated the case. Corrected
> summary: 22/23 on the 23-task suite, R5 4-6/6 across 6 runs (mean ~4.8),
> with interrupt_replan failing ~5/6 on runaway verbosity. Loops occur in ~3/6
> runs on varying tasks — not the one fixed search_navigate loop claimed below.
First run measured R5 4/6 with a loop on search_navigate (worst-repeat 4,
verified via recompute_r5_redundancy.py, which also cleared 3 false positives,
22.5% -> 15.0%). Failures were runaway verbosity, not confusion:
interrupt_replan burned 16,682 tokens, expr_eval the full 16,000 cap.
The direction holds — the 23-task score says top-tier and round 5 disagrees, the
pattern MODEL-RANKING's headline calls out for coder. **Route long-document
reading to north-mini; route agentic work to [[pocket-35b]] / [[kimi-distill]]**
(both 22/23 + clean 6/6 at ~102 t/s).
Heavy reasoner: 659 tok / 2438 chars of thought for a trivial task, and at
max_tokens=700 it hits the cap MID-THINK and returns an EMPTY content with
finish=length. Short-budget clients will look broken.
, -ub 512)loaded but printed compute buffer allocation failed, retrying without
pipeline parallelism and left 180MiB free on CUDA1. Fixes, both needed:
-ub 512 -> 256 (vocab 262144 makes the logit buffer ~536MB at ub 512 vs
~268MB at ub 256 — the dominant term) and split 45,55 -> 50,50. Then clean:
1060/1104 MiB free, static across generation.
[[jinja-system-guard-tool-parser]] guard, unlike the Qwen-family templates),
prose-with-tools 200 (not [[llama-cpp-peg-native-tool-parser-500]]). No
--chat-template-file needed.
three files all named IQ4_XS at 4.14/4.27/5.64 bpw. IQ4_XS is a fixed
~4.25bpw format, so the names do not describe the contents and an -hf
repo:IQ4_XS selector is ambiguous across three matches.
The first needle run died with KeyError: 'choices', which reads like a model
failure and was not:
1. The ~13 tok/line constant is tokenizer-specific. This tokenizer gives
16.00 tok/line, so needle.py 450000 built 34,615 lines = ~554k actual
tokens and overflowed the 500k window. Measure first, then
TARGET = (desired / measured_tok_per_line) * 13.
2. /tokenize 404s through llama-swap (8090) — only OpenAI routes are
proxied. tok_count() swallows the exception and returns -1, silently
disabling the size check that would have caught it. Point NEEDLE_BASE at
the backend port.
Both recorded as MODEL-RANKING caveat 13.
The "R5 4/6 with a genuine loop on search_navigate" above came from one run.
Six runs of identical weights (3 solo, 1 x2 single-stream, 2 x2 concurrent):
4/6, 6/6, 5/6, 5/6, 5/6, 4/6 — mean ~4.8/6, spread 4-6, loops in 3 of 6 runs
landing on *different* tasks.
What actually reproduces: interrupt_replan fails ~5 of 6 runs, always by
runaway verbosity (16,682 / 3,920 / 8,445 / 8,679 / 3,805 output tokens).
search_navigate fails 2/6. So the durable signal is *which task fails
repeatedly*, not the aggregate or which run tripped the loop detector.
Conclusion unchanged in direction (still not the agentic pick — [[pocket-35b]] is
22/23 with a clean 6/6 at 102 t/s) but the reason is verbosity-driven task
failure, not a specific loop. See MODEL-RANKING caveat 14: **a single R5 number
is a sample, not a measurement** — which also qualifies every other single-run
R5 recorded that day.
north-mini-x2 (2-slot, 250112 tok/slot, port 9116): loads clean with MORE
headroom than solo (1468/1512 MiB free — halving per-slot ctx shrinks compute
buffers), 1.38x aggregate throughput on straight generation. Grill 21/23, R5 6/6
single-stream; concurrent R5 5/6 + 4/6 — inside the solo range, so **no
concurrency-specific failure mode**. Scope it to parallel long-context READING;
send concurrent agentic work to [[kimi-distill-x2]].
bench/metric_sweep.py north-mini --rounds 5 --repeats 3 →
bench/metric-sweep-20260801-211211.jsonl. Base tier 4/6, 4/6, 5/6; full
base+hard mean 7.7/10, 1.3 loops per repeat, 3 flaky tasks (interrupt_replan,
registry_cascade, selective_rename).
Running tally is now 9 runs: 4,6,5,5,5,4,4,4,5 → mean ~4.7/6. The per-task
signal SHIFTED and is worth noting: search_navigate failed 3/3 here (was
2/6 before → 5/9 overall), twice with stop=loop at 5 and 6 redundant
calls; interrupt_replan failed 2/3 (→ 7/9 overall), once looping and once
running away to 8780 tokens. So BOTH tasks are unreliable, and the earlier
reading that search_navigate was the milder of the two no longer holds.
Verdict unchanged and now well-supported at n=9: not the agentic pick. The
6 tasks that pass in every repeat do so cleanly — 38 steps, 5208 tokens
(4552-5847), 0.0 redundant, 67s — so when it works it is lean; it is the
tail that disqualifies it.
PROCESS NOTE (cost real time this session): I proposed this re-grill arguing
the dismissal "rested on n=1 that a later run contradicted." That was WRONG —
this file already documented 6 runs. I had read the north-mini-x4 yaml comment
instead of this memory file. CLAUDE.md says read the fact file before touching
the area it covers; doing so would have reframed the work as "add repeats to an
existing 6-run tally", not "overturn a single-run verdict".
north-mini-x2 was RESTORED to llama-swap.yaml on 2026-08-01 (it had been
replaced by an unvalidated north-mini-x4` in b9bccfe1). Re-load-tested: cold
load 85.6s from ssd, idle 1454/1500 MiB free, **static 1454/1498 under two
concurrent generations**, both slots finish=stop — consistent with the 1468/1512
recorded above. Both x2 and x4 now coexist as a menu choice.
"north-mini":
aliases: [north-mini-code-1.0-cohere2moe-488k]
# unsloth/North-Mini-Code-1.0-GGUF:UD-Q4_K_XL (17.93GiB) — TRIAL 2026-07-28.
# CohereLabs North-Mini-Code-1.0. ARCH: cohere2moe (Cohere2MoeForCausalLM) —
# SUPPORTED in this build: LLM_ARCH_COHERE2MOE is registered in
# llama.cpp/src/llama-arch.cpp with a dedicated src/models/cohere2moe.cpp.
# Real sparse MoE: 128 experts, 8 per token. 49 layers, 32 q-heads / 4 kv-heads,
# head_dim 128, vocab 262144, rope_theta 50000.
#
# THE POINT OF THIS ENTRY IS THE CONTEXT. max_position_embeddings is 500000 and
# the architecture actually pays for it: layer_types interleaves 1 full_attention
# layer per 3 sliding_attention layers — only **13 of 49** layers grow with
# context; the other 36 are capped at sliding_window=4096. With 4 kv-heads that
# makes long context extraordinarily cheap:
# KV/token/layer = 4 x 128 x 2 = 1024 elements
# q8_0 @ 500k = 13 layers x 1024 x 1.0625 x 500000 = 6.59GiB
# + 36 sliding layers capped at 4096 = 0.15GiB -> ~6.74GiB
# So 17.93GiB weights + ~6.74GiB KV = ~24.7GiB, leaving ~6GiB in the 31GiB pool.
# That is why this runs the FULL native context at **q8_0** KV rather than
# dropping to q4 — quality-first per [[vram-spend-priority]], for once affordable
# at 488k. Longest context on the box by ~2x (previous best 262144).
#
# VERIFY AT LOAD: the numbers above assume llama.cpp applies its SWA KV
# optimisation to this arch. If it instead allocates full KV for all 49 layers,
# that is 49 x 1024 x 1.0625 x 500000 = ~26.6GiB and it will NOT fit. Check
# measured free VRAM against the ~6.7GiB prediction; if it is ~26GiB, SWA is not
# being applied — drop -c to 262144 rather than fighting it.
#
# MEASURED 2026-07-28 — SWA IS applied, the full 500000 ctx loads at q8_0 KV.
# But the FIRST attempt (45,55 / -b 2048 -ub 512) was a DEGRADED load: it printed
# "compute buffer allocation failed, retrying without pipeline parallelism" and
# left only 180MiB free on CUDA1 vs 2890MiB on CUDA0. Two fixes, both needed:
# * -ub 512 -> 256. vocab is 262144, so the logit buffer is vocab x ub x 4B =
# ~536MB at ub 512 vs ~268MB at ub 256 — the single biggest compute-buffer term.
# * split 45,55 -> 50,50 to correct the starved CUDA1.
# Result: CLEAN load (no pipeline-parallelism warning), 1060/1104 MiB free,
# static across generation. 94.1 t/s decode, ~356 t/s prompt.
# TOOLS: all three probes clean — single-turn 200, MULTI-turn non-first system
# 200 (no [[jinja-system-guard-tool-parser]] guard, unlike the Qwen-family
# templates), and prose-with-tools 200 (not affected by
# [[llama-cpp-peg-native-tool-parser-500]]). No --chat-template-file needed.
# HEAVY REASONER: budget max_tokens generously. Answering the merge_intervals
# task took 659 completion tokens with 2438 chars of reasoning; at max_tokens=700
# it hit the cap MID-THINK and returned an EMPTY `content` with finish=length.
#
# SOURCE NOTE: use unsloth, NOT byteshape/North-Mini-Code-1.0-GGUF — that repo
# ships three files all named "IQ4_XS" at 4.14/4.27/5.64 bpw (IQ4_XS is a fixed
# ~4.25bpw format, so 5.64 is mislabelled), which also makes an -hf `:IQ4_XS`
# selector ambiguous.
# STORAGE (2026-08-03): switched from `-hf` to the NVMe copy by absolute path,
# matching what north-mini-x2 and north-mini-x8 already did. The `-hf` form
# resolved through ~/.cache/huggingface/hub, whose repo dir is a SYMLINK to
# /mnt/ssk500 (the SSK ssd) — so the solo entry was the only one of the three
# still cold-loading 17.93GiB at the ssd's 0.23 GiB/s (85.6s measured) while
# its siblings read the identical file from nvme at 1.08 GiB/s (16.6-16.9s
# measured). Same weights: both copies verified byte-identical 2026-08-03
# (equal size, head-512MB `cmp`, tail-512MB sha256 both 6de8554fb30827c1).
# Bonus: -m sidesteps HF resolution entirely, so this entry can no longer hit
# the offline dangling refs/main death in [[llama-cpp-offline-hf-refs]].
# The ssd blob is left in place (18GB) — deleting it is a separate decision;
# nothing references it once this entry is repointed.
# STORAGE TIER: moved off the NVMe to /mnt/ssk500 on 2026-08-08. This is a
# spare, not a driver, so it pays the slow-disk penalty: measured 380 MB/s
# on ssk500 vs 2.5 GB/s on the NVMe, i.e. ~+60-100s on a cold load (a 16 GB
# vLLM model took 196s from ssk500 vs 98-136s from the NVMe).
# NVMe RESIDENCY as of 2026-08-12: `coder-agentic` + `fable-fusion` (both
# promoted back today). muse-glimmer was moved OFF to ssk500 deliberately.
cmd: |
/usr/bin/env
CUDA_DEVICE_ORDER=PCI_BUS_ID
${llama_bin}
-m /mnt/ssk500/gguf/North-Mini-Code-1.0-UD-Q4_K_XL.gguf
--alias north-mini
--jinja -ngl 99 -c 500000 -fa on
--tensor-split 50,50
-b 1024 -ub 256
--temp 0.2 --top-p 0.9 --top-k 40
--cache-type-k q8_0 --cache-type-v q8_0
--host 127.0.0.1 --port 9117 --parallel 1
proxy: http://127.0.0.1:9117