gpt-oss-20b

Promoted active in llama-swap.yaml · aliases: gpt-oss-20b-mxfp4-4060ti-131k

Grill run history

Run	Score	tok/step	R5	Log
No results-*.log found for this id/alias.

Memory notes

gpt-oss-20b (gpt-oss-20b.md)

gpt-oss-20b — UN-RETIRED 2026-08-29 (one-shot coder only)

RESOLVED 2026-09-01: the loops are REAL, but a CRASH was inflating the score

Tested properly — FULL R5, n=3 per arm, c=32768, single GPU, only --swa-full

differing. Two separate effects, and they point opposite ways:

|---|---|---|---|

| --swa-full | 0/3 | 4/6, 4/6, 4/6 | search_navigate x3, +multifile_refactor, +interrupt_replan |

1. --swa-full eliminates a crash that was inflating the failure count. Every

rolling run died with an illegal CUDA access —

ggml_cuda_mul_mat_q (mmq.cu:239) → NVRM Xid 31, see

[[nvrm-xid31-mmu-fault-4060ti]] — and reported err=2, i.e. **2 of 6 tasks never

reached the model**. The recorded 4/6, 3/6, 3/6 was very likely measuring the

same crashes, not the model.

2. The loops are REAL and survive the fix — THE VERDICT STANDS.

search_navigate looped in 6 of 6 runs across both arms. "Never wire it to

tool loops" is correct; only the *score* was wrong.

FIXED 2026-09-01 — --swa-full IS NOW IN THE ENTRY, AT THE FULL 131072. I

predicted it would not fit; measured, it does, and it costs nothing:

| ctx | free VRAM | decode |

|---|---|---|

| 131072 | 1030 MiB | 87.71 tok/s |

| 98304 | 1974 | 87.45 |

| 65536 | 2918 | 87.93 |

| 49152 | 3390 | 87.59 |

Decode is identical everywhere — the window costs VRAM only. 1030 MiB is under

the ~1400 comfort line but 2.2x the 466 MiB where cuBLAS actually died, so it was

validated under load rather than assumed: full R5 x2 at 131072, 0 crashes,

free VRAM steady at 1014 MiB through both rounds, no err=. Verified live through

llama-swap at 87.58 tok/s. Fall back to 98304 (1974 MiB free, same speed) if

that margin ever proves thin.

A LESSON ABOUT n=1, AT MY OWN EXPENSE: while diagnosing

[[gpt-oss-120b-cpu-moe-spill]] I ran this model *once* on interrupt_replan, saw

no crash, and concluded "the 20b does not crash, which implicates the -ncmoe

spill path." Wrong. It runs single-GPU with zero CPU offload and crashes 3/3.

The bug is in gpt-oss rolling-SWA handling generally. One run of a stochastic

failure is not evidence of absence — the same rule

[[round5-is-a-sample-not-a-measurement]] states for scores applies to crashes.

---

History below — the 2026-07-27 retirement and the original entry

Removed from the box 2026-07-27 ([[2026-07-27-model-cleanup]]): both the

coder and coder-x4 entries deleted from llama-swap.yaml, the 14GB F16

weights wiped from the HF cache. Rationale: grill round 5 (long-horizon

agentic) caught genuine tool-call loops on both entries — no longer safe for

agentic work despite the strong 23-task score. coder-quick and

qwen3-coder-next-80b-160k both beat it on single-shot coding with clean R5

runs. Everything below is history.

Original entry (ADDED, top performer)

OpenAI's open MoE, added to llama-swap.yaml (port 9105), models.json, statusline-command.sh 2026-07-10.

Architecture

MoE 21B total / 3.6B active per token (32 experts, top-4 routing) — "gpt-oss" arch
F16 GGUF is 13.8GB but MoE layers are natively MXFP4, so all quants barely shrink

(Q2_K 11.5GB vs F16 13.8GB). Pick F16 — near-unquantized for only ~2GB more.

OpenAI "Harmony" channel format (analysis/commentary/final); EOS <|return|>.
Reasoning effort low/medium/high in the Harmony system prompt (default medium).

Config (chosen)

Context 128K = n_ctx_train, the CEILING (256k gets capped to 128k). VRAM is NOT the

limit — 128k q8_0 uses only ~9.5GB/GPU, ~13GB free. No --tensor-split (cards equal 16+16).

KV q8_0 (quality-first; spare VRAM affords it). Loads ~7s, no pipeline fallback.
Sampler temp 0.2 for coding determinism. Card's native is temp 1.0/top-p 1.0/top-k 0,

but 0.2 gave the perfect coding score.

Decode ~70-82 t/s (MoE-class fast, same league as the qwen3.6/ornith/agents 35B MoEs).

Grill 2026-07-10 (temp 0.2)

Single direct run (port 19999): PERFECT 23/23 — FIRST on this box.

Then HEAVY 3-pass run via the real swap endpoint (port 8090) to test stability:

Pass 1: r1 7/8, r2 5/5, r3 5/5, r4 5/5 = 22/23
Pass 2: r1 8/8, r2 5/5, r3 5/5, r4 5/5 = 23/23
Pass 3: r1 7/8, r2 5/5, r3 5/5, r4 5/5 = 22/23
r2/r3/r4 are a ROCK-SOLID 5/5 EVERY pass (algorithmic-2, tool-call/format, agentic loop).

Agentic loop lean & consistent: ~900 out_tok / 10 steps / ~14s wall each pass.

TWO flaky tasks, measured over 7 full runs (see the 2026-07-26 section below).

1. expr_eval — always the SAME bug: unary-minus vs power precedence, -2**2

returns 4.0 instead of -4.0 (Python evals -(22)). 3 passes in 7** across

both builds: coin-flip, and the reason this model reads 22-23/23 rather than a

solid 23.

2. median_two_sorted — the single-element edge case (a=[-16],

b=[-13,-4,8,11] -> returns -13.0, expected -4). Never failed in July; **2

failures in 9 observations on 2026-07-26**. Rarer than expr_eval but real.

DEAD HEURISTIC — do not repeat it: an earlier version of this note claimed

expr_eval "burns way more tokens when it passes (5516/79s) than when it fails

early (2488/35s)". That came from a 3-run sample and does NOT hold. Full table:

pass 2 FAILED at 4800 tok — more than pass 1 spent PASSING at 4458 — and

pass 3 passed at just 2375. Token count predicts nothing about the outcome.

Net: ~22-23/23 every pass. Best pure coder AND agentic generalist on this hardware. Compare to [[deepseek-r1-distill-llama-70b]]

and [[kimi-dev-72b]] (both REJECTED dense models) — this MoE dominates on every axis.

2026-07-26 — 3-pass single-slot re-grill on b9870 (`70543a4d`)

The July passes were on the PRE-b9870 binary, and uncensored demonstrably changed

behaviour across that upgrade, so they could not serve as a same-build baseline.

Full re-run, 1 slot / 128k, nothing else on the box:

|---|---|---|---|---|

| 1 | 23/23 | 17024 | 3.8 min | — |

| 2 | 21/23 | 17685 | 4.0 min | median_two_sorted, expr_eval |

| 3 | 23/23 | 15598 | 3.4 min | — |

**Mean 22.3/23 — identical to July's 22.3 (22, 23, 22). The build change neither

helped nor hurt this model.** Rounds 2-4 were 5/5 in all three passes, as in every

run on record; ALL variance is round 1, in the two tasks above.

4th data point, 2026-07-26 — run as `coder-x4` (4 slots x 32k)

Full grill on the parallel entry ([[parallel-agent-slots]]):

bench/results-coder-x4-20260726-140322.log = 21/23 (r1 6/8, r2 5/5, r3 5/5,

r4 5/5), 17.3k tokens / 4.7 min.

expr_eval failed again on the exact documented bug (-2**2 -> 4.0). Across all

seven runs it is 3 passes in 7, so quote this model as **22/23 typical (22.3

mean), 23/23 at best** — NOT a reliable 23/23.

median_two_sorted failed once here (a=[-16] b=[-13,-4,8,11] -> -13.0, expected

-4) — a NEW task to see fail. Re-ran it 5x: 5/5 PASS, so not a 32k-window

regression. NOTE: the single-slot passes later failed it too, so it is a genuine

second flaky task at ~2-in-9, NOT the one-off it looked like here.

Conclusion: the 32k slot window is FREE — 21/23 sits inside the single-slot

distribution (23/21/23) measured the same hour on the same build.

NOTE: llama-server is a SYSTEM service, but a restart is NOT needed to pick up config

changes — kill -HUP $(systemctl show llama-server -p MainPID --value) reloads in

place without sudo (verified 2026-07-26). See [[llama-swap-setup]] and

[[vram-spend-priority]].

llama-swap.yaml entry

  "gpt-oss-20b":
    aliases: [gpt-oss-20b-mxfp4-4060ti-131k]
    # OCCASIONAL-USE CODER, not a daily driver — lives on /mnt/models (cold tier)
    # per [[storage-tiers]], not /mnt/ssk500.
    #
    # ggml-org MXFP4 GGUF (12.11 GB) = the checkpoint's NATIVE precision. The
    # whole unsloth quant ladder is pointless here: Q2_K_L 11.76 GB through F16
    # 13.79 GB barely differ, because the experts are ALREADY MXFP4 and only the
    # non-expert tensors requantize. Take the native build.
    #
    # PREFER THIS OVER gpt-oss-20b-ft. Measured 2026-08-29, identical prompt, n=3:
    #     llama.cpp here   87.53 tok/s   (87.59 / 87.38 / 87.63)
    #     FreeToken :9162  60.17 tok/s   (-31%)
    # The model is 12 GB and fits ENTIRELY in one 16 GB card, so llama.cpp keeps
    # every expert resident while FreeToken streams them from pinned host banks —
    # machinery that only pays off for models BIGGER than VRAM. The -ft entry
    # exists for parity, but this is the fast path.
    #
    # 13,187 MiB on the 4060 Ti at the FULL 131072 window with q8_0 K AND V
    # (this arch accepts q8_0 V — no f16 fallback needed), ~3.1 GB headroom.
    # Pinned CUDA_VISIBLE_DEVICES=1 = the 4060 Ti ([[cuda-device-order-pci-bus-id]]),
    # single-GPU, so it coexists with a FreeToken driver on the 5060 Ti.
    #
    # *** OUTSTANDING ONE-SHOT CODER, POOR AGENT *** — grilled 2026-08-29
    # (bench/grill-gpt-oss-20b-20260829-1046): coding 22/23, realcase 7/7,
    # KLayout+ref 7/8. But R5 run 3x gave complete 4/6, 3/6, 3/6 with ONE GENUINE
    # LOOP (search_navigate, 14 steps, 8 redundant) — this is the model that was
    # RETIRED from this box for exactly that. Promoted on coding strength with the
    # agentic weakness known. Use for generate/refactor/explain; do NOT wire it to
    # long-horizon tool loops. Also: one 19,845-token runaway (expr_eval) and 2
    # <|constrain|> control-token leaks in 186 turns.
    #
    # *** --swa-full IS MANDATORY: WITHOUT IT THIS ENTRY CRASHES THE GPU ***
    # Added 2026-09-01 after a FULL R5 A/B, n=3 per arm, c=32768, only this flag
    # differing:
    #     rolling SWA (as this entry shipped)  CRASHED 3/3, complete 3/6 x3, err=2 each
    #     --swa-full                           CRASHED 0/3, complete 4/6 x3
    # Every rolling run died with an illegal CUDA access in ggml_cuda_mul_mat_q
    # (mmq.cu:239) -> NVRM Xid 31, losing 2 of 6 tasks to err=2. THE HISTORICAL
    # 4/6, 3/6, 3/6 BELOW WAS VERY LIKELY MEASURING THOSE CRASHES, not the model.
    # See [[nvrm-xid31-mmu-fault-4060ti]]; this is NOT the -ncmoe spill path (this
    # entry has no CPU offload at all, single GPU, 13 GB fully resident).
    #
    # THE LOOPS ARE STILL REAL — the verdict below STANDS. search_navigate looped
    # in 6 of 6 runs across BOTH arms. --swa-full fixes the crash, not the looping.
    #
    # IT COSTS NOTHING HERE. --swa-full doubles KV (18 of 36 layers are full-attn
    # -> all 36), but the FULL 131072 window still fits and decode is unchanged.
    # Measured on the 4060 Ti, q8_0 KV, free VRAM after load:
    #     131072  1030 MiB free  87.71 tok/s   <- this entry, VALIDATED
    #      98304  1974 MiB       87.45
    #      65536  2918 MiB       87.93
    #      49152  3390 MiB       87.59
    # 1030 MiB is under the ~1400 comfort line but 2.2x the 466 MiB where cuBLAS
    # actually died, so it was VALIDATED UNDER LOAD rather than assumed: full R5
    # x2 at 131072, 0 crashes, free VRAM steady at 1014 MiB through both rounds,
    # no err=. Drop to 98304 if that margin ever proves too thin.
    #
    # SAMPLER — ALL THREE VALUES ARE MEASURED, NOT INHERITED.
    # --top-k 0 is EXPLICIT: llama.cpp's default top-k is 40, so omitting the
    # flag does NOT disable it.
    # --top-p 1.0 (disabled) is worth 12%: this vocab is 201,088 tokens and
    # top-p sorts/cumsums all of it every token, while 1.0 skips the path.
    #     top-p 0.9  77.0 tok/s      top-p 1.0  86.2 tok/s
    # Temperature is free (86.3 at temp 0.7 vs 86.0 at 0.2), so --temp 0.2 costs
    # nothing and keeps decoding tight.
    # THE TAIL-RISK WORRY WAS TESTED AND REJECTED. Concern was that disabling
    # top-p would let rare tail draws seed runaways/loops on a model that has
    # both. A/B'd round1+round5 x3 per arm, identical servers, top-p the only
    # variable:
    #     top-p 0.9   R5 complete 13/18  looped 4   r1 cap-hits 1/3  score 7/8 x3
    #     top-p 1.0   R5 complete 14/18  looped 2   r1 cap-hits 1/3  score 7/8 x3
    # Each arm hit the 32,000-token cap exactly once; scores identical in all six
    # runs. No detectable difference, so top-p is NOT the lever for this model's
    # runaways -- do not re-add it as a guard without new evidence.
    # (--repeat-penalty is also absent: the 22/23 grill was measured without it.)
    # REASONING MODEL (harmony analysis channel): budget max_tokens generously or
    # a short cap returns EMPTY content. --reasoning-format deepseek splits it
    # into message.reasoning_content.
    cmd: |
      /usr/bin/env
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      CUDA_VISIBLE_DEVICES=1
      ${llama_bin}
      -m /mnt/models/gguf/gpt-oss-20b-MXFP4.gguf
      --alias gpt-oss-20b
      --jinja
      -ngl 99 -c 131072 -fa on
      -b 2048 -ub 512
      --temp 0.2 --top-p 1.0 --top-k 0
      --cache-type-k q8_0 --cache-type-v q8_0
      --swa-full
      --reasoning-format deepseek
      --host 127.0.0.1 --port 9174 --parallel 1
    proxy: http://127.0.0.1:9174

Generated 2026-09-12 07:52 by web-server/models.coolboxapp.com/generate.py from llama-swap.yaml, bench/results-*.log, bench/grill_out/lmeval-mini/ and auto-memory/ on server.