Active active in llama-swap.yaml · aliases: gpt-oss-120b-mxfp4-cpumoe-131k
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| No results-*.log found for this id/alias. | ||||
gpt-oss-120b-MXFP4.gguf, 63.39 GB, on /home/giovanni/models-nvme (real
NVMe — the ONLY viable tier, since host-resident experts page off this file on
every forward pass; /mnt/models and /mnt/ssk500 are USB, see
[[storage-tiers]]). Entry on port 9177. NOT YET GRILLED for capability —
speed and fit only, and this lineage was retired once for R5 loops
([[gpt-oss-20b]]).
Arch: 36 layers, 128 experts top-4 (5.1B active), **18 full-attn + 18 sliding
with a 128-token window, vocab 201,088. Native context 131072** (GGUF
gpt-oss.context_length) — but it is YaRN x32 from an original 4096, so the
deep end is extrapolated, NOT trained. Retrieval past ~32k is UNTESTED here;
[[qwen36-27b-deep-retrieval-fails]] is the precedent (3/3 at 32k, 0/3 at 222k).
Combined with ~19 min to prefill 131k, treat the top of this window as
theoretical until a needle run says otherwise. Experts are 1.585 GiB/layer
(128 x 3 x 2880^2 params at 4.25 bpw) — the unit of all the arithmetic here.
VRAM: (36-ncmoe) x 1.585 GiB + ~3.2 GiB fixed <= ~30 GiB usable
RAM: ncmoe x 1.585 GiB <= ~27 GiB page cache
VRAM wants ncmoe >= 20, page cache wants ncmoe <= 15. **They are ~5 layers
apart**, so at EVERY working config several GB of experts page off NVMe
continuously. Not tunable away — it is the model against this box. ncmoe 18
is arithmetically impossible (the constraints cross) and OOMs.
Measured ladder (ctx 16384, warm decode): ncmoe 36 = 5.05 tok/s (3.7 GB VRAM,
28 GB idle — the runbook's start point wastes the whole box), 30 = 7.53,
26 = 10.84, 24 = 11.76, 21 = 19.72/20.70/18.49 (n=3), 20 = 21.14 (fastest,
needs q8_0 KV). Reference the same hour: [[gpt-oss-20b]] = 86.0 tok/s.
1. --tensor-split is NOT a layer split under -ncmoe. The first N layers
are nearly weightless, AND ~3.2 GiB of non-layer tensors (embeddings, output
head, KV, compute buffers) land ENTIRELY on CUDA0. So CUDA0 needs **~2 FEWER
heavy layers** than CUDA1. An even-looking 82,18 wasted 4.3 GB
(13511 vs 9235 MiB); rebalancing to 75,25 took decode 11.76 -> 19.72 at the
SAME ncmoe. Same trap as [[laguna-s-2-1-cpu-spill]], which is where to look
first next time.
2. q8_0 KV bought COMPUTE, not context (still true, but see --swa-full
below: the ncmoe=20 config it unlocked is NOT the served one). Half the layers use a 128-token
sliding window, so KV is cheap and context is nearly free (32k cost nothing
over 16k: 18.78 vs 19.72; full 131072 costs only ~2 expert layers) — cf.
[[kv-sizing-full-attention-layers]]. What q8_0 ACTUALLY freed was the ~300 MiB
of cuBLAS headroom that let ncmoe drop a whole layer. At f16 KV, ncmoe=20
LOADED and then died on the first decode:
CUDA error: the resource allocation failed
current device: 1, in function cublas_handle
A health check passes on a config that cannot generate — same class as the
lazy ViT buffer in [[qwen35moe-35b-vram-ceiling]]. Always fire a real request.
3. Prefill measured on a short prompt is MEANINGLESS. An 80-token prompt
reads ~3-25 tok/s here; a 3,264-token prompt reads ~113 tok/s, flat across
every window. The short number is first-touch page faults, not throughput.
Nearly published "prompt processing is unusable" off the wrong one.
The served entry is ncmoe=24 --tensor-split 78,22 -c 131072 q8_0 KV --swa-full
at 10.95 tok/s, NOT the ncmoe=22 the speed ladder alone picks. Reason:
Without --swa-full this model CRASHES THE GPU. Half the layers are sliding
-window (128 tok); llama.cpp's ROLLING SWA cache on the -ncmoe spill path makes
an illegal CUDA access when conversation history is rewritten mid-run, surfacing
as NVRM Xid 31 — see [[nvrm-xid31-mmu-fault-4060ti]] for why that is nearly
undiagnosable without root.
Isolation on R5 interrupt_replan (the task that injects a requirement
mid-trajectory) — this is the reusable part:
| arm | crash | task |
|---|---|---|
| default (rolling SWA) | 5/5 at 131k AND 32k | no data |
| KV q8/q8, f16/f16, q8/f16 | 3/3 | not the cache type |
| --no-context-shift | none | FAIL — loop, 4 redundant |
| --swa-full | none | PASS 3/3, 0 redundant, 4-5 steps |
Two lessons beyond this model: (1) --no-context-shift stops the crash but
leaves the cache WRONG — the model then loops, which reads exactly like a model
weakness. Only --swa-full gives correct context. (2) It fired at a 40-token
prompt with <1k context, so "raise ncmoe" and "widen the split" were both
wrong fixes; **a crash on a spilled config is not automatically a memory
problem.**
Cost: KV at 131072/q8_0 goes 2.25 -> 4.50 GiB (~1.5 expert layers). At 131k,
ncmoe=23 has NO safe split (75,25 -> 756 MiB free; 78,22 -> 450 MiB; 80,20 will
not load) and cuBLAS died at 466 MiB, so 24 is the floor. ncmoe=22 -c 65536
is the fast alternative at 17.83 tok/s if the full window is ever not needed.
On the committed config through llama-swap, budget raised to 1800s/task (the
600s default is calibrated for 80-140 tok/s models; this one runs ~11):
FAIL multifile_refactor 15 steps 4 redundant LOOP
PASS test_driven_loop 5 steps 0
PASS search_navigate 3 steps 0
PASS constraint_recovery 6 steps 0
PASS interrupt_replan 4 steps 0
PASS already_done 4 steps 0
R5SCORE complete=5/6 looped=1/6 redundant=4 redundant_pct=10.8
Above [[gpt-oss-20b]] (4/6, 3/6, 3/6) and below the 6/6 tier ([[pocket-35b]],
[[glm-4.7-flash]], [[thomson-1.0-small-trial]]). **n=1, so per
[[round5-is-a-sample-not-a-measurement]] this is inside the noise band of a 6/6
model — do NOT quote it as a verdict.** multifile_refactor PASSED (13 steps,
1 redundant) in the pre---swa-full attempt and looped here, so the
looping-task IDENTITY — the part that is signal — is not established. Needs 2-3
runs.
Decode inside R5 ran 3.8-8.2 tok/s, not the 10.95 benchmark, because agent
turns carry accumulated context. That is the number real use feels.
ggml-org/gpt-oss-120b-GGUF also ships eagle3-gpt-oss-120b-Q8_0.gguf
(849 MB, arch eagle3, 1 block, NVIDIA licence) — it fits in the ~2 GB/card
that is left over. Arm set 1, --spec-draft-ngl left at its auto default:
| arm | decode (2 prompts) |
|---|---|
| base | 10.45 / 12.39 |
| -md, n-max 3 | 11.41 / 12.68 |
| n-max 5 | 11.41 / 12.50 |
| n-max 8 | 10.75 / 12.00 (WORSE) |
Forcing placement with -ngld all changed nothing: gpu3 10.42/12.12,
gpu5 9.38/12.96, and the decisive gpu5b — ncmoe raised to 25, freeing an
extra 1618 MiB on the 4060 Ti (2068 -> 3686) — still 10.45/11.08 with the
drafter declining to take the room. **Drafted mean 10.64/12.22 vs base
10.45/12.39 over 7 arms: NO GAIN. Do not wire it.**
The drafter never reaches VRAM under either auto or all; free memory is
byte-identical in all six drafted arms. It does load
(common_speculative_init_result: loading draft model, no error), but the
server also logs failed to measure the memory of the extra model, fitting
without it and eagle3 requires ctx_other to be set.
Why it probably cannot help here regardless (hypothesis, but it predicts the
data): speculative decoding amortizes COMPUTE, while this config is bottlenecked
on EXPERT PAGING. Verifying k draft tokens activates up to k*4 different experts
instead of 4, so host->GPU traffic GROWS with draft depth — which is exactly why
n-max 8 (10.75/12.00) is worse than n-max 3 and worse than base. Expect the same
for any drafter on any -ncmoe spill; this is a property of the spill, not of
eagle3.
SEQUENTIALLY for one request (no overlap, both ~10% util), so total decode
time is the SUM and the optimum equalizes TIME, not bytes. At fixed
ncmoe=22/131k: 78,22 = 13.63 tok/s and 7.11 after a 3.3k prompt; **75,25
= 14.31 and 10.93 (+54% on the long case)**; 72,28 OOMs. User's suggestion,
confirmed by measurement.
[[gpt-oss-20b]] sampler win does NOT transfer. --top-p 1.0 --top-k 0is worth 12% there on the same 201,088 vocab; here 18.49 vs 19.72 = nothing.
This config is bound by expert paging, not by sorting the vocab. Measure the
flag on THIS model, every time.
[[single-run-scores-are-unreliable]] applies to throughput too.
--reasoning-format deepseek works: content + reasoning_content splitcleanly through llama-swap.
--no-mmap): host-side experts exceedRAM, so they must stay file-backed and evictable or the box swaps.
pkill -f 'llama-server.*port 8080' **inside a Bash tool call kills its own shell** — the harness runs bash -c '<script text>', so the pattern matches
the caller's cmdline. Exit 144, instantly. Kill by the PID bound to the port
instead. Recurrence of [[pgrep-f-self-match-deadlock]] in a new place.
assumed had died had actually fired), and each one's startup pkill killed
the other's server — three mutual kills read exactly like a config failure.
Use flock. Cf. [[background-task-silent-kills]].
bench/ is gitignored but 144 result logs are tracked; new logs need git add -f, same as [[auto-memory-gitignore-force-add]].
Weights had been parked on /mnt/wd1tb (dead USB HDD, [[storage-tiers]]); that
drive died and was replaced by /mnt/ssd1700, a genuinely fast USB SSD
(~1.8 GB/s). Re-downloaded byte-exact from ggml-org/gpt-oss-120b-GGUF,
CUDA_DEVICE_ORDER=PCI_BUS_ID added to the entry per
[[gpu-card-assignment-policy]] (tensor-split 78,22 left UNCHANGED, per that
doc's tested rule — inverting is wrong), uncommented in llama-swap.yaml,
and verified standalone: loaded in ~44s (partly cache-assisted), free VRAM
2422/2613 MiB (comfortably above the ~1.4 GiB floor), generated correctly
once max_tokens was raised past the harmony-reasoning-channel floor. HUP-
reloaded into production llama-swap. Not re-grilled — the 2026-08-31
measurements above are assumed to transfer since nothing about the model
config changed, only the storage tier and GPU index mapping.
# ===== RE-ENABLED 2026-09-11 — weights moved off the dead /mnt/wd1tb HDD
# onto /mnt/ssd1700 (fast USB SSD, confirmed non-rotational, ~1.8 GB/s
# read; re-downloaded byte-exact from ggml-org/gpt-oss-120b-GGUF). This was
# a genuine storage-tier block: the -ncmoe spill path pages ~2/3 of the
# experts from disk on EVERY forward pass, and wd1tb's ~120-140 MB/s HDD
# was ~20x slower than the NVMe this entry was originally measured on
# ([[storage-tiers]], [[gpt-oss-120b-cpu-moe-spill]]). ssd1700 should be
# close to that original NVMe baseline — re-verify decode tok/s against
# the measured ladder below rather than assuming.
#
# CUDA_DEVICE_ORDER=PCI_BUS_ID ADDED (box-wide convention since 2026-09-01,
# [[gpu-card-assignment-policy]]) and the --tensor-split NUMBERS ARE
# DELIBERATELY UNCHANGED (78,22) — do NOT invert them. The policy doc
# tested inverting on this exact class of entry and it was WRONG: the
# ~3.2 GiB of non-layer tensors follows CUDA0 regardless of which physical
# card that is, so inverting hands the wrong card both the bigger weight
# share AND that overhead (measured 162 MiB min-free vs 1048 MiB
# unchanged, on a different entry — crash territory). Net effect here:
# CUDA0 is now the 5060 Ti, so the 78% share that used to sit on the
# (slower, 288 GB/s) 4060 Ti now sits on the (faster, 448 GB/s) 5060 Ti —
# which is the SAME direction this entry's own "SPLIT BIASED TOWARD THE
# FASTER CARD" measurement below argues for, so this is not just inertia.
# Still: RE-VERIFY THE FIT before trusting it under load, this has a
# documented crash history (Xid 31, see --swa-full note below) and the
# margins were measured in MiB, not GiB.
"gpt-oss-120b":
aliases: [gpt-oss-120b-mxfp4-cpumoe-131k]
# 116.8B MoE (128 experts, top-4, 5.1B active), 36 layers, native MXFP4.
# 63.39 GB — the host-resident experts are paged from this file on every
# forward pass, so it needs a genuinely fast tier; see the RE-ENABLED
# note above for why /mnt/ssd1700 now qualifies where /mnt/wd1tb didn't.
#
# *** THIS MODEL DOES NOT FIT. IT IS SERVED BY DELIBERATE SPILL. ***
# 63.39 GB against 32 GB VRAM + ~27 GB usable RAM. --n-cpu-moe keeps the MoE
# tensors of the first N layers on CPU; attention/embeddings/norms stay on the
# cards. Experts are 1.585 GiB/layer (128 x 3 x 2880^2 params at 4.25 bpw),
# which is the unit of ALL the arithmetic below.
#
# THE TWO CONSTRAINTS NEVER BOTH RELAX — measured 2026-08-31:
# VRAM: (36-ncmoe) x 1.585 GiB + ~3.2 GiB fixed <= ~30 GiB usable
# RAM: ncmoe x 1.585 GiB <= ~27 GiB page cache
# They are ~5 layers apart, so at EVERY working config several GB of experts
# page off disk continuously. This is not tunable away; it is the model.
#
# --tensor-split IS LOAD-BEARING AND IS **NOT** A LAYER-COUNT SPLIT HERE.
# -ncmoe makes the first N layers nearly weightless, so an even-looking split
# is wildly uneven in bytes ([[laguna-s-2-1-cpu-spill]] documents the same
# trap). On top of that ~3.2 GiB of non-layer tensors (embeddings, output
# head, KV, compute buffers) land ENTIRELY on CUDA0, so CUDA0 must be given
# ~2 FEWER heavy layers than CUDA1 to balance. Measured at ncmoe=24, split
# 82,18: 13511 MiB on CUDA0 vs 9235 on CUDA1 — 4.3 GB of waste.
# A = 78 originally mapped to the 4060 Ti; since the 2026-09-11 re-enable
# it maps to the 5060 Ti (CUDA0 flipped, split numbers didn't — see above).
#
# q8_0 KV IS BUYING COMPUTE, NOT CONTEXT. Half the layers use a 128-token
# SLIDING window (18 full-attn + 18 sliding), so KV is cheap and context is
# nearly free — 32k cost nothing over 16k (18.78 vs 19.72 tok/s). What q8_0
# actually bought was the ~300 MiB of cuBLAS headroom that let ncmoe drop a
# whole layer: at f16 KV, ncmoe=20 LOADED and then died on the first decode
# with "CUDA error: the resource allocation failed ... in cublas_handle"
# (466 MiB free). Health checks pass on a config that cannot generate.
#
# MEASURED LADDER (ctx 16384 unless noted, warm decode, n>=1, on NVMe):
# ncmoe 36 3.7 GB VRAM 5.05 tok/s <- runbook's start; 28 GB idle
# ncmoe 30 13.0 GB 7.53
# ncmoe 26 19.5 GB 10.84
# ncmoe 24 22.7 GB 11.76
# ncmoe 21 27.6 GB 19.72 / 20.70 / 18.49 (n=3)
# ncmoe 20 28.9 GB 21.14 (q8_0 KV, ts 74,26 — fastest overall)
# ncmoe 18 OOM, arithmetically impossible (constraints cross)
# Prefill is FLAT at ~113 tok/s on a 3.3k prompt regardless of window.
# NOTE: prefill measured on an 80-token prompt reads ~5-25 tok/s and is
# MEANINGLESS — it is dominated by first-touch page faults, not throughput.
#
# THIS ENTRY IS THE FULL-CONTEXT CONFIG, NOT THE FASTEST ONE. ncmoe=20 at
# 16k is 21.1 tok/s but 16k is useless for agent work; 131072 costs two
# expert layers and ~27% decode to gain 8x the window. Per [[vram-spend-priority]].
#
# *** --swa-full IS MANDATORY. WITHOUT IT THIS ENTRY CRASHES THE GPU. ***
# Half the layers are sliding-window (128 tokens). llama.cpp's ROLLING SWA
# cache, on the -ncmoe spill path, makes an ILLEGAL CUDA MEMORY ACCESS when a
# conversation's history is rewritten mid-run:
# CUDA error: an illegal memory access was encountered
# current device: 0, in ggml_backend_cuda_synchronize -> llama_decode
# which the kernel logs as NVRM Xid 31 MMU fault on the 4060 Ti. See
# [[nvrm-xid31-mmu-fault-4060ti]] — from userspace it looks EXACTLY like an
# OOM (llama-swap 502, "upstream process exited unexpectedly") and ONLY
# `sudo dmesg -T` distinguishes it.
#
# Reproducer + isolation, R5 `interrupt_replan` (the one task that injects a
# new requirement mid-trajectory), 2026-08-31:
# default (rolling SWA) CRASHED 5/5 -- at 131k AND at 32k
# KV q8/q8, f16/f16, q8/f16 all 3 CRASHED -> NOT the cache type
# --no-context-shift no crash, but task FAILS (loop, 4 redundant)
# --swa-full PASS 3/3, 0 redundant, 4-5 steps <- this entry
# Not memory pressure: it fired at a 40-token prompt step with <1k context,
# so raising ncmoe or widening the split would both have been wrong fixes.
# gpt-oss-20b (no spill) does NOT crash on the same task, which is what
# implicates the -ncmoe path rather than the arch or the build.
#
# COST: --swa-full gives the 18 sliding layers a FULL-SIZE KV cache, so KV at
# 131072/q8_0 goes 2.25 -> 4.50 GiB, about 1.5 expert layers. That is why
# ncmoe is 24 here and not the 22 the speed ladder alone would pick.
# ncmoe=23 has NO safe split at 131k: 75,25 leaves 756 MiB free on CUDA1 and
# 78,22 leaves 450 MiB on CUDA0 — both under the ~1 GiB cuBLAS needs (it died
# at 466 MiB) — and 80,20 will not load. Measured, --swa-full, 131072:
# ncmoe 24 ts 78,22 free 2078/2068 10.95 tok/s <- this entry
# ncmoe 23 ts 75,25 free 756/2661 13.90 REJECTED, no margin
# ncmoe 22 c 65536 free 1556/3075 17.83 (65k alternative)
# Chosen for the full native window; 65k would be 63% faster if that is ever
# worth revisiting. NOTE prefill is ~113 tok/s, so FILLING 131k costs ~19 min
# before the first token — the deep end of this window is theoretical.
#
# SPLIT BIASED TOWARD THE FASTER CARD — MEASURED, not assumed. llama.cpp
# runs the two cards SEQUENTIALLY for one request (no overlap; both sit near
# 10% util), so total decode time is the SUM and the optimum equalizes TIME,
# not bytes. The 5060 Ti (~448 GB/s) should therefore hold MORE weight than
# the 4060 Ti (~288 GB/s) — at ncmoe=22/131072/q8_0 KV (old convention,
# CUDA0=4060Ti): 78,22 gave the 4060Ti 78% and read 13.63 tok/s warm /
# 7.11 after a 3.3k prompt; 75,25 read 14.31 / 10.93 (+54% post-prompt);
# 72,28 OOM'd on CUDA1's kv cache buffer. Since the 2026-09-11 re-enable
# CUDA0=5060Ti, so this entry's unchanged 78,22 now gives the FASTER card
# the bigger share directly — the thing this measurement was arguing for.
# SAMPLER: the gpt-oss-20b top-p 1.0 / top-k 0 win (worth 12% there, same
# 201,088 vocab) DOES NOT TRANSFER — measured 18.49 vs 19.72 tok/s, i.e.
# nothing. This config is bound by expert paging, not by sorting the vocab.
# Do not inherit it; [[gpt-oss-20b]] is the cautionary note in both directions.
#
# --reasoning-format deepseek: harmony analysis channel -> reasoning_content.
# Budget max_tokens generously or a short cap returns EMPTY content.
cmd: |
/usr/bin/env
CUDA_DEVICE_ORDER=PCI_BUS_ID
${llama_bin}
-m /mnt/ssd1700/models/gpt-oss-120b/gpt-oss-120b-MXFP4.gguf
--alias gpt-oss-120b
--jinja
-ngl 99 --n-cpu-moe 24 -c 131072 -fa on
--tensor-split 78,22
-b 2048 -ub 512
--temp 0.2
--cache-type-k q8_0 --cache-type-v q8_0
--swa-full
--reasoning-format deepseek
--host 127.0.0.1 --port 9177 --parallel 1
proxy: http://127.0.0.1:9177