laguna-s

laguna-s-2-1-cpu-spill (laguna-s-2.1-cpu-spill.md)

Laguna-S-2.1 (118B) — RETIRED 2026-08-03, entry + 34.7GB deleted

RE-ADDED 2026-09-01 as laguna-s at UD-Q3_K_M — and this file's quant ceiling was WRONG

**The claim below — "Next real expert step is IQ3_XXS at 41.2 GiB — unservable on

a 31 GiB pool. No quant of this model that fits this box will fix expr_eval" —

is FALSE and blocked a fair retest for a month.** It treated host RAM as the

ceiling. [[gpt-oss-120b-cpu-moe-spill]] served 59.0 GiB on this box, so

41 GiB was never the limit. unsloth's UD-Q3_K_M (50.3 GiB) is smaller than

that, and its bulk experts ARE IQ3_XXS (92 tensors) with **Q8_0 attention

throughout** — better than either quant tested in August.

*Generalisable:* a "will not fit" verdict is only as good as the pool it assumed.

Re-check it whenever the envelope moves.

Served entry (verified end-to-end through llama-swap):

-ngl 99 --n-cpu-moe 31 -c 131072 -fa on --tensor-split 76,24

-b 2048 -ub 512 -rea on --temp 0.6 --cache-type-k q8_0 --cache-type-v q8_0

BATCH IS WORTH 2.7x ON PREFILL — the first version of this entry got it wrong.

It shipped -b 512 -ub 128, chosen after ncmoe=32 failed on compute buffers at

-b 1024 -ub 256, then carried to ncmoe=33 without re-testing. That limit

does not bind at 31 or 33. Measured on one 11,539-token prompt, fresh server each:

| ctx | batch | prefill | free (5060/4060) |

|---|---|---|---|

| 262144 | 512/128 | 36.7 tok/s | 1878/1394 |

| 262144 | 2048/512 | 100.5 | 1650/1156 |

| 131072 | 2048/512 | 114.8 (134.7 via llama-swap) | 2774/2490 |

| 65536 | 2048/512 | 125.1 | 3426/1012 |

**BATCH dominates (2.7x); CONTEXT is minor (1.24x over a 4x range) and there is

NO paging cliff.** 131072 chosen over the native 262144: faster AND the best

margins of all four. 65k is 9% faster again but drops a card to 1012 MiB.

"Decode is flat, so context is free" was TRUE BUT MISLEADING — decode IS flat

~9.6-11 tok/s at every context (paging-bound), which is why I first took the

whole 262k window. But PREFILL is the agent metric and it is NOT flat. A 30k

turn went ~14 min -> ~3.7 min from this fix. *Never choose a serving config on

decode alone.*

### DO NOT ADD --swa-full (tested, n=3)

Laguna is a sliding-window arch (36 of 48 layers, window 512), so the

[[nvrm-xid31-mmu-fault-4060ti]] crash that makes --swa-full MANDATORY for

gpt-oss-120b was the obvious worry. It does not transfer. On R5

interrupt_replan — the reproducer that crashed gpt-oss 5/5:

| arm | crash | task |

|---|---|---|

| rolling SWA, n=3 | 0/3 | PASS, PASS, PASS, 0 redundant |

| --swa-full, n=1 | 0/1 | PASS, 0 redundant (no benefit) |

And it would cost 4x KV here (12 of 48 full-attn, vs gpt-oss's 18 of 36):

7.28 -> 29 GiB at 262144, making this window impossible. Same flag, opposite

verdict on two models — test it per model, never inherit it.

### expr_eval RE-TESTED at 3-bit, n=20 — STILL FAILS. The prediction was WRONG.

The A/B below localised expr_eval to the 2-bit experts and predicted a 3-bit

step would fix it. Measured 2026-09-01: it does not. Two builds that differ

ONLY in 92 expert tensors (attention held at Q8_0 in both — the exact complement

of the August A/B):

| build | experts | -2**2 bug | pass |

|---|---|---|---|

| Aug IQ2_XXS | 2-bit | 3/3 deterministic | 0/3 |

| Aug IQ2_M | 2-bit, identical experts | 3/3 deterministic | 0/3 |

| UD-Q3_K_M | IQ3_XXS | 1/10 sporadic | 0/10 |

| UD-IQ4_XS | IQ3_S | 5/10 sporadic | 0/10 |

What changed is the failure MODE, not the outcome: from one deterministic

arithmetic bug to assorted code-quality defects (nonlocal pos UnboundLocalError,

over/under-strict malformed-input handling, once syntactically invalid Python).

UD-IQ4_XS IS NOT BETTER — do not pull it again. Despite +3.3 GiB and a

better 3-bit expert (IQ3_S vs IQ3_XXS) it shows the August bug 5x more often

(5/10 vs 1/10, one-sided Fisher p~0.07, not significant at n=10) and passes no

more often. Note also that the build NAMED "IQ4_XS" is not 4-bit in its bulk

experts — 92 of them are IQ3_S. Read the tensor types, not the filename.

Temperature was a real confound and was controlled: grill_round1 defaults

to GRILL_TEMP=0.2 (what August measured) and the first arm was mistakenly run

at 0.6 to match the served sampler. Re-run at 0.2: still 0/5. Temperature moved

the failure MIX, not the outcome.

**So this file's conclusion — "No quant of this model that fits this box will fix

expr_eval" — is RIGHT, but its reasoning was WRONG.** It said no such quant

would FIT. They fit fine (50.3 and 53.6 GiB both serve). They just don't fix it.

*A correct conclusion resting on a false premise still needs re-testing when the

premise falls* — it took serving three quants to learn that.

### The split trap, refined

--tensor-split 76,24 was BISECTED at ncmoe=33/262144 (free MiB, 5060Ti/4060Ti):

70,30 no load | 74,26 804/2470 | 76,24 1878/1394 | 78,22 2954/318 |

80,20 no load. One expert layer is 995 MiB = ~2 points of A, which is why

the window is this narrow. And CUDA1's share is fixed by the SPLIT ALONE:

ncmoe 33->34 left CUDA1 at *exactly* 804 MiB and gave all 996 MiB to CUDA0. A

tight CUDA1 can only be relieved by raising A — raising ncmoe does nothing for it.

Target ~1.4 GiB free per card; cuBLAS died at 466 MiB on gpt-oss.

-b 512 -ub 128 is load-bearing at 262144 (the compute-buffer lesson below, again).

**CLIENT max_tokens FLOOR IS ~4000 — the "MIN CLIENT max_tokens = 1200" below is

WRONG.** Measured n=2 per rung on one identical prompt:

| max_tokens | emits content | tokens used |

|---|---|---|

| 1400 | 0/2 (finish=length; one gave content_len=0 with 4606 chars of reasoning) | 1400 cap |

| 2000 | 2/2 | 1136 / 1153 |

| 4000 | 2/2 | 1894 / 1310 |

| 8000 | 2/2 | 1964 / 1398 |

Natural usage spans 1136-1964 tok (1.7x) on the SAME prompt and never exceeds

~2000 even when given 8000 — so a generous cap costs NOTHING while a tight one

truncates intermittently. The intermittency is the trap: it reads as a broken

model rather than a truncated one, and is very likely what the August trial logged

as "~30% of requests returned EMPTY bodies" and blamed on the sampler.

Cf. [[kimi-code-max-output-size-zero-margin]].

NOT overturned: the score-per-resource case that retired it. R5

interrupt_replan x4 all PASS, but expr_eval still fails 0/20 at 3-bit (see

above) — so the capability gap the retry was meant to close is STILL THERE.

Trial ran ~3 hours and ended the same day it started. Entry removed from

llama-swap.yaml, UD-IQ2_M weights deleted. The findings below are all still

valid and are why this file is kept -- the METHOD (CPU spill) is a keeper even

though the model is not.

Ran the first --n-cpu-moe spill experiment on this box. [[laguna-xs-2.1]] said

"S-2.1 is REJECTED — unservable here, don't re-derive this". That verdict was

wrong on its stated grounds and is now corrected.

**STATUS: RETIRED. I concluded "reject"; the user overrode it for a real-use

trial (shipped 4ca478f6), and ~3h of real use ended in retirement anyway --

so the original cost case was right, but ONLY the trial produced the findings

below, none of which the grill could have surfaced.** The reject was

defensible on score-per-resource (below) but it leaned on a single grill, and

the one thing this model is unambiguously good at — a clean long-horizon

agentic loop — is the axis the 23-task score cannot see. Worth remembering as a

pattern: *a mediocre aggregate score plus a spotless round 5 is not the same

model as a mediocre score alone.*

Served config (VERIFIED end-to-end, not predicted): -c 262144 -ncmoe 24

--tensor-split 68,32 -b 1024 -ub 256 --no-mmap, port 9122, 15.70 t/s, cold

load 30.4s, free 1433/2422 MiB, ~15.5GiB host RAM. Alias laguna-s — NOT

laguna-s-2.1, which is the pre-existing poolside CLOUD provider in

models.json (full precision, their hardware). Benchmarking the wrong one

would look like a spectacular quant result.

The triage rule that was wrong

laguna-xs-2.1 and [[model-triage-checklist]] both said --n-cpu-moe spill is

not a way out: "31GB host RAM with ~19GB in use, and 10-of-256 expert routing

per token means CPU-resident experts get hit constantly." **Measured, it is

fine: 7.4-8.1 GiB spilled to host RAM, 24 t/s decode** — not the ~10 t/s

that made [[kimi-dev-72b]] and [[devstral-2-123b-iq1s]] ungradeable. The whole

grill completed. *Spill cost ~2.5x throughput, not viability.*

Also: the "19GB in use" that made RAM look scarce was **an idle ollama holding a

retired gpt-oss-20b at keep_alive: -1** (see [[ollama-keepalive-ram-pin]]) —

a fixable leak, not a real constraint. Check what is actually resident before

concluding RAM is short.

The -ncmoe / --tensor-split trap (cost the most time)

-ncmoe N strips MoE from the first N layers, making them nearly weightless

(0.05 GiB vs 0.72 GiB). --tensor-split still divides by layer count, so an

even-looking split is wildly uneven in bytes. At -ncmoe 12 --tensor-split 48,52

(inherited from XS) CUDA1 got layers 23-47 = 25 full layers = **18.1 GiB on a

16 GiB card**. The aggregate fit at 27.0 GiB; only the split was wrong.

Two asymmetries load CUDA0 that layer-count ignores:

1. Non-layer tensors (embeddings/output, 0.32 GiB) all land on CUDA0.

2. KV follows full-attention layers, not layer count. Laguna is 1-in-4 full

attention (layers 0,4,8,...), so the first-N-layers half holds 8 of 12 of

them — KV splits ~2:1, not by layer count. Compounds with a measured

~1069 MiB fixed CUDA0 overhead (compute buffers + ctx + logits) vs

~-146 MiB on CUDA1.

Calibrating that overhead from one observed run made the model predictive to

4-12 MiB. Working configs (34.6-34.7 GiB weights, 32k ctx, q8_0 KV):

| quant | -ncmoe | --tensor-split | CUDA0 free | CUDA1 free | CPU | t/s |

|---|---|---|---|---|---|---|

| IQ2_XXS | 13 | 63,37 | 1183 MiB | 2548 MiB | 8.09 GiB | 21-23 |

| IQ2_M | 12 | 60,40 | 1239 MiB | 1818 MiB | 7.41 GiB | 24-25 |

--no-mmap is correct here (llama.cpp warns): CPU tensors become anonymous RAM

instead of file-backed, and there is headroom once ollama is not squatting.

IQ2_XXS vs IQ2_M — a clean controlled A/B, 3 runs each

Do not pick IQ2_XXS. IQ2_M costs 76 MiB (+0.2%) and is measurably better.

The two GGUFs have byte-identical expert tensors (IQ2_XXS x92 + IQ3_XXS x45

+ IQ4_XS x2 + IQ2_S x2 in both — 92% of weights, +0 MiB). All 76 MiB buys

attention Q5_K->Q6_K (94 tensors) and output head Q4_K->Q5_K. So the A/B

isolates attention/output precision with expert precision held constant.

``

IQ2_XXS round1: 5, 6, 5 mean 5.33/8 full grill 20/23 R5 6/6 0-loop 0/34-redundant

IQ2_M round1: 6, 7, 7 mean 6.67/8 full grill 21/23 R5 6/6 0-loop 0/34-redundant

`

The result that matters — two failure modes, cleanly separated:

• lru_ttl 3/3 FAIL -> 3/3 PASS. Experts identical, so this failure was

*purely* attention/output-head precision. A discrimination failure, echoing

[[ternary-quant-costs-little-on-vision]]: extreme quant degrades

DISCRIMINATION, not knowledge.

• expr_eval 3/3 FAIL in BOTH (three different errors: unexpected token /

-2**2 precedence / unexpected character). Untouched by attention precision

=> it lives in the 2-bit experts. Next real expert step is IQ3_XXS at

41.2 GiB — unservable on a 31 GiB pool. **No quant of this model that fits

this box will fix expr_eval.**

Rounds 2-5 were identical across quants (5/5, 5/5, 5/5, R5 6/6). The entire

21-vs-20 gap is round 1, whose own spread is 5-7/8.

The cost case against it (why I said reject)

~21.7/23 projected, but it needs **both GPUs entirely plus 7.4-15.5 GiB host

RAM** at 15-24 t/s. [[pocket-35b]] scores 22/23 + R5 6/6 at 102 t/s fully in

VRAM; [[coder-agentic-default-driver]] does 31.0/33 with genuine 237k retrieval.

On score-per-resource it buys nothing and costs the whole box. That case still

stands — it is why this is a *trial*, not a promotion.

THE COMPUTE BUFFER CAPS CONTEXT, NOT KV — and it is invisible at low ctx

The single most expensive mistake of the session, and a repeat of what

[[laguna-xs-2.1]] already recorded ("262144 is unreachable at ANY split ...

2104.70 MiB compute buffer, not KV"). I quoted that memory and then walked into

it anyway.

I calibrated per-card overhead at 32k (CUDA0 +1069 MiB, CUDA1 -146 MiB),

which predicted the *next 32k config* to within 4-12 MiB — genuinely

accurate, which is what made it seductive. Extrapolated to 256k it was garbage:

the load died in 30s on allocating 1323.17 MiB on device 1 ... failed to

allocate compute pp buffers. That term is ~0 at 32k and ~1.3 GiB per card

at 256k.

Rule: never predict a high-context fit from a low-context calibration. The

buffer scales with BATCH, so -b 1024 -ub 256 (from 2048/512) roughly

halves it and was load-bearing — cf. [[north-mini-code]], which needed

-ub 256 for the same class of reason (there it was the logit buffer; laguna's

vocab is only 100352, so here it is the pp activations).

Measured context ladder (all loaded and generation-tested, not modelled):

| ctx | -ncmoe | split | -b/-ub | t/s | VRAM free | host RAM left |

|---|---|---|---|---|---|---|

| 32k | 12 | 60,40 | 2048/512 | 24.5 | 1239/1818 | ~24 GB |

| 131k | 16 | 62,38 | 2048/512 | 20.1 | 1260/1353 | 16 GB |

| 131k | 14 | 60,40 | 1024/256 | 22.1 | 610/817 (tight) | 17 GB |

| 256k | 24 | 68,32 | 1024/256 | 15.7 | 1433/2422 | 9 GB |

256k costs ~22% throughput and ~6 GB more host RAM than 131k. **Host RAM, not

VRAM, is the binding constraint at 256k** — 9 GB left for everything else.

Note 256k is reachable here *only because of the spill*: laguna-xs could not

load 262144 at any split, because it had no way to move weights off the GPU.

Genuinely good, and worth remembering: R5 **6/6, 0 loops, 0/34 redundant on

BOTH quants** — two independent full grills, 0/68 redundant calls combined. That

is the axis that killed [[north-mini-code]], coder and kimi-linear. Its

long-horizon agentic loop is clean.

Two claims of mine that n=1 falsified — read [[single-run-scores-are-unreliable]] first

1. "Leanest thinking model on the box, 10,018 tokens for the whole grill."

False — that was the lowest of three. Round-1 totals ran **4,326 / 34,968 /

18,651 (8.1x spread)**. Corrected: ~4x leaner than XS in tokens but 3.75x

slower per token = rough wall-clock parity, not the 2.5x win claimed.

2. Runaways. 3 of 24 XXS task-runs hit the exact 16,000-token cap; IQ2_M 2

of 24 — including on tasks it *passed*. Same signature that rejected

[[gemma-4-26b-a4b-moe]]. Higher attention precision fixed a correctness mode,

not the verbosity instability.

Facts worth not re-deriving

• Arch laguna (LLM_ARCH_LAGUNA) covers S-2.1 as well as XS — same string, no

new llama.cpp work. Clean on the jinja guard; round 3 5/5 both quants, so the

[Patch: Laguna] tool parser works out of the box, no chat-templates/ patch.

• 48 layers, 256 experts top-10 + 1 shared, 12 full-attention + 36 sliding(512).

MoE = 31.99 of 34.63 GiB (92%), ~0.674 GiB/layer; blk.0 is dense (no experts).

• KV is cheap: ~468 MiB at 16k, ~936 MiB at 32k (q8_0), because only 12 layers

are full-attention — cf. [[kv-sizing-full-attention-layers]].

• -rea on mandatory (GGUF template defaults enable_thinking=false).
• Logs: bench/results-laguna-s-20260803-143757.log (IQ2_XXS),

bench/results-laguna-s-iq2m-20260803-152627.log (IQ2_M).

Shell trap that cost two stalls

pgrep -f / pkill -f match any process whose cmdline contains the pattern

— including the script doing the matching. pkill -f 'port 9190' killed its

own launcher shell before exec; a wait loop on pgrep -f 'grill_round[0-9].py'

blocked forever on itself because its own body contained that literal string.

Match the real process: pgrep -af 'python3 grill_round' | grep -v 'bash -c'.

What the real-use trial actually cost (2026-08-03, ~3h)

The grill said 21/23 + R5 6/6 + 0 loops. Real use said otherwise, and none of

these were visible to the harness:

1. ~30% of Claude Code requests returned EMPTY bodies (9-36B after 5-6

MINUTES) because the entry served poolside's temp 1.0 / top-p 1.0 -- a

sampler the grill never exercises, since it always sends its own. Fixed by

temp 0.6, but see [[grill-does-not-validate-real-use]]; this is the general

lesson and it outlives the model.

2. --reasoning-budget does NOT fix that -- it only engages ABOVE its own

value, so 4096 does nothing for a client sending max_tokens=400. Measured

floor after fixing: MIN CLIENT max_tokens = 1200.

3. Prompt processing is the real tax at long context. ~708 tok/s, so a

183k-token agent prompt costs ~4 MINUTES before the first token, repaid

on every cache miss. Inherent to CPU-resident experts. This, not decode

speed, is what makes a spilled MoE painful for agentic work.

4. Context whiplash: dropped 256k->131k to relieve RAM pressure while

misdiagnosing the empty responses as "slow", which then broke a real 183k

job with "request exceeds available context size". Diagnose before

re-tuning; the RAM was never the problem.

Final verdict: the cost case in "why I said reject" was correct. 15.7 t/s,

both GPUs, 15.5GiB host RAM and a 4-minute prompt tax, to score under

[[pocket-35b]] (22/23 at 102 t/s, fully in VRAM). Its one genuine strength --

a spotless long-horizon agentic loop -- never got to matter, because the

per-turn latency made long-horizon work impractical.

Would try again only if: a sparse MoE lands that fits in VRAM outright, or

this box gets enough RAM/bandwidth that the prompt-processing tax disappears.

**(SUPERSEDED 2026-09-01 — retried on the third route neither condition

anticipated: the servable ENVELOPE grew. See the top of this file.)**

The -ncmoe technique itself is proven and reusable -- see the config table

above.

RE-ENABLED 2026-09-11

Weights had been parked on /mnt/wd1tb (dead USB HDD, [[storage-tiers]]); that

drive died and was replaced by /mnt/ssd1700, a genuinely fast USB SSD

(~1.8 GB/s). Re-downloaded byte-exact from unsloth/Laguna-S-2.1-GGUF

(UD-Q3_K_M), CUDA_DEVICE_ORDER=PCI_BUS_ID added per

[[gpu-card-assignment-policy]] (tensor-split 76,24 left UNCHANGED), uncommented

as laguna-s in llama-swap.yaml, and verified standalone: loaded in ~36s,

free VRAM 2882/3369 MiB, generated correctly at the documented temp 0.6 /

-rea on settings. HUP-reloaded into production llama-swap`. Not

re-grilled beyond the standalone smoke test. Note: the entry's own

"measured at ncmoe=33" split-bisection table labels its free-MiB columns

"(5060Ti/4060Ti)" in a way that doesn't cleanly reconcile with the rest of

the entry's card assumptions — flagged but not resolved during the

2026-09-11 re-enable; re-derive from a fresh sweep if the split ever needs

re-tuning rather than trusting that table's card labels.