Rejected no longer in llama-swap.yaml — entry and/or weights removed after this verdict.
| Run | Score | tok/step | Log |
|---|---|---|---|
| No results-*.log found under this name. | |||
/mnt/ssk500/gguf/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q5_K_M.gguf, 25.1 GiB,
MoE 30B-A3B, arch nemotron_h_moe (hybrid Mamba-2 / MoE / attention) — day-0
supported by the 2026-08-11 llama.cpp build 6e62ba538, no fork needed.
On ssk500, NOT the NVMe (moved 2026-08-13, freeing 25.11 GiB). Single-role
spare, so it pays the slow disk: 76 s to load (~330 MB/s) vs ~25 s on NVMe.
Irrelevant here — its role costs ~27 min of prefill on the first query, so +50 s
of load is noise. Copy verified byte-identical before deleting the original.
Wired into llama-swap.yaml on port 9121 on 2026-08-12, with a
models.json alias generated by .claude/sync-models.sh. Before that it had
been run by hand for the trial and torn down, so nothing in the toolchain could
reach it. Batteries run through llama-swap on :8090 like every other model.
**Run KEEP bartowski Q5_K_M, at a client-side max_tokens/max_output_size of
32000, not 16000.** llama-swap.yaml carries no token-budget flag of its own —
GRILL_MAX_TOKENS/max_tokens is a per-request body parameter, so this is a
kimi-code / pi / omp model-config change, not a yaml edit.
Triggered by Giovanni asking to compare the installed bartowski Q5_K_M
against unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:UD-Q4_K_XL. Full
battery x3 repeats, then a 16k-vs-32k budget split on R1 (the suspected
budget-sensitive suite) extended to the whole battery + realcase once R1
confirmed the effect was real. Scripts: bench/grill-nemotron-q5km.sh (16k),
bench/grill-nemotron-q5km32k.sh (32k), bench/report-nemotron-ab.py.
Artifacts: bench/grill-nemotron-q5km-20260815/, bench/grill-nemotron-q5km32k-20260815/.
### bartowski Q5_K_M, 16k vs 32k budget (3 reps each)
| suite | 16k | 32k | read |
|---|---|---|---|
| R1 | 7,7,7/8 raw; honest 19/24 (~79%) | 7,7,7/8 raw; honest 21/24 — clean | lru_ttl rescued at 32k every rep (11.7-17k tok, well clear of either cap); expr_eval FAILS regardless of budget (27k-32k tok) — field-universal hard task |
| R2-R4 | 5/5 nearly always | 5/5 nearly always | unchanged (one 16k rep1 bounded_blocking_queue FAIL was a genuine deadlock in the CANDIDATE's own threading code — see [[klayout-office-grill-rounds]] harness note) |
| R5 | 4,5,4/6, loops every rep | 5,4,4/6, loops every rep (1 rep had 2) | loops in 6/6 reps total, both budgets — never fixed, budget-independent |
| klayout | 0/8 x3 | 0/8 x3 | field norm, unchanged |
| klayout-ref | 3,4,6/8 (mean 4.3) | 3,5,3/8 (mean 3.7) | noisy both ways, ~50% true rate across all 6 samples — NOT the 75% the old single Aug-12 sample suggested |
| office (unassisted) | 2,2,4/9 = 2.7/6 reachable | 4,4,6/9 = 4.7/6 reachable | genuine budget effect, same shape as R1's lru_ttl |
| office+ref | 6,6,6/9 (6/6 reachable) | 6,4,6/9 (one isolated IndentationError in generated code, not reproduced) | effectively unchanged |
| realcase | PASS 7/7 (128s,12k tok), FAIL 0/7 RUNAWAY (170s,16k tok CAP-HIT), PASS 7/7 (128s,12k tok) — 2/3 | PASS 7/7 x3 (125-259s, 11.7-24.2k tok) — 3/3, zero runaways | the ONE 16k failure was a genuine budget cap, not a capability gap — this is the suite that resembles Giovanni's actual KLayout+Excel work |
| throughput | median 94.5 tok/s, n=206, range 49.9-95.4 | (same server flags) | consistent with 2026-08-12's 91.9-95.4 reading |
| needle @260k | 15/15 PASS (5 depths x3 reps), median 83.4s cold | n/a — needle.py reads its own NEEDLE_MAX_TOKENS (held at 12000), never GRILL_MAX_TOKENS, so budget has no effect on retrieval | matches/exceeds the 2026-08-12 5/5 x2 |
### unsloth UD-Q4_K_XL, 16k only (3 reps; 32k/1M/needle never ran — trial stopped early)
25,505,724,480 B = 23.75 GiB (1.36 GiB smaller than bartowski's 25.10 GiB).
**GGUF tensor read overturned the naming assumption 2026-08-15: on
nemotron_h_moe the K-quant path is nearly dead — expert tensor shapes are
not divisible by the 256-element K-quant superblock, so BOTH quantizers fall
back to legacy types. unsloth "UD-Q4_K_XL" contains zero Q4_K tensors**
(composition Q5_0 9.40 / Q5_1 8.92 / Q8_0 5.39 GiB, 6.19 bpw effective).
bartowski "Q5_K_M" contains Q5_1/Q8_0/Q4_0 plus 18 real K-quant tensors
(0.13 GiB of 25.10), 6.55 bpw effective — a 5.5% bit gap, not a quant-tier
gap. unsloth spends its bits differently, not just fewer: it holds
ssm_in/ssm_out/token_embd at Q8_0 (bartowski: Q5_1) and economises on
the MoE experts (Q5_0/Q5_1 vs bartowski's Q4_0-mixed). Despite being smaller,
it was ~5% SLOWER (89.6 vs 94.5 tok/s median) — the always-read
SSM/embedding path costs more per token than the routed-expert savings buy,
on a memory-bandwidth-bound decode. Inferred mechanism, not profiled directly.
| suite | result | vs bartowski @16k |
|---|---|---|
| R1 | 7,7,7/8 raw; honest 17/24 (~71%) | worse — 3-4 tasks/rep hit the cap (vs bartowski's ~2), incl. articulation_points and wildcard_match which never capped for bartowski |
| R2-R4 | 5/5 nearly always | tie |
| R5 | 5,4,3/6, loops every rep (one rep had 2) | tie — same 6/6-loops pattern |
| klayout | 0/8 x3 | tie |
| klayout-ref | 4,3,3/8 (mean 3.3) | tie within noise |
| office (unassisted) | 4,6,4/9 = 4.7/6 reachable | better than bartowski's 16k number (2.7) — but statistically ≈ bartowski's OWN 32k number (4.7), i.e. matched once bartowski gets the same effective budget headroom |
| office+ref | 6/9 x3 | tie |
| realcase | FAIL 0/7 RUNAWAY x2 (179.4s/179.5s, both capped at exactly 16,000 tok) — 0/2, rep3 cancelled | worse — never produced a single passing artifact in 2 tries, vs bartowski's 2/3 at the same budget |
| throughput | median 89.6 tok/s, n=203 | ~5% slower |
| needle | never run | n/a |
REJECTED. Even before accounting for the missing 32k/1M data, its 16k
realcase and R1 rates were both worse than bartowski's at the same budget,
and it was slower doing it. The asymmetry matters for future re-evaluation:
this build was never given the same 32k chance that rescued bartowski's
weak points, so "REJECTED" here means "lost the fair comparison it got," not
"proven incapable of the same budget-driven fix." **GGUF file deleted
2026-08-16** after this verdict was recorded (re-fetch: hf download
unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-UD-Q4_K_XL.gguf, ~6 min at the ~73
MiB/s measured this session). The 30 raw suite logs from its 3-repeat 16k
battery are kept at bench/grill-nemotron-ud-q4kxl-20260815/.
### Corrections to the 2026-08-12 single-sample claims below
The n=1 numbers in the next section were the reason this A/B was run at all —
several turned out to be lucky or unlucky draws, not the model's true rate:
16k is 2/3 PASS; at 32k it is 3/3 PASS. The Aug-12 FAIL 5/7 [GEOMETRY]
sample was itself an unlucky draw — see [[realcase-spec-plus-apiref-multiplicative]],
which should be re-read with this correction in mind.
True rate across 6 samples (both budgets) is ~50%, with a 3-6/8 spread.
reps at 16k average 19/24 (~79%), and at 32k it is a clean 21/24 — but the
qualitative point (don't trust the raw 7/8, lru_ttl's pass is often
truncation luck at 16k) was correct and is now confirmed at n=3.
reps) — that one was not a fluke, but it IS a budget artifact fixable at 32k.
bench/grill-nemotron-lightning-20260812.sh, 1 repeat, via llama-swap :8090.
Artifacts in bench/grill-nemotron-lightning-20260812/.
| suite | result | field position |
|---|---|---|
| coding R1-R4 | 21/23 | level with glm-flash / kimi-distill; below gemma 44/46 |
| R5 agentic | 4/6, 1 LOOPED | interrupt_replan, 5 redundant calls, aborted step 12 |
| tool calling (R3) | 3/3 clean | structured calls + correct restraint |
| klayout from memory | 0/8 | field norm — every model is 0 here |
| klayout + API ref | 6/8 (75%) | = gemma 75%, < coder-agentic 81%, < fable-711 94% |
| office unassisted | 2/9 = 2 of 6 reachable | see [[office-suite-vision-floor]] |
| office + API ref | 6/9 = 6 of 6 reachable | PERFECT on every stage it can attempt |
| real-case gate | FAIL 5/7 [GEOMETRY] | the only model ever to fail this arm |
| needle @ 260,841 tok | 5/5, twice | NOT the box record — north-mini did 440,181 3/3 |
| needle @ 997,115 tok | PASS (depth 0.5 only) | deepest IN-SPEC retrieval ever measured here |
| throughput | 91.9-95.4 tok/s | flat |
| cap-hits | 2, reproducible | expr_eval, lru_ttl |
| modality | TEXT-ONLY | no mmproj |
Against it, decisively: it FAILS the real-case gate that coder-agentic,
gemma-awq and fable-fusion all pass 7/7 — see
[[realcase-spec-plus-apiref-multiplicative]], which this run also refuted on
bimodality. That gate is the only suite resembling Giovanni's actual
KLayout+Excel work, and the failure mode is the bad one: valid GDS, correct
layers, correct bbox, wrong cell count and wrong array — output that
survives a casual look. It also has a genuine R5 loop, the single most common
cause of retirement here ([[gpt-oss-20b]], [[tess-4-27b]],
[[devstral-small-2-24b]], [[laguna-xs-2.1]] all died on one).
For it: exactly one thing — long-context depth. 260,841 verified beats
gemma's 247,592, and the GGUF declares 1M (above). Everything else it does,
gemma does better while ALSO having vision and passing the real-case gate.
THE 1M CAME BACK PASS, so the narrow role is real. Verified 997,115 tokens
(below). Prior in-spec box record was north-mini at 440,181 3/3, so this is
2.3x the deepest anything here had done. **Its role is 440k-1M offline
retrieval, and it is the only option for that band** — but batch-only at 27
min/query, and it does NOT displace north-mini below 440k (3/3 at 700 s) or
gemma below 248k (3/3 at 150 s).
Keep it ONLY if that band is a real need. Everything else about it — realcase
FAIL, an R5 loop, text-only, two 16k runaways — is a reason not to point
ordinary work at it.
R1 was run twice on identical weights. Both runs hit the cap on the SAME two
tasks, to within 0.5 s:
expr_eval FAIL 169.7s/16000 tok | FAIL 169.8s/16000 tok
lru_ttl PASS 169.5s/16000 tok | FAIL 169.3s/16000 tok
lru_ttl's first "PASS" was luck: it never finishes, it gets truncated at the
cap, and whether the truncated text still executes is a coin flip. **Do not
record 7/8; the honest R1 is 6/8 with two runaways.** An earlier note in this
file called 7/8 "effectively clean at field standard" on the strength of the
single run — that was too generous and is withdrawn.
expr_eval alone WOULD be excusable: it is the field-universal failure that
only [[thinkingcap-awq-trial]] has ever passed. Its artifact shows a structural
omission rather than truncation — r1_expr_eval.py defines parse_unary,
parse_power, parse_term, parse_expr, calls parse_primary() at line 91,
never defines it, and still ends cleanly at return result.
UNTESTED ALTERNATIVE: this model is a heavy reasoner (213 tokens to answer
"reply with exactly: WIRED"), so 16000 may simply be too small a budget for
those two tasks rather than a true runaway. [[budget-sensitivity-is-a-model-property]]
shows that distinction is real and model-specific. A 32k arm on just those two
tasks would separate the two explanations cheaply. Not yet run.
5/5 at 260,841 real tokens — the deepest verified retrieval measured on
this box, past gemma-awq's 247,592 ([[gemma-4-26b-awq-vllm]]) and bonsai's
224k ([[bonsai-awq-trial]]). Verified twice on a cold cache, ~88s/query.
This is a 30B MoE holding 260k in 32 GiB alongside 25.1 GiB of weights, so the
KV must be dominated by a small full-attention layer count — the
[[kv-sizing-full-attention-layers]] pattern, consistent with the "Lightning"
hybrid-attention naming. Notably it does NOT share the Qwen-lineage deep
retrieval weakness in [[qwen36-27b-deep-retrieval-fails]].
Beware the measurement trap that bit this run: see
[[needle-warm-cache-invalidates-repeats]].
NVIDIA's card advertises "up to 1M tokens". For [[nemotron-3-nano-30b-a3b]] the
identical claim was fake — that GGUF exposed 262144. This one does not
repeat it. Read from the GGUF metadata 2026-08-12:
nemotron_h_moe.context_length = 1048576 <- genuinely declared
nemotron_h_moe.block_count = 53
attention.head_count_kv nonzero on 7 blocks: [5,12,19,26,33,42,52]
(52 = unused MTP head -> 6 active)
rope.freq_base = 10000.0 rope.scaling.finetuned = False
-c 262144 in llama-swap.yaml is a CHOICE, not the ceiling. It fits, too —
at 6 active attention layers, kv_heads 2, head_dim 128 the KV is 6 KiB/token:
f16 @262144 = 1.50 GiB f16 @1M = 6.00 GiB (25.11+6.00 = too tight)
q8_0 @262144 = 0.80 GiB q8_0 @1M = 3.19 GiB (25.11+3.19 = 28.3, FITS)
### VERIFIED 2026-08-12: it really does retrieve at ~1M, 2/2
depth 0.5: PASS ctx=997,115 tok 1609.8s in[reasoning]
depth 0.9: PASS ctx=997,115 tok 1610.1s in[reasoning]
Depth 0.9 is the one that matters: [[nemotron-3-nano-30b-a3b]] FAILED 0.5 AND
0.9 at both 698k and 831k while lightning passes both at ~1M. Late-context
recall is where a long-context claim usually dies, and it holds here. The two
timings are within 0.3 s of each other — cost is pure prefill and does not
depend on needle position.
Served standalone on 9122: -c 1048576 --cache-type-k q8_0 --cache-type-v q8_0
--tensor-split 50,50 -b 1024 -ub 128. **The rope.freq_base doubt below is
empirically answered — extrapolation is not broken at 1M.**
GETTING IT TO LOAD IS THE HARD PART, AND IT IS NOT A KV PROBLEM. The KV
landed exactly on the 3.19 GiB prediction. What blocks the load is the COMPUTE
buffer, per [[kv-sizing-full-attention-layers]]:
-ub 512, split 48,52 -> OOM, wanted 5181 MiB on device 0
-ub 128, split 48,52 -> OOM, wanted 1290 MiB on device 1
-ub 64, split 48,52 -> OOM, wanted 1157 MiB on device 1 <- ub is NOT the lever
-ub 128, split 50,50 -> LOADS (falls back to no pipeline parallelism)
Halving ub 128->64 moved the buffer only 1290->1157 MiB, so at this depth it is
context-dominated, not ub-dominated. The fix is BALANCE: at 48,52 CUDA1 carried
52% of the 28.3 GiB of weights+KV = 14.72 GiB and had no room left for its
compute buffer. Same starved-device failure north-mini hit at 500k. Budget
weights + KV + compute buffer PER DEVICE, not just weights.
COST: 1609.8 s — 26.8 minutes — for the FIRST query, ~620 tok/s prefill.
Comparable to nano's 1329 s at 831k, so it is the box's speed, not a model
defect.
BUT THE PREFIX CACHE HOLDS AT 1M, AND THAT IS THE WHOLE WORKFLOW. Measured
2026-08-13 (bench/warmcache-1m-probe.sh, bench/warmcache-1m-20260813.log),
identical prompt twice on one loaded server:
cold PASS ctx=997,115 1610.8s
warm PASS ctx=997,115 2.0s <- 805x, same answer, not degraded
So the real cost model is **27 min ONCE per corpus, then ~2 s per follow-up
question** — interactive, not batch. This is the same mechanism that makes a
needle *repeat* worthless as a measurement
([[needle-warm-cache-invalidates-repeats]]): a trap for benchmarking, the
entire value proposition for use.
Conditions it depends on:
* the corpus prefix must be BYTE-IDENTICAL — send [corpus][question] and
never edit the corpus half
* moving content invalidates everything after it: depths 0.5 and 0.9 ran
back-to-back on one loaded server and BOTH cost ~1610 s, sharing nothing
* --parallel 1 = one slot, so one corpus at a time
* llama-swap evicts on any other model request to :8090, discarding the
cache — for a real session run it STANDALONE on 9122 via
bench/launch-nemotron-lightning-1m.sh
2 depths, not 5. 0.5 and 0.9 both PASS; 0.1/0.25/0.75 at 1M are untested,
though 0.9 is the hard one and 0.1 is the easy one nano passed even out of
spec. Good enough to call the capability real; not a 5/5 sweep.
Original doubt, now resolved by the above. Two reasons had been to doubt 1M:
rope.freq_base is 10000.0 with no rope scaling — the Llama-2 default, where a
genuine 1M model normally carries a large rope base or YaRN, so either the long
range lives in the Mamba backbone or the rope config did not survive GGUF
conversion. And retrieval is verified only to 260,841. Nothing above that
has been measured, and this box has already been burned once by a vendor 1M
claim. Test with needle.py at label ~600000 (the ~1.7x filler ratio puts that
near 1M actual) on a server started -c 1048576 --cache-type-k q8_0
--cache-type-v q8_0 before believing it.
[[nemotron-3-nano-30b-a3b]] (Nemotron-3-Nano-30B-A3B) was REJECTED 2026-07-29
for scoring 20/23 while dominated by pocket-35b, and for an advertised 1M
context that was really 262144 with 3/5 retrieval. Same vendor, same 30B-A3B
shape, different model and a materially better result — 5/5 where Nano got
3/5. Don't let the earlier rejection triage this one away.
R2-R5 + klayout(+ref) + office + loop, per the standard runbook, and the
[[grill-does-not-validate-real-use]] client-path check. Until then there is no
promote/reject verdict here — only a strong partial. Its plausible role is the
long-context slot, where it beats the current driver outright on depth; that
claim is untested against gemma's 44/46 coding and 18/18 office.