Kept / trial active in llama-swap.yaml · aliases: nemotron-3.5-lightning-30b-a3b-262k
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| 20260812-183657 | 7/8 | — | — | results-nemotron-lightning-20260812-183657.log |
"nemotron-lightning":
aliases: [nemotron-3.5-lightning-30b-a3b-262k]
# bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:Q5_K_M (25.11 GiB) —
# TRIAL 2026-08-12, PARTIAL. ARCH: nemotron_h_moe (hybrid Mamba-2 / MoE /
# attention), day-0 supported by the 2026-08-11 build 6e62ba538 —
# LLM_ARCH_NEMOTRON_H_MOE is in llama-arch.cpp:93. No fork needed.
#
# NOT the REJECTED Nemotron-3-Nano ([[nemotron-3-nano-30b-a3b]]). Same vendor
# and same 30B-A3B shape, different model, and it BEATS nano on nano's own
# protocol (same port, same f16 KV arm):
# needle @262k nano 3/5 -> lightning 5/5 (nano FAILED depths 0.5, 0.9)
# R1 nano 6/8 -> lightning 7/8
# Do not triage it away on the earlier rejection.
#
# KV IS ALMOST FREE, hence f16 rather than q8_0. Read from the GGUF metadata
# 2026-08-12: block_count 53, and attention.head_count_kv is nonzero on only
# **7 blocks — [5, 12, 19, 26, 33, 42, 52]** — of which blk.52 is the unused
# MTP/nextn head, so **6 attention layers are active**. Every other block is
# Mamba/MoE, whose SSM+conv state is per-SEQUENCE not per-token (~48 MiB
# total). At kv_heads 2 / head_dim 128 that is 6 KiB/token:
# f16 @ 262144 = 1.50 GiB q8_0 @ 262144 = 0.80 GiB
# f16 @ 1M = 6.00 GiB q8_0 @ 1M = 3.19 GiB
# So this runs native context at FULL f16 precision — quality-first per
# [[vram-spend-priority]] — and sidesteps the q8_0-V-cache rejection trap.
# See [[kv-sizing-full-attention-layers]].
#
# -c 262144 IS A CHOICE, NOT THE MODEL'S CEILING. The GGUF declares
# context_length = **1048576**, and unlike [[nemotron-3-nano-30b-a3b]] — whose
# card claimed 1M while its GGUF exposed 262144 — that 1M is real in the
# weights. q8_0 KV would make 1M fit (25.11 + 3.19 = 28.3 GiB). NOT RAISED
# because it is UNVERIFIED and there is reason to doubt it: rope.freq_base is
# 10000.0 with rope.scaling.finetuned = False, the Llama-2 default, where a
# genuine 1M model normally carries a large rope base or YaRN. Retrieval is
# verified only to 260,841 tokens. Test with needle.py at label ~600000
# before trusting anything above 262144.
#
# Q5_K_M is the ceiling for weights: Q6_K is 31.95 GiB and Q8_0 32.60 GiB
# against a 31.0 GiB usable pool. Q5_K_M at 25.11 leaves ~5.9 GiB.
#
# Expect these WARNINGS at load, both benign: "failed to fit params to free
# device memory: n_gpu_layers already set by user to 99" (we set -ngl
# deliberately), and a run of "model has unused tensor blk.52.nextn.*" — that
# is the MTP/speculative head, which this build does not use.
#
# STATUS: verdict NOT reached. R1 (7/8, ~92-95 tok/s) and the needle are the
# ONLY suites ever run; R2-R5, klayout(+ref), klvision, office and loop have
# never executed. Its one R1 miss is `expr_eval`, the field-universal failure
# that only thinkingcap has ever passed — so 7/8 is a clean R1 at field
# standard. See auto-memory nemotron-35-lightning-30b.
# STORAGE TIER: /mnt/ssk500, NOT the NVMe — moved 2026-08-13. It is a
# single-role spare (440k-1M retrieval), not a driver, so it pays the slow-disk
# penalty: ~330 MB/s measured on ssk500 vs ~1.08 GiB/s on the NVMe, i.e. ~76 s
# to read 25.11 GiB instead of ~25 s. That +50 s is noise against the ~27 min
# first-query prefill its actual role costs. Copy verified byte-identical
# before the NVMe original was deleted (size, head-512MB cmp, tail-512MB
# sha256 44fd8f528177340f). NVMe working set was coder-agentic +
# fable-fusion only — STALE as of 2026-08-14, it is now those two plus
# thinkingcap, gemma-awq and fable-711-gptq. See [[nvme-working-set]].
cmd: |
/usr/bin/env
CUDA_DEVICE_ORDER=PCI_BUS_ID
${llama_bin}
-m /mnt/ssk500/gguf/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q5_K_M.gguf
--alias nemotron-lightning
--jinja -ngl 99 -c 262144 -fa on
--tensor-split 48,52
-b 2048 -ub 512
--host 127.0.0.1 --port 9121 --parallel 1
proxy: http://127.0.0.1:9121