llama4-scout-17b-16e-tq1

Llama-4-Scout-17B-16E-Instruct @ UD-TQ1_0 — REJECTED 2026-08-16

Triggered by "find the largest MoE model that would fit this box." Real

published quant sizes (not estimates — burned once already the same night on

a bad DeepSeek-V4-Flash size estimate) ruled out every other large-MoE family

checked: DeepSeek-V4-Flash smallest published 76.86 GiB, Qwen3-235B-A22B

79.8 GiB, gpt-oss-120b 58.3 GiB, Kimi-K2 226.9 GiB (even at TQ1_0!),

MiniMax-M2 52.5 GiB, GLM-5.2 ~700B total. **UD-TQ1_0 of Llama-4-Scout,

27.25 GiB, was the only one that cleared this box's ~30 GiB VRAM ceiling with

real headroom.** Base repo (meta-llama/Llama-4-Scout-17B-16E-Instruct) is

gated; the unsloth GGUF re-upload is not, so no HF token was needed.

~17B active of ~109B total (public knowledge — direct config.json fetch

failed against the gated original repo). llama.cpp has real support

(LLM_ARCH_LLAMA4).

The strong prior going in, and what it actually predicted

[[devstral-2-123b-iq1s]] (dense 123B @ UD-IQ1_S, 26.49 GiB, nearly the same

size class and same "forced to ~1-bit to fit VRAM" shape) scored **2/23 with

ZERO working tool calls** — total collapse. TQ1_0 is an even more extreme

quant tier than what killed that trial. The one genuine unknown going in:

Devstral was dense (every parameter active every token, so quantization

noise hits every computation); Scout is MoE (~17B of ~109B active) — whether

sparse activation is more or less robust to extreme quantization than a

dense forward pass had never been tested on this box.

Answer: MoE is dramatically more robust. R1 quick-check produced

coherent, CORRECT code (sum_of_list, clean and working) and 4-5/8 on real

tasks — not the near-total collapse Devstral showed. This is the one

genuinely new, reusable finding from this trial, independent of whether the

model itself earns a role.

Full 3-repeat battery result

bench/grill-llama4-scout.sh, GRILL_MAX_TOKENS=16000 (this model's outputs

were consistently short — 200-1200 tokens even on its hardest task — no

evidence a bigger budget would help, unlike nemotron-lightning the same

night). 27 minutes total for the whole battery — fast, as expected for a

MoE with only ~17B active params (vs. dense Qwen3.8-27B's ~16 tok/s the same

night; this ran 34-36 tok/s).

| suite | rep1 | rep2 | rep3 | combined |

|---|---|---|---|---|

| R1 | 4/8 | 4/8 | 5/8 | 13/24 (54%) |

| R2 | 3/5 | 2/5 | 3/5 | 8/15 (53%) |

| R3 | 2/5 | 2/5 | 2/5 | UNMEASURED — see harness note below |

| R4 | 1/5 | 1/5 | 1/5 | UNMEASURED |

| R5 | 0/6 | 0/6 | 0/6 | UNMEASURED |

| klayout (memory) | 0/8 | 0/8 | 0/8 | 0/24 — field norm |

| klayout + ref | 5/8 | 6/8 | 5/8 | 16/24 (66.7%) |

| office (unaided) | 0/9 | 0/9 | 0/9 | 0/27 |

| office + ref | 4/9 | 4/9 | 4/9 | 12/27 (44.4%) |

| realcase | FAIL [NO_GDS] | FAIL [SPEC_PARSE] | FAIL [RUNAWAY] | 0/3, three different failure modes |

R3/R4/R5 are a harness limitation, not a genuine score. See

[[llama-cpp-peg-native-tool-parser-500]] for the full mechanism — Llama-4's

native tool-call syntax ([func(args)], bracket-wrapped) crashed

llama-server's auto-derived tool-call parser, and the crash corrupted

ongoing server state badly enough that EVERY suite after the first crash

failed too, including klayout/office/realcase which never pass a tools

parameter. Fixed with --skip-chat-parsing, which stopped the crash but

means message.tool_calls is always empty — the grill's fallback extractor

only recognizes Qwen/Hermes <tool_call>{...}</tool_call> syntax, not this

model's bracket format, so R3/R4/R5 read near-zero regardless of whether the

model is actually calling tools correctly (it is — verified directly via

curl, e.g. [get_weather(location='Paris', unit='celsius')], a fully

correct call).

Comparisons against other extreme-resource models on this box

vs. coder-agentic ([[coder-agentic-regrilled]], 2-bit UD-IQ2_XXS,

30B-A3B, ~3B active — itself an extreme low-bit MoE, the fairest comparison

available): coder-agentic wins on every axis that's fairly comparable —

coding 93.5% vs 54%, klayout+ref 81.3% vs 66.7%, office unaided 33% vs 0%,

office+ref 50% vs 44% (close), throughput 70 vs ~35 tok/s. Likely mechanism:

coder-agentic's 2 bits per active parameter beats Llama-4-Scout's sub-2-bit

ternary per active parameter, and per-active-parameter precision seems to

matter more here than total capacity (~30B vs ~109B).

vs. Laguna-S-2.1 ([[laguna-s-2.1-cpu-spill]], 118B, CPU-spilled MoE,

RETIRED 2026-08-03): Laguna's coding was clearly better (21/23 ≈ 91% vs

Llama-4-Scout's ~53%) — R5 not comparable (Laguna genuinely 6/6 zero-loop;

Llama-4-Scout's 0/18 is the harness artifact above, not a real result).

Llama-4-Scout's one real structural advantage: it's VRAM-only, so it can't

hit Laguna's actual cause of death — a 4-minute prompt-processing tax from

CPU-resident experts at long context, plus ~30% of real Claude Code requests

silently returning empty from a sampler mismatch the grill never exercised.

That advantage is architectural, not proven — Llama-4-Scout never had a

real-use trial the way Laguna did, and its coding score is weaker to begin

with.

Verdict: REJECTED, no role

Doesn't beat either comparison model on quality, and realcase — the suite

that most resembles Giovanni's actual work — never once succeeded across 3

reps, failing a different way each time (NO_GDS, SPEC_PARSE, RUNAWAY).

That pattern (0/3, three distinct failure modes) reads as a genuinely low

true rate, not an unlucky sample of something like coder-agentic's ~30-40%

(P(0/3) at a 30% true rate is still ~34%, so it's not statistically

decisive alone, but combined with the weaker coding score it doesn't

overcome the prior).

The trial's actual value: a real, reusable finding, not a promoted model.

MoE-at-extreme-quantization surviving where dense-at-extreme-quantization

collapsed is new information for this box, gathered nowhere else tonight or

before. Worth remembering if a future genuinely-good MoE model needs to be

squeezed into an unusually tight VRAM budget — extreme quant is a real lever

for MoE in a way it was already known NOT to be for dense

([[devstral-2-123b-iq1s]]).

Cleanup

GGUF deleted 2026-08-16 (27.25 GiB reclaimed). Re-fetch:

hf download unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF

Llama-4-Scout-17B-16E-Instruct-UD-TQ1_0.gguf. Trial/grill scripts kept at

bench/trial-llama4-scout.sh / bench/grill-llama4-scout.sh (with

--skip-chat-parsing already wired in) in case of a future re-test —

e.g. if grill_round3.py's extractor is ever taught Llama-4's bracket

syntax, or a future llama.cpp release fixes the underlying peg-native

grammar for this template.