llama-3-3-70b-iq3xxs

llama33-70b — Llama-3.3-70B-Instruct UD-IQ3_XXS — REJECTED TWICE, REMOVED 2026-07-29

unsloth/Llama-3.3-70B-Instruct-GGUF:UD-IQ3_XXS, 25.76 GiB. Dense 70B, llama

arch: 80 layers, 64 q-heads / 8 kv-heads, head_dim 128, vocab 128256, 128k

native ctx, rope_theta 500000. Meta, Dec 2024.

Status: REMOVED 2026-07-29 (entry + template + 26GB weights deleted). It was

re-downloaded that morning at the user's request, fully grilled for the first

time, and removed the same day on the result. Do not re-add without new evidence.

The re-trial settled it: 16/23 and the worst R5 on the box

First run where rounds 3-5 were VALID (the peg-native 500s were fixed by the

template patch below), so this is the only complete measurement of this model:

| round | score |

|---|---|

| R1 algorithmic | 6/8 |

| R2 algorithmic | 4/5 |

| R3 tool-call/format | 5/5 — clean, zero REQ-ERRs |

| R4 agentic loop | 1/5 (+1 unrenderable, see below) |

| 23-task | 16/23 |

| R5 long-horizon | 1/6 complete, 5/6 LOOPED, 46% redundant calls (29/63) |

R5 is the worst recorded here — for scale, coder's 4/6 with 2 loops is the

repo's headline finding of agentic unfitness. Decode measured ~12 t/s.

CAVEAT — the agentic rounds are confounded by the no-mix guard. R1-R3 are

single-turn and clean (15/18); the collapse is confined to exactly the rounds

needing in-turn reasoning, which the guard forbids. A control run against the

stock template was offered and NOT done. Treat 15/18 as firm and R4/R5 as

directionally-bad-but-unproven. It does not change the verdict: even crediting

R4 a perfect 5/5 gives 20/23, still under kimi-distill (21/23) and pocket-35b

(22/23) at ~1/8th the speed and 1/8th the context.

Llama-3.3 cannot express parallel tool calls at all. The template raises

"This model only supports single tool-calls at once!" when re-rendering an

assistant message holding 2+ tool_calls — instant failure, 0 tokens (400 via

/apply-template, surfaced as 500 through the chat path). That is what killed

R4 gather_aggregate, a 3-city fan-out. Any fan-out agentic task is

structurally unrenderable on this template.

Config used in the re-trial (entry now deleted): port 9120, -c 32768,

q4_0 KV, --tensor-split 50,50, -b 1024 -ub 256, and

--chat-template-file chat-templates/llama33-nomix.jinja (load-bearing — see

below). Standalone load verified: 14958/16311 + 15144/16380 MiB, ~1.2 GiB

headroom per GPU, decode ~12 t/s. Re-confirming the quant survey: IQ3_XXS is

still the largest that fits — Q3_K_S is 28.79 GiB, leaving too little for KV.

The service unit is llama-server.service, not llama-swap.service

(CLAUDE.md's name is wrong); restarting it needs sudo, which the Claude Code

classifier blocks — the user must run it.

Why it was run, and what it settled

Four dense 70B+ candidates were rejected on triage in one day

([[model-triage-checklist]]) with the reasoning "dense 70B is the wrong shape

for this box". This one was downloaded specifically to replace that inference

with a measurement, because no dense 70B was being served at the time and

[[kimi-dev-72b]]'s rejection was a different model a full bit lower (IQ2_XXS).

Answer: dense 70B at ~3bpw LOSES to a 35B sparse MoE at ~4bpw, on both axes.

| | rounds 1-2 (valid) | decode |

|---|---|---|

| llama33-70b (dense 70B, IQ3_XXS) | 9/13 (r1 6/8, r2 3/5) | 10.7 t/s |

| kimi-distill (35B-A3B MoE, IQ4_XS) | 11/13 (r1 7/8, r2 4/5) | 102.5 t/s |

Lower score and ~10x slower. Prefer active parameter count over total.

Quant damage is NOT the explanation — and this is the important control.

Unlike [[devstral-2-123b-iq1s]] (IQ1_S, syntactically invalid output), IQ3_XXS

produced clean, correct, well-documented code on the first real generation. So

the 9/13 is a genuine capability result at a healthy quant, not an artifact.

Round-1 misses were median_two_sorted and expr_eval — the two documented

flakiest tasks in the suite.

The FIRST trial's rounds 3-5 were INVALID (superseded by the re-trial above)

The tools rounds hit [[llama-cpp-peg-native-tool-parser-500]]: HTTP 500s

("does not match the expected peg-native format"). The harness logs those as

REQ-ERR, which is indistinguishable from a real miss in the summary table.

Round 3 still scored 4/5 — the tool-CALLING tasks passed, so the model calls

tools fine.

Corrected 2026-07-29: the trigger is not "prose while tools are present"

(prose alone parses fine, verified end-to-end) — it is mixed output, a tool

call and prose in the same turn. So the consequence is narrower than first

recorded: Llama-3.3-template models are not categorically unusable for agentic

work on b10133, and a system-prompt guard forbidding mixed output clears the

500. The rounds 3-5 numbers here remain invalid regardless, since the runs were

made without that guard.

Serving facts (if ever revisited)

• 25.76 GiB weights in the 31.0 GiB pool → q4_0 KV mandatory; KV is

90 KiB/token (80 x 8 x 128 x 2 x 0.5625B), so 32k context costs 2.81 GiB.

q8_0 KV would be 170 KiB/token (5.31 GiB at 32k) and does NOT fit. The

advertised 128k needs 11.5 GiB of KV — unreachable.

• Loaded at -c 32768 --tensor-split 50,50 -b 1024 -ub 256 → 1028/1096 MiB

free, and the footprint stayed static across generation (1026/1094 after).

• -ub 256 chosen deliberately: vocab 128256 makes the logit buffer

vocab x ub x 4B ≈ 262 MiB at ub 512 vs 131 MiB at ub 256.

• Template is CLEAN of the [[jinja-system-guard-tool-parser]] guard — both

single-turn and multi-turn system+tools returned 200.

The template patch that made rounds 3-5 valid (file deleted with the entry)

chat-templates/llama33-nomix.jinja (REMOVED with the entry; recreate from

this recipe if ever needed) = the stock Meta template plus a

"never mix a tool call with prose in one reply" instruction injected into

both tool branches (the tools_in_user_message one is the default and the

one that actually fires). This is the durable form of the mitigation, since

llama-swap cannot inject a per-request system prompt.

Verified 2026-07-29 against the real 70B: prose-only, prose-with-tools,

single-tool call, multi-tool call, and tool_restraint (the task the bug

originally wrecked) all return 200 with correct behaviour. On the stock

template the same prose-with-tools case reliably 500s.

How to apply

Stop re-litigating dense 70B+ on this box; it is measured now, not assumed. And

per [[llama-cpp-peg-native-tool-parser-500]], never score a tools round without

first curling a failing request and reading the error body.