← all models

devstral-small-2-24b

Retired  no longer in llama-swap.yaml — entry and/or weights removed after this verdict.

Grill run history

RunScoretok/stepLog
No results-*.log found under this name.

Memory note

devstral — RETIRED 2026-07-27, NO LONGER SERVABLE

Removed from the box 2026-07-27 ([[2026-07-27-model-cleanup]]): entry

deleted from llama-swap.yaml, 19GB Q6_K weights wiped from the HF cache.

Rationale: a genuine grill-round-5 miss, and being the Mistral-lineage

novelty didn't justify a slot at 6x kimi-distill's decode speed for the same

21/23. Everything below is history.

Original entry — mistralai/Devstral-Small-2-24B-Instruct-2512 (Q6_K, 128k, port 9116)

Added 2026-07-26 (commit 0bb5d30e). Found by the "anything new worth testing?"

scan — the one model that brought a genuinely new capability rather than a

variant of what we run (see [[davidau-catalog-verdicts]] for the rejects that

scan also produced). The box's ONLY non-Qwen/GLM/gpt-oss lineage.

  • arch mistral3 (Mistral-Small-3), runs on b9870 with no new build. Dense

23.6B, no MTP head (Mistral doesn't ship one) → ~17 t/s, the slowest

coder here. Every other coder on the box is a fast MoE; this is the speed cost.

  • Purpose-built AGENTIC CODER: post-trained with All Hands AI for OpenHands-style

repo agents, ~68% SWE-bench Verified (2512/v2, up from v1's 46.8%). Apache-2.0.

  • Vision-capable (mmproj exists) but served text-only here — NOT in

sync-models VISION_IDS. Add --mmproj later to enable.

  • Quant Q6_K 18GB (near-lossless per VRAM policy). Q8_0 23.3GB also fits.
  • Sampler temp 0.15 (Mistral's card coding default, lower than the box's 0.2),

top-p 0.95.

Grill 21/23 (bench/results-devstral-20260726-154318.log, temp 0.15)

r1 7/8, r2 5/5, r3 4/5, r4 5/5.

  • THE defining trait: terse, non-thinking. 150-900 output tok/task vs

[[gpt-oss-20b]]'s 4-16k and the DavidAU expansions' 6-16k. Whole grill finished

in ~2 min despite 17 t/s. r4 agentic = 374 tok total / 37 tok/step (leanest

measured on this box).

  • Zero runaways — expr_eval failed CLEANLY at 903 tok (a real code bug,

UnboundLocalError), never the 16k-cap loop the DavidAU models hit. Rock stable.

  • r2 is top-tier: passes BOTH weighted_interval_scheduling AND text_justify

— [[glm-4.7-flash]] fails both; tess/vision-coder fail text_justify. Its raw

algorithmic coding (r1+r2 = 12/13) beats glm-flash and matches the champions.

  • Two misses: expr_eval (lineage-wide hard case, ~everyone fails it) and

json_strict — emitted correct JSON but WRAPPED IN A ``json markdown fence

instead of bare. Watch this: it may markdown-fence structured output.

Repo-fix probe (real agentic loop: file tools + a self-contained test runner,

multi-file bug traced test -> cart.py -> pricing.py; harness in scratchpad

repo_fix_probe.py). devstral PASS in 7 steps / 13.3s, one-shot correct edit.

Head-to-head [[gpt-oss-20b]] (coder) also PASS but 9 steps / 11.2s, introducing a

syntax error then self-recovering. Single sample, but on its designed task

devstral was the cleaner of the two.

Placement / the kimi-distill rivalry (IMPORTANT for keep/drop)

21/23 ties [[kimi-distill]] — which runs **~6x faster (100 t/s) with 2x the

context (256k)** at the same score. On the pure grill number kimi-distill

dominates it. Devstral's case rests ENTIRELY on the agentic-coding axis the grill

under-tests: terse one-shot repo edits, zero-runaway stability, SWE-bench pedigree,

Mistral-lineage diversity, and a 384k context ceiling (128k practical at q8 here).

KEPT as the "repo-scale agentic coder," NOT a gpt-oss replacement. If VRAM/disk

tightens, it's the first coder to reconsider (the kimi-distill repo-fix probe was

offered but not yet run — that would settle whether the clean one-shot edit is a

real edge or noise).

Ops gotchas

  • Split --tensor-split 47,53 is MEASURED to put the faster 5060 Ti (CUDA1)

as the heavier card per user request: 15509MiB/802 free vs 4060's 14981/1399.

At 128k q8 both cards are near the ceiling — 50,50 leaves the 4060 heavier,

45,55 OOMs the 5060 (its ~1.1GB compute buffer overflows). 47,53 is the

crossover and matches [[gpu-device-ordering]]'s recorded balance point.

  • jinja system-guard PATCHED — the embedded Mistral template only consumes

messages[0] as system; any later system message hit raise_exception('Only

user, assistant and tool roles are supported') → 500. Patched copy

chat-templates/devstral-small-2-24b-2512.jinja renders a non-first system

message as its own [SYSTEM_PROMPT]…[/SYSTEM_PROMPT] block (Mistral's

native delimiter, NOT ChatML — this is the Mistral variant of the

[[jinja-system-guard-tool-parser]] fix), wired via --chat-template-file.

Verified: non-first system → 200 & obeyed; system+non-first-system+tools → 200

& clean tool_call. Extraction gotcha here: tokenizer.chat_template` value

length is u64 and the key is at byte ~8.38M (large F16-ish header).

  • 256k needs q4 KV (128k q8 already fills the cards). Raise -c only for a

real long-context job, and drop cache-type with it.

  • Card's own caveat "current llama.cpp implementations may not be accurate" —

tool-calling parsed fine in practice; the json_strict fencing is the one

formatting wart seen.