mellum2-thinking-vllm-entry

RETIRED 2026-08-26 — entry REMOVED, 23 GB BF16 safetensors DELETED. The

vLLM BF16 Thinking entry (renamed mellum2-thinker during the 2026-08-26 rename)

was removed: dominated by coder-agentic on coding (43/46 vs 17.7/23), KLayout

(13/16 vs 2.3/8), office (ceiling-with-ref vs ~2/9); its one edge — realcase 7/7×3

— is on a noisy ~40%-baseline axis ([[coder-agentic-regrilled]]: p=0.65), and it

costs both cards (TP=2, evicts freetoken-driver + the 4060 Ti pool). The vLLM BF16

*Instruct* sibling (mellum2, formerly mellum2-instruct) was grilled 2 reps the

same day and removed too — it tracked the Thinking (~17.5/23 coding, 0 R5 loops but

runaways, klayout+ref ~0.5/8, office ~1/9, realcase 0/2) and was WORSE than the

Q8_0 llama.cpp mellum on coding. The Mellum2 build that stays is mellum —

the Q8_0 GGUF Instruct on llama.cpp, single-card, ~21/23 coding ([[mellum2-12b-a25b-trial]]).

Everything below is history; re-adding needs a 23 GB re-download from HF.

Wired 2026-08-25 as llama-swap entry mellum2 on :9163. Distinct from the

existing mellum entry, which is the *Instruct* sibling at Q8_0 on llama.cpp —

JetBrains ships Thinking and Instruct as separate checkpoints (Base -> SFT ->

RLVR), not two quants of one model. Vendor numbers, Thinking vs Instruct:

LiveCodeBench v6 69.9 vs 37.2, BFCL v3 69.4 vs 66.3.

Chosen because the Instruct was grilled n=13 and DEMOTED for exactly the

long-horizon axis a reasoning variant might fix ([[mellum2-12b-a25b-trial]]:

R5 loops 31%, realcase 23%, KLayout+ref a hard 3/8 ceiling).

vLLM BEAT llama.cpp AT SINGLE STREAM — first time on this box

The standing assumption here is "llama.cpp for solo, vLLM for concurrency"

([[x8-concurrency-is-nearly-free]], the -x2 entries). **That is false for this

model.** Same BF16 weights, same 131072 window, same prompts, unquantized KV on

both sides (bench/conc_bench.py, raw rows in bench/mellum2-concurrency.jsonl):

| N | llama.cpp BF16 --parallel 4 | vLLM BF16 TP=2 | ratio |

|---|---|---|---|

| 1 | 66.4 tok/s | 97.5 | 1.47x |

| 2 | 96.1 | 136.8 | 1.42x |

| 4 | 133.1 | 198.3 | 1.49x |

| 8 | 131.9 (walled) | 288.7 | 2.19x |

| TTFT | 0.45 s | 0.08 s | |

llama.cpp saturates at N=4 and the 8th slot buys literally nothing (2.01x ->

1.99x) — the wall [[x8-concurrency-is-nearly-free]] documents. vLLM keeps

scaling to 2.96x at N=8. So the engine choice here is not a trade-off at all;

vLLM wins on every measured axis at identical precision.

Do not generalise this to the whole box. It is a 2.5B-active MoE at BF16 —

lots of small GEMMs, which is where vLLM's batching and CUDA graphs pay off

most. The dense entries may still favour llama.cpp.

The KV surprise: 131072 is FREE at BF16 on 2x16 GiB

BF16 weights are 11.4 GiB per card after TP=2, leaving only ~2.2 GiB each

for KV. The naive conclusion (mine, before measuring) was that max-model-len

would have to drop to ~65536. Wrong: vLLM funds 291,966 KV tokens, i.e.

2.23x concurrency at the full 131072.

Reason: Mellum2 is 3 sliding (window 1024) : 1 full over 28 layers, and

vLLM's hybrid allocator only pays full length on the 7 full-attention

layers ([[kv-sizing-full-attention-layers]]). ~15.7 KiB/token, not the

~56 KiB/token an all-layers estimate gives.

Consequence: do NOT add --kv-cache-dtype fp8 to this entry. Nearly every

other vLLM entry here uses it to buy context. Here KV is not the constraint, so

fp8 would cost accuracy for headroom nothing needs.

Tensor split, measured (kept for the record — that GGUF is now deleted)

bench/mellum2_split_sweep.sh, log bench/mellum2-split-sweep-20260825-1852.log:

| split | tok/s | 5060 Ti | 4060 Ti |

|---|---|---|---|

| default / 50,50 | 64.5 | 12649 MiB | 13557 MiB |

| 42,58 | 66.3 | 14249 | 11953 |

| 38,62 | 67.1 | 15047 | 11153 |

Loading the faster 5060 Ti harder wins, as [[decode-bandwidth-ceiling]]

predicts. Took 42,58 over the marginally faster 38,62 because 38,62 leaves only

~1.2 GB free on the 5060 Ti for a 1.2% gain. Remember the llama.cpp entry sets

no CUDA_DEVICE_ORDER, so bare ordering applies and the second number

is the 5060 Ti's share ([[gpu-device-ordering]]) — the opposite of the

neighbouring mellum / glm-ocr entries, which set PCI_BUS_ID

([[cuda-device-order-pci-bus-id]]).

At BF16 there IS no single-card option — 22.64 GiB will not fit a 16 GiB card,

so "both GPUs or just the 5060 Ti" only has one answer at this fidelity. Q8_0

(12.04 GiB) is the single-card build if speed ever outranks accuracy.

A false-failure grep that reported a healthy server as dead

vllm/trial-mellum2-thinking.sh originally grepped the startup log for

free memory as a failure signature. vLLM prints

Free memory on device (15.31/15.58 GiB) on startup on every healthy boot,

so the script declared FAILED to start on a server that had come up fine with

the full 131072 window — and the engine kept running, unowned, behind the

"failure". Fixed by anchoring every pattern to something that only appears when

the engine actually dies (Traceback (most recent call last), raise ValueError,

torch.OutOfMemoryError, Engine core initialization failed).

Same family as [[eval-false-pass-adjacent-data]], inverted: a false FAILURE

rather than a false pass. When a start-detector fires, confirm against the port

and the KV-cache line before believing it.

Verified, not assumed

• Entry command run verbatim (macros expanded, ${vllm_wrap_bin}, quoted

--override-generation-config) on :9163 — serves, 291,966 KV tokens.

• vllm/toolcheck.py 5/5 with --tool-call-parser hermes (vendor's).
• Reasoning arrives under reasoning, reasoning_content absent —

confirming [[vllm-reasoning-key-not-reasoning-content]]. All 11 grill suites

read both keys, so the harness is thinking-model-ready.

• Sampling is the vendor's temp 0.6 / top-p 0.95 / top-k 20.
• No orphaned VLLM::Worker_TP* after teardown

([[vllm-orphan-worker-vram-squat]] did not trigger).

Constraints

Needs BOTH cards, so it cannot coexist with freetoken-driver (owns the

5060 Ti) — sudo systemctl stop freetoken-driver first, same as gemma-awq /

qwen3-coder / fable-fusion. It also evicts the whole 4060 Ti swap pool

(mellum, bonsai, glm-ocr, qwen3vl-8b).

mellum2 is in sync-models.sh's SKIP_IDS (vLLM entries size with

--max-model-len, not -c) and hand-added to models.json's vllm provider,

text-only. The statusline's existing *mellum* glob already resolves it to

131072 ([[sync-models-hook]]).

Downloads: BF16 safetensors in the HF cache. The BF16 GGUF was benchmarked and

then DELETED 2026-08-25 (22.6 GB reclaimed) — vLLM beat it on every axis, so

a llama.cpp fallback did not earn the disk. If vLLM's mellum support regresses on

upgrade ([[vllm-0271-upgrade]] is the precedent), re-pull

JetBrains/Mellum2-12B-A2.5B-Thinking-GGUF-BF16; the sizing work is recorded

above. The first GGUF download died silently at 18.3/24.3 GB — the

[[background-task-silent-kills]] pattern — and a hf download retry loop resumed

it from the .incomplete file at no cost.

GRILL n=3 (2026-08-25, 98 min, 33 suites, 324 turns)

bench/grill-mellum2-thinking.sh, GRILL_TEMP=0.2, GRILL_MAX_TOKENS=32000 via

:9163 — same settings as the Instruct's n=13 run, so these compare directly.

Text-only, so vision suites were correctly omitted and office caps at 6/9

([[office-suite-vision-floor]]).

| suite | rep1 / rep2 / rep3 | Thinking | Instruct (n=13) |

|---|---|---|---|

| coding r1-r4 | 17 / 19 / 17 | ~17.7/23 | ~21/23 |

| R5 long-horizon | 5/6, 5/6, 5/6 complete | looped 0/6 ALL 3 REPS | LOOPED 31% |

| klayout unaided | 0/8 x3 | 0/8 | ~0.4/8 |

| klayout+ref | 3/8, 2/8, 2/8 | ~2.3/8 | 3/8 ceiling |

| office | 2/9, 0/9, 4/9 | ~2/9 | ~3.6/9 |

| office+ref | 0/9, 2/9, 0/9 | ~0.7/9 | ~4.8/9 |

| realcase gate | PASS 7/7 x3 | 100% | 23% |

| envelope | 0 leaks, 0 tool problems | CLEAN | CLEAN |

| stream parity | 0 stream-only leaks x3 | clean | clean |

| speed | ~90-97 tok/s | | ~130 |

**VERDICT: KEEP as the multi-step / agentic trial pick. It is NOT a coding-round

upgrade and NOT a KLayout or office model.**

It fixes exactly the two things that demoted the Instruct, and nothing else.

realcase went 23% -> 3/3, and R5 looping went 31% -> **0/6 in all three

reps** — that is the axis that retired [[gpt-oss-20b]], [[tess-4-27b]],

[[devstral-small-2-24b]] and [[laguna-xs-2.1]]. On the multi-step task that

actually resembles real work here, it is a different and better model.

The cost is real and was not predicted from the vendor's benchmarks. Short

single-shot coding DROPPED (~17.7/23 vs the Instruct's ~21/23) despite

LiveCodeBench v6 69.9 vs 37.2. office+ref collapsed to ~0.7/9 from ~4.8/9.

A vendor benchmark gap of that size predicted the realcase win but was actively

misleading about r1-r4 — one more instance of [[grill-does-not-validate-real-use]]

pointed the other way.

THE FAILURE MODE CHANGED FROM LOOPS TO RUNAWAYS. 6 cap-hits at the full

32000-token budget across the battery (2.2% of 324 turns had an abnormal

finish_reason), on DIFFERENT tasks each time (expr_eval, bounded_blocking_queue,

grow_shapes...). The Instruct's failure was repeating itself; this one's is

thinking without terminating. Cap-hit COUNT is the leading indicator

([[qwen38-ridge-rejected]]) — watch it if this is ever promoted, and note that

R5's clean 0/6 does NOT cover it, because R5 measures redundancy, not budget

exhaustion.

KLayout+ref is unchanged at ~2.3/8 (Instruct 3/8, if anything marginally

worse). So the 3/8 ceiling is a **pya API-knowledge limit, not a reasoning

limit** — reasoning cannot substitute for the API recall that

[[klayout-api-recall-vs-composition]] describes. Still far short of qwen36-35b's

15/16 ([[klayout-office-model-comparison]]).

n=3, not n=13. The Instruct looked fine at n=3 too

([[round5-is-a-sample-not-a-measurement]]). But R5 was identical in all three

reps here (5/6 complete, 0 looped, 0.0% redundant), which is a tighter

distribution than the Instruct ever showed.