Active active in llama-swap.yaml
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| No results-*.log found for this id/alias. | ||||
gemma12 = cyankiwi/gemma-4-12B-it-qat-AWQ-INT4 (10.45 GiB weights), served
single-card via vLLM TP=1 on the 4060 Ti, 131072 ctx. Grilled n=3 on 2026-08-23,
156 min, 42 suites. Its llama-swap entry said "NOT GRILLED YET" until now.
| suite | rep1 | rep2 | rep3 |
|---|---|---|---|
| r1 | 7/8 | 8/8 | 7/8 |
| r2 / r3 / r4 | 5/5 | 5/5 | 5/5 |
| klayout (unaided) | 0/8 | 0/8 | 0/8 |
| klayout + ref | 2/8 | 2/8 | 2/8 |
| klvision | 4/6 | 4/6 | 4/6 |
| closedloop | 1/4 | 0/4 | 0/4 |
| closedloop + ref | 0/4 | 0/4 | 0/4 |
| office (unaided) | 0/9 | 0/9 | 0/9 |
| office + ref | 9/9 | 9/9 | 9/9 |
| R5 | 5/6 0 loops | 4/6 0 loops | 5/6 0 loops |
| realcase | PASS 7/7 | FAIL 5/7 | PASS 7/7 |
Envelope CLEAN: 376 turns, 0 control-token leaks, 0 tool-envelope problems,
0 stream-only leaks, 4 cap-hits (1.1%). ~26 tok/s single-card on the 4060 Ti
(it did ~43 at TP=2 across both cards — a dense 12B genuinely gains from the
split, unlike the 2.5B-active [[mellum2-12b-a25b-trial]]).
office 0/9 -> 9/9, three times each, nothing in between. Unaided it invents
plausible method names — cell.formula (does not exist; it is
cell.value = "=..."), add_data(titles=...) (it is titles_from_data). All
three struct stages die on that, cascading their pdf+vision stages, hence a total
zero rather than a partial score. Hand it the real signatures and it takes ALL
NINE stages including vision, reading its own rendered chart correctly
({'grand_total': 260, 'tallest': 'Gadget'} — verified, not a false pass per
[[eval-false-pass-adjacent-data]]).
But the same help does NOT work for KLayout: +9 on office vs +2 on KLayout.
Since the model demonstrably CAN use an adequate reference, the likeliest reading
is that the grill's 40-line pya signature list is insufficient for it, not that it
is incapable — the shape [[realcase-spec-plus-apiref-multiplicative]] predicts
(partial help 0-2/7, complete help 7/7). **Untested and worth testing: gemma12
driving KLayout with the full klayout-classes skill loaded.**
klvision 4/6 three times shows vision READING is fine. The closed loop is gated at
the code stage, so it never exercises that — consistent with
[[closed-loop-never-reaches-vision]].
Zero R5 loops in all three reps (completion drifted 5/6, 4/6, 5/6). It fails by
not finishing, never by running away. Contrast mellum: loops in ~31% of runs.
A genuine R5 loop is what ended gpt-oss-20b, tess-4-27b, devstral-small-2 and
laguna-xs.
realcase 7/7, 5/7, 7/7 — two clean passes, and the miss still produced a valid
executing GDS scoring 5 of 7 checks. NOT mellum's bimodal pattern (7/7 or
near-zero across varying failure stages, ~31% pass over 16 samples).
| | gemma12 | mellum |
|---|---|---|
| office ceiling | 9/9 (with ref) | 6/9 — text-only, cannot attempt vision |
| realcase | 2 of 3 pass | ~31% of 16 |
| R5 loops | 0 of 3 | ~31% of 13 |
| KLayout + ref | 2/8 | 3/8 |
| tok/s | 26 (1 card) | 137 |
| vision | yes | no |
Slower and slightly worse at pya composition; better everywhere else that matters
for real work. If you need vision or a loop-free agent, gemma12; if you need speed
on short turns, mellum.
CAVEAT: n=3. realcase is the highest-variance suite here — mellum's spanned 0/7
to 7/7 over sixteen samples ([[round5-is-a-sample-not-a-measurement]]).
"gemma12":
# SAME WEIGHTS AS gemma12-solo, CONFINED TO ONE CARD so glm-ocr can sit on the
# other. Ask for this id (not `gemma12-solo`) when you need OCR and chat at the
# same time; the `coserve` group keeps both resident.
# CARD ORDERING (verified empirically 2026-08-22, do NOT swap blindly):
# with CUDA_DEVICE_ORDER=PCI_BUS_ID the indices match nvidia-smi, which is
# the OPPOSITE of llama.cpp's bare CUDA ordering in [[gpu-device-ordering]]:
# CUDA_VISIBLE_DEVICES=0 -> RTX 5060 Ti (36 SMs, cc12.0 Blackwell) STRONGER
# CUDA_VISIBLE_DEVICES=1 -> RTX 4060 Ti (34 SMs, cc8.9 Ada) weaker
# REPINNED to the 4060 Ti 2026-08-23: the 5060 Ti is held permanently by the
# `mellum` driver peer, so the whole llama-swap pool now lives on the 4060.
# glm-ocr also takes the 4060 Ti -- it is 2.8 GB and
# ~95% prefill (measured 1952 ms prompt-eval vs 97 ms decode), so it is the
# right tenant for the weaker card.
#
# WHY NOT JUST CO-LOCATE THE 26B gemma-awq: TP=2 allocates SYMMETRICALLY, so
# a 2-GPU model co-resident with glm-ocr is squeezed on BOTH cards by
# glm-ocr's footprint on one, and the other card's spare ~12 GB is wasted.
# Measured 2026-08-22: vLLM refused at --gpu-memory-utilization 0.74 ("Free
# memory on device cuda:0 (11.3/15.48 GiB) ... less than desired"), and at
# 0.70 it loaded the weights but had under 0.91 GiB left for KV and aborted.
# One model per card sidesteps all of it: measured 14.8 GB + 2.96 GB,
# 263,563 KV tokens, GPU0 at 85% while GPU1 idled at 0% -- genuinely
# parallel, not time-sliced.
#
# FULL 131072 WINDOW EVEN ON ONE CARD. A 65536 cap was set here first out of
# caution and then REMOVED as unnecessary -- measured 2026-08-22:
# gemma12-solo TP=2, both cards, 131072 -> 846,018 KV tokens, 6.45x concurrency
# gemma12 TP=1, 5060 Ti, 131072 -> 378,419 KV tokens, 2.89x concurrency
# vLLM only needs the KV pool to hold ONE max-length request, and 378k >>
# 131k. The single-card cost is concurrency (2.89x vs 6.45x), NOT context.
# (An earlier 263,563-token figure in notes was the 65536 config -- obsolete.)
cmd: |
/usr/bin/env
HF_HOME=/home/giovanni/.cache/huggingface
CUDA_DEVICE_ORDER=PCI_BUS_ID
CUDA_VISIBLE_DEVICES=1
PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
${vllm_bin} serve cyankiwi/gemma-4-12B-it-qat-AWQ-INT4
--served-model-name gemma12
--tensor-parallel-size 1
--host 127.0.0.1 --port 9157
--max-model-len 131072
--max-num-seqs 8
--gpu-memory-utilization 0.92
--kv-cache-dtype fp8
--enable-auto-tool-choice
--tool-call-parser gemma4
--reasoning-parser gemma4
proxy: http://127.0.0.1:9157