← all models

qwen3vl

Promoted  active in llama-swap.yaml · aliases: qwen3vl-30b-instruct

Grill run history

RunScoretok/stepR5Log
No results-*.log found for this id/alias.

Memory notes

qwen3vl-tool-call-loops-real-use (qwen3vl-tool-call-loops-real-use.md)

qwen3vl (remodlai Qwen3-VL-30B-A3B-Instruct AWQ) was PROMOTED and made the

box-wide default on 2026-08-24 on the strength of a clean n=3 grill (R5 5/6

complete, 0 loops in 3/3 reps, coding ~21/23). Within hours of real use in

pi, it produced a 21-call identical tool-call loop (skill_manage

repeated until pi's loop-detector blocked it, then retried the same call

forever with the same narration). The user stopped it the same day.

The grill's "0 loops" measured TEXT runaways, not TOOL-CALL fixation.

Round 5 counts cap-hits/redundancy in generated prose; it never exercises a

long multi-turn tool loop against a real harness. The signal that WAS there and

was underweighted: closedloop 0/4 unaided across all 3 reps (the only tool-use

arm in the battery) — read at the time as "weak agentic", not as "will fixate".

Why: a model can generate clean prose and still fixate on re-emitting one

tool call, because the failure needs harness state (a blocked/failed call fed

back) that the text suites never produce. Same structural gap as

[[grill-does-not-validate-real-use]] (gemma leaked <|channel> every turn and

24 grill runs saw nothing) — the battery does not validate the deployed path.

How to apply:

  • Never promote a model to DEFAULT on text-suite loop metrics alone. Treat

closedloop (the tool-use arm) as the loop gate: 0/4 unaided is a BLOCKER for

a default, whatever R5 says.

  • Before making any model the box-wide default, run it in a real harness (pi /

kilo / opencode) on an actual multi-step task with tools and watch for repeated

identical calls. Grill first, then deploy-test — n=3 clean is not sufficient.

  • Untested hypothesis for THIS model: its llama-swap entry serves

repetition_penalty: 1.0 (off) with temperature: 0.7. No repeat-penalty +

high temp is the profile most prone to tool-call fixation. Worth testing

(repetition_penalty ~1.1, lower temp) BEFORE concluding the weights are at

fault — but a config fix must be proven in a real harness, not the grill.

  • qwen3vl remains fine as a SELECTABLE vision/speed option (129 tok/s, vision,

clean text). The defect is specifically long-horizon agentic tool use.

  • RESOLVED 2026-08-24 evening: all six harnesses repointed to qwen36-35b

(Qwen3.6-35B-A3B AWQ, 88 tok/s, vision, 180224 ctx verified same day). Chosen

over the faster gemma-awq (93 tok/s, 44/46 coding) because gemma scores 0/8 on

the UNAIDED closed loop while qwen36-35b is the second model ever non-zero

there (4/8) — picking a 0/8 model right after abandoning one for tool-call

looping would repeat the mistake. fable-fusion is the only better loop-closer

(6/8) but runs at ~33 tok/s. Costs accepted: 9 cap-hits (2nd worst), verbose

reasoning model at ~71 s/task.

See [[qwen3vl-30b-instruct-awq-trial]] for the grill numbers,

[[round5-is-a-sample-not-a-measurement]] and

[[single-run-scores-are-unreliable]] for why n=3 still was not enough here.

qwen3vl-30b-instruct-awq-trial (qwen3vl-30b-instruct-awq-trial.md)

remodlai/Qwen3-VL-30B-A3B-Instruct-AWQ (W4A16, group 128, gemm; vision tower

+ mlp.gate left unquantized). **VERDICT 2026-08-24: PROMOTE as a selectable

vision option.** Meets the user's spec — vision + 2-GPU + good coding +

Mellum-fast — and is the only model on the box combining Mellum-class speed +

vision + solid coding + clean R5 (0 loops). Not the strongest coder/klayout

on the box ([[coder-agentic-regrilled]] and [[qwen36-35b-awq-trial]] beat it

there); it's the vision + speed pick.

Lives at /home/giovanni/models-nvme/Qwen3-VL-30B-A3B-Instruct-AWQ (17 GB, 6

shards, ~51 GB on disk because hf download --local-dir keeps the blob cache —

the model itself is 17 GB). Downloaded via hf download (huggingface-cli is

DEPRECATED on this box's hub 1.28.0 — it just prints help).

Why this model

User wants vision + 2-GPU + good coding + Mellum-fast. Mellum (2.5B active,

text-only) hit ~130 tok/s. This is 3.3B active with a vision tower, so text

decode should be Mellum-class; the ViT only costs on image prefill. The

Thinking sibling was already grilled here and NOT promoted

([[qwen3-vl-thinking-trial]]: closed-loop false-pass; vision 7/7 but realcase

only via API ref). This is the Instruct variant — terse, no reasoning-token

tax, the closer match to "fast like Mellum". Apples-to-apples vs

[[gemma-4-26b-awq-vllm]] (44/46, 18/18 office, 93 tok/s) and

[[qwen36-35b-awq-trial]] (KLayout+ref 15/16, realcase 7/7).

Serving — VERIFIED

vLLM 0.26.0 + transformers 5.14.1 (vllm-env). TP=2 mandatory: 17.9 GiB AWQ

overflows one 16 GiB card (vram_fit says TP=1 is 5 GiB short). Script:

bench/serve-qwen3vl-30b-instruct-awq.sh, port 9166, served-model-name

qwen3vl-30b-instruct.

``

CUDA_DEVICE_ORDER=PCI_BUS_ID vllm serve <path> \

--served-model-name qwen3vl-30b-instruct --tensor-parallel-size 2 \

--quantization awq --kv-cache-dtype fp8 --max-model-len 180000 \

--max-num-seqs 8 --gpu-memory-utilization 0.97 \

--enable-auto-tool-choice --tool-call-parser hermes --trust-remote-code

`

  • --tool-call-parser hermes (NOT qwen3_coder): Qwen3-VL lineage per the

vllm-serving skill parser table.

  • Loads in ~16s (6 shards) + ~2min CUDA-graph profiling → startup ~132s.
  • Available KV cache memory: 4.14 GiB; GPU KV cache size: 180,768 tokens.
  • Evict the resident driver first (qwen38-mtp on :9165 holds both cards) +

reap VLLM::EngineCore/VLLM::Worker orphans ([[vllm-orphan-worker-vram-squat]]).

The Claude Code auto-classifier BLOCKS killing qwen38-mtp (a process not

started this session) — user ran the eviction via !.

Funded context is 180K, NOT 262K — three numbers that disagree

| source | funded ctx |

|---|---|

| model native (max_position_embeddings) | 262144 (no YaRN — 1M needs rope scaling, not configured) |

| vram_fit.py @0.9 FP8 KV | 205,346 (12% conservative, usually right) |

| vLLM actual @0.97 FP8 KV | 180,768 ← the real number |

vram_fit was over-optimistic here because vLLM's CUDA-graph memory profiling

(default since v0.21) reserves extra: util 0.97 is *effectively* 0.9373. vLLM

refused --max-model-len 234000 with the exact ceiling (180768) in the error.

Trust the vLLM error, not vram_fit, for the final number — but vram_fit still

correctly proved TP=2 was needed. Still ~1.4× Mellum's 131K.

Speed — VERIFIED Mellum-class

  • 129 tok/s decode (59 out-tok / 0.46s, 13-tok prompt → essentially pure

decode). Confirms 3.3B-active vision MoE decodes at Mellum speed. The user's

"fast like Mellum" target is MET.

  • Prefill ~1,000 tok/s (158K in ~155s) — slower than Mellum's 4,800 (30B MoE,

TP=2 comm, AWQ dequant) but irrelevant at the grill's 32K budget.

Needle — VERIFIED 4/4 PASS to 177K

bench/needle.py, label→actual ratio 1.58× (label 100000 → 158,487 actual;

needle.py docstring's 1.7× is an upper bound). Instruct model answers in

content (no reasoning_content).

| label | actual | depth | result | wall |

|---|---|---|---|---|

| 100000 | 158,487 | 0.1 | PASS | 162.7s |

| 100000 | 158,487 | 0.5 | PASS | 159.2s |

| 100000 | 158,487 | 0.9 | PASS | 114.1s |

| 112000 | 177,492 | 0.5 | PASS | 140.3s |

Retrieval works right to the 180K edge. ctx=0tok in the output is a

/tokenize format mismatch (vLLM vs llama-server), NOT a failure — pp=158487

is the real prefill count. Compare [[mellum2-12b-a25b-trial]] 9/9 @122K — this

beats it on window (177K vs 122K) at the same decode speed, with vision added.

Vision — VERIFIED

DADa.png (276 KB) → "Orange." via Python requests. NOTE: a 276 KB image

base64 (368 KB) exceeds shell ARG_MAX as a curl -d argv ("Argument list

too long") — the grill's pre-battery vision probe was rewritten to use Python

requests (the real suites grill_klayout_vision.py already use requests, so

they were always fine).

Grill — n=3, 2026-08-24 (bench/grill-qwen3vl-30b-instruct-awq-20260824-1626)

bench/grill-qwen3vl-30b-instruct-awq.sh, GRILL_REPS=3, GRILL_TEMP=0.2,

GRILL_MAX_TOKENS=32000, port 9166. 14 suites × 3 reps, ~27 min total.

Coding (r1-r5 aggregate): 20/23 · 21/23 · 21/23 — ~90%, stable.

r1 6/7/7 of 8, r2 4/4/4 of 5, r3 5/5/5 of 5, r4 5/5/5 of 5.

R5 — the decisive finding: 5/6 complete, 0/6 looped in ALL 3 reps

(redundant 3.1 / 6.2 / 0.0%). [[mellum2-12b-a25b-trial]]'s killer was 31% R5

loops; this model has zero in three independent runs. The blocker is fixed.

| suite | rep1 | rep2 | rep3 |

|---|---|---|---|

| klayout | 0/8 | 1/8 | 2/8 (unaided ~1/8, weak) |

| klayout+ref | 5/8 | 4/8 | 4/8 (~4.3/8, 54%) |

| klvision | 5/6 | 5/6 | 5/6 (stable 83% — good vision discrimination) |

| closedloop | 0/4 | 0/4 | 0/4 (unaided = 0) |

| closedloop+ref | 4/4 | 4/4 | 2/4 (83% with ref) |

| office | 0/9 | 0/9 | 0/9 (unaided = 0; office vision floor + API recall) |

| office+ref | 6/9 | 6/9 | 9/9 (78% with ref) |

| realcase | 0/7 | 4/7 | 0/7 (weak; 0 full passes, 1 rep partial 4/7) |

Envelope: CLEAN — 371 turns, 0 leaks, 0 tool problems. 4 abnormal

finish_reason (1.1%) = runaways, all length (32K cap hit): expr_eval-rep1

(271s), office-rep1, office-rep3, closedloop-ref-rep3. Nondeterministic, on

failing tasks, NOT on R5 agentic loops. n=1 looked perfectly clean; n=3

surfaced these — [[single-run-scores-are-unreliable]] vindicated again.

vs coder-agentic (the comparison the user asked for)

| dimension | qwen3vl-30b-instruct | coder-agentic (Qwen3-Next 80B IQ2_XXS) |

|---|---|---|

| coding | ~21/23 (90%) | 43/46 (94%) — slightly ahead |

| R5 | 5/6, 0 loops | 6/6, 0 loops — edges it |

| KLayout+ref | 4.3/8 (54%) | 13/16 (81%) — much better |

| office+ref | 78% | perfect |

| realcase | weak (0 full/3) | ~40% (3/10) |

| speed | 129 tok/s | 70 tok/s — qwen3vl 1.8× faster |

| context | 180K | 262K |

| vision | YES | no |

Different roles: coder-agentic = stronger pure-coding/klayout/office/realcase

and stays the text default. qwen3vl-30b-instruct = faster + vision. Among

vision models, [[qwen36-35b-awq-trial]] is stronger on KLayout/realcase (15/16,

7/7) but qwen3vl-30b is ~1.4× faster and has stable 5/6 klvision.

Restore after serving

The box's default driver qwen38-mtp is DOWN while this is served. Restore it

after: re-run its original vllm serve` (TP=2, :9165, MTP, qwen3_coder parser).

qwen3vl-30b-instruct should be added to llama-swap.yaml / models.json as a

SELECTABLE entry (port 9166, hermes parser) — it does not replace the driver.

llama-swap.yaml entry

  "qwen3vl":
    aliases: [qwen3vl-30b-instruct]
    # remodlai build of the SAME base as qwen3-vision above (Qwen3-VL-30B-A3B-Instruct),
    # but a DIFFERENT quantizer — and it grills BETTER than the QuantTrio build.
    # The QuantTrio `qwen3-vision` comment above calls that build "the LEAST
    # repeatable model measured here (spread 6)" and 2/4 closed-loop. This remodlai
    # build (W4A16, group 128, gemm; vision tower + mlp.gate left unquantized),
    # grilled n=3 on 2026-08-24, is the OPPOSITE on the metric that retired Mellum:
    #   R5 5/6 complete with ZERO loops across all 3 reps (Mellum2 was 31% loops).
    # coding r1-r5 ~20-21/23 (90%), klvision 5/6 STABLE (3/3 reps), closedloop-ref
    # 10/12, office-ref 21/27 (78%), KLayout+ref 4.3/8, realcase weak (0/7,4/7,0/7).
    # Envelope CLEAN (4 runaways = 32K cap-hits on failing tasks, NOT on R5).
    # See auto-memory/qwen3vl-30b-instruct-awq-trial.md. PROMOTED as the
    # vision + speed pick: 129 tok/s (Mellum-class, 1.8x coder-agentic), vision
    # works, klvision stable. Faster than qwen36-35b; qwen36-35b beats it on
    # KLayout/realcase. NOT a driver replacement — a selectable vision option.
    #
    # SHORT primary name `qwen3vl` (user preference); long form `qwen3vl-30b-instruct`
    # is an ALIAS above so the descriptive name still routes. DISTINCT from
    # `qwen3-vision` (the QuantTrio build of the same base) per the file convention:
    # a client config must never be ambiguous about which artifact answered —
    # same convention as fable-fusion vs fable-711-gptq.
    # LOCAL PATH (not HF id): the remodlai weights live on the NVMe at
    # /mnt/models/Qwen3-VL-30B-A3B-Instruct-AWQ (17 GB, 6 shards),
    # downloaded via `hf download remodlai/Qwen3-VL-30B-A3B-Instruct-AWQ --local-dir`.
    # -m-style absolute path also sidesteps HF resolution / offline -hf failures.
    #
    # CONTEXT 180000 (not 176128 like the QuantTrio build): this build funds
    # 180,768 KV tokens at 0.97 fp8 KV (vLLM's own startup ceiling — verified by
    # needle 4/4 PASS to 177,492 actual). vram_fit over-predicted 205K; CUDA-graph
    # profiling reserves extra so util 0.97 is effectively 0.9373. Trust the vLLM
    # error, not vram_fit, for the final number.
    # PARSERS hermes (JSON-in-XML, Qwen3-VL lineage), NO reasoning-parser (Instruct,
    # no think block — same as the qwen3-vision sibling).
    # ${vllm_wrap_bin} (not bare ${vllm_bin}): orphan-reaping wrapper so a mid-run
    # decode crash self-heals instead of leaving VLLM::Worker_TP* pinning ~14 GiB
    # per GPU — auto-memory/vllm-orphan-worker-vram-squat.md. Matches the 2026-08-20
    # qwen3-thinking/qwen3-instruct entries.
    cmd: |
      /usr/bin/env
      HF_HOME=/home/giovanni/.cache/huggingface
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      VLLM_STDERR=/tmp/qwen3vl-via-swap.err
      NCCL_ASYNC_ERROR_HANDLING=1
      ${vllm_wrap_bin} serve /mnt/models/Qwen3-VL-30B-A3B-Instruct-AWQ
      --served-model-name qwen3vl
      --tensor-parallel-size 2
      --host 127.0.0.1 --port 9166
      --max-model-len 180000
      --max-num-seqs 8
      --gpu-memory-utilization 0.97
      --kv-cache-dtype fp8
      --override-generation-config '{"temperature":0.7,"repetition_penalty":1.0}'
      --enable-auto-tool-choice
      --tool-call-parser hermes
      --trust-remote-code
    proxy: http://127.0.0.1:9166