qwen3-vl-thinking

qwen3-vl-thinking-trial (qwen3-vl-thinking-trial.md)

Trialled 2026-08-07. Same base, publisher and sampler as the Instruct variant

already measured as qwen3-vision, one variable changed (Instruct →

Thinking, hence --reasoning-parser qwen3), so this is a sound A/B — unlike

the cross-build comparison rejected in [[genesis-hermes-v5]].

Served: --max-model-len 131072 (NOT 204800 — at 204800 it refuses to

start: 4.69 GiB KV needed vs 3.01 GiB available, vLLM's own estimated ceiling

is 131392), TP=2, --max-num-seqs 16, --kv-cache-dtype fp8,

--tool-call-parser hermes, --reasoning-parser qwen3. ~140 tok/s.

| Suite | thinking | qwen3-vision (Instruct) | fable-711-gptq |

|---|---|---|---|

| KLayout from memory | 0/8 | 1/8 | 0/8 |

| KLayout + API ref | 6/8 | 5/8 | 7/8 |

| Vision ground truth | 4/6 | 5/6 | 4/6 |

| Closed loop | 0 real (see below) | 0/2 | 2/3 — now suspect |

| Office | 3/5 | 0/3 | 2/5 |

| Coding R1 | 3/8 @16k → 6/8 @32k | — | — |

| R2 / R3 / R4 | 5/5 / 5/5 / 5/5 | — | — |

THE HYPOTHESIS FAILED. The trial existed to test whether explicit

reasoning closes the KLayout closed loop (write geometry code → render → look

→ judge), where Instruct scores 0/2. It appeared to score 2/3. It did not:

• loop_array — code wrote int(2 * 1000 / layout.dbu), i.e. µm→nm and THEN

divided by dbu, producing squares of 2000 µm instead of 2 µm and a bbox of

17000x12000 µm instead of 17x12. The error is uniform, so KLayout

auto-zooms and the render is pixel-identical to a correct one; the vision

question only asks count/rows/columns, so the model correctly described a

layout that was entirely wrong. Graded PASS.

• loop_ring — failed outright at the code stage (kdb.Rectangle, which does

not exist). Never reached vision.

So zero of two scenes produced correct geometry. Grader hole and fix

recorded in [[eval-false-pass-adjacent-data]] and

[[klayout-office-grill-rounds]]. **fable-711-gptq's 2/3 was graded by the

same one-sided check and its artifacts are gone — treat the closed-loop column

in [[klayout-office-model-comparison]] as unverified until re-run.**

Reasoning disengages on perception. Vision scenes cost 69–508 tokens and

0.9–4.0 s; single KLayout coding tasks cost 8,000–12,000. Whatever the extra

capacity buys, it is not applied to looking at an image — which is exactly why

the closed-loop hypothesis was wrong in the first place.

The 16k coding score is a budget artifact — a genuinely new finding. R1

scored 3/8 at GRILL_MAX_TOKENS=16000 with 3 tasks pinned at the cap. At

32000 it scores 6/8: median_two_sorted (9,666 tok), articulation_points

(10,172) and lru_ttl (14,684) all recover — they needed 10–15k and 16k was

cutting them off mid-answer. But expr_eval and wildcard_match are TRUE

runaways: 12,108→32,000 and 16,000→32,000, expanding to fill whatever cap

they are given, at ~270 s each. So: 6/8 with a ~25% runaway rate, not 3/8.

Contrast [[kimi-distill]], where a 32k budget scored WORSE — a thinking model's

budget sensitivity is per-model and must be measured, not assumed either way.

The real gain is office. 3/5, and formula_totals completed the full

chain — real spreadsheet formulas → structural check → LibreOffice PDF export

→ read {"grand_total": 260, "tallest": "Gadget"} off the rasterised page.

openpyxl never evaluates formulas (only LibreOffice does, during export), so

reading the total off the render is the only proof the chain worked. Office

was previously a clean failure sweep for every model tried here.

Verdict: not a promotion. It does not beat qwen3-vision at the thing it

was brought in for, and the one clear win (office) is a niche this box rarely

exercises. The 6/8 vs 5/8 KLayout edge is inside the variance documented in

[[single-run-scores-are-unreliable]] — one run, and not enough to act on.

Worth a re-run under the FIXED grader before any promotion is considered.

RE-MEASURED 2026-08-08 ([[regrill-20260808-four-model-trust]]), 2 repeats,

and the verdict hardens to REJECT on trust grounds: 17 cap-hits, 8.5x every

other model in the field (4/8 of R1 and 3/5 of R2 burned the full 16k budget).

R5 came in 7/12, the worst agentic score of the four. Its office 3/5 "only real

win" did not survive either — 2/7 on the non-vision subset, against glm-flash's

12/12. The R1 budget-artifact story still stands (a 32k diagnostic is owed, see

[[regrill-20260808-four-model-trust]]), but no budget change fixes 17 runaways.

Its closed-loop 2/4 false pass is now understood: [[closed-loop-never-reaches-vision]].

REVERSED IN PART 2026-08-08 by [[closed-loop-works-with-api-ref]]. Given the

KLayout API reference, this model scores a verified 8/8 on the closed loop

(4/4 code, 4/4 vision, zero runaways, 136 tok/s) — the ONLY model of three that

closes it at all; the other two manage 1/4 and 2/4. The hypothesis this trial

was built on (explicit reasoning closes the loop) was never falsified, it was

never TESTED: the suite had no API-reference mode, so stage 1 always ran from

memory where everyone scores ~0 ([[closed-loop-never-reaches-vision]]).

Its 17 cap-hits stand and are disqualifying for long algorithmic coding, but

they do NOT appear here — 0 across all four loop attempts. **Use it for KLayout

closed-loop work, with a reference loaded; keep it off R1/R2-style tasks.**

32k DIAGNOSTIC, 2026-08-08 ([[budget-sensitivity-is-a-model-property]]):

15/26 -> 19/26 at a doubled budget, for 1.25x the tokens. Partly vindicated

(6 tasks were genuinely STARVED) but partly damned — **3 tasks that PASSED at

16k regressed to full-cap runaways at 32k**, and its median output is ~12.8k

against a 16k cap, i.e. it expands to fill whatever ceiling it is given.

19/26 is still far below fable's 28/29. The coding verdict stands; the VISION

verdict ([[closed-loop-works-with-api-ref]], 8/8, zero cap-hits) is unaffected.

CORRECTED 2026-08-10: the "R1 3/8@16k is a budget artifact — 6/8@32k" claim

does not survive a fixed classifier. All four tasks that recovered at 32k passed

using FEWER than 16000 tokens (13856, 7252, 8037, 6780), so the 16k budget was

never binding — they are flaky, not starved. See [[office-loop-denominator-fix]]

sibling finding in [[budget-sensitivity-is-a-model-property]].