ornith-awq-trial

ornith-awq-probe — ulkaa/Ornith-1.5-35B-A3B-AWQ-INT4 on vLLM, TRIAL 2026-08-21 (REJECTED)

The vLLM counterpart to the served GGUF [[ornith-15-35b-trial]] (same base

Ornith-1.5-35B-A3B, arch qwen3_5_moe). Standalone vLLM 0.26.0, TP=2, port

9148, --max-model-len 131072 --max-num-seqs 8 --gpu-memory-utilization 0.93

--kv-cache-dtype fp8 --reasoning-parser qwen3 --tool-call-parser qwen3_coder.

The ulkaa AWQ is compressed-tensors W4A16 INT4 (group_size 32, MSE

observer, asymmetric, Marlin WNA16 MoE kernels) — NOT classic

AWQ-with-smoothing. The vision tower is baked in as BF16 (the quantizer's

ignore list excludes model.visual.blocks.*), unlike the llama.cpp mmproj.

Shipped a recipe.yaml + coherence-results.json (all_pass). 0 downloads,

unknown quantizer — but the GO/NO-GO probe cleared it.

GO/NO-GO probe: GO (refutes [[fable711-awq-self-quantize-todo]])

Loaded cleanly (3 shards 24.33 GiB, Marlin WNA16 MoE backend, FLASH_ATTN for

the vision tower, FLASHINFER main attn, Triton GDN prefill). Three coherence

probes all correct + non-empty: "OK", "65" (7×24−38÷2), and a clean correct

is_prime. **This REFUTES the concern that AWQ on qwen3_5_moe is always

broken** — fable711's four builds all emitted empty output (the (1+weight)

RMSNorm breaks classic AWQ smoothing), but compressed-tensors W4A16 with

vision-tower exclusion works. A build can load cleanly AND be coherent; the

grill is still needed to see if it holds capability.

The reasoning-in-content serving difference (a confound to watch)

reasoning_content came back empty on every probe, including a

"explain step by step" prompt — the full reasoning landed in content as

markdown, then the answer. The GGUF used a patched --chat-template-file

that pre-opens the think block, so its reasoning went to reasoning_content

and content held clean code. The AWQ uses the repo's **default chat

template** (no --chat-template override), which apparently does NOT pre-open

the block, so the qwen3 reasoning parser has nothing to split. **Consequence

for the grill:** max_tokens limits TOTAL generated tokens (reasoning +

content combined) regardless of the split, so cap-hit RISK is unchanged —

BUT the realcase gate hit the 32000 cap with truncated code

(SyntaxError: unterminated string literal (line 9)). Both the AWQ and the

GGUF hit 32000 on realcase; the GGUF's code *completed* (reasoning was

separate), the AWQ's did not (CoT mixed into content left less clean budget

for the code). This is a **serving-config artifact, possibly fixable with a

--chat-template override**, not a clean quant defect — but it cost the AWQ

the gate as-served. The clearer quant-quality signals are the wrong-answer

fails below (office struct, klayout+ref), which are NOT runaways.

Grill (1 rep, 49 min, budget 32000, temp 0.2 — driven direct at :9148)

bench/grill-ornith-15-35b.sh parameterized via GRILL_ALIAS/GRILL_PORT/

GRILL_BASE env (defaults unchanged → committed GGUF behavior preserved).

(The raw battery.log + the 25 GiB AWQ weights were removed 2026-08-21 after the

rejection — the scores below are the record; the model is gone from the box.)

| suite | AWQ (vLLM INT4) | GGUF (Q4_K_M) | |

|---|---|---|---|

| coding r1-4 | 21/23 | 21/23 | tie (diff tasks fail: AWQ expr_eval+text_justify) |

| R5 agentic | 5/6 clean | 5/6 clean | tie (0 loops, 0% redundant both) |

| klayout (no ref) | 0/8 | 0/8 | tie (AWQ rambles to 32000 cap; GGUF fails faster) |

| klayout +ref | 5/8 | 7/8 | AWQ −2 (rotate_place runaway, grow_shapes wrong) |

| klvision | 5/6 | 5/6 | tie (same nested_rings shape-detection miss) |

| closedloop (no ref) | 0/4 | 0/4 | tie |

| closedloop +ref | 2/4 | 3/4 | AWQ −1 (loop_array:code runaway) |

| office (no ref) | 0/9 | 8/9 | AWQ −8 — all 3 struct (openpyxl) FAIL → 6 vision BLOCKED |

| office +ref | 9/9 | 9/9 | tie (ref fully recovers the AWQ, incl. vision) |

| realcase | 0/7 FAIL | 7/7 PASS | AWQ loses the gate (runaway + truncated code) |

| decode | ~86 tok/s | ~93 tok/s | AWQ ~8% slower single-stream |

The pattern is weaker unaided API recall/assembly from the INT4 quant.

Coding, R5, and vision-with-ref hold (the quant doesn't break basic

codegen/agentic/tool-use). But unaided office struct collapses 0/3 (the GGUF

got 3/3 without any API ref), klayout+ref drops 2, and realcase — the

high-value multi-step gate — fails outright. WITH an API ref the AWQ recovers

to near-perfect (office+ref 9/9, klayout+ref 5/8). So the quant costs the

model's ability to assemble openpyxl/KLayout API from memory, not its ability

to follow a provided signature.

Concurrency load-test (the vLLM-specific value)

vllm bench serve --backend openai --tokenizer ulkaa/Ornith-1.5-35B-A3B-AWQ-INT4

--random-input-len 1024 --random-output-len 256 --max-concurrency 8

--num-prompts 64, 64/64 succeeded:

• Output throughput 294 tok/s aggregate (peak 408) at 8-way concurrency.
• Per-stream TPOT 24.2 ms → ~41 tok/s/stream at 8 concurrent.
• Total throughput 1470 tok/s; TTFT median 647 ms (1024-tok prefill), P99 1.8 s.

The GGUF serves --parallel 1 (single-stream only) — the AWQ's 8-way

concurrency is a capability the GGUF cannot offer. But [[qwen36-35b-awq-trial]]

already fills the concurrent-qwen3_5_moe-on-vLLM slot at higher quality

(klayout+ref 15/16, no realcase fail), so the AWQ's concurrency doesn't earn it

a role.

Verdict — REJECTED as a served replacement; do not wire

Not promoted. It is worse than the GGUF it would replace on every

quality-sensitive axis (realcase, office-unaided, klayout+ref) AND slightly

slower single-stream. Its one advantage (8-way concurrency at 294 tok/s) is

already covered by qwen36-35b at higher quality. The GGUF ornith-15-35b

remains the served ornith (a selectable trial, per [[ornith-15-35b-trial]]).

The AWQ files are NOT wired into llama-swap.yaml or any agent config — this

was a standalone probe only; nothing to remove.

Lessons worth keeping (beyond this model):

1. **compressed-tensors W4A16 (MSE observer, no smoothing) LOADS and is

coherent on qwen3_5_moe** — refutes "AWQ is systemically broken on this

arch" (fable711). The breakage was classic AWQ *smoothing* + the

(1+weight) RMSNorm, not AWQ-on-qwen3_5_moe per se. A compressed-tensors

W4A16 build is a viable vLLM path for this family — but grill it, because

"loads + coherent" ≠ "holds capability" (this one lost realcase).

2. **Reasoning-in-content is a serving-config trap for thinking models at a

token budget.** When the chat template doesn't pre-open the think block,

reasoning leaks into content, and at a tight max_tokens the code can

truncate mid-generation (unterminated string literal). The total-token

budget is unchanged — the failure mode is content-shape, not budget-size.

Fix = a --chat-template/--chat-template-file that pre-opens the block

(the qwen3-thinking stream-leak fix, applied to the GGUF, is the analogue).

If re-evaluating this AWQ, override the template first and re-run realcase

before concluding the quant itself can't do it.

3. **vLLM bench serve needs --tokenizer <real-repo> when --model is a

served-model-name** (not a HF repo) — otherwise tokenization for the

random dataset fails. --backend openai to hit an external server (default

vllm starts its own).

4. Track the real child PID, not the setsid launcher PID — the setsid

wrapper exits immediately after detaching; monitoring it reports a false

"died." Variant of [[background-task-silent-kills]]. Used a self-PID-write

(echo $$ > pidfile; exec ...) here.