Kept / trial active in llama-swap.yaml · aliases: qwen3.8-27b-gsq-rco-iq3s-256k, gsq-rco
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| 20260911-143105 | 21/23 | 129.8 | — | results-qwen38-gsq-20260911-143105.log |
qwen38-gsq — ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, IQ3_S-mtp (3.50 bpw, 11.29 GiB) + mmproj-Qwen3.8-27B-BF16 (0.87 GiB). A quant of the REAL Qwen/Qwen3.8-27B base (declared in the GGUF general.base_model, unlike fable-turbo which is a merge). Method: GSQ (Gumbel-Softmax Quantization, arXiv 2604.18556) + RCO (Riemannian Constrained Optimization, arXiv 2605.00649) = a different quant type PER TENSOR under a size budget. Standard GGUF (file_type 26), arch qwen35, 866 tensors, byte-identical hyperparams to qwen38/fable-turbo, no fork. Weights on /mnt/models (cold tier). Port 9197, aliases qwen38-gsq / gsq-rco / qwen3.8-27b-gsq-rco-iq3s-256k. Dual-GPU, CUDA_DEVICE_ORDER=PCI_BUS_ID (CUDA0=5060Ti), -sm tensor with NO --tensor-split (the pair aborts with a bogus ~20 GB OOM, [[sm-tensor-rejects-tensor-split]]), --spec-type draft-mtp, q8_0 KV @262144. Patched template chat-templates/gsq-rco-qwen38-262k.jinja strips the non-first-system guard (byte-identical to qwen3.8-27b-nonfirst-system.jinja apart from that one line, [[jinja-system-guard-tool-parser]]). ~52 tok/s, MTP draft accept 22/26.
Grill 2026-09-09 (bench/gsq-arms-20260909-2218, artifacts kept locally):
[[tommy2-iq2m-grill]]). The GSQ-RCO claim (3.5-bit preserves API recall) held on OUR axes, not the card's AIME/GPQA zero-shot average (which it claims is ABOVE bf16 — a sign of fitting the allocation objective, not a selling point).[[klayout-api-recall-vs-composition]]).The point of the entry: q8_0 KV at the FULL 262144 — fable-fusion CAN'T (17.23 GiB weights OOM at q8_0@262144 → forced 196608, confounding its own KV experiment [[fable-fusion-regrilled]]). Smaller weights buy the clean q8 run, serving [[vram-spend-priority]] (upgrade KV q4→q8 before chasing tok/s). Budget: 11.29 + 0.87 + 8.00 + ~5.65 overhead = ~26.2 of 31.9 GiB, ~6 spare. Single-card NOT on (mmproj loads wholly on CUDA0 ignoring --tensor-split, [[mmproj-caps-cuda0-tensor-split]]).
Backend: ONLY llama.cpp. vLLM can't do IQ3_S GGUF; FreeToken wants NVFP4 + its expert-offload trick is meaningless for a DENSE 27B; TabbyAPI needs EXL3 (re-quant from the 51 GB bf16, discarding the GSQ work) + has no /v1/messages ([[tabbyapi-no-anthropic-messages-endpoint]]).
Status: KEPT as trial, NOT promoted to driver. Counterweight is unaided-KLayout verbosity/runaway; the recall result is genuine.
Why: a genuinely new data point — sub-4-bit quants don't have to collapse API recall, and q8_0 KV at 262k is reachable when the weights are small enough.
How to apply: when triaging a low-bit quant, grade the UNAIDED office + KLayout arms FIRST (that's where 2-bit damage shows, [[tommy2-iq2m-grill]]); don't trust the card's AIME/GPQA numbers. The patched template must ship with the entry — the file was untracked (a fresh-clone gap), now committed alongside the alias regen.
"qwen38-gsq":
aliases: [qwen3.8-27b-gsq-rco-iq3s-256k, gsq-rco]
# TRIAL 2026-09-09 — ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, file
# Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf (3.50 bpw, 11.29 GiB) + the repo's own
# mmproj-Qwen3.8-27B-BF16 (0.87 GiB). Weights on /mnt/models (cold tier, trial).
#
# *** WHY THIS ONE IS DIFFERENT FROM THE OTHER Qwen3.8-27B ENTRIES ***
# It is a QUANTIZATION OF THE REAL BASE, declared in the GGUF itself:
# general.base_model.0.repo_url = https://huggingface.co/Qwen/Qwen3.8-27B
# (contrast `fable-turbo`, which carries NO base_model fields and turned out to
# be a merge, and `qwen38`, which is an uncensored FINE-TUNE of the same base.)
# apache-2.0, ungated, 479k downloads. ISTA-DASLab = the GPTQ/SpQR/QuIP/AQLM lab.
# METHOD: GSQ (Gumbel-Softmax Quantization, arXiv 2604.18556) for the scalar
# grids + RCO (Riemannian Constrained Optimization, arXiv 2605.00649) to assign
# a DIFFERENT quant type PER TENSOR under a size budget. Output is standard
# GGUF (`general.file_type 26`, quantization_version 2) and needs no fork —
# arch qwen35, 866 tensors, byte-identical hyperparameters to qwen38 and
# fable-turbo, so it runs on the pinned ${llama_bin}.
#
# *** THE POINT OF THIS ENTRY IS q8_0 KV AT THE FULL 262144. ***
# This arch has full_attention_interval 4, so only 16 of 65 layers carry KV:
# 4.00 GiB at q4_0 @262144, 8.00 GiB at q8_0. At 11.29 GiB of weights the
# q8_0 budget is 11.29 + 0.87 + 8.00 + ~5.65 measured overhead = ~26.2 of
# 31.9 GiB, leaving ~6 GiB spare. **fable-fusion CANNOT reach this config** —
# its 17.23 GiB of weights OOM at q8_0 @262144 and force 196608, which
# CONFOUNDED its KV experiment (depth and window changed together, see
# auto-memory/fable-fusion-regrilled.md). Smaller weights buy the clean run.
# This directly serves auto-memory/vram-spend-priority.md: "upgrade KV q4->q8
# before chasing tok/s".
#
# SINGLE-CARD IS NOT ON: 11.29 + 0.87 + 4.00 + 5.65 = ~21.8 GiB vs a 16 GiB
# card, and the mmproj tower loads WHOLLY on CUDA0 ignoring --tensor-split
# ([[mmproj-caps-cuda0-tensor-split]]). Dual-GPU with q8_0 KV is the right use.
#
# WHAT MUST BE MEASURED, and why the vendor numbers do NOT answer it:
# the card reports AIME25 / GPQA-Diamond / LiveCodeBench and a 5-task zero-shot
# average, claiming "recovery 100.3-101.8%" (i.e. ABOVE its own BF16 source —
# treat that as a sign of fitting the allocation objective, not as a selling
# point). NONE of those are this box's axes. **The risk is low-bit API-RECALL
# damage**: tommy2 at IQ2_M scored office 0/9 UNAIDED vs 9/9 with a reference
# while looking fine on benchmarks ([[tommy2-iq2m-grill]]). So grade the
# UNAIDED office + KLayout arms FIRST — that is where 2-bit damage showed up,
# and no amount of KV precision can fix a weights defect.
#
# TEMPLATE: embedded template carries the non-first-system guard
# (raise_exception 'System message must be at the beginning.'), which is a
# turn-one 500 in Claude Code. Extracted and diffed: it is byte-identical to
# the proven chat-templates/qwen3.8-27b-nonfirst-system.jinja apart from that
# one line. Patched copy kept separately so a re-pull of either repo cannot
# desync the other entry. See [[jinja-system-guard-tool-parser]].
#
# -sm tensor: qwen35 is NOT arch-gated for it (fable-fusion +49.5%,
# kat-coder-fast +18.7%). Do NOT also pass --tensor-split — the pair aborts
# with a bogus ~20 GB "cudaMalloc failed" ([[sm-tensor-rejects-tensor-split]]).
# SAMPLER: the base Qwen3.8-27B card's "thinking, precise coding" preset.
cmd: |
/usr/bin/env
CUDA_DEVICE_ORDER=PCI_BUS_ID
${llama_bin}
-m /mnt/models/gguf/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
--mmproj /mnt/models/gguf/mmproj-Qwen3.8-27B-BF16.gguf
--alias qwen38-gsq
--jinja --chat-template-file chat-templates/gsq-rco-qwen38-262k.jinja
-ngl 99 -c 262144 -fa on
-sm tensor
--spec-type draft-mtp --spec-draft-n-max 2
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0
--cache-type-k q8_0 --cache-type-v q8_0
--host 127.0.0.1 --port 9197 --parallel 1
proxy: http://127.0.0.1:9197