fable711-awq-self-quantize-todo

Verdict: REJECTED, twice independently.

• Attempt 1 (.../Qwen3.6-27B-Fable-Fusion-711-AWQ-INT4, first build,

18GB): quantized all of in_proj_qkv/z/b/a. Tool-call go/no-go 1/5,

grill 0/8, 0/5, 2/5, 0/5 — most tasks burned the full 16,000-token cap

on empty/garbage output.

• Attempt 3 (same output path, second build, 20GB): quantized only

{in_proj_qkv, in_proj_z} together (required — see the vLLM fusion note

below), excluded {in_proj_b, in_proj_a, out_proj} entirely. **Also

broken, same signature: tool-call go/no-go 1/5**, stuck generating

empty output at ~27 tok/s for a very long time on the very first grill

task before being killed. (Attempt 2, quantizing only qkv alone and

excluding z, never got graded — vLLM rejected it outright, see below.)

• Attempt 4 (26GB): excluded the ENTIRE linear_attention block

(in_proj_qkv/z/b/a + out_proj), quantizing only full_attention's

q/k/v/o (16 of 64 layers) + MLP (all 64) — the most conservative possible

configuration. STILL COMPLETELY BROKEN: 0/5 tool calls, and a plain

curl "Say hello in exactly three words" at temperature 0 returned

finish_reason: stop with content: None and reasoning_content: None

— literally zero tokens of output. Also note this build is **too large to

serve normally** (26GB → ~13GB/card of 16GB leaves no KV cache room; vLLM

refuses with "No available memory for the cache blocks"). It only started

under a diagnostic squeeze (--max-model-len 8192 --max-num-seqs 1

--gpu-memory-utilization 0.96 --enforce-eager, yielding a mere 20,935-token

KV cache) — so even if it HAD worked, this configuration is not viable on

this box.

**Attempt 4 is the decisive result: THE PROBLEM IS NOT THE LINEAR-ATTENTION

BLOCK.** Quantizing only the bog-standard parts (full_attention q/k/v/o +

MLP, both covered by llmcompressor's own unmodified default_mappings) still

produces a model that emits nothing at all. Whatever is wrong is **systemic

to this llmcompressor-AWQ-on-qwen3_5 pipeline**, not to the hand-written

hybrid-attention mapping — which means the three earlier attempts' failures

were probably never about in_proj_* at all, and all four builds share one

undiagnosed root cause.

**Leading remaining suspect (UNVERIFIED) — the (1 + weight) RMSNorm offset

convention.** Qwen3_5RMSNorm applies (1 + weight), not weight (see

_init_weights: init.zeros_(module.weight) with the comment "we initialize

with 0s to be 1 centered as the RMSNorm here does (1 + weight)"). AWQ

smoothing works by dividing a norm's weight by scale s and multiplying the

following Linear by s — which is only valid if the norm applies weight

directly; under (1 + weight), dividing w by s does NOT divide the

effective gain (1 + w) by s, silently corrupting every smoothed layer.

llmcompressor is aware of this class of problem and logs Found 161

offset-norm modules to convert / Restoring 161 norm modules to offset

convention — 161 = 128 decoder norms + 1 final norm + 32 q/k norms on the 16

full_attention layers, i.e. it appears to cover the text model's plain

RMSNorms. Whether that conversion is actually CORRECT for this architecture

(or silently mis-handles something) was not verified, and is where a future

investigation should start. **Before any further attempt, first validate the

whole pipeline end-to-end on a model that HAS a registry-supported AWQ

mapping and a conventional RMSNorm** — if that also comes out broken, the

problem is the environment/pipeline, not qwen3_5 at all.

vLLM structural constraint discovered along the way: vLLM's own qwen3_5

serving implementation FUSES in_proj_qkv + in_proj_z into one physical

tensor (in_proj_qkvz) at load time, and refuses to load a checkpoint where

the two halves have different quantization schemes ("Found a different

quantization schemes for ['in_proj_qkv', 'in_proj_z'] ... vLLM requires all

to use the same scheme"). Presumably in_proj_b+in_proj_a fuse into

in_proj_ba the same way. This means {qkv, z} and {b, a} must each be

quantized-or-excluded as a PAIR — you cannot quantize one half of either

fused pair alone, regardless of what the quality experiment calls for.

**Don't re-trust this recipe or any of its output files without independently

re-validating** — the 2-sample smoke test used throughout development never

catches this (only checks "does the pipeline complete," not "is the output

any good"); both broken builds passed it cleanly.

Harness instability hit while re-running this (2026-08-06/07, unrelated

to the model): Bash calls with run_in_background: true, and even

foreground calls that auto-converted to background after their timeout, got

silently killed mid-run four times in a row with empty output logs despite

the underlying process making real progress (confirmed via ps/nvidia-smi

mid-run). Foreground execution and a fully-detached nohup ... & disown

process (tracked via the Monitor tool polling kill -0 <pid> instead of

the Bash background mechanism) both worked reliably. If a background

quantization/training job mysteriously dies with zero output again, try the

detached nohup+Monitor pattern before assuming the job itself is broken.

No AWQ of DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP

exists on HF (checked 2026-08-06). What exists: ~20 GGUF mirrors, MLX 4/6/8bit

(Apple only), one GPTQ-INT4 (cjxzdzh/...-GPTQ-INT4, the working 21/23

candidate), one FP8, two NVFP4 (Blackwell-only). Source used for

self-quantizing: philipjohnbasile/Qwen3.6-27B-Fable-Fusion-711-bf16 (55.6GB,

still on disk at /home/giovanni/models-nvme/staging/ along with the broken

AWQ output — ~70GB total, safe to delete unless retrying).

Script: vllm/quantize-fable711-awq.py (llmcompressor 0.12.0,

transformers 5.10.1, dedicated ~/quant-env venv). Getting it to RUN

end-to-end took five real, non-obvious bugs, all fixed — but "runs without

crashing" turned out not to mean "produces a usable model":

1. **No AWQ mapping exists for this hybrid architecture, and llmcompressor's

own default mapping is silently broken for it.** Qwen3.5 alternates

linear_attention (Qwen3_5GatedDeltaNet: fused in_proj_qkv/z/b/a +

out_proj after conv1d+gated-delta-rule+gated-RMSNorm) and

full_attention (standard q/k/v/o_proj) layers, 3:1 ratio (48:16 of 64).

llmcompressor's match_modules_set() requires a mapping's smooth_layer

regex to resolve to the SAME COUNT as its balance_layer regex —

input_layernorm exists on all 64 layers but q/k/v_proj only on 16, so

the library's own default q/k/v mapping throws "AWQ needs to match a

single smoothlayer" on this model as shipped. Worked around by anchoring

two separate mappings to explicit full_attention/linear_attention

layer indices from config.text_config.layer_types. **This hand-built

mapping for in_proj_qkv/z/b/a (no registry precedent to check against)

is the leading suspect for the broken output** — out_proj was left

entirely unsmoothed (no valid candidate), which may also be a bigger

quality hit than assumed.

2. AutoModelForCausalLM silently loads the WRONG, vision-less class.

qwen3_5 resolves to TWO different classes depending on which Auto

mapping you use: AutoModelForCausalLM → Qwen3_5ForCausalLM (text-only,

flatter model.layers.N.* tree, 851 tensors, no error, silently drops the

vision tower); AutoModelForImageTextToText → the real

Qwen3_5ForConditionalGeneration (model.language_model.layers.N.* +

model.visual.*, 1184 tensors). Loading via the wrong class also breaks

save_pretrained with a confusing, seemingly-unrelated

AttributeError: Qwen3_5TextModel has no attribute 'language_model'.

3. **31GB system RAM can't hold a 52GB bf16 model even via llmcompressor's

own "sequential" pipeline** (sequential_offload_device only accepts

cpu/cuda:N, no disk option) — got OOM-killed (empty stdout = SIGKILL)

despite 27GB free swap at the time. Fixed by loading via

AutoModelForImageTextToText.from_pretrained(device_map="auto",

max_memory={0: "2GiB", 1: "2GiB", "cpu": "20GiB"}, offload_folder=...,

offload_state_dict=True) and passing the model OBJECT into oneshot()

(skips its internal path-based loader), letting accelerate's own

AlignDevicesHook disk-stream layers from NVMe. Peak RAM stayed ~5-6GB.

4. **Saving a disk-offloaded model OOMs a single GPU, unrelated to model

size.** accelerate.utils.offload.OffloadedWeightsLoader fetches each

offloaded tensor onto its hook's execution_device (a GPU) by default,

and save_pretrained never frees tensors between shard writes, so one

16GB card fills mid-save even though the final output is only 18GB. Fixed

by walking model.modules() and forcing every

_hf_hook.weights_map's (OffloadedWeightsLoader, possibly wrapped in

PrefixedDataset) .device = "cpu" before calling save.

5. **GPU OOM during REAL calibration (512 samples) that the 2-sample smoke

test never revealed, and it's driven by sample COUNT, not per-sample

size.** Failed at the same ~14.3-14.7GB point regardless of halving

max_seq_length (2048→1024), setting sequential_targets=

["Qwen3_5DecoderLayer"], or PYTORCH_CUDA_ALLOC_CONF=

expandable_segments:True — none of the "obvious" fixes helped, because

AWQ's grid search needs every calibration sample's activations held

simultaneously, so memory scales with num_calibration_samples, not

sequence length. Only dropping to 128 samples (from 512) got it through.

This is itself a quality concern: 128 samples at max_seq_length=1024

is a thin calibration set for a 27B model, and may be part of why the

scales came out badly — compounding whatever the mapping issue is.

Lesson for next time: a pipeline that "completes without error" on a

smoke test is necessary but nowhere near sufficient — always run at least a

handful of GRILL tasks against a freshly self-quantized model (not just

toolcheck.py's tool-call go/no-go) before considering the quantization job

"done," matching [[grill-does-not-validate-real-use]]'s point but one level

earlier in the pipeline (before serving even starts, quantization itself can

silently produce garbage).