Kept / trial active in llama-swap.yaml · aliases: glm-4.7-flash-awq-cyankiwi
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| No results-*.log found for this id/alias. | ||||
Added zai-org/GLM-4.7-Flash (unsloth GGUF, UD-Q4_K_XL) to llama-swap.yaml as model key
glm-flash (port 9107, alias glm-4.7-flash-30b-a3b-198k), 2026-07-15.
Architecture gotcha: despite the "GLM" name, this is NOT the glm4moe arch
(src/models/glm4-moe.cpp) already in this box's llama.cpp build. Its HF class is
Glm4MoeLiteForCausalLM, which the local checkout's converter (conversion/glm.py:204-206)
maps to MODEL_ARCH.DEEPSEEK2 — it loads via the DeepSeek-V2/V3 MLA attention path instead.
Before adding any new "GLM" or oddly-named model to this box, check conversion/glm.py (or
the relevant conversion/*.py) for the actual arch mapping rather than assuming from the name.
MLA payoff: because of the DeepSeek2/MLA path, KV cache per token is much lighter than the
GQA models on this box. Full native 202752-token context loads clean at q8_0 KV with
~3.8-4.0GB free per GPU (50/50 split, 17.5GB weights) — no split tuning needed, unlike the
GQA-based 30B-class entries (gemma-31b, qwen3-vl, vision-coder) which need q4_0 KV and/or
asymmetric splits to fit similar context.
**Grill (temp 0.7, the model card's tool-calling profile — NOT the grill's default 0.2, which
this model card doesn't recommend): 20/23.** r1 7/8, r2 3/5 (weighted_interval_scheduling and
text_justify are genuine bugs, reproduced at both 0.2 and 0.7), r3 5/5, r4 5/5 and notably lean
(1462 out_tok / 17.5s wall / 162 tok-per-step across 5 agentic tasks — best agentic efficiency
measured on this box so far). Mid-pack on raw algorithmic coding (weaker than gpt-oss-20b
22-23/23 and vision-coder 22/23 — see [[gpt-oss-20b]]), but flawless tool-calling/agentic and
the roomiest context-per-GB on the box.
Needle recall: PASS at depth 0.1/0.5 (~154k real ctx), inconclusive at depth 0.9 — the
model was still mid-<think> when it hit the needle harness's fixed 2000-token budget, not a
wrong answer. Same overshoot-the-token-budget pattern as the agent (Agents-A1) entry.
See [[vram-spend-priority]] for the general quality-over-speed KV policy applied here.
"glm-flash-ck":
aliases: [glm-4.7-flash-awq-cyankiwi]
# CONTROLLED QUANTIZER A/B against `glm-flash-awq` — TRIAL 2026-08-31.
# SAME base model (zai-org/GLM-4.7-Flash), DIFFERENT quantizer:
# glm-flash-awq = QuantTrio/GLM-4.7-Flash-AWQ 18.41 GiB, 145k dl
# glm-flash-ck = cyankiwi/GLM-4.7-Flash-AWQ-4bit 18.78 GiB, 344k dl
# EVERY OTHER FLAG IS IDENTICAL to the glm-flash-awq entry on purpose
# (TP=2, 163840, max-num-seqs 16, gpu-mem-util 0.90, fp8 KV, temp 1.0,
# glm47 tool+reasoning parsers) so the quantizer is the ONLY variable.
# Only the model path, --served-model-name and --port differ.
#
# WHY THIS IS WORTH A TRIAL AT ALL: quantizer choice has changed behaviour
# on this box before, on the SAME base weights — `qwen3vl` (remodlai) vs
# `qwen3-vision` (QuantTrio) graded R5 0/3 loops vs spread-6
# ([[qwen3vl-30b-instruct-awq-trial]]). glm-flash is the box's best AGENT
# ([[glm-4.7-flash]]: R5 6/6, best agentic efficiency 162 tok/step), so a
# free upgrade on that axis is worth 20 GB of disk to test.
# cyankiwi also has the better track record here — qwen38-awq and gemma-awq
# are both cyankiwi builds ([[qwen38-awq-vllm-trial]], [[gemma-4-26b-awq-vllm]]).
#
# NOTE the HF arch is `Glm4MoeLiteForCausalLM` / compressed-tensors (NOT
# classic AWQ despite the repo name); llama.cpp GGUF conversions of the same
# model report DEEPSEEK2/MLA, which is a different naming of the same thing.
# MIT licence, max_position_embeddings 202752.
#
# *** REQUIRES --no-enable-prefix-caching. WITHOUT IT THE ENGINE DIES. ***
# ROOT CAUSE ISOLATED 2026-08-31 by a 4-way test — it is PREFIX CACHING,
# NOT memory:
# 155000 ctx, caching ON -> 1 request then dead
# 131072 ctx, caching ON -> 1 request then dead
# 92000 ctx, caching ON -> 1 request then dead (KV 4.05 GiB, ample)
# 131072 ctx, caching OFF -> 6/6 survived
# Context varied by 63k tokens with ZERO effect; one flag fixed it. The
# first request populates the cache, the second hits it, the request goes
# missing from the model runner output and EngineCore dies:
# scheduler.py update_from_output -> req_id_to_index[req_id]
# KeyError: 'chatcmpl-...' -> EngineDeadError -> every later req 500s
# vLLM PR #53799 guards this KeyError but is still OPEN/unmerged (checked
# 2026-08-31), so it is in NO release incl. 0.28.0 — an upgrade does NOT
# fix it, and [[vllm-0271-upgrade]] already rolled one back. That PR blames
# a worker OOM; here there is no OOM, so this is the same KeyError from a
# DIFFERENT upstream cause. QuantTrio's awq build on the SAME engine and
# identical flags is unaffected -> compressed-tensors takes a different
# scheduler path.
#
# CAVEAT ON ANY FUTURE RESULT: prefix caching is a real feature the
# incumbent uses (it contributes to glm-flash-awq's 3.18x at 8 concurrent),
# so a grill of this entry runs in a NON-PRODUCTION config. Needing a
# disabled feature the incumbent relies on is itself a mark against it.
#
# STILL NO QUALITY VERDICT — it has never completed a suite.
# It loads (143 s) and answers correctly — 3 back-to-back requests clean,
# reasoning properly separated — then EngineCore dies and every subsequent
# request 500s until the process restarts:
# File ".../vllm/v1/core/sched/scheduler.py", line 1670, in update_from_output
# req_index = model_runner_output.req_id_to_index[req_id]
# KeyError: 'chatcmpl-...' -> EngineDeadError
# A vLLM SCHEDULER bug, not a model/config fault. It is also what made
# llama-swap report "upstream process exited unexpectedly" and restart into
# the same crash. QuantTrio's build on the SAME engine and identical flags is
# unaffected, so the likely trigger is compressed-tensors vs awq taking a
# different scheduler path. The upstream card says GLM-4.7-Flash is only
# supported on vLLM/SGLang MAIN branches; 0.26.0 is a release, and
# [[vllm-0271-upgrade]] records an upgrade already rolled back once.
#
# -c 155000 (not 163840) is REQUIRED and unrelated to the crash: this build
# is 18.78 GiB vs QuantTrio's 18.41, leaving 3.99 GiB for KV where 163840
# needs 4.13 ("estimated maximum model length is 158096").
#
# RETRY WHEN vLLM MOVES. Weights kept at /mnt/models/... per
# [[ask-before-deleting-weights]]. To grill it then: R5 + agentic efficiency
# are the deciding axes (glm-flash is the box's best AGENT), plus
# bench/grill_textturn.py since QuantTrio is one of the 3/3 CLEAN vLLM
# entries ([[textturn-leak-comparison]]) and a quantizer swap could change that.
cmd: |
/usr/bin/env
HF_HOME=/home/giovanni/.cache/huggingface
CUDA_DEVICE_ORDER=PCI_BUS_ID
PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
${vllm_bin} serve /mnt/models/GLM-4.7-Flash-AWQ-4bit-cyankiwi
--served-model-name glm-flash-ck
--tensor-parallel-size 2
--host 127.0.0.1 --port 9176
# 155000, not 163840: this build is 18.78 GiB (QuantTrio's is 18.41) and
# vLLM refuses 163840 with "4.13 GiB KV needed / 3.99 available, estimated
# maximum model length is 158096". Matched to glm-flash-awq for the A/B.
--max-model-len 155000
--max-num-seqs 16
--gpu-memory-utilization 0.90
--kv-cache-dtype fp8
--override-generation-config '{"temperature":1.0}'
--no-enable-prefix-caching
--enable-auto-tool-choice
--tool-call-parser glm47
--reasoning-parser glm47
proxy: http://127.0.0.1:9176