glm-flash-sglang

Active active in llama-swap.yaml · aliases: glm-4.7-flash-sglang-tp2

Grill run history

Run	Score	tok/step	R5	Log
No results-*.log found for this id/alias.

Memory notes

glm-4-7-flash (glm-4.7-flash.md)

Added zai-org/GLM-4.7-Flash (unsloth GGUF, UD-Q4_K_XL) to llama-swap.yaml as model key

glm-flash (port 9107, alias glm-4.7-flash-30b-a3b-198k), 2026-07-15.

Architecture gotcha: despite the "GLM" name, this is NOT the glm4moe arch

(src/models/glm4-moe.cpp) already in this box's llama.cpp build. Its HF class is

Glm4MoeLiteForCausalLM, which the local checkout's converter (conversion/glm.py:204-206)

maps to MODEL_ARCH.DEEPSEEK2 — it loads via the DeepSeek-V2/V3 MLA attention path instead.

Before adding any new "GLM" or oddly-named model to this box, check conversion/glm.py (or

the relevant conversion/*.py) for the actual arch mapping rather than assuming from the name.

MLA payoff: because of the DeepSeek2/MLA path, KV cache per token is much lighter than the

GQA models on this box. Full native 202752-token context loads clean at q8_0 KV with

~3.8-4.0GB free per GPU (50/50 split, 17.5GB weights) — no split tuning needed, unlike the

GQA-based 30B-class entries (gemma-31b, qwen3-vl, vision-coder) which need q4_0 KV and/or

asymmetric splits to fit similar context.

**Grill (temp 0.7, the model card's tool-calling profile — NOT the grill's default 0.2, which

this model card doesn't recommend): 20/23.** r1 7/8, r2 3/5 (weighted_interval_scheduling and

text_justify are genuine bugs, reproduced at both 0.2 and 0.7), r3 5/5, r4 5/5 and notably lean

(1462 out_tok / 17.5s wall / 162 tok-per-step across 5 agentic tasks — best agentic efficiency

measured on this box so far). Mid-pack on raw algorithmic coding (weaker than gpt-oss-20b

22-23/23 and vision-coder 22/23 — see [[gpt-oss-20b]]), but flawless tool-calling/agentic and

the roomiest context-per-GB on the box.

Needle recall: PASS at depth 0.1/0.5 (~154k real ctx), inconclusive at depth 0.9 — the

model was still mid-<think> when it hit the needle harness's fixed 2000-token budget, not a

wrong answer. Same overshoot-the-token-budget pattern as the agent (Agents-A1) entry.

See [[vram-spend-priority]] for the general quality-over-speed KV policy applied here.

llama-swap.yaml entry

  "glm-flash-sglang":
    aliases: [glm-4.7-flash-sglang-tp2]
    # FIRST SGLANG ENTRY — a FOURTH backend after llama.cpp, vLLM and
    # TabbyAPI/EXL3. Wired like the vLLM and Tabby entries: llama-swap owns the
    # lifecycle, so selecting this EVICTS the resident model instead of
    # fighting it for VRAM. Runbook: ~/Documents/install-sglang.md.
    #
    # SAME WEIGHTS AS `glm-flash-awq` (QuantTrio/GLM-4.7-Flash-AWQ, 19 GB, on
    # NVMe). That is the point: it makes a clean BACKEND A/B possible on the
    # box's best agent ([[glm-4.7-flash]]) without a second download. QuantTrio
    # and NOT the cyankiwi build, deliberately — cyankiwi needs prefix caching
    # disabled or its engine dies, and SGLang's RadixAttention is on by default,
    # so cyankiwi would confound every result here.
    #
    # *** FLASHINFER_USE_CUDA_NORM=1 IS MANDATORY AND NOT A TUNING KNOB ***
    # Without it TP=2 dies during prefill CUDA-graph capture on the sm_89 rank:
    #     Architecture: Ada (sm_89) | Compatible SM archs: sm_89, sm_86
    #     Target SM ARCH: unknown (unspecified)
    #     cudaErrorNoKernelImageForDevice (209)
    #       flashinfer/norm/kernels/rmsnorm.py:1323, in rmsnorm_cute
    # It is NOT NCCL and NOT attention — it is ONE norm kernel. FlashInfer's
    # CuTe-DSL backend has a PROCESS-GLOBAL compile target, and
    # is_cute_dsl_arch_supported() returns True for BOTH sm_120 and sm_89, so
    # its own auto-fallback never fires. One global target, two architectures;
    # the other rank then launches a mismatched cubin. Proven not to be a
    # mixed-arch impossibility by a 3-way test: shape A on GPU 0 serves, shape A
    # on GPU 1 (sm_89) ALONE serves, only TP=2 dies.
    # `--attention-backend triton` does NOT fix it (rmsnorm is not attention;
    # tested, identical crash). See auto-memory/sglang-tp2-mixed-arch-norm-kernel.md.
    #
    # checkEndpoint IS LOAD-BEARING. SGLang's /health returns 200 LONG before
    # it can serve — measured ~5 minutes early on this entry, and a chat request
    # sent in that window HUNG with both GPUs at 0% util. llama-swap defaults to
    # /health, which would hand traffic to a server that cannot answer.
    # /health_generate runs a real generation, so it cannot pass early.
    #
    # PARSERS ARE PASSED EXPLICITLY ON PURPOSE. SGLang auto-DETECTS them and
    # logs "reasoning_parser=glm45, tool_call_parser=glm47" — then leaves
    # server_args.reasoning_parser=None, so GLM's raw chain-of-thought comes
    # back inside message.content. Detected is not applied. Given this box's
    # history with think-tag leakage ([[aquila-think-tag-leak-real-use]]), never
    # rely on the auto-detection line.
    #
    # *** GRILLED AND BENCHMARKED HEAD-TO-HEAD vs THE INCUMBENT, 2026-09-04.
    # QUALITY AND THROUGHPUT ARE BOTH A WASH. THE ONLY REAL LOSS IS CONTEXT. ***
    #
    # 23-task suite, n=3 PER BACKEND, same weights/cards/day
    # (bench/grill-backend-ab.sh, results in bench/n3-{sglang,vllm}-rep*):
    #     glm-flash-sglang : 19, 21, 20  -> mean 20.0   R5 6/6 6/6 6/6
    #     glm-flash-awq    : 18, 20, 21  -> mean 19.67  R5 6/6 5/6 6/6
    #     R5 LOOPED 0/6 ON ALL SIX RUNS, both backends.
    # A 0.33 gap in means against a 3-POINT SPREAD WITHIN each arm — the two are
    # INDISTINGUISHABLE ([[single-run-scores-are-unreliable]]). An n=1 pass had
    # shown 22 vs 20 and that reversed at n=3; do not quote a single run here.
    #
    # Throughput, matched protocol (1024-in/256-out, warm, n=2 at 8 concurrent):
    #     sglang 78.2 / 120.7 / 187.0 / 271.6 tok/s at c=1/2/4/8  -> 3.47x
    #     vllm   79.9 / 121.0 / 182.7 / 263.7 tok/s               -> 3.30x
    # Within ~3% everywhere. vLLM wins TTFT (102 vs 120 ms at c=8) and loads 3x
    # faster (110 s vs ~350 s).
    #
    # *** AN EARLIER REVISION OF THIS COMMENT SAID SGLANG SCALES WORSE
    # (2.6x vs 3.18x) AND WAS "STRICTLY WORSE ON BOTH AXES". BOTH WRONG. ***
    # The cause is worth keeping: THE FIRST BENCH RUN AFTER A SERVER STARTS IS
    # ~12% LOW on both backends (sglang 239.9 -> 271.8, 271.5; vllm 234.6 ->
    # 263.8, 263.7; reps 2-3 agree to 0.1%). A warm SGLang run had been compared
    # against a remembered vLLM figure, then a cold one against nothing. Always
    # discard run 1 — bench/grill_prefix_cache.py does this automatically.
    #
    # Where SGLang genuinely wins: PREFIX CACHE. On shared-prefix traffic
    # (bench/grill_prefix_cache.py) it cuts TTFT 3.22x vs vLLM's 2.77x — but it
    # is SLOWER IN ABSOLUTE TTFT on both arms (867->269 ms vs 547->198 ms), so
    # the better ratio partly reflects a worse uncached baseline. Judge the
    # cache by the ratio, the backend by the absolute.
    #
    # *** DO NOT DRIVE AN AGENT CLIENT (Claude Code, pi, kilo) WITH THIS ENTRY.
    # PREFILL IS 2.8x SLOWER THAN vLLM AND GETS WORSE WITH LENGTH. ***
    # Found 2026-09-04 when a Claude Code client on this entry sat at
    # "Waiting for API response ... retry in 51s" and looked like a network
    # fault. It was not: the server was READY and chunk-prefilling a ~64k prompt
    # at ~400 tok/s in 2048-token chunks (#pending-token 62528 falling by 2048 a
    # batch). ~3 min to first token, so the client times out and RETRIES, which
    # queues a second 64k prefill behind the first and compounds.
    #
    # Mean TTFT, n=3, concurrency 1, same weights/cards (both measured):
    #     in= 8192   sglang  2370 ms (2592 tok/s)   vllm 1233 ms (5022 tok/s)
    #     in=32768   sglang 18166 ms (1727 tok/s)   vllm 6454 ms (4706 tok/s)
    # SGLang's prefill DEGRADES with length (2592 -> 1727); vLLM's holds flat.
    # Extrapolated to a 64k agent prompt: ~60 s vs ~14 s.
    #
    # ROOT CAUSE: GLM-4.7-Flash is MLA (kv_lora_rank 512, v_head_dim 256) and
    # SGLang has no fast MLA kernel for these cards — it logs "Attention backend
    # not specified. Use triton backend by default", the generic fallback.
    # NEITHER LEVER WORKS HERE, both tested:
    #   --chunked-prefill-size 8192  -> OOM AT SERVING TIME. This entry runs at
    #       0.12 GiB free, so there is no room for bigger prefill chunks; the
    #       same tightness that caps context at 131072 blocks the prefill fix.
    #   --attention-backend flashinfer -> crashes in prefill graph capture, and
    #       warns first: "backend='auto' selected 'fa2' on SM120, which is not
    #       Blackwell-native and gives poor MLA decode performance".
    #   flashmla / cutlass_mla / trtllm_mla target sm_90/sm_100 datacenter parts.
    # Triton is the only working MLA path on sm_89+sm_120. Hardware/maturity
    # limit, not a misconfiguration — do not go looking for a flag.
    #
    # The 1024-token benchmarks below MISSED THIS ENTIRELY. At short prompts the
    # backends look identical; the gap only appears at agent-scale context.
    # ALWAYS MEASURE PREFILL AT A REALISTIC PROMPT LENGTH.
    #
    # NET: keep vLLM as the driver, and send agent clients to glm-flash-awq.
    # This entry is for short prompts and shared-prefix work. Two axes separate
    # them: context (163840 vs 131072) and PREFILL (2.8x).
    #
    # ~6 MIN TO READY (355 s measured), almost all CUDA-graph capture; weight
    # load is only 12 s of it. healthCheckTimeout is 1200 globally, so it fits.
    cmd: |
      /usr/bin/env
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      FLASHINFER_USE_CUDA_NORM=1
      HF_HOME=/home/giovanni/.cache/huggingface
      # PATH is load-bearing for the same reason as the vLLM/Tabby entries:
      # systemd hands llama-swap a minimal PATH and the CUDA JIT paths shell out
      # to a toolchain. See auto-memory/llama-swap-setup.md.
      PATH=/home/giovanni/ai/sglang-venv/bin:/usr/local/cuda-13.0/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      SGLANG_STDERR=/tmp/glm-flash-sglang-via-swap.err
      ${sglang_wrap_bin}
      --model-path /home/giovanni/models-nvme/hf/models--QuantTrio--GLM-4.7-Flash-AWQ/snapshots/88e3d3d913c0d97c8f505cdc03433c48226bedc3
      --served-model-name glm-flash-sglang
      --tp 2
      --host 127.0.0.1 --port 9190
      # *** 131072 IS A MEASURED CEILING, NOT A CONSERVATIVE GUESS. ***
      # The vLLM incumbent glm-flash-awq runs this SAME checkpoint at 163840.
      # SGLang CANNOT reach that here, and it is worth knowing why before
      # someone "fixes" this number upward. Three attempts, all OOM in CUDA
      # graph capture (RuntimeError: markCaptureEnd called with no captures in
      # progress — a capture aborted by OOM, not a torch bug):
      #     0.90, pool auto  -> pool 184941 tok / 4.66 GiB, 0.24 GiB free, OOM
      #     0.90, pool 163840-> pool 163840 tok / 4.13 GiB, 0.70 GiB free, OOM
      #     0.85, pool 163840-> pool fell to 154960 (BELOW the context!),
      #                         0.98 GiB free, OOM
      #     0.85, ctx 131072 -> pool 154960 tok / 3.91 GiB, 1.23 GiB free, OK
      # THE NON-OBVIOUS PART: long context costs VRAM *beyond* the KV pool.
      # The identical 3.91 GiB pool left 2.08 GiB free at ctx 16384 but only
      # 0.98 GiB at ctx 163840 — the attention workspaces scale with the
      # declared window. So lowering context helps TWICE. SGLang's graph
      # capture wants ~1.2 GiB free, which is more than vLLM needs; that, not
      # KV, is what caps this entry below the incumbent.
      --context-length 131072
      # fp8 KV halves cost/token (50.5 -> 25.2 KB/token/rank) and is what makes
      # 131072 reachable at all; the incumbent uses fp8 too. e4m3 not e5m2 —
      # 3 mantissa bits vs 2, and KV needs precision over range
      # (auto-memory/vram-spend-priority.md: spend spare VRAM on KV PRECISION).
      --kv-cache-dtype fp8_e4m3
      # 0.90 does NOT work (see the ladder above). Runtime free memory is thin
      # at 0.85 — the log warns "free device mem: 0.12 GiB" when a Triton kernel
      # lazy-loads after serving starts — but it is STABLE: 64/64 requests at 8
      # concurrent, zero OOM. If a mid-run OOM ever appears, drop to 0.80 or cut
      # context; do not raise this.
      --mem-fraction-static 0.85
      --reasoning-parser glm45
      --tool-call-parser glm47
    proxy: http://127.0.0.1:9190
    checkEndpoint: /health_generate

# ---------------------------------------------------------------------------
# ---------------------------------------------------------------------------

# ---------------------------------------------------------------------------
# PEERS — remote OpenAI-compatible servers proxied transparently, OUTSIDE the
# swap pool. Requesting a peer model does NOT start/stop any local backend and
# does NOT evict the resident GPU model, because peers are routed by the router
# without the swap lifecycle. That is exactly what makes them right for Ollama's
# *cloud* tags: they run on Ollama's servers, use no local VRAM, and answer in
# 1-9s vs ~83s for a cold local 24B. See auto-memory ollama-cloud-tags-retire-silently.
#
# AUTH: no apiKey field on purpose. Ollama ignores the Authorization header a
# client sends and authorises *-cloud with its OWN OLLAMA_API_KEY baked into
# coolbox/docker-compose.yml (auto-memory ollama-cloud-auth-lives-in-daemon).
# So llama-swap needs no key; the daemon handles it. If cloud calls start
# returning 401, rotate the daemon's key and `docker compose up -d ollama`
# (recreate, not restart — env is baked at container-create time).
#
# RETIREMENT RISK: cloud tags are server-side pointers. A retired tag keeps
# listing at 0.0 GB and fails with HTTP 410; `ollama pull` cannot fix it. All
# four below were probed alive 2026-08-19. kimi-k3:cloud was probed the same
# day and returns a BILLING error ("extra usage balance is empty") — alive but
# unusable on this plan, so deliberately omitted. If you rely on any cloud tag,
# probe it on a schedule (curl /v1/chat/completions, watch for 410) — nothing
# else will tell you when one dies.
#
# DO NOT add ollama LOCAL models here (gemma4:31b, gemma4-31b-168k, ...). They
# allocate GPU VRAM that ollama manages independently of llama-swap, so with
# OLLAMA_KEEP_ALIVE pinning they can hold VRAM a llama-swap load needs -> OOM.
# Cloud tags are safe precisely because they use no local VRAM.
#
# Addressed as: ollama/<tag>  e.g. ollama/gemma4:31b-cloud
# ---------------------------------------------------------------------------
peers:
  # THE ONLY ALWAYS-ON DRIVER. Gemma-4-26B-A4B-NVFP4 served by FreeToken
  # (systemd: gemma4-ft-driver.service) pinned to the RTX 5060 Ti on :9162.
  # 131072 ctx (declared == funded), hybrid CPU+PCIe expert execution,
  # ~26.9 tok/s warm. The unit is `enabled`, so IT COMES BACK ON EVERY BOOT.
  #
  # *** THE PEER KEY BELOW IS AN ADDRESS PREFIX, NOT A SYSTEMD UNIT NAME. ***
  # /v1/models advertises `freetoken-driver/gemma4-ft`,
  # `freetoken-driver/qwen36-ft` and `freetoken-driver/gpt-oss-20b-ft` --
  # renaming the key `freetoken-driver:` RENAMES EVERY ONE OF THOSE IDS for
  # every client. Do NOT "tidy" it to match the unit name; the resemblance is
  # historical and it has already cost one debugging session (2026-09-03:
  # `systemctl {stop,start,enable} freetoken-driver` typed five times, all
  # failing silently, against the wrong thing entirely).
  #
  # ONLY TWO OF THE THREE UNITS ARE INSTALLED (symlink in /etc/systemd/system
  # -> systemd/ in this repo). Check with `ls -l /etc/systemd/system/*ft*`:
  #     gemma4-ft-driver.service     INSTALLED, enabled  -> gemma4-ft
  #     gpt-oss-ft-driver.service    INSTALLED           -> gpt-oss-20b-ft
  #     freetoken-driver.service     *** NOT INSTALLED *** -> qwen36-ft
  # `systemd/freetoken-driver.service` exists in this repo but has no symlink,
  # so systemd does not know it: every `systemctl ... freetoken-driver` fails
  # with "unit not found" and **qwen36-ft is currently UNSERVABLE**. Symlink it
  # into /etc/systemd/system and `daemon-reload` to bring that model back.
  #
  # --moe-backend is PINNED to `hybrid` in the unit (2026-08-28), not left on
  # `auto`: auto reads ~/.cache/freetoken/benchbw.json, and a bench run with
  # `--model X` (no --dtype) silently WIPES the per-format verdicts, dropping
  # nvfp4 back to offload with no warning. Controlled A/B that day, same box:
  # offload 33.8 vs hybrid 45.7 tok/s = +35.3% (n=3 each, spreads <1 tok/s).
  # Pinning also turns a broken CPU-MoE path into a loud start failure instead
  # of a quiet one-third throughput loss.
  #
  # LAYOUT as of 2026-08-23: this driver owns the 5060 Ti; the 4060 Ti is the
  # swap pool that llama-swap manages itself (mellum, gemma12, bonsai, glm-ocr —
  # only ONE fits at a time, which is why there are no groups any more).
  # mellum used to be a second always-on driver here and is now an ordinary
  # model entry instead; mellum-driver.service is stopped and should stay so.
  #
  # *** WHILE THIS RUNS, NO TP=2 ENTRY CAN LOAD *** — gemma-awq, qwen3-coder,
  # qwen3-thinking, qwen38-awq, fable-fusion, thinkingcap, qwen36-35b and omni
  # all need BOTH cards and the 5060 Ti is taken. To use one:
  #     sudo systemctl stop  gemma4-ft-driver   (frees the 5060 Ti)
  #     sudo systemctl start gemma4-ft-driver   (~70 s to "API server is ready")
  #
  # *** THE UNIT IS A COPY IN /etc/systemd/system, NOT A SYMLINK TO THIS REPO,
  # SO EDITING systemd/gemma4-ft-driver.service HERE CHANGES NOTHING LIVE. ***
  # After editing it (or gpt-oss-ft-driver.service), re-install with:
  #     sudo bash systemd/install-units.sh
  # Switched to copy-install 2026-09-03 because the old symlink form was a
  # `link`-form unit, and `systemctl disable` REVERSES `link` -- it deleted
  # /etc/systemd/system/gemma4-ft-driver.service outright and every later
  # command answered "Unit file ... does not exist". With a real file,
  # enable/disable only touch the .wants/ link and are safe again. See
  # [[systemctl-disable-deletes-symlinked-unit]].
  #
  # NEITHER FreeToken unit is enabled at boot any more (no .wants/ link, as of
  # 2026-09-03) -- gemma4-ft is ON DEMAND now, so an agent hitting
  # `freetoken-driver/gemma4-ft` on a fresh boot gets a connection error until
  # someone starts it. That is deliberate: it keeps the 5060 Ti free for TP=2.
  # Then VERIFY WITH nvidia-smi, NOT `systemctl status` -- the frontend can
  # hang reporting active(running) while holding 0 MiB, or leave an orphan
  # worker squatting ~15 GB. See the zombie note in the unit file.
  # Bring it back with: sudo systemctl start gemma4-ft-driver  (~70 s to ready)
  #
  # COSTS ~18 GB OF HOST RAM (pinned MoE expert banks) on top of ~15.7 GB VRAM --
  # mellum2 uses none. On a 31 GB box that leaves ~13 GB for everything else.
  freetoken-driver:
    proxy: http://127.0.0.1:9162
    # responseHeader is PER-PEER and takes an INT of SECONDS (a "3600s" string is
    # rejected: `cannot unmarshal !!str into int`; the same key at TOP LEVEL is
    # silently IGNORED). The default gave up while the model was still working —
    # a ~120k-token prompt takes ~190 s and the client saw
    # `502 peer proxy error: net/http: timeout awaiting response headers`.
    timeouts:
      responseHeader: 3600
      upstream: 3600
      idleConn: 3600
    models:
      - qwen36-ft
      - gemma4-ft
      # gpt-oss-20b-ft (systemd: gpt-oss-ft-driver) — third mutually-exclusive
      # FreeToken model on this same :9162 / 5060 Ti slot. offload backend
      # PINNED (auto picks hybrid and costs 32%); 60.2 tok/s at 131k, the
      # fastest model on the box. Outstanding one-shot coder (22/23, realcase
      # 7/7), POOR AGENT (R5 complete 3-4/6, 1 genuine loop in 3 runs).
      - gpt-oss-20b-ft

  ollama:
    proxy: http://127.0.0.1:11434
    models:
      - gemma4:31b-cloud
      - glm-5.2:cloud
      - mistral-large-3:675b-cloud
      - kimi-k2.7-code:cloud

Generated 2026-09-12 07:52 by web-server/models.coolboxapp.com/generate.py from llama-swap.yaml, bench/results-*.log, bench/grill_out/lmeval-mini/ and auto-memory/ on server.