glm-flash-awq

Grill run history

llama-swap.yaml entry

# 2026-07-19: raggo.net backend retired (only the SearXNG web search at
# search.raggo.net remains). The embed entry stays available on demand like
# any other model, but no longer runs persistently — the former
# "embed-persistent" group (swap:false, exclusive:false, persistent:true)
# is gone; restore it from git history if raggo comes back.

  # ==========================================================================
  # vLLM BACKENDS — added 2026-08-08.
  #
  # WHY: the vllm-*.service units each own :8200 and are mutually exclusive, so
  # switching models meant `sudo systemctl start X` plus a full reload. Measured
  # swap cost is 58-63s (glm-flash) to ~99s (the VL models), and it cannot be
  # avoided — two models do not co-reside: weights alone are 8.68 + 8.54 =
  # 17.22 GiB/GPU against a 16 GiB card, before any KV cache. llama-swap cannot
  # make the reload cheaper, but it makes it AUTOMATIC and sudo-free: ask for
  # the model by name and it tears down the previous backend first.
  #
  # MUTUALLY EXCLUSIVE WITH THE vllm-*.service UNITS. Both want the whole GPU
  # pool. Pick one manager: either llama-swap owns everything (these entries),
  # or systemd does (vllm/systemd/*.service). Running both will fight over VRAM.
  #
  # KEYLESS on purpose — llama-swap is the front door and these bind 127.0.0.1
  # only. The systemd units serve WITH --api-key because they are exposed
  # through litellm directly; here llama-swap is the thing litellm talks to.
  #
  # Env is set via `/usr/bin/env` as the exec target rather than a per-model
  # `env:` key, which nothing else in this file uses. HF_HOME must be explicit
  # (the service user's default is wrong) and CUDA_DEVICE_ORDER=PCI_BUS_ID is
  # the box-wide invariant — see auto-memory gpu-device-ordering.
  #
  # --override-generation-config IS NOT OPTIONAL. Added 2026-08-11 to all ten
  # vLLM entries after finding every one of them was SERVED HOT and GRADED COLD.
  #
  # vLLM defaults to `--generation-config auto`, which means an omitted sampling
  # parameter falls back to the MODEL VENDOR's generation_config.json. Those are
  # chat/creative defaults, and they are wild:
  #     gemma-awq 1.0/0.95/64   glm-flash-awq 1.0/0.95/50
  #     thinkingcap 1.0/0.95/20 qwen36-35b 1.0/0.95/20  bonsai-awq 1.0/0.95/20
  #     qwen3-coder 0.7/0.8/20  omni (none -> vLLM neutral, also 1.0)
  # Meanwhile EVERY published grill number was measured at temperature 0.2,
  # because bench/grill_round*.py always sends an explicit temperature. So the
  # scores describe a model nobody was actually talking to: real clients (Claude
  # Code, open-webui, pi/omp) normally omit temperature and got 1.0.
  #
  # The llama.cpp entries never had this bug — they carry `--temp 0.2` in their
  # cmd, which llama.cpp applies exactly when the client omits one. That is why
  # coder-agentic behaved BETTER than gemma-awq in real coding work despite a
  # tied grill score: coder-agentic was served at the temperature it was
  # measured at, and gemma-awq was served at 5x it.
  #
  # This is [[grill-does-not-validate-real-use]] repeating. Its rule — "the yaml
  # sampler must be a sane serving default, not a vendor benchmark profile" —
  # was written for llama.cpp and never carried over, because vLLM takes no
  # --temp flag and the equivalent knob went unnoticed for three months.
  #
  # THE VALUES ARE THE DEVELOPER'S, NOT A HOUSE DEFAULT — revised 2026-08-11.
  # These were first pinned to a flat temperature 0.2 (matching what the grill
  # measures). That was reverted the same day, for two reasons:
  #   * bench/sampler_sweep.py found NO correctness signal across 0.2/0.4/0.6/0.8
  #     — 11-13 of 13 in every cell, which is noise. The premise that 0.2 buys
  #     accuracy was never actually measured, only assumed.
  #   * gemma-awq was reported LOOPING right after the pin. Not reproducible in
  #     150+ requests ([[gemma-looping-not-reproducible]]), but with no upside
  #     demonstrated there was nothing to weigh against a live complaint.
  # So each entry now serves its AUTHOR'S recommendation: the model card where
  # one exists, else the shipped generation_config.json (also authored by them).
  #
  #     gemma-awq 1.0   glm-flash-awq 1.0   thinkingcap 1.0   omni 0.6
  #     qwen3-coder 0.7 + rep 1.05          qwen3-vision 0.7 + rep 1.0
  #     qwen3-vl-thinking 0.8 + rep 1.0     fable-711-gptq 0.6
  #     qwen36-35b 0.6 (card's THINKING preset; its generation_config says 1.0)
  #     bonsai-awq 0.7 (card profile; its generation_config says 1.0)
  #
  # THEY ARE STATED EXPLICITLY EVEN WHERE THEY MATCH generation_config.json.
  # Seven of the ten do match, so the flag is redundant for those — that is the
  # point. Silent inheritance is exactly how a 1.0 serving default went unnoticed
  # for three months while every published score was measured at 0.2. An
  # explicit value is auditable; an inherited one is invisible.
  #
  # Only THREE actually change behaviour vs the shipped config: omni (which ships
  # no sampling at all, so vLLM's neutral 1.0 applied), qwen36-35b and bonsai-awq
  # (both ship 1.0 while their cards recommend 0.6 / 0.7).
  #
  # top_p/top_k are deliberately NOT set: they already come from the same
  # developer config, so restating them would add noise without changing values.
  # QUOTING: llama-swap strips bare double quotes ({"a":1} reaches the child as
  # {a:1}, invalid JSON). The single-quoted form is verified to survive intact.
  #
  # PATH IS LOAD-BEARING AND WAS THE FIRST THING TO BREAK under systemd.
  # flashinfer JIT-compiles attention kernels at startup and shells out to
  # `ninja`, which lives in ~/vllm-env/bin and ~/.local/bin — NOT on systemd's
  # minimal PATH. Hand launches worked only because the trial scripts export
  # PATH="$HOME/vllm-env/bin:...". Without it the worker dies with
  #     FileNotFoundError: [Errno 2] No such file or directory: 'ninja'
  # ~40s in, and llama-swap reports the useless "upstream command exited
  # prematurely". Reproduced and fixed 2026-08-08 under `env -i`.
  # ==========================================================================

  "glm-flash-awq":
    # THE EVERYDAY DRIVER. 26-27/29 coding over 2 repeats, R5 6/6 then 5/6,
    # 82 tok/s, and 13/16 KLayout WITH an API reference despite being TEXT-ONLY
    # (Glm4MoeLiteForCausalLM, zero vision tensors). Its output length ignores
    # the token budget — median 2467t at a 16k cap, 2655t at 32k — which is why
    # its rare cap-hits are isolated loops rather than a distribution pressing
    # against the ceiling. See auto-memory regrill-20260808-four-model-trust.
    # 163840 ctx depends on a LOCAL vLLM fp8-MLA patch that SILENTLY REVERTS on
    # every vLLM upgrade — auto-memory vllm-mla-fp8-smem-patch.
    cmd: |
      /usr/bin/env
      HF_HOME=/home/giovanni/.cache/huggingface
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      ${vllm_bin} serve QuantTrio/GLM-4.7-Flash-AWQ
      --served-model-name glm-flash-awq
      --tensor-parallel-size 2
      --host 127.0.0.1 --port 9130
      --max-model-len 163840
      --max-num-seqs 16
      --gpu-memory-utilization 0.90
      --kv-cache-dtype fp8
      --override-generation-config '{"temperature":1.0}'
      --enable-auto-tool-choice
      --tool-call-parser glm47
      --reasoning-parser glm47
    proxy: http://127.0.0.1:9130