gemma-awq

Grill run history

llama-swap.yaml entry

  "gemma-awq":
    # NON-QWEN. gemma-4-26B-A4B-it at AWQ-4bit, 16 GiB, 26B-A4B MoE.
    # TIED BEST coding on the box (44/46) at 93 tok/s — 3x the fables' speed —
    # plus tied-best vision (10/12) answering in 7-15 tokens, the fastest here.
    # Its GGUF sibling was REJECTED 2026-07-31 for deterministic 16k runaways
    # that DO NOT occur on this build; see auto-memory gemma-4-26b-awq-vllm.
    #
    # --tool-call-parser gemma4 is LOAD-BEARING. `hermes` returns
    # tool_calls: null with NO error, which silently nulls every agentic round.
    #
    # --reasoning-parser gemma4 IS ALSO REQUIRED, which is easy to miss: gemma4
    # emits no <think> block, but it does wrap reasoning in CHANNEL markers.
    # tokenizer_config.json spells the format out:
    #   (<|channel>thought\n(?P<thinking>.*?)<channel|>)?(?P<tool_calls>...)
    # Without the parser that text lands in `content` verbatim — measured
    # 2026-08-10, the turn AFTER a tool_result returned
    #   "<|channel>thought\n<channel|>The task has been completed successfully."
    # Plain turns look clean, so this only shows up in agentic use.
    #
    # Office is 18/18 UNASSISTED, spread 0 — joint-best with glm-flash-awq, and
    # the only model that manages it while also having vision. (Its first
    # measurement was 0/18: gemma writes `if __name__ == "__main__":` and the
    # grill exec'd with __name__ == "builtins", silently skipping every program.
    # See auto-memory exec-main-guard-defect.)
    #
    # 262144 is its NATIVE max_position_embeddings, and KV is nearly free here:
    # only 5 of 30 layers are full_attention (the rest sliding-window at 1024),
    # so fp8 KV costs 20 KiB/token = 2.50 GiB/GPU at the full 262k. Measured at
    # 131072 the server had 4.16 GiB KV free (597,899 tokens) — it was using a
    # quarter of its headroom. See auto-memory kv-sizing-full-attention-layers.
    #
    # SAMPLER: temperature 1.0, no repetition penalty — google's own card values,
    # which are also what this build ships in generation_config.json (1.0 / 0.95
    # top_p / 64 top_k). A repetition_penalty of 1.05 was briefly added here on
    # 2026-08-11 against a looping report and then REMOVED: it was never shown to
    # help (see [[gemma-looping-not-reproducible]]) and the developer specifies
    # none. Do not re-add it without a reproducible failing case — it has a real
    # cost on CODE, where tokens legitimately repeat (indentation, `self.`,
    # recurring identifiers).
    #
    # THE REAL GAP IS KLAYOUT: 0/16 from memory, 12/16 with the API reference,
    # and 0/8 on the closed loop. Point KLayout work at fable-fusion (15/16 and
    # 8/8 on the loop, the only model that closes it unaided).
    cmd: |
      /usr/bin/env
      HF_HOME=/home/giovanni/.cache/huggingface
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      ${vllm_bin} serve cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit
      --served-model-name gemma-awq
      --tensor-parallel-size 2
      --host 127.0.0.1 --port 9135
      --max-model-len 262144
      --max-num-seqs 8
      --gpu-memory-utilization 0.90
      --kv-cache-dtype fp8
      --override-generation-config '{"temperature":1.0}'
      --enable-auto-tool-choice
      --tool-call-parser gemma4
      --reasoning-parser gemma4
    proxy: http://127.0.0.1:9135