gemma12-solo

Grill run history

llama-swap.yaml entry

  "gemma12-solo":
    aliases: [gemma-4-12b-qat]
    # THE TWO-HARNESS ENTRY. gemma-4-12B-it QAT AWQ-INT4 (cyankiwi, same
    # quantizer as the gemma-awq driver), 11.2 GB, added 2026-08-22.
    # TP=2 ACROSS BOTH CARDS. Use this when you want two coding harnesses on one
    # model: vLLM continuous batching already interleaves requests from
    # different clients (--max-num-seqs 8), so a SECOND INSTANCE IS NOT NEEDED
    # for a second client -- only for hard isolation. Splitting the weights
    # 5.6 GB/card instead of duplicating them 11.2+11.2 spends the difference on
    # KV cache (~2x the pool) and lets every request use both GPUs, so
    # single-stream latency is better for both harnesses too.
    # The cost, accepted deliberately: a burst from one harness can occupy batch
    # slots the other wanted. If that ever bites, the alternative is two
    # single-GPU instances (one per card) at the price of losing OCR co-residency.
    #
    # THIS ENTRY IS EXCLUSIVE (no group): it uses both cards, so it evicts
    # everything, glm-ocr included. When you need OCR alongside a chat model,
    # ask for `gemma12` instead -- same weights, single card, co-resident.
    #
    # 131072 is its NATIVE max_position_embeddings (NOT 262144 like the 26B).
    # KV is cheap here: only 8 of 48 layers are full_attention (the other 40 are
    # sliding at window 1024), so the full window fits easily at fp8.
    # QAT, not post-training quant: Google's quantization-AWARE-trained
    # checkpoint holds far more quality at INT4 than a PTQ build.
    # GRILLED n=3 2026-08-23 (156 min, 42 suites) -- see [[gemma12-grill]]:
    #   coding 22-23/23, R5 ZERO LOOPS all 3 reps, realcase PASS 2 of 3,
    #   klvision 4/6 x3, envelope CLEAN over 376 turns.
    #   office 0/9 UNAIDED vs 9/9 WITH the API ref, three times each -- its
    #   weakness is API RECALL, not capability. But the same help is worth only
    #   +2 on KLayout (0/8 -> 2/8 x3), so pya composition is a hard ceiling here;
    #   worth retrying with the full klayout-classes skill rather than the
    #   grill's 40-line signature list.
    # ~26 tok/s single-card; it did ~43 at TP=2, so a dense 12B gains from the
    # split (unlike the 2.5B-active mellum, which loses).
    cmd: |
      /usr/bin/env
      HF_HOME=/home/giovanni/.cache/huggingface
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      ${vllm_bin} serve cyankiwi/gemma-4-12B-it-qat-AWQ-INT4
      --served-model-name gemma12-solo
      --tensor-parallel-size 2
      --host 127.0.0.1 --port 9156
      --max-model-len 131072
      --max-num-seqs 8
      --gpu-memory-utilization 0.90
      --kv-cache-dtype fp8
      --enable-auto-tool-choice
      --tool-call-parser gemma4
      --reasoning-parser gemma4
    proxy: http://127.0.0.1:9156