gemma-exl3

Grill run history

llama-swap.yaml entry

  "gemma-exl3":
    aliases: [gemma-4-26b-a4b-it-exl3-6bpw]
    # THIRD EXL3 ENTRY — and the first that is a PRECISION UPGRADE OF THE DRIVER
    # rather than a new model. `turboderp/gemma-4-26B-A4B-it-exl3` @ 6.10bpw,
    # 22 GB on /mnt/models/exl3. SAME BASE WEIGHTS as `gemma-awq`
    # (cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit), which is the box's DRIVER at
    # 44/46 coding and 18/18 office ([[gemma-4-26b-awq-vllm]]).
    #
    # THE TRADE, measured 2026-09-04:
    #     |            | this entry (EXL3 6.10bpw) | gemma-awq (INT4)  |
    #     | context    | 262144  (PARITY)          | 262144            |
    #     | KV         | Q8, 5.00 GiB              | fp8               |
    #     | weights    | 6.10 bpw                  | INT4              |
    #     | decode     | 51.1 tok/s                | 93 tok/s          |
    #     | vision     | YES, 5.0 s, correct       | YES               |
    # ~1.5 bits more weight precision for ~45% of the throughput, at IDENTICAL
    # context. [[vram-spend-priority]] says spend spare VRAM on quality, so this
    # is a legitimate trade — but the incumbent is at 44/46, which leaves almost
    # no room to gain and plenty to lose. NO QUALITY VERDICT YET.
    #
    # 262144 FITS, and the reason is worth keeping: only 5 of 30 layers are
    # full attention (25 are sliding_attention, window 1024), so KV is just
    # 20.0 KiB/token at Q8 -> 5.00 GiB for the full window. 21.95 GiB weights +
    # 5.00 KV = 26.95 GiB of ~30.3 usable; measured 27.4 GiB resident.
    # Contrast the DENSE gemma-4-31b-it, which has 16 KV heads and costs
    # 80 KiB/token — it caps near 104k and would decode ~4x slower.
    # See [[kv-sizing-full-attention-layers]].
    #
    # *** VISION IS REAL HERE — 355 vision_tower tensors verified in
    # model.safetensors.index.json BEFORE downloading. *** That check exists
    # because `kat-coder`'s quant declared a multimodal config and shipped ZERO
    # visual tensors, making --vision true a hard crash
    # ([[kat-coder-exl3-trial]]). Never trust a vision claim without grepping
    # the index. This matters for gemma specifically: office needs vision for
    # 3 of 9 stages ([[office-suite-vision-floor]]).
    #
    # *** TOOL CALLING WAS THE GATE, AND IT NEEDED ONE FLAG: --tool-format
    # gemma4 (below). *** TabbyAPI DOES have a gemma4 tool format — the earlier
    # note here said it did not, which was wrong. What it does not do is
    # auto-detect it. Without the flag the first n=3 scored 17/23, 16/23, 16/23
    # with tool_weather and tool_select failing 3/3 and R5 completing 0/6; pure
    # coding was untouched at r1 8/8, 7/8, 7/8. A config gap, not a model
    # defect. `--reasoning true` is harmless (gemma-4 is NON-thinking:
    # reasoning_content comes back empty, answer lands in content).
    #
    # *** OFFICE/KLAYOUT VERDICT: PARITY WITH THE DRIVER ON ALL FOUR ARMS. ***
    # This was the measurement meant to settle promotion, on the driver's
    # strongest ground. It settles it as a TIE:
    #     office unaided  9/9, 9/9      vs driver 9/9 per rep   TIE
    #     office + ref    9/9, 6/9, 6/9 vs driver 6/9 per rep   TIE
    #     klayout memory  0/8           vs driver 0/8           TIE
    #     klayout + ref   6/8           vs driver 6/8           TIE
    # The 6.10bpw precision buys NOTHING measurable here. Vision works (office
    # needs it for 3 of 9 stages), so this is apples-to-apples. A +3 "win" on
    # office+ref was reported at n=1 and was WRONG — two fresh-server reps both
    # gave 6/9; the 9/9 outlier followed an unaided pass on the same server, so
    # warm prefix cache is the likely cause. Driver totals in auto-memory are
    # 2 REPS — halve them before comparing.
    #
    # *** IT HITS THE EXLLAMAV3 RECURRENT-STATE SLOT LEAK. RESTART BETWEEN
    # HEAVY RUNS. *** After ~40 requests:
    #     AssertionError: Cannot create new state: no available slots
    #     -> 503 "Chat completion ... aborted. Maybe the model was unloaded?"
    # `num_slots = max_batch_size` (default 16) and slots are never returned.
    # THE PROCESS STAYS ALIVE AND /v1/models STILL RETURNS 200 WITH THE MODEL
    # LISTED — only completions fail, so a health check cannot see it. It
    # produced three bogus 0/9 grill runs that looked like a quality collapse.
    # gemma-4 runs 25 of 30 layers as sliding_attention, which is the suspected
    # trigger (kat-coder did 3 full reps without it). This alone blocks
    # promotion. See [[exl3-tabbyapi-backend]].
    #
    # TP setup is identical to the other two EXL3 entries: nccl backend (no P2P
    # on these cards) + `cuda_malloc_async: false` in ~/ai/tabbyAPI/config.yml.
    # ALSO: after a `kill -9` of a TP server, WAIT ~25 s before relaunching —
    # the pinned 64 MB host arena is not released instantly and a fast restart
    # fails with the same cudaHostRegister error the allocator setting fixes.
    # Loads in 85 s.
    #
    # ON /mnt/models (cold tier) NOT /mnt/ssk500 — ssk500 had only 21 GB free
    # against a 21.95 GiB quant. If this ever becomes a driver, move it to NVMe
    # first: llama-swap swaps it constantly and USB costs cold-load time
    # ([[storage-tiers]]).
    cmd: |
      /usr/bin/env
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      HF_HOME=/home/giovanni/.cache/huggingface
      PATH=/home/giovanni/ai/tabbyAPI/venv/bin:/usr/local/cuda-13.0/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      TRITON_CACHE_DIR=/home/giovanni/.cache/triton
      TABBY_HOME=/home/giovanni/ai/tabbyAPI
      TABBY_STDERR=/tmp/gemma-exl3-via-swap.err
      ${tabby_wrap_bin}
      --host 127.0.0.1 --port 9192
      --model-dir /mnt/models/exl3
      --model-name gemma-4-26B-A4B-it-exl3-6.10bpw
      --max-seq-len 262144
      --cache-size 262144
      --cache-mode Q8
      # 355 vision_tower tensors verified present — unlike kat-coder.
      --vision true
      --reasoning true
      # *** LOAD-BEARING. WITHOUT IT EVERY TOOL TASK FAILS AND THE SCORE
      # COLLAPSES. *** TabbyAPI ships endpoints/OAI/utils/toolcall_formats/
      # gemma4.py, but tool_format is NOT auto-detected — it is read from
      # kwargs/tabby_config.yml, and THIS QUANT SHIPS NO tabby_config.yml
      # (kat-coder's does, which is why that one logged a tool format and this
      # one silently did not). Symptom when missing: "no tool call emitted",
      # tool_weather + tool_select FAIL 3/3, round 4 collapses to 1/5 and R5
      # completes 0/6 — while looping 0/6, i.e. it never calls a tool rather
      # than looping. Measured: round 3 went 3/5 -> 5/5 the moment this was
      # added. Confirm at startup with "Using tool format: gemma4".
      --tool-format gemma4
      --tensor-parallel true
      --tensor-parallel-backend nccl
      --gpu-split-auto false
      --chunk-size 2048
    proxy: http://127.0.0.1:9192