coder-agentic-q3

Grill run history

llama-swap.yaml entry

  "coder-agentic-q3":
    aliases: [qwen3-coder-next-80b-q3]
    # THIRD rung of the Qwen3-Coder-Next 80B-A3B quality ladder, added 2026-08-24.
    # All three entries are the SAME base weights, kept side by side on purpose:
    #   coder-agentic     UD-IQ2_XXS 21.71 GiB + q4_0 KV @262k  (~72 t/s, 22/23)
    #   coder-agentic-q3  UD-IQ3_XXS 26.53 GiB + q8_0 KV @131k  (this entry, ~67 t/s)
    # *** SPEED FIGURES RECONCILED 2026-09-09 — THREE conflicting numbers were in
    # circulation for this one entry, all real but measured differently:
    #   49.8 t/s  ladder grill 2026-08-24, LAYER split      <- STALE, superseded
    #   ~61 t/s   KV-isolation 2026-08-26, LAYER, in-suite  <- matches 61.5 below
    #   64.4 -> 72.5 t/s  2026-09-02 microbenchmark, layer -> TENSOR
    #   61.5 -> 67.3 t/s  2026-09-02 in-suite,       layer -> TENSOR
    # This entry runs `-sm tensor`, so the CURRENT numbers are ~72 t/s on a clean
    # microbenchmark and ~67 t/s in-suite. Quote the in-suite one for planning.
    # A middle rung `coder-agentic-hq` (UD-IQ2_M 23.25 GiB + q8_0 KV @262k, 62 t/s)
    # existed 2026-08-24 and was DELETED the next day: the ladder grill (n=3, same
    # harness, bench/grill-ladder-20260824-2057) had it MATCHING the baseline on
    # every suite -- coding 22/23, klayout+ref 5/8, office 0/9 -- while costing 14%
    # decode. It bought nothing. Weights deleted too; ssk500 was at 97%.
    # This is the first TRUE 3-BIT rung: every entry above is a 2-bit quant, and
    # 2-bit is the band that turned DeepSeek-R1-Distill-70B to garbage
    # ([[deepseek-r1-distill-llama-70b]], grill 0/3). The Unsloth DYNAMIC quants
    # dodge that (coder-agentic scores 22/23 at UD-IQ2_XXS), but 3-bit should
    # need no dodging at all -- that is the hypothesis this entry tests.
    #
    # CONTEXT 131072, NOT 262144 -- this is the real cost of the extra 3.28 GiB.
    # Sized from the MEASURED overhead of the -hq load (28.91 GiB used for 23.25
    # weights + 3.00 KV@262k => overhead 2.66 GiB), not a guess:
    #   -c 131072 -> 26.53 + 2.66 + 1.50 KV = 30.69 / 31.92, spare 1.23 GiB  <- chosen
    #   -c 196608 -> spare 0.48 GiB  (TIGHT)
    #   -c 262144 -> DOES NOT FIT
    # 1.23 GiB spare is deliberate. qwen25vl-72b loaded at 31.1/32 GiB with ~0.9
    # GiB spare and then HARD-ABORTED on its first image; an 80B MoE with a 2048
    # batch has no reason to be run on that knife-edge. Raise -c only after
    # watching a real long-context run, never on arithmetic alone.
    #
    # GRILLED 2026-08-24 (bench/grill-ladder-20260824-2057, n=3, verdict in
    # [[coder-agentic-quant-ladder]]). MORE BITS IS NOT A STRICT WIN: vs the 2-bit
    # baseline this 3-bit rung GAINS on reference-driven suites (office unaided
    # 2->12, office+ref 12->18, klayout+ref 16->18) but REGRESSES R5 long-horizon
    # (looping 1->4 across 3 reps, complete 6/6->4/6) — the dealbreaker axis on
    # this box. It also costs half the context (131k vs 262k). *** THE OLD "30%
    # SLOWER (49.8 vs 72 t/s)" CLAIM IS DEAD as of the 2026-09-02 -sm tensor
    # switch: this rung now runs ~67 t/s in-suite against the 2-bit's ~72, i.e.
    # near parity, so speed is NO LONGER a reason to prefer the 2-bit. *** Note
    # the asymmetry — the 2-bit `coder-agentic` is still on --tensor-split 48,52
    # (layer) and has NOT been re-measured under -sm tensor, so it may gain too;
    # the two figures also come from different protocols. Pick the rung on R5
    # looping and context, not on decode. REACH FOR THIS RUNG for office / KLayout-with-a-reference
    # task; for agentic / driver use the 2-bit coder-agentic stays better (clean
    # R5 + 262k). The middle rung coder-agentic-hq (IQ2_M) was DELETED — it
    # matched baseline on every suite at 14% slower and bought nothing.
    #
    # FOLLOW-UP 2026-08-26 (KV-isolation, see [[coder-agentic-quant-ladder]]): the
    # looping is the q8_0 KV, NOT the 3-bit weights — a matched 3-bit + q4_0 KV +
    # 131k run loops only 1× (vs 4× here). q8_0 KV is a double-edged sword: +4
    # office+ref (18 vs 14) AND +3 loops. q4_0 KV gives NO speed gain (~61 tok/s
    # either way — the 26.5 GiB weights dominate bandwidth, not the ~1.5 GiB KV).
    # q4_0 KV + 262k OOM-CRASHES under load (context-length runtime overhead breaches
    # the ~1.2 GiB spare on the large-prompt klayout/office requests), so 131k is the
    # STABILITY ceiling for this 3-bit rung — q8_0 KV + 131k is load-bearing for
    # stability, not just context. Kept q8_0 KV + 131k (reverted from a q4_0 trial).
    #
    # Batch/sampler held IDENTICAL to the other rung. SPLIT MODE IS NOT — see
    # below; that divergence is deliberate and measured.
    #
    # *** -sm tensor, AND THEREFORE NO --tensor-split. DO NOT "RESTORE" IT. ***
    # Passing --tensor-split with -sm tensor does not start: llama_params_fit is
    # not implemented for SPLIT_MODE_TENSOR, and the load then dies on a
    # 20006.64 MiB "cudaMalloc failed" that is NOT an OOM. Tensor mode splits
    # itself, and exactly even (15305|15305 MiB) — there is nothing to tune.
    # See auto-memory/sm-tensor-rejects-tensor-split.md.
    #
    # WHY tensor: layer split runs the cards SEQUENTIALLY, so the faster 5060 Ti
    # idles waiting its turn (util 34%/50%). Tensor mode runs them concurrently
    # (79%/83%) and is worth +12.5% decode standalone, +9.4% in-suite. It costs
    # -7.4% PREFILL (already compute-parallel; tensor mode only adds all-reduce
    # there), so quote the in-suite number, not the microbenchmark.
    #
    # GRILLED 2026-09-02, BOTH ARMS SAME-DAY (bench/grill-smt-ab-20260902-0727,
    # r1-r4 n=2 + R5 n=3 per arm, same port/cmd so -sm is the only variable):
    #   coding    22/23, 22/23 BOTH arms, same expr_eval loss
    #   R5 loops  tensor 1,1,2 | layer 1,1,1 — interrupt_replan 3/3 IN BOTH
    #   decode    67.3 vs 61.5 t/s in-suite
    # The loop FINGERPRINT is identical, which is the R5 signal; the 4-vs-3 count
    # is sampling (this rung's own layer baseline is 1,1,2 = 4, and the loops are
    # the q8_0 KV per the follow-up above, not the split mode).
    # PER-MODEL VERDICT: -sm tensor was quality-regressive on gemma-4-26b in
    # 2026-07 (temp-0.4 repetition loop). Grill any other entry before switching
    # it, and note NO --mmproj entry has been tested under tensor mode.
    # STORAGE NVMe — PROMOTED off ssk500 2026-08-25 once this became a model the
    # user actually reaches for. Worth ~47 s per cold load (nvme 2.4 GB/s vs
    # ssk500 409 MB/s, [[nvme-working-set]]). The counter-example is
    # coder-agentic, which was promoted then DEMOTED because its cold load was
    # dominated by 262144-context KV allocation rather than disk read; this rung
    # is 131072, so KV alloc is cheaper and disk is a larger share of load time.
    cmd: |
      /usr/bin/env
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      ${llama_bin}
      -m /home/giovanni/models-nvme/Qwen3-Coder-Next-UD-IQ3_XXS.gguf
      --alias coder-agentic-q3
      --jinja -ngl 99 -c 131072 -fa on
      -sm tensor
      -b 2048 -ub 1024
      --temp 0.2 --top-p 0.9 --top-k 40 --repeat-penalty 1.05
      --cache-type-k q8_0 --cache-type-v q8_0
      --host 127.0.0.1 --port 9172 --parallel 1
    proxy: http://127.0.0.1:9172