tommy

Grill run history

llama-swap.yaml entry

  "tommy":
    aliases: [thomson-1.0-small-qwen35-256k]
    # thomsonreuters/Thomson-1.0-Small — TRIAL 2026-08-26. Thomson Reuters'
    # "SovereignAI" value-aligned model. SAME arch as qwen36-35b: Qwen3.6-35B-A3B
    # (qwen3_5_moe, Qwen3_5MoeForConditionalGeneration), 35B total / 3B active,
    # 256 experts 8/tok, 40 layers (full_attention_interval 4 = 10 full-attn),
    # 2 kv_heads, head_dim 256, 262144 ctx. VISION-capable (qwen3_5_moe_vision,
    # depth 27 — mmproj WIRED, like ornith, unlike text-only pocket-35b).
    # Multistage pipeline over the Qwen3.6-35B-A3B base: value re-alignment
    # (Snowdon1.1-Small) + continual pre-training + post-training, tuned for
    # legal / tax / journalism + deep-research + political neutrality. The card
    # claims "general capabilities largely preserved" vs the base — this grills
    # that head-to-head vs the served qwen36-35b (41/46 coding, R5 12/12,
    # KLayout+ref 15/16, office+ref 12/12). Triage check 8 ([[model-triage-checklist]]):
    # derivative of a served base, so it must justify on the grill; expect parity
    # (= redundant) unless the tune shifted something. NOT a coder model — graded
    # on the standard grill for a controlled "did the tune preserve the base?" test.
    # LICENSE: polyform-strict-1.0.0 (not open for commercial use). Homelab /
    # personal use is the use case here; note before any production reuse.
    #
    # QUANT: bartowski Q4_K_M (19.92 GiB) + mmproj-BF16 (0.84 GiB), from
    # bartowski/thomsonreuters_Thomson-1.0-Small-GGUF. Q4_K_M is the proven 4-bit
    # for this arch (pocket-35b 22/23 R5 6/6, ornith vision, fable-fusion all run
    # it at 262k). No AWQ repo exists and self-quant AWQ is REJECTED here (4x,
    # [[fable711-awq-self-quantize-todo]]), so GGUF / llama.cpp is the only path
    # without a 70 GB BF16 pull. Q5_K_M (23.3 GiB) would fit text-only but is
    # tight with the ViT buffer (first-image OOM risk, [[qwen38-27b-install]]);
    # Q6_K (28 GiB) is over the 27 GiB ceiling. Q4_K_M + mmproj @ 262k leaves
    # ~6 GiB headroom for the lazily-allocated ViT buffer (fable-fusion proves
    # Q4_K_M + vision @ 262k survives the largest image at ~1.1 GB spare).
    #
    # TEMPLATE: ships the Qwen3.5/3.6-MoE family template (7764 chars, verified
    # byte-identical to pocket-35b / ornith / kat-coder) WITH the non-first-system
    # raise_exception guard (line 85). Same one-line patch as every qwen3_5_moe
    # entry — chat-templates/qwen35moe-nonfirst-system-256k.jinja is the shipped
    # template with line 85 replaced by a ChatML system render (diffed + confirmed
    # 2026-08-26: ONLY line 85 differs). --chat-template-file is LOAD-BEARING
    # ([[jinja-system-guard-tool-parser]]); a single-turn smoke test does NOT
    # catch the multi-turn system+tools 400.
    #
    # SERVING mirrors ornith-15-35b exactly (same arch, same Q4_K_M, same
    # mmproj-BF16 + q8_0 KV @ 262k shape): split 42,58 (retuned for ornith because
    # mmproj + q8 KV puts ~4 GB of non-weight memory on device0 = the 4060 Ti;
    # 42,58 relieves the slow card, [[gpu-device-ordering]]). q8_0 KV accepted by
    # qwen3_5_moe (pocket-35b / ornith prove it). Sampler = Qwen3 thinking preset
    # (0.6 / 0.95 / 20 / min-p 0); the grill overrides temp to 0.2.
    #
    # *** --reasoning-format deepseek IS LOAD-BEARING — DO NOT DROP IT. ***
    # Without it aquila leaks `</think>` into message.content on PLAIN-TEXT
    # answer turns (the summarising message after a tool returns), and Claude
    # Code renders it inline then repeats the whole answer. Found in REAL USE
    # 2026-08-30 AFTER a 125-turn grill scored envelope CLEAN — the grill never
    # exercises that turn ([[aquila-think-tag-leak-real-use]],
    # [[grill-does-not-validate-real-use]]). tommy on the byte-identical
    # template does NOT leak, so this is the MODEL, not the template.
    # *** RE-VERIFIED AT n=8 (bench/grill_textturn.py): THE FLAG IS NOT ENOUGH. ***
    # 6/24 turns still leak (25%). It DOES fix the tools-bound shapes (16/16
    # clean: text answer + post-tool summary), but with NO tools bound it leaks
    # 6/8 — and one rep was a RUNAWAY, not a cosmetic leak: finish=length,
    # 16177 chars, `</think>` x734 and `<tool_call>` x732 repeated until the cap.
    # DO NOT DRIVE CLAUDE CODE (or any agent) WITH THIS ENTRY. The user's real
    # Claude Code transcript leaked even though that client binds tools, so the
    # clean B/C scenarios do not model it fully. Keep aquila for text/RAG use.
    # Speed: ~80.7 tok/s mean (median 84) vs tommy's ~93.4 — the bandwidth cost
    # of Q5_K_M over Q4_K_M ([[decode-bandwidth-ceiling]]).
    #
    # RESIDUAL (accepted): with NO tools bound it still emits
    # `<tool_call><function=bash>` as prose — aquila's trained tool format
    # differs from this template's. Claude Code always binds tools, so that
    # path is not hit; do not use aquila tool-less for agentic prompts.
    #
    # NOT A FAMILY-WIDE FIX: this flag works here only because aquila emits the
    # DeepSeek `<think>` tag. `tommy` emits `<thinking>`, which no llama.cpp
    # reasoning-format parses — tested n=5, it leaks 2/5 without the flag and
    # 3/5 with it (noise; no improvement). Do NOT copy this flag to tommy and
    # assume it fixed anything. Both models also FABRICATE tool execution ~1/5
    # on tool-less turns, which no flag addresses.
    #
    # REASONING MODEL (family template defaults reasoning_effort xhigh) — CoT
    # goes to reasoning_content, answer to content. Budget max_tokens generously:
    # at a short cap it hits the limit mid-think and returns EMPTY content with
    # no error (same trap as pocket-35b / ornith). STORAGE /mnt/ssk500 (trial
    # tier); ~+47 s cold load vs NVMe ([[nvme-working-set]]) — cheap for a trial.
    # *** -sm tensor (was `--tensor-split 42,58`). DO NOT RESTORE --tensor-split: passing it
    # with -sm tensor does not start -- llama_params_fit is unimplemented for
    # SPLIT_MODE_TENSOR and the load dies on a bogus ~20 GB "cudaMalloc failed"
    # that is NOT an OOM. Tensor mode splits itself, evenly, with nothing to tune.
    # Layer split runs the cards SEQUENTIALLY so the faster 5060 Ti idles waiting;
    # tensor runs them concurrently. auto-memory/sm-tensor-rejects-tensor-split.md
    #
    # GRILLED 2026-09-02 BOTH ARMS SAME-DAY (bench/grill-smt-tommy-*,
    # r1-r4 n=2 + R5 n=3 per arm, same port + same cmd so -sm is the ONLY variable):
    #   coding  21/23 22/23 tensor | 21/23 22/23 layer
    #   R5      6/6 6/6 6/6 tensor | 6/6 6/6 5/6 layer
    #   R5 LOOPS 0/6 IN EVERY REP OF BOTH ARMS  <- the signal that matters
    #   vision  5/6 tensor | 5/6 layer  (klvision, real rendered images)
    #   decode  111.0 vs 90.0 tok/s in-suite over 46 tasks/arm = +23.3%
    # Differences are within this box's known R5/coding sampling spread.
    cmd: |
      /usr/bin/env
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      ${llama_bin}
      -m /mnt/ssk500/gguf/thomsonreuters_Thomson-1.0-Small-Q4_K_M.gguf
      --mmproj /mnt/ssk500/gguf/mmproj-thomsonreuters_Thomson-1.0-Small-bf16.gguf
      --alias tommy
      --jinja --chat-template-file chat-templates/qwen35moe-nonfirst-system-256k.jinja
      -ngl 99 -c 262144 -fa on
      -sm tensor
      -b 2048 -ub 512
      --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0
      --cache-type-k q8_0 --cache-type-v q8_0
      --host 127.0.0.1 --port 9170 --parallel 1
    proxy: http://127.0.0.1:9170