qwen3-thinking

Grill run history

llama-swap.yaml entry

  "qwen3-thinking":
    aliases: [qwen3-30b-a3b-thinking-2507-196k]
    # Qwen3-30B-A3B-Thinking-2507 (Alibaba, the July-2025 "2507" thinking refresh of
    # the Qwen3-30B-A3B MoE), served as a vLLM AWQ via the QuantTrio build.
    # TRIAL 2026-08-20: the box's one open gap is a FAST TEXT-ONLY THINKING MoE --
    # existing thinking models are dense/slow (qwen38 ~28 tok/s) or vision+rejected
    # (qwen3-vl-thinking, false-passed on 1000x-oversize geometry). This is 30.5B
    # total / 3.3B active, same qwen3_moe arch as qwen3-coder/qwen3-vision, so ~140
    # tok/s with 262k native context. The 2507 refresh posted big gains over the
    # original Qwen3-30B-A3B (AIME25 70.9->85.0, LiveCodeBench 57.4->66.0, GPQA
    # 65.8->73.4). Candidate for the R5 long-horizon loop + the real-case gate.
    #
    # ARCH qwen3_moe (Qwen3MoeForCausalLM): 48 layers, hidden 2048, 32 attn / 4 kv
    #   heads (GQA, head_dim 128), 128 experts top-8, moe_int 768. PURE MoE -- all
    #   48 layers full-attention, NO linear-attention/DeltaNet (unlike qwen3_5 /
    #   qwen38-awq). Standard qwen3_moe handling, same as qwen3-coder.
    # QUANT: AWQ 4-bit, group 128, gemm, zero_point. modules_to_not_convert =
    #   ['.mlp.gate'] only -- experts ARE quantized, a clean standard AWQ MoE (NOT
    #   the broken avyukth RTN that wrecked qwen3_5). QuantTrio = proven publisher,
    #   same as the served qwen3-coder / qwen3-vision AWQs. 16.8 GiB on disk
    #   (verified from the HF API blob listing).
    # CONTEXT 196608, NOT the 262144 native. vram_fit.py at util 0.97: 16.8 GiB
    #   weights + 1.5 overhead leaves 12.6 GiB KV pool = 276k tokens at fp8 (48
    #   KiB/token). 196608 uses 9.4 GiB for one seq, leaving ~3 GiB for a second
    #   concurrent seq -- safe for the grill's 1-2 concurrent suites. 262144 would
    #   monopolize the whole KV pool (max-num-seqs effectively 1). Step to
    #   131072 before touching the quant if a big think-block OOMs.
    # PARSERS: --tool-call-parser hermes (JSON-in-XML dialect, same as the
    #   qwen3-vl-thinking / qwen3-vision siblings on this box) + --reasoning-parser
    #   qwen3 (splits the think block into reasoning_content). The repo's
    #   tokenizer_config.json ships a chat_template instructing JSON tool calls
    #   wrapped in the 151657/151658 special tokens -- hermes_tool_parser.py
    #   matches that EXACTLY (same markers + json.loads). NOT qwen3_coder: that
    #   is the XML <function=>/<parameter=> dialect of the Qwen3-CODER line
    #   (qwen3-coder, qwen38-awq), which this model does NOT emit -- verified
    #   2026-08-20, qwen3_coder leaked the raw call to content (0 parsed).
    #   NO --enable-expert-parallel: 128 experts divides evenly over TP=2
    #   (64/rank), and qwen3-coder/qwen3-vision run fine WITHOUT it on 0.26.0.
    # SAMPLER: temp 0.6 = the Qwen3 thinking preset (shipped generation_config).
    # LOAD-BEARING (2026-08-21): QuantTrio's shipped template ends
    # '<|im_start|>assistant\n' with NO pre-opened <think>, but vLLM's qwen3
    # reasoning parser starts in ParserState.REASONING assuming there is one.
    # Under STREAMING that mismatch dumped the whole think block into visible
    # content on 42% of turns (0% non-streaming — invisible to any
    # non-streaming test). This template is QuantTrio's own with the single
    # change '...assistant\n' -> '...assistant\n<think>\n'. Verified 0/12 after
    # vs 5/12 before; tool calling re-checked intact. Do NOT drop this flag,
    # and note the flag is --chat-template: --chat-template-file does not
    # exist in this vLLM and kills the model at launch with a bare HTTP 500.
    cmd: |
      /usr/bin/env
      HF_HOME=/home/giovanni/.cache/huggingface
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      VLLM_STDERR=/tmp/qwen3-thinking-via-swap.err
      NCCL_ASYNC_ERROR_HANDLING=1
      ${vllm_wrap_bin} serve QuantTrio/Qwen3-30B-A3B-Thinking-2507-AWQ
      --served-model-name qwen3-thinking
      --tensor-parallel-size 2
      --host 127.0.0.1 --port 9143
      --max-model-len 131072
      --max-num-seqs 8
      --gpu-memory-utilization 0.95
      --kv-cache-dtype fp8
      --override-generation-config '{"temperature":0.6}'
      --enable-auto-tool-choice
      --tool-call-parser hermes
      --reasoning-parser qwen3
      --chat-template /home/giovanni/coolbox/templates/qwen3-thinking-2507.jinja
    proxy: http://127.0.0.1:9143