qwen3-instruct

Grill run history

llama-swap.yaml entry

  "qwen3-instruct":
    aliases: [qwen3-30b-a3b-instruct-2507-131k]
    # Qwen3-30B-A3B-Instruct-2507 (Alibaba, the July-2025 "2507" NON-thinking
    # refresh of the Qwen3-30B-A3B MoE), served as a vLLM AWQ via the Sophia-AI
    # W4A16 build. HEAD-TO-HEAD sibling of qwen3-thinking above: SAME base, SAME
    # 48-layer/128-expert/top-8 qwen3_moe arch. Heavier + slower than Thinking:
    # Sophia-AI keeps attn + .mlp.gate + lm_head at BF16 (18.15 GiB, 241 ignored
    # modules) vs QuantTrio Thinking's router-only BF16 (16.81 GiB), and decodes
    # ~108 tok/s (BF16 attention read) not ~146. See the verdict memory file.
    # The ONLY differences: NO think block (Instruct, not
    # Thinking) so NO --reasoning-parser, and the default temp is 0.7 (Instruct
    # preset) not 0.6 (thinking preset). This grill answers whether dropping the
    # think block keeps the quality while saving the reasoning tokens, or
    # whether the Thinking tune's CoT buys the score. TRIAL 2026-08-20.
    #
    # ARCH/QUANT/CONTEXT: SAME base/arch as qwen3-thinking, but a HEAVIER build.
    #   Sophia-AI compressed-tensors INT4 W4A16 group 128, modules_to_not_convert =
    #   attn projections + .mlp.gate + lm_head -- a BIGGER carve-out than QuantTrio
    #   Thinking (router-only), so 18.15 GiB not 16.81. That extra ~1.34 GiB eats
    #   KV headroom: at 196608/0.97 it OOMs during load (recurring "upstream exited
    #   unexpectedly" + orphan worker, 2026-08-20), so ctx is 131072 not 196608.
    # PARSERS: --tool-call-parser hermes (JSON-in-XML, same as qwen3-thinking /
    #   qwen3-vl-thinking / qwen3-vision -- this 30B-A3B-2507 line emits JSON tool
    #   calls, NOT the XML <function=> dialect of the Coder line). NO
    #   --reasoning-parser: Instruct emits no think block (matches the
    #   qwen3-vision Instruct sibling, which also has none).
    # SAMPLER: temp 0.7 = the Qwen3 Instruct preset. The grill overrides to 0.2.
    cmd: |
      /usr/bin/env
      HF_HOME=/home/giovanni/.cache/huggingface
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      VLLM_STDERR=/tmp/qwen3-instruct-via-swap.err
      NCCL_ASYNC_ERROR_HANDLING=1
      ${vllm_wrap_bin} serve Sophia-AI/Qwen3-30B-A3B-Instruct-2507-AWQ-W4A16
      --served-model-name qwen3-instruct
      --tensor-parallel-size 2
      --host 127.0.0.1 --port 9144
      --max-model-len 131072
      --max-num-seqs 8
      --gpu-memory-utilization 0.95
      --kv-cache-dtype fp8
      --override-generation-config '{"temperature":0.7}'
      --enable-auto-tool-choice
      --tool-call-parser hermes
    proxy: http://127.0.0.1:9144