qwen38-exl3

Grill run history

llama-swap.yaml entry

  "qwen38-exl3":
    aliases: [qwen3.8-27b-exl3-6bpw]
    # FIRST EXL3 ENTRY ON THE BOX — TabbyAPI + ExLlamaV3 backend, neither
    # llama.cpp nor vLLM. Wired exactly like the vLLM entries: llama-swap owns
    # the lifecycle, so selecting this model EVICTS the resident one instead of
    # fighting it for VRAM. Install runbook: install-tabbyapi-exllamav3.md.
    #
    # MODEL: Qwen3.8-27B, turboderp's EXL3 at 6.00bpw with a 6-bit head and
    # 6-bit vision tower (branch SC_6.00bpw_H6_V6, 21.07 GiB on disk).
    # Arch Qwen3_5ForConditionalGeneration — in exllamav3's supported list.
    # Effectively lossless weights vs the INT4 AWQ `qwen38-awq` already here.
    #
    # *** WHY 6bpw AFFORDS A LONG WINDOW AT ALL ***
    # This is a HYBRID: only 16 of its 64 layers are full attention; the other
    # 48 are linear (Gated DeltaNet, whose state is per-SEQUENCE and fixed, not
    # per-token). KV is therefore just 32 KiB/token at Q8.
    # DO NOT size this model from num_hidden_layers — that overestimates KV 4x
    # and would talk you down to ~65k. Use `python bin/vram_fit.py --hf ...`.
    #
    # CONTEXT IS 131072, NOT THE MODEL'S FULL 262144, and the reason is a trap
    # worth remembering: nvidia-smi reports 16311+16380 MiB, but torch reports
    # only 15.48+15.58 GiB actually allocatable — ~0.9 GiB less across the pair.
    # Sizing off nvidia-smi says 262144 fits; sizing off the real figure gives a
    # Q8 ceiling of ~240k, i.e. JUST under. 131072 leaves genuine headroom.
    # To go longer, drop KV to Q6 (24 KiB/tok puts 262144 at ~6.0 GiB) rather
    # than shaving the split — but measure before trusting it.
    #
    # SPLIT: --gpu-split is GB PER DEVICE in CUDA_DEVICE_ORDER=PCI_BUS_ID order,
    # so 14 -> CUDA0 = 5060 Ti and 15 -> CUDA1 = 4060 Ti. CUDA0 deliberately
    # gets the SMALLER budget: embeddings, output head and compute buffers land
    # there regardless of the split. Same rule as the llama.cpp entries above —
    # auto-memory/gpu-card-assignment-policy.md.
    #
    # --vision true is REQUIRED. TabbyAPI defaults vision to FALSE and would
    # load this multimodal model as text-only, silently and without error.
    #
    # the CLIENT's headers, which never match a key TabbyAPI generated for
    # itself. Safe because the wrapper binds it to loopback only.
    #
    # *** CHAT TEMPLATE: THE OFFICIAL ONE REJECTS MULTI-SYSTEM PROMPTS ***
    # Qwen's own chat_template.jinja calls raise_exception("System message must
    # be at the beginning.") whenever a system message is NOT the first message,
    # or when there are two of them. Tested here on 5 message shapes before the
    # first load: official = 2 of 5 RAISE, turboderp's shipped
    # chat_template.jinja.fixed = 5 of 5 OK.
    # This is the same failure class as auto-memory/jinja-system-guard-tool-parser
    # and it is exactly what 400s agent clients that inject a mid-conversation
    # system turn. This entry currently runs the OFFICIAL template (TabbyAPI's
    # default, and what the model was trained against). If an agent client
    # starts 400ing on system-message ordering, the tested remedy is:
    #     --prompt-template /mnt/ssk500/exl3/Qwen3.8-27B-exl3-6.0bpw/chat_template.jinja.fixed
    # NOT adopted by default because .fixed is a THIRD-PARTY template
    # ("qwen3.8-froggeric-v22.3", 26 KB vs the official 9 KB) that also changes
    # thinking control and defaults tool calls to an xml format — a behaviour
    # change that needs its own grill before it drives anything.
    #
    # TABBY_STDERR is load-bearing for debugging: under llama-swap the journal
    # shows only "upstream exited unexpectedly" and the loader traceback (bad
    # split, unsupported arch, OOM) is otherwise LOST.
    cmd: |
      /usr/bin/env
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      HF_HOME=/home/giovanni/.cache/huggingface
      # PATH IS LOAD-BEARING, exactly as for the vLLM entries. systemd hands
      # llama-swap a minimal PATH; flash-linear-attention backs this model's 48
      # Gated DeltaNet layers and JIT-compiles them through Triton, which shells
      # out to a toolchain (ptxas/ld/gcc). Without PATH the entry dies with
      # llama-swap's generic "upstream command exited prematurely" and nothing
      # in the journal names the real cause. See auto-memory/llama-swap-setup.md
      # ("a service environment is not a login environment").
      PATH=/home/giovanni/ai/tabbyAPI/venv/bin:/usr/local/cuda-13.0/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      TRITON_CACHE_DIR=/home/giovanni/.cache/triton
      TABBY_HOME=/home/giovanni/ai/tabbyAPI
      TABBY_STDERR=/tmp/qwen38-exl3-via-swap.err
      ${tabby_wrap_bin}
      --host 127.0.0.1 --port 9182
      --model-dir /mnt/ssk500/exl3
      --model-name Qwen3.8-27B-exl3-6.0bpw
      --max-seq-len 196608
      --cache-size 196608
      # 196608 AT A FLAT Q8, chosen over 262144 at K8/V6. Both were loaded and
      # measured here: the full 262144 window needs 8.0 GiB of KV on top of
      # ~20.4 GiB of weights and exllamav3 refuses it ("Insufficient VRAM in
      # split for model and cache") even at a 29.5 GiB split, so it only fits by
      # dropping V to 6 bits. 196608 costs 6.0 GiB at a full Q8 and fits with
      # headroom. auto-memory/vram-spend-priority.md says spend spare VRAM on KV
      # PRECISION before context, so the shorter window at Q8 wins.
      # To take the full 262144 instead, set --cache-mode 8,6 (verified working).
      --cache-mode Q8
      --vision true
      # REASONING PARSER IS REQUIRED. Default is false, and Qwen3.8 is a
      # thinking model: without this it emits its chain-of-thought straight into
      # `content` ("We need to respond to user...") with reasoning_content null,
      # then hits max_tokens before answering. Verified on this box 2026-09-02.
      # start_in_reasoning defaults to `auto`, which is what handles this
      # template — it pre-opens <think> at the end of the prompt, so there is no
      # opening tag in the output for a naive parser to key on.
      --reasoning true
      # *** TENSOR PARALLEL, ENABLED 2026-09-04 — WORTH +48%, AND IT NEEDS BOTH
      # FLAGS BELOW *PLUS* A config.yml CHANGE. ALL THREE OR IT FAILS. ***
      #     layer split (old)  15.04 tok/s   GPU util 51% / 73%  (SEQUENTIAL)
      #     tensor parallel    22.27 tok/s   GPU util 97% / 97%  (n=3, +48%)
      # Closes most of the gap to qwen38-awq (~28 t/s): 86% slower -> 21% slower,
      # at 6bpw instead of INT4.
      #
      # THE OLD "tensor_parallel HANGS" VERDICT WAS A BACKEND CHOICE, NOT A
      # HARDWARE LIMIT. That attempt used `native`, which upstream recommends
      # "for PCIe" — but that assumes P2P-capable PCIe, and
      # torch.cuda.can_device_access_peer is FALSE BOTH WAYS on this pair. A
      # native all-reduce does direct peer copies, so with no P2P it spins
      # forever: 100% util, frozen memory, SIGTERM-immune. A deadlock waiting on
      # data that can never arrive — not a slow path.
      # TabbyAPI DEFAULTS tp_backend to `native` (backends/exllamav3/model.py)
      # even though exllamav3's own default is `nccl`, so nccl must be EXPLICIT.
      --tensor-parallel true
      --tensor-parallel-backend nccl
      #
      # *** REQUIRES `cuda_malloc_async: false` UNDER `memory:` IN
      # ~/ai/tabbyAPI/config.yml — NOT SETTABLE FROM THIS FILE. ***
      # With TabbyAPI's default cudaMallocAsync allocator, TP dies at load:
      #   cuda_host_register(...) -> RuntimeError: cudaHostRegister(0x.., 67108864)
      #   failed: invalid argument
      # That 64 MB pinned host arena is the staging buffer NCCL needs BECAUSE
      # these cards have no P2P, so it is on the critical path. TabbyAPI's own
      # docstring warns the allocator "may cause issues with certain workloads".
      # If this entry ever dies at load with cudaHostRegister, that key was reset.
      # It is a GLOBAL TabbyAPI setting shared by every Tabby entry.
      #
      # --gpu-split is INTENTIONALLY GONE: TP splits automatically and ignores it,
      # so the old 13.5/15 tuning is moot. Vision still works (verified with a
      # real image: 9.5 s, correct description), but the tower still lands WHOLLY
      # on CUDA0 (auto-memory/mmproj-caps-cuda0-tensor-split.md) and peaks at
      # 15682 MiB of 16311 — only 629 MiB spare. A much larger image could OOM;
      # if that happens drop --cache-size before anything else.
      --gpu-split-auto false
      --chunk-size 2048
    proxy: http://127.0.0.1:9182