← all models

omni

Active  active in llama-swap.yaml

Grill run history

RunScoretok/stepR5Log
No results-*.log found for this id/alias.

llama-swap.yaml entry

  "omni":
    # THE ONLY MODEL ON THIS BOX THAT CAN HEAR. Qwen3-Omni-30B-A3B AWQ, 25.7 GiB,
    # the largest model served here (15.2 of 16.3 GiB per card). Verified
    # 2026-08-10: espeak-ng speech in -> 8/8 words back verbatim. vLLM reports
    # Supported tasks: ['generate','transcription']. Every other model in the HF
    # cache has ZERO audio tensors. See auto-memory qwen3-omni-audio-works.
    #
    # AUDIO NEEDS librosa/soundfile/av IN THE VENV, and the server must be
    # started AFTER they are installed — vLLM caches a placeholder for a missing
    # module and then 500s with "PlaceholderModule should not be used...".
    # Install with /home/giovanni/vllm-env/bin/pip, NOT the shell's pip (that one
    # is ~/.local/bin/pip and targets SYSTEM python). Never `pip install
    # vllm[audio]`: it can upgrade vLLM and silently revert the MLA fp8 patch.
    #
    # 65536 is its NATIVE max_position_embeddings and fits easily — measured
    # 3.86 GiB KV per GPU (168,464 tokens) against the 1.50 GiB total that 65k
    # needs. 132 tok/s. Grill 42/46 with the LOWEST spread in the table (3), and
    # 4/16 KLayout from memory, 2nd only to fable-fusion.
    #
    # ALWAYS give it the openpyxl reference for spreadsheet work: office is 0/18
    # unassisted with FOUR 16k runaways, and 18/18 with the reference and none.
    cmd: |
      /usr/bin/env
      HF_HOME=/home/giovanni/.cache/huggingface
      CUDA_DEVICE_ORDER=PCI_BUS_ID
      PATH=/home/giovanni/vllm-env/bin:/home/giovanni/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      ${vllm_bin} serve cyankiwi/Qwen3-Omni-30B-A3B-Instruct-AWQ-4bit
      --served-model-name omni
      --tensor-parallel-size 2
      --host 127.0.0.1 --port 9136
      --max-model-len 65536
      --max-num-seqs 8
      --gpu-memory-utilization 0.93
      --kv-cache-dtype fp8
      --override-generation-config '{"temperature":0.6}'
      --enable-auto-tool-choice
      --tool-call-parser hermes
    proxy: http://127.0.0.1:9136