← all models

embed

Kept / trial  active in llama-swap.yaml

Grill run history

RunScoretok/stepR5Log
No results-*.log found for this id/alias.

llama-swap.yaml entry

  "embed":
    # raggo.net embeddings (raglib.py, model id "embed"). Qwen3-Embedding-0.6B,
    # 1024-dim. CPU-only (-ngl 0) so it never competes with the resident GPU chat
    # model for VRAM. --embedding + --pooling last (Qwen3-Embedding uses last-token
    # pooling); the asymmetric query-instruction prefix is applied by raglib, not here.
    # Kept resident by the embed-persistent group below (never swaps out).
    cmd: |
      ${llama_bin}
      -hf Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
      --alias embed
      --embedding --pooling last
      -ngl 0 -c 8192
      --host 127.0.0.1 --port 9108 --parallel 2
    proxy: http://127.0.0.1:9108