Kept / trial active in llama-swap.yaml · aliases: xyz-aquila-mini-qwen35-256k
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| No results-*.log found for this id/alias. | ||||
Found in REAL USE, not by the grill: Claude Code (v2.1.251) pointed at aquila
rendered </think> inline and then repeated its whole answer, spent **1m21s
reasoning on ls -ltr**, and told the user the listing was "displayed above"
when no listing had ever been produced. See [[xyz-aquila-mini-trial]] for the
model's grill record — which shows NONE of this.
Plain request, system + user, no tools bound, via llama-swap :8090:
content: '<tool_call>\n</think>\n\n<tool_call>\n</think>\n\nThe output of
ls -ltr is:\n\n``\ntotal 8\ndrwxr-xr-x 2 user group ... dir1'
Two separate defects in one response:
1. </think> and <tool_call> leak into message.content (x2 each).
2. It FABRICATED the ls output — total 8, dir1, a wholly invented
listing — instead of calling a tool. This is why the user kept saying "I do
not see it": there was never a real listing to see.
A second run leaked a raw tool call as prose:
'<tool_call>\n<function=bash>\n<function=shell>\nls -ltr\n</function>\n</tool_call>'
Aquila emits <tool_call><function=NAME>...</function></tool_call>. The box
serves it with the Qwen3.5/3.6-MoE family template
(chat-templates/qwen35moe-nonfirst-system-256k.jinja), whose parser expects a
different shape, so the markers pass through unparsed and the answer boundary is
lost (hence the duplicated answer).
⚠️ CORRECTED 2026-08-30 — THE "ONLY AQUILA LEAKS" CONTROL WAS WRONG (n=1).
The first draft of this file claimed tommy was clean and therefore the defect was
aquila-specific. That rested on ~3 lucky draws. **Re-tested at n=5 on the SAME
no-tools text turn, tommy LEAKS 2/5** — <think>/</think> in run 1,
<thinking>/</thinking> in run 2 — and FABRICATES tool execution 1/5
([tool]bash: ls -ltr followed by an invented total 12 … dir1 listing).
Two other runs emitted pseudo-tool prose (<bash>ls -ltr</bash>, a `bash block)
without calling anything.
**So this is NOT an aquila model defect — it is a CONFIG GAP across the whole
qwen3_5_moe family on this box.** None of tommy, tommy2, ornith-15-35b
(or any other entry on qwen35moe-nonfirst-system-256k.jinja) sets
--reasoning-format. aquila simply leaks more often and adds a tool-format mess
on top, which is why it surfaced first in real use.
The lesson repeats [[single-run-scores-are-unreliable]]: a 3-sample "clean"
control is not a control. Leak rates need n>=5 before any per-model claim.
With tools BOUND the tool path is clean (finish=tool_calls, empty content).
The leak lives on plain-text answer turns — i.e. exactly the summarising
message an agent emits after a tool returns, which is most of real agent use.
With tools bound it also tends to re-call the tool instead of answering, matching
the user's "let me re-run it" loop.
bench/grill-aquila-20260830-1103/envelope-report.txt reads **"125 turns across
14 suites … CONTROL-TOKEN LEAK 0 (0.0%) … VERDICT: CLEAN"**, and
bench/grill_envelope.py _SOFT does include r"</?think>" — the soft scan
also found nothing. The grill genuinely never saw it, because its rounds either
bind tools (clean path) or never exercise the plain-text summarising turn a real
agent hits constantly.
This is a direct re-run of [[grill-does-not-validate-real-use]] (gemma leaked
<|channel> every turn while 24 grill runs saw NOTHING). **An envelope CLEAN
verdict does not license wiring a model to a client. Send one no-tools text turn
through the actual client before trusting it.**
Third time in one session a clean grill number failed on contact with reality —
see also the realcase correction in [[thomson-1.0-small-trial]] and the R5
n=1 draw in [[round5-is-a-sample-not-a-measurement]].
(TESTED 2026-08-30, now in the entry)Added to the aquila llama-swap entry and verified on the real path:
| turn | without flag | WITH flag |
|---|---|---|
| no tools, text answer | </think> x2 + <tool_call> x2, fabricated ls output | </think> GONE (reasoning -> reasoning_content), <tool_call> remains |
| tools bound, text answer | — | CLEAN (finish=stop, "Four.") |
| tools bound, post-tool summary (the failing case) | leaked | CLEAN, summarises the REAL tool output, no fabrication |
</think> leaking was the user-visible defect and the flag eliminates it: the
think block is now extracted into reasoning_content where it belongs.
RESIDUAL, accepted: with no tools bound aquila still emits
<tool_call><function=bash>... as prose (3/3 runs, one malformed with an
unbalanced </function>). That is the format mismatch and the flag does not
touch it. Claude Code always binds tools so the path is not hit — but **do not
use aquila tool-less for agentic-style prompts**. A template matching aquila's
<function=> format would be the real fix if that ever matters.
) predates this finding; that commit's"envelope CLEAN" line must be read with this file.
(not <think>) in content on the same request. It is not in grill_envelope.py's _HIGH/_SOFT patterns so nothing
flags it, and tommy's entry has no --reasoning-format. Unexamined; may be the
same class of problem hiding behind a different tag.
Earlier sections called --reasoning-format deepseek a fix on n=1-3 samples.
Re-measured with the new suite, 8 reps x 3 scenarios, flag ACTIVE:
| scenario | leaks |
|---|---|
| A no tools bound | 6/8 |
| B tools bound, text answer | 0/8 |
| C tools bound, post-tool summary | 0/8 |
| overall | 6/24 = 25% — LEAKY |
The flag DID fix the tools-bound shapes (16/16 clean). The tool-less path is
still badly broken, and one rep is a RUNAWAY, not a cosmetic leak:
finish=length content=16177 chars 33.4s
</think> x734 <tool_call> x732
It repeated the tag pair 734 times until the token cap. Another rep emitted a
full pseudo tool call as prose:
<tool_call><function=shell><function_id>shell-1</function_id><parameter=command.
Do NOT use aquila as a Claude Code / agent driver. The user's real transcript
leaked even though Claude Code binds tools, so scenarios B/C do not fully model
that client — the 25% rate and the 734x runaway are disqualifying regardless.
Speed for the record: aquila Q5_K_M ~80.7 tok/s mean (median 84, 58-90 over
45 grill samples) vs tommy Q4_K_M ~93.4 — about 14% slower, the expected
bandwidth cost of the extra bits ([[decode-bandwidth-ceiling]]).
Same no-tools text turn, same template, -c 65536, temp 0.2:
| model | flag | tag leaks | fabricated execution |
|---|---|---|---|
| tommy | none (production) | 2/5 | 1/5 |
| tommy | --reasoning-format deepseek | 3/5 | 1/5 |
| aquila | none (production) | 2/2 observed | yes |
| aquila | --reasoning-format deepseek | </think> GONE (n=3) | no |
WHY THE FLAG FIXES AQUILA BUT NOT TOMMY — the reusable bit:
/ </think> reasoning-format parses it -> extracted to reasoning_content, leak gone.
/ </thinking> reasoning-format recognises -> passes into content whatever the flag.
reasoning_content IS populated (126-129 chars) at the same time, so the
<thinking> block leaks *in addition to* correctly-extracted reasoning.
2/5 -> 3/5 on tommy is noise at n=5; the honest reading is no improvement,
not "made it worse". tommy has no fix yet. Options untested: a template that
emits/strips <thinking>, or a sanitizer rule at the 8091 proxy.
BOTH models fabricate tool execution ~1/5 on the no-tools turn — tommy wrote
[tool]bash: ls -ltr + an invented total 12 … dir1 listing; aquila invented
total 8 … dir1. That is model behaviour, NOT fixable by a reasoning-format flag,
and it is the actual cause of the user's "I do not see it".
⚠️ THE AQUILA FIX IS UNDER-VERIFIED. It was validated at n=1 on the
tools-bound paths and n=3 on the no-tools turn. Tommy's 2/5 base rate proves 3
clean draws are not evidence ([[single-run-scores-are-unreliable]]).
Re-verify aquila at n>=5 on the tools-bound turn before trusting the entry.
Do not SIGTERM a llama-swap child to free VRAM while someone is using the box.
Doing so here left the process hung mid-shutdown still holding all 29 GB, so
llama-swap could not start a replacement and every request 502'd (including the
user's live claude-cli session) for ~5 minutes. kill -9 <pid> cleared it and
llama-swap cold-loaded cleanly. Test on a spare port instead, or stop the service
deliberately first.
pkill -f self-match, third variant ([[pgrep-f-self-match-deadlock]]): the
bracket trick ([X]YZAILab) protects the PATTERN, but the same command line also
carried -m /mnt/models/gguf/XYZAILab_..._Q5_K_M.gguf` as a PAYLOAD, which the
regex matched — killing the invoking shell (exit 144) so the nohup never ran.
Match on something that cannot appear in your own argv, or kill by PID.
"aquila":
aliases: [xyz-aquila-mini-qwen35-256k]
# XYZAILab/XYZ-Aquila-mini — TRIAL 2026-08-30. Candidate REPLACEMENT for
# `tommy` in the deep-research / grounded-RAG slot, found by an HF sweep of
# the Qwen3.6-35B-A3B derivative space (see [[thomson-1.0-small-trial]] for
# what it has to beat). "An open-weight thinking model for Deep Search",
# apache-2.0 — 440 likes, the most-liked 35B-A3B derivative on the Hub, and
# bartowski's GGUF has 478k downloads.
#
# WHY THIS IS A DROP-IN, NOT A NEW SHAPE: verified against Thomson before
# download — SAME arch (Qwen3_5MoeForConditionalGeneration, 40 layers,
# 256 experts 8/tok, 2 kv_heads, head_dim 256, full_attention_interval 4 =
# 10 full-attn layers, 262144 ctx, vision depth 27) and the chat template is
# BYTE-IDENTICAL to Thomson's (md5 52b6d51ae5b2, 7764 chars) — so the box's
# existing chat-templates/qwen35moe-nonfirst-system-256k.jinja patch applies
# verbatim ([[jinja-system-guard-tool-parser]]; --chat-template-file is
# LOAD-BEARING).
#
# QUANT = Q5_K_M (23.30 GiB), NOT Q4_K_M — the largest rung MEASURED to fit
# at FULL 262144 with vision, 2026-08-30. Do not "correct" it down to match
# tommy: tommy's own comment predicted Q5 would OOM on the ViT buffer, and
# that prediction is WRONG for this box. Measured on the real hardware
# (32,691 MiB total across both cards), all four numbers from live probes:
# weights 23,859 + mmproj 860 + q8_0 KV @262k 2,700 + compute 1,973
# + ViT 472 (3072x3072 image) = 29,864 MiB peak, 2,827 MiB SPARE.
# Verified NOT just to load but to SURVIVE: a 3072px image answered with
# finish=stop, and a 221,978-token prompt processed with NO peak growth —
# the compute buffer is reserved at load time from -c/-b and does not grow
# with real prompt length, so the [[coder-agentic-quant-ladder]] "loads on
# arithmetic, OOMs on a real request" failure mode does NOT apply here.
# Q5_K_L (23.59, +297 MiB) is the next rung and should fit on these numbers
# but is UNVERIFIED. Q6_K (27.99, +4,802 MiB) does NOT fit at any useful
# context — buying it back needs -c 32768 or dropping vision, neither of
# which is worth it for a Deep Search model.
#
# --tensor-split 45,55 IS LOAD-BEARING and was tuned by measurement, not
# inherited. tommy's 42,58 strands ~1 GiB on the wrong card. Note the
# NON-WEIGHT allocations (mmproj + KV + compute + ViT) land preferentially
# on CUDA0 = the 4060 Ti ([[gpu-device-ordering]]), so moving TOWARD 50,50
# makes the 4060 Ti the binding card, not the 5060 Ti: at 47,53 the sibling
# bigbang trial (REJECTED+deleted 2026-08-30) left only 543 MiB on the 4060 Ti and HARD-CRASHED on the
# first image (GGML_ASSERT in ggml_backend_tensor_alloc). 45,55 gives
# ~1,112 / ~2,187 MiB free and survives.
#
# WHY IT MIGHT BEAT tommy: Thomson only TIES its own base ([[model-triage-checklist]]
# check 8 — a derivative must justify itself), and its one distinct win is
# RAG fidelity (pj.sg 22/28, 0 hedged / 0 invented). Aquila-mini is post-trained
# for exactly that axis rather than for legal-domain alignment, and it is
# apache-2.0 vs Thomson's polyform-strict-1.0.0 (which bars commercial use —
# a real constraint if pj.sg ever ships this).
#
# STORAGE /mnt/models (cold tier, 195 GB free) NOT ssk500 — ssk500 is at 90%
# / 46 GB free and [[storage-tiers]] says the two are within ~3% on
# throughput, so there is no speed argument for spending the last of ssk500
# on a trial. Promote to ssk500 only if it beats tommy.
#
# REASONING MODEL (family template defaults reasoning_effort xhigh) — CoT to
# reasoning_content, answer to content. Budget max_tokens generously: at a
# short cap it hits the limit mid-think and returns EMPTY content with no
# error (same trap as tommy / pocket-35b / ornith). UNGRILLED at write time.
cmd: |
/usr/bin/env
CUDA_DEVICE_ORDER=PCI_BUS_ID
${llama_bin}
-m /mnt/models/gguf/XYZAILab_XYZ-Aquila-mini-Q5_K_M.gguf
--mmproj /mnt/models/gguf/mmproj-XYZAILab_XYZ-Aquila-mini-bf16.gguf
--alias aquila
--jinja --chat-template-file chat-templates/qwen35moe-nonfirst-system-256k.jinja
--reasoning-format deepseek
-ngl 99 -c 262144 -fa on
--tensor-split 45,55
-b 2048 -ub 512
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0
--cache-type-k q8_0 --cache-type-v q8_0
--host 127.0.0.1 --port 9175 --parallel 1
proxy: http://127.0.0.1:9175