Active active in llama-swap.yaml · aliases: glm-ocr-0.9b
| Run | Score | tok/step | R5 | Log |
|---|---|---|---|---|
| No results-*.log found for this id/alias. | ||||
Chosen 2026-08-22 for the alzheimer_dataset corpus (7 scanned PDFs, 241 pages
with no text layer). VERIFIED WORKING, not just selected.
Why GLM-OCR (zai-org/GLM-OCR, GGUF at ggml-org/GLM-OCR-GGUF):
94.62 on OmniDocBench V1.5 = #1 overall, only 0.9B params, MIT. Beat
baidu/Unlimited-OCR (3.34B), datalab-to/chandra-ocr-2 (5.3B),
deepseek-ai/DeepSeek-OCR-2 (3.39B), baidu/Qianfan-OCR (4.74B).
VRAM is NOT a selection criterion for OCR — every serious OCR model is
0.7-5.3B and fits 16 GB trivially. Pick on accuracy + servability instead.
Files: /mnt/ssk500/gguf/GLM-OCR-Q8_0.gguf (907 MB) +
mmproj-GLM-OCR-Q8_0.gguf (462 MB). GGUF arch is glm4, projector glm4v
— both present in the box's llama.cpp b10369, so it loads in <1 s with no
new dependency. Output is clean markdown (italics, LaTeX preserved).
Do NOT reach for vLLM here: GLM-OCR needs vLLM *nightly* + transformers
from git; the box serves 0.26.0.
--device CUDA1 DOES NOT CONFINE A VISION MODEL. With it, one process
held memory on BOTH GPUs (2990 + 1636 MiB) and each card sat at ~30% util —
the vision tower straddled the pair with cross-device sync overhead. There
is no --mmproj-device flag. CUDA_DEVICE_ORDER=PCI_BUS_ID
CUDA_VISIBLE_DEVICES=<n> confines it properly, and the confined instance
then hits 86% util — faster AND on one card.
In-process concurrency buys NOTHING; separate processes buy 1.81x.
Measured on disjoint cold page sets: conc 1/2/4/8 = 0.78 / 0.73 / 0.76 /
0.67 pages/s — flat, slightly declining, because mtmd serialises image
encoding. Two GPU-confined instances (ports 9199 + 9198), pages fanned
round-robin: 0.74 -> 1.33 pages/s. OCR is ~95% prefill (prompt eval
1952 ms vs decode 97 ms), so it is COMPUTE-bound — the opposite of
[[decode-bandwidth-ceiling]], which is about decode and does not apply.
MEASUREMENT TRAP that produced a wrong answer first: llama.cpp caches
image embeddings. Re-running the same pages across a concurrency sweep gave
prompt eval = 5.87 ms / 1 tokens and a fake 2.1x scaling curve. Any
throughput sweep on a vision model MUST use disjoint, never-seen images per
level, and a fresh process to clear the cache.
"glm-ocr":
aliases: [glm-ocr-0.9b]
# CARD ASSIGNMENT (verified empirically 2026-08-22, do NOT swap blindly):
# with CUDA_DEVICE_ORDER=PCI_BUS_ID the indices match nvidia-smi, which is
# the OPPOSITE of llama.cpp's bare CUDA ordering in [[gpu-device-ordering]]:
# CUDA_VISIBLE_DEVICES=0 -> RTX 5060 Ti (36 SMs, cc12.0 Blackwell) STRONGER
# CUDA_VISIBLE_DEVICES=1 -> RTX 4060 Ti (34 SMs, cc8.9 Ada) weaker
# its coserve partner (now mellum2) takes the 5060 Ti: decode is
# memory-bandwidth-bound and Blackwell
# has far more of it. glm-ocr takes the 4060 Ti: it is 3 GB and ~95% prefill,
# so it is the right tenant for the weaker card.
# DOCUMENT OCR. GLM-OCR (zai-org), 0.9B decoder + CogViT encoder, MIT.
# 94.62 on OmniDocBench V1.5 = #1 OVERALL, beating models 4-6x its size
# (Unlimited-OCR 3.34B, chandra-ocr-2 5.3B, DeepSeek-OCR-2 3.39B). Added
# 2026-08-22 after it recovered 7 scanned PDFs / 241 pages in the
# alzheimer_docs corpus that had no text layer at all. Clean markdown out,
# italics and LaTeX preserved. See auto-memory/glm-ocr-verdict.md.
#
# VRAM IS NOT THE CONSTRAINT FOR OCR — every serious OCR model is 0.7-5.3B.
# Pick on accuracy and servability. This one loads in UNDER A SECOND
# (1.37 GB of weights), so unlike the big vLLM entries it is nearly free to
# swap in and straight back out.
#
# NOT ON vLLM ON PURPOSE: GLM-OCR needs vLLM *nightly* + transformers from
# git. The box serves 0.26.0, and 0.27.1 was already rolled back once
# ([[vllm-0271-upgrade]]). The GGUF arch is `glm4` and the projector
# `glm4v`, both present in llama.cpp b10369 — zero new dependencies.
#
# CUDA_VISIBLE_DEVICES IS LOAD-BEARING, and `--device CUDA1` IS NOT A
# SUBSTITUTE. Measured 2026-08-22: with `--device` the process still held
# memory on BOTH cards (2990 + 1636 MiB) and each sat at ~30% util, because
# the vision tower straddled the pair and paid cross-device sync. There is
# no --mmproj-device flag. Confined with CUDA_VISIBLE_DEVICES it reaches
# 86% util on one card and is FASTER. CUDA_DEVICE_ORDER=PCI_BUS_ID makes
# the index match nvidia-smi (0 = 5060 Ti), NOT llama.cpp's CUDA ordering.
#
# THROUGHPUT COMES FROM PROCESSES, NOT --parallel. mtmd serialises image
# encoding, so concurrency inside one server is flat: 0.78 / 0.73 / 0.76 /
# 0.67 pages/s at 1 / 2 / 4 / 8. Two GPU-confined instances instead give
# 0.74 -> 1.33 pages/s (1.81x). For a big batch, run a second copy by hand
# on the other card and fan pages across both:
# CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 \
# llama.cpp/build/bin/llama-server -m <gguf> --mmproj <mmproj> \
# --port 9149 -ngl 99 -c 16384 -fa on --parallel 2
# Deliberately NOT a second llama-swap entry: it would clutter every agent's
# model list for a batch-only mode, and llama-swap is exclusive by default
# so the pair would evict each other anyway.
cmd: |
/usr/bin/env
CUDA_DEVICE_ORDER=PCI_BUS_ID
CUDA_VISIBLE_DEVICES=1
${llama_bin}
-m /home/giovanni/models-nvme/GLM-OCR-Q8_0.gguf
--mmproj /home/giovanni/models-nvme/mmproj-GLM-OCR-Q8_0.gguf
--alias glm-ocr
-ngl 99 -c 32768 -fa on
--temp 0.0
--host 127.0.0.1 --port 9148 --parallel 4
proxy: http://127.0.0.1:9148