v3.9.0 — new service llamacpp / llamacpp-cuda (Surya OCR 2)
Minor release. Adds a second on-prem VLM serving lane to aigate — a
supervised `llama-server` wrapper that mirrors the vllm-wrap lifecycle
surface (/v1/chat/completions, /api/ps, DELETE /api/ps/{model_id},
POST /unload) but serves GGUF weights and supports vision models via
mmproj. Why this exists: vllm-CPU's vision-VLM support is weak, and
even on GPU the upstream-blessed inference target for Surya is GGUF.
llama.cpp does Qwen3-VL-class document models cleanly on both CPU
and GPU.
First model shipped: Surya OCR 2 (datalab-to/surya-ocr-2-gguf), a
~650M-param document VLM that does OCR / layout / table-recognition
through one model — behaviour switches on the prompt (training-time
contracts, see CHANGELOG for the exact strings).
Pieces:
- llamacpp + llamacpp-cuda wrapper services (~750 LOC Python,
mirrors vllm-wrap 1-for-1). Base images pinned by sha256 digest.
- llamacpp-pull sidecar — reads both models.{cpu,cuda}.json,
huggingface-cli download --local-dir, exits 0. Wrapper depends_on
completion, runs HF_HUB_OFFLINE=1.
- HTTP-URL → data-URL rewriting in the proxy so OpenAI's
`image_url.url: https://...` works transparently (llama-server's
mtmd doesn't fetch URLs natively).
- LiteLLM provider entries + new resource_manager groups
(cpu-llamacpp / cuda-llamacpp) so sibling eviction now covers
llamacpp under contention with ollama / sdcpp / talkies / vllm.
- tests/test_llamacpp.sh — 8 cases (4 per variant) verifying
models-list + captcha OCR (data: URL) + PDF OCR (full page) +
captcha OCR via hybrids3 presigned URL. All 8 pass end-to-end
through LiteLLM → wrapper → llama-server → GGUF VLM.
Two new fixtures committed:
tests/.fixtures/captcha.png (18 KB, contains "AIGATE2026")
tests/.fixtures/doc.pdf (1 KB, two lines of known prose)
See CHANGELOG.md for the full file-by-file breakdown.