vh
e376d0aec9
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
infinity
OpenAI-compatible embeddings + reranker server. One container serves both embedding and reranker models simultaneously.
Server: ana-ml2
Port: 7997 (infinity default)
GPU: pinned to GPU 1 by default (configurable via .env)
What it replaces / supersedes
qwen3-embedding-0.6Bentry in llama-swap (llama.cpp GGUF → infinity transformer)qwen3-reranker-0.6Bentry in llama-swap
Once infinity is verified stable, retire those two entries from stacks/llama-swap/config.yaml.
Deploy
# On ana-ml2:
sudo mkdir -p /opt/docker/compose/infinity
sudo chown $USER /opt/docker/compose/infinity
cd /opt/docker/compose/infinity
# Copy compose.yaml + .env.example here (e.g. via scp from this workspace)
# Then:
cp .env.example .env
# edit .env — pick GPU, models, etc.
# Pre-download models into the shared HF cache (optional, speeds first boot)
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)"
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)"
# Dry-parse
docker compose config
# Launch
docker compose up -d
docker compose logs -f
Verify
# Health
curl -s http://localhost:7997/health
# Embedding
curl -s http://localhost:7997/embeddings \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .
# Reranker
curl -s http://localhost:7997/rerank \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .
# Listed models
curl -s http://localhost:7997/models | jq .
Scaling knobs
BATCH_SIZEin.env— bigger = higher throughput, more VRAM. 32 is safe; try 64 or 128 if you have headroom.- Model size — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Pick based on quality-vs-latency tradeoff. On RTX 6000 Ada 46 GB, the 8B pair fits easily (~20 GB VRAM).
ENGINE=optimum— uses ONNX runtime, sometimes faster. Requires the model to have ONNX weights available; fall back totorchif it errors on startup.