vh
e376d0aec9
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
66 lines
2.1 KiB
Markdown
66 lines
2.1 KiB
Markdown
# infinity
|
|
|
|
OpenAI-compatible embeddings + reranker server. One container serves both embedding and reranker models simultaneously.
|
|
|
|
**Server:** ana-ml2
|
|
**Port:** 7997 (infinity default)
|
|
**GPU:** pinned to GPU 1 by default (configurable via `.env`)
|
|
|
|
## What it replaces / supersedes
|
|
|
|
- `qwen3-embedding-0.6B` entry in llama-swap (llama.cpp GGUF → infinity transformer)
|
|
- `qwen3-reranker-0.6B` entry in llama-swap
|
|
|
|
Once infinity is verified stable, retire those two entries from `stacks/llama-swap/config.yaml`.
|
|
|
|
## Deploy
|
|
|
|
```bash
|
|
# On ana-ml2:
|
|
sudo mkdir -p /opt/docker/compose/infinity
|
|
sudo chown $USER /opt/docker/compose/infinity
|
|
cd /opt/docker/compose/infinity
|
|
|
|
# Copy compose.yaml + .env.example here (e.g. via scp from this workspace)
|
|
# Then:
|
|
cp .env.example .env
|
|
# edit .env — pick GPU, models, etc.
|
|
|
|
# Pre-download models into the shared HF cache (optional, speeds first boot)
|
|
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)"
|
|
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)"
|
|
|
|
# Dry-parse
|
|
docker compose config
|
|
|
|
# Launch
|
|
docker compose up -d
|
|
docker compose logs -f
|
|
```
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
# Health
|
|
curl -s http://localhost:7997/health
|
|
|
|
# Embedding
|
|
curl -s http://localhost:7997/embeddings \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .
|
|
|
|
# Reranker
|
|
curl -s http://localhost:7997/rerank \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .
|
|
|
|
# Listed models
|
|
curl -s http://localhost:7997/models | jq .
|
|
```
|
|
|
|
## Scaling knobs
|
|
|
|
- **`BATCH_SIZE`** in `.env` — bigger = higher throughput, more VRAM. 32 is safe; try 64 or 128 if you have headroom.
|
|
- **Model size** — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Pick based on quality-vs-latency tradeoff. On RTX 6000 Ada 46 GB, the 8B pair fits easily (~20 GB VRAM).
|
|
- **`ENGINE=optimum`** — uses ONNX runtime, sometimes faster. Requires the model to have ONNX weights available; fall back to `torch` if it errors on startup.
|