# infinity

OpenAI-compatible embeddings + reranker server. One container serves both embedding and reranker models simultaneously.

**Server:** ana-ml2
**Port:** 7997 (infinity default)
**GPU:** pinned to GPU 1 by default (configurable via `.env`)

## What it replaces / supersedes

- `qwen3-embedding-0.6B` entry in llama-swap (llama.cpp GGUF → infinity transformer)
- `qwen3-reranker-0.6B` entry in llama-swap

Once infinity is verified stable, retire those two entries from `stacks/llama-swap/config.yaml`.

## Deploy

```bash
# On ana-ml2:
sudo mkdir -p /opt/docker/compose/infinity
sudo chown $USER /opt/docker/compose/infinity
cd /opt/docker/compose/infinity

# Copy compose.yaml + .env.example here (e.g. via scp from this workspace)
# Then:
cp .env.example .env
# edit .env — pick GPU, models, etc.

# Pre-download models into the shared HF cache (optional, speeds first boot)
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)"
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)"

# Dry-parse
docker compose config

# Launch
docker compose up -d
docker compose logs -f
```

## Verify

```bash
# Health
curl -s http://localhost:7997/health

# Embedding
curl -s http://localhost:7997/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .

# Reranker
curl -s http://localhost:7997/rerank \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .

# Listed models
curl -s http://localhost:7997/models | jq .
```

## Scaling knobs

- **`BATCH_SIZE`** in `.env` — bigger = higher throughput, more VRAM. 32 is safe; try 64 or 128 if you have headroom.
- **Model size** — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Pick based on quality-vs-latency tradeoff. On RTX 6000 Ada 46 GB, the 8B pair fits easily (~20 GB VRAM).
- **`ENGINE=optimum`** — uses ONNX runtime, sometimes faster. Requires the model to have ONNX weights available; fall back to `torch` if it errors on startup.