docs: delete stacks/infinity (retired, replaced by vllm)
Stack was retired and replaced by the vllm stack (originally vllm-qwen3, renamed 2026-05-13). Its README still framed it as a current solution while ana-ml2's README + vllm's README both documented the retirement. stacks/vllm/README.md "Migrating off Infinity" step 3 explicitly said "Delete stacks/infinity/ from this workspace" — actioning that now. No backwards-compat shims (PRACTICES §4): contract of a deleted system has no historical value the next contributor needs; the replacement path is documented in stacks/vllm/README.md. Surfaced by /tend-docs audit 2026-05-14.
This commit is contained in:
@@ -1,30 +0,0 @@
|
||||
# Infinity stack tunables. Copy this to `.env` on the server before deploying.
|
||||
#
|
||||
# cp .env.example .env
|
||||
# # edit .env with real values
|
||||
# docker compose up -d
|
||||
|
||||
# Image version — pin for reproducibility (`latest` for edge)
|
||||
INFINITY_VERSION=latest
|
||||
|
||||
# Port exposed on host
|
||||
INFINITY_PORT=7997
|
||||
|
||||
# GPU assignment (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work)
|
||||
GPU_ID=1
|
||||
|
||||
# Models — both served simultaneously; reference by the full repo name in requests
|
||||
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
||||
RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
|
||||
|
||||
# Inference engine: torch (widest support) or optimum (ONNX, sometimes faster)
|
||||
ENGINE=torch
|
||||
|
||||
# Batch size — 32 is a safe default; bump for throughput if VRAM allows
|
||||
BATCH_SIZE=32
|
||||
|
||||
# Optional API key — leave blank for no auth (fine on the internal network)
|
||||
API_KEY=
|
||||
|
||||
# HuggingFace token — only needed for gated models
|
||||
HF_TOKEN=
|
||||
@@ -1,65 +0,0 @@
|
||||
# infinity
|
||||
|
||||
OpenAI-compatible embeddings + reranker server. One container serves both embedding and reranker models simultaneously.
|
||||
|
||||
**Server:** ana-ml2
|
||||
**Port:** 7997 (infinity default)
|
||||
**GPU:** pinned to GPU 1 by default (configurable via `.env`)
|
||||
|
||||
## What it replaces / supersedes
|
||||
|
||||
- `qwen3-embedding-0.6B` entry in llama-swap (llama.cpp GGUF → infinity transformer)
|
||||
- `qwen3-reranker-0.6B` entry in llama-swap
|
||||
|
||||
Once infinity is verified stable, retire those two entries from `stacks/llama-swap/config.yaml`.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
# On ana-ml2:
|
||||
sudo mkdir -p /opt/docker/compose/infinity
|
||||
sudo chown $USER /opt/docker/compose/infinity
|
||||
cd /opt/docker/compose/infinity
|
||||
|
||||
# Copy compose.yaml + .env.example here (e.g. via scp from this workspace)
|
||||
# Then:
|
||||
cp .env.example .env
|
||||
# edit .env — pick GPU, models, etc.
|
||||
|
||||
# Pre-download models into the shared HF cache (optional, speeds first boot)
|
||||
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)"
|
||||
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)"
|
||||
|
||||
# Dry-parse
|
||||
docker compose config
|
||||
|
||||
# Launch
|
||||
docker compose up -d
|
||||
docker compose logs -f
|
||||
```
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
# Health
|
||||
curl -s http://localhost:7997/health
|
||||
|
||||
# Embedding
|
||||
curl -s http://localhost:7997/embeddings \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .
|
||||
|
||||
# Reranker
|
||||
curl -s http://localhost:7997/rerank \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .
|
||||
|
||||
# Listed models
|
||||
curl -s http://localhost:7997/models | jq .
|
||||
```
|
||||
|
||||
## Scaling knobs
|
||||
|
||||
- **`BATCH_SIZE`** in `.env` — bigger = higher throughput, more VRAM. 32 is safe; try 64 or 128 if you have headroom.
|
||||
- **Model size** — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Pick based on quality-vs-latency tradeoff. On RTX 6000 Ada 46 GB, the 8B pair fits easily (~20 GB VRAM).
|
||||
- **`ENGINE=optimum`** — uses ONNX runtime, sometimes faster. Requires the model to have ONNX weights available; fall back to `torch` if it errors on startup.
|
||||
@@ -1,63 +0,0 @@
|
||||
# Infinity — OpenAI-compatible embeddings + reranker server.
|
||||
#
|
||||
# Serves embedding and reranker models simultaneously from one container
|
||||
# on port 7997 (HTTP). Consumers: AIPA agents (search/retrieval), LibreChat
|
||||
# RAG, anything that needs vector embeddings.
|
||||
#
|
||||
# All tunables live in .env — edit that, not this file.
|
||||
#
|
||||
# Pre-download models to avoid first-run delay:
|
||||
# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Embedding-0.6B
|
||||
# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Reranker-0.6B
|
||||
|
||||
services:
|
||||
infinity:
|
||||
image: michaelf34/infinity:${INFINITY_VERSION}
|
||||
container_name: infinity
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${INFINITY_PORT}:7997"
|
||||
volumes:
|
||||
- /tank/aimodels/huggingface:/hfcache
|
||||
environment:
|
||||
- HF_HOME=/hfcache
|
||||
- HF_HUB_CACHE=/hfcache/hub
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
command: >
|
||||
v2
|
||||
--model-id ${EMBED_MODEL}
|
||||
--model-id ${RERANK_MODEL}
|
||||
--engine ${ENGINE}
|
||||
--device cuda
|
||||
--batch-size ${BATCH_SIZE}
|
||||
--host 0.0.0.0
|
||||
--port 7997
|
||||
--api-key ${API_KEY:-}
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${GPU_ID}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:7997/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 120s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI Systems
|
||||
- homepage.name=Infinity
|
||||
- homepage.icon=mdi-vector-arrange-below
|
||||
- homepage.description=Embeddings + Reranker API (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:7997/docs
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
Reference in New Issue
Block a user