vh
e376d0aec9
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
41 lines
1.4 KiB
Bash
41 lines
1.4 KiB
Bash
# vllm-qwen3 stack tunables. Copy this to `.env` on the server before deploying.
|
|
#
|
|
# cp .env.example .env
|
|
# # edit .env with real values
|
|
# docker compose up -d
|
|
|
|
# Image version — pin for reproducibility (`latest` for edge)
|
|
VLLM_VERSION=latest
|
|
|
|
# Host ports (container always listens on 8000 internally)
|
|
EMBED_PORT=8001
|
|
RERANK_PORT=8002
|
|
|
|
# GPU assignment — both services share this GPU
|
|
# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work)
|
|
GPU_ID=1
|
|
|
|
# Models — reference by full repo name in API requests
|
|
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
|
RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
|
|
|
|
# GPU memory split — fractions are of TOTAL GPU memory, not free memory.
|
|
# When two vLLM services share a GPU, each profiler needs its own slice to
|
|
# fit both the model and KV cache, so small values cause the second-to-start
|
|
# service to OOM on KV cache allocation. 0.40 + 0.40 leaves ~20% headroom
|
|
# and is comfortably above the minimum for two 0.6B Qwen3 models at 8k ctx.
|
|
EMBED_GPU_MEM_UTIL=0.40
|
|
RERANK_GPU_MEM_UTIL=0.40
|
|
|
|
# Context length caps — lower these if VRAM is tight.
|
|
# Qwen3-Embedding supports up to 32k; reranker up to 32k.
|
|
EMBED_MAX_MODEL_LEN=8192
|
|
RERANK_MAX_MODEL_LEN=8192
|
|
|
|
# Optional API key — leave blank for no auth (fine on the internal network).
|
|
# If set, both services require `Authorization: Bearer <key>`.
|
|
API_KEY=
|
|
|
|
# HuggingFace token — only needed for gated models
|
|
HF_TOKEN=
|