vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

Browse Source 
Captures the full workspace state built up to this point:

  - CLAUDE.md + README.md describing conventions and the four-host fleet
    (ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
  - Per-host notes under servers/<host>/ with ssh-target fallback files
    and latest system-details snapshots (two in-compose credential leaks
    scrubbed; the upstream compose files still need to move those to .env).
  - scripts/: server_inspect.sh (read-only remote diagnostic),
    refresh-server-info.sh (dir-driven discovery + snapshot capture with
    validation warnings), add-host.sh, sync-stacks.sh (pull
    compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
  - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
    rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
    infinity reference. All use the .env-driven + traefik-net + homepage
    label pattern.
  - configs/restic/ana-docker/: first resticprofile config + pre-backup
    hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
    for the other three hosts to come.
  - docs/pfi/: general infrastructure reference carried over.
  - .gitignore excludes .env, stacks-mirror/, and assorted secret/state
    filenames to prevent re-leaks on later commits.
This commit is contained in:
vh
2026-04-20 14:29:48 -07:00
commit e376d0aec9
55 changed files with 9101 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/llama-swap/.env.example
+18
View File
@@ -0,0 +1,18 @@
# llama-swap stack tunables. Copy to `.env` on ana-docker before deploying.
#
# cp .env.example .env
# # edit if needed
# docker compose up -d
# Image tag. `cuda` is the CUDA-enabled build; pin to a specific release
# (e.g. `cuda-v0.0.6`) for reproducibility once upstream tags stabilize.
LLAMA_SWAP_VERSION=cuda
# Host port for the OpenAI-compatible API (container listens on 8080)
LLAMA_SWAP_PORT=9292
# Host paths
# --- Legacy GGUF models referenced by config.yaml as `-m /models/<file>`
MODELS_DIR=/tank/aimodels/llm
# --- Shared HuggingFace cache used by `-hf` model entries
HF_CACHE_DIR=/tank/aimodels/huggingface
stacks/llama-swap/README.md
+52
View File
@@ -0,0 +1,52 @@
# llama-swap
GGUF model server with on-demand model swapping. Served via llama.cpp's `llama-server` under the llama-swap proxy.
**Server:** ana-ml2
**Port:** 9292 (configurable via `.env`)
**GPU:** both (unpinned — `runtime: nvidia` grants access to all devices; per-model GPU selection happens inside `config.yaml`)
## Files
- **`compose.yaml`** — canonical compose. Deployed to `/opt/docker/compose/llama-swap/compose.yaml` on ana-ml2.
- **`.env.example`** — template for the per-host `.env`. Copy to `.env` on the server and tweak.
- **`config.yaml`** — model definitions and groups. Deployed to `/opt/docker/conf/llama-swap/config.yaml` on the server.
Homepage labels are in the compose file under the `AI Systems` group, matching the convention used by `vllm-qwen3` and `infinity`.
## Deploy a fresh install
```bash
scripts/deploy-stack.sh ana-ml2 llama-swap
ssh ana-ml2 '
cd /opt/docker/compose/llama-swap && \
cp -n .env.example .env && \
docker compose config && \
docker compose up -d && \
docker compose logs --tail=30
'
```
## Model reference conventions
- **Modern entries:** use `-hf <user>/<repo>[:<quant>]` — reads from the shared HF cache, nothing to pre-stage outside `hf download`
- **Legacy entries:** use `--model /models/<dir>/<file>.gguf` — reads GGUFs from `/tank/aimodels/llm/` (pre-HF-cache era, gradually being migrated)
New models should prefer the `-hf` pattern.
## Deploy updates to config only
```bash
# After editing config.yaml here:
scp config.yaml ana-ml2:/opt/docker/conf/llama-swap/config.yaml
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose restart'
```
## Deploy updates to compose only
```bash
# After editing compose.yaml or .env.example here:
scripts/deploy-stack.sh ana-ml2 llama-swap
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose up -d'
```
stacks/llama-swap/compose.yaml
+50
View File
@@ -0,0 +1,50 @@
# llama-swap — GGUF model server with on-demand model swapping.
#
# Proxies OpenAI-compatible API requests to llama.cpp server instances
# and swaps which model is loaded into VRAM per request. Runs on
# ana-ml2 using both GPUs dynamically (no explicit device pinning —
# llama-swap picks per-model-definition).
#
# Model definitions live in /opt/docker/conf/llama-swap/config.yaml on
# the server. Canonical copy of that config is config.yaml in this
# workspace; deploy with scp + `docker compose restart` or the script
# at the bottom of README.md.
#
# All tunables live in .env — edit that, not this file.
services:
llama-swap:
image: ghcr.io/mostlygeek/llama-swap:${LLAMA_SWAP_VERSION}
container_name: llama-swap
restart: unless-stopped
stdin_open: true
tty: true
runtime: nvidia
ports:
- "${LLAMA_SWAP_PORT}:8080"
volumes:
- /opt/docker/conf/llama-swap/config.yaml:/app/config.yaml
- ${MODELS_DIR}:/models
- ${HF_CACHE_DIR}:/hfcache
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
healthcheck:
test: ["CMD-SHELL", "curl -fsS http://localhost:8080/ >/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
networks:
- tnet
labels:
- homepage.group=AI Systems
- homepage.name=llama-swap
- homepage.icon=mdi-swap-horizontal
- homepage.description=GGUF model swapper (llama.cpp; ana-ml2)
- homepage.href=http://10.250.50.54:${LLAMA_SWAP_PORT}
networks:
tnet:
name: traefik-net
external: true
stacks/llama-swap/config.yaml
+463
View File
@@ -0,0 +1,463 @@
# ============================================================================
# llama-swap configuration for PFI-ANA
# Optimized and synchronized with /models disk inventory
# Last updated: 2026-04-10
#
# KB Sources:
# - reference/nemotron-3-super-running-parameters.md
# - reference/nemotron-3-nano-running-parameters.md
# - reference/qwen3.5-running-parameters.md
# - reference/qwen3-coder-next-running-parameters.md
# - reference/gemma-4-running-parameters.md
# - reference/qwen3-embedding-running-parameters.md
# - reference/qwen3-reranker-running-parameters.md
#
# Changelog:
# 2025-07-22: Removed jina-reranker-v3 (unused, out of rotation).
# 2026-04-10: Fixed Qwen3-Embedding pooling (mean→last; causal LM uses last-token
# pooling). Fixed ctx-size 4096→8192 for embedding+reranker. Fixed
# reranker: removed --embeddings flag (not an embedding model).
# 2026-04-17: Added Qwen3.6-35B-A3B Abliterated Heretic Q8_0 via -hf syntax.
# Requires HF_HOME=/hfcache in compose (see docker-compose.yml).
# New convention: use -hf repo[:quant] instead of --model /path.
# ============================================================================
# Default 1200 seconds (20 min) to wait for model to be available to load.
healthCheckTimeout: 1200
# logLevel: sets the logging value
# - optional, default: info
# - Valid log levels: debug, info, warn, error
logLevel: info
# metricsMaxInMemory: maximum number of metrics to keep in memory
# - optional, default: 1000
metricsMaxInMemory: 1000
# startPort: sets the starting port number for the automatic ${PORT} macro.
# - optional, default: 5800
# - the ${PORT} macro can be used in model.cmd and model.proxy settings
# - it is automatically incremented for every model that uses it
# startPort: 10001
models:
# ==========================================================================
# QWEN 3.5 MODELS (KB-recommended settings)
# - Thinking mode: temp 1.0, top-p 0.95, top-k 20, min-p 0.0, presence_penalty 1.5
# - Coding (precise): temp 0.6, top-p 0.95, top-k 20, min-p 0.0, presence_penalty 0.0
# - Non-thinking general: temp 0.7, top-p 0.8, top-k 20, min-p 0.0, presence_penalty 1.5
# - Context: 256K native (start 16K-32K for responsiveness)
# - Gibberish fix: add --cache-type-k bf16 --cache-type-v bf16
# - No Ollama support for Qwen3.5 GGUFs — use llama.cpp only
# ==========================================================================
"qwen3.5-35-a3b":
name: "Qwen 3.5 35B-A3B Thinking"
description: "MoE reasoning model. 3B active params, general-purpose thinking/chat."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.5
--chat-template-kwargs '{"enable_thinking":true}'
"qwen3.5-122b-a10b":
name: "Qwen 3.5 122B-A10B UD-Q4_K_XL"
description: "Large MoE reasoning model. 10B active params, heavy reasoning tasks."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_Qwen3.5-122B-A10B-GGUF/UD-Q4_K_XL/Qwen3.5-122B-A10B-UD-Q4_K_XL-00001-of-00003.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.5
--chat-template-kwargs '{"enable_thinking":true}'
"qwen3.5-9b":
name: "Qwen 3.5 9B UD-Q4_K_XL"
description: "Dense 9B model. Lightweight general-purpose chat and reasoning."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_Qwen3.5-9B-GGUF/Qwen3.5-9B-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.5
--chat-template-kwargs '{"enable_thinking":true}'
# --------------------------------------------------------------------------
# Qwen 3.6 — uses -hf syntax, reads from HF_HOME=/hfcache (host pre-download)
# --------------------------------------------------------------------------
"qwen3.6-35-a3b-abliterated":
name: "Qwen 3.6 35B-A3B Abliterated Heretic Q8_0"
description: "Qwen3.6 MoE, 3B active. Abliterated/heretic variant of BF16 quantized to Q8_0. ~38GB."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--jinja
-hf IIEleven11/Qwen3.6-35B-A3B-Abliterated-Heretic-BF16-Q8_0-GGUF
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.5
--repeat-penalty 1.0
--reasoning on
--reasoning-format deepseek
# ==========================================================================
# NEMOTRON MODELS (KB-recommended settings)
# - General Chat: temp 1.0, top-p 1.0, min_p 0.01
# - Tool Calling: temp 0.6, top-p 0.95, min_p 0.01
# - NoPE architecture: no YaRN needed
# - DEPRECATED --special flag for reasoning tokens
# - --special flag causes issues.
# - Start ctx 16K-32K, increase cautiously
# ==========================================================================
"nemotron-3-super-120b":
name: "NVIDIA Nemotron 3 Super 120B-A12B UD-Q4_K_XL"
description: "Flagship NVIDIA reasoning model. 12B active of 120B, MoE. 64-72GB VRAM at Q4."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 16384
--flash-attn on
--temp 1.0
--top-p 1.0
--min-p 0.01
--seed 3407
"nemotron-3-nano-30b":
name: "NVIDIA Nemotron 3 Nano 30B-A3B UD-Q4_K_XL"
description: "Compact Nemotron. 3B active of 30B, MoE. ~24GB at Q4. Best performance/size on 24GB GPUs."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--special
--model /models/unsloth_Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 1.0
--min-p 0.01
--seed 3407
# ==========================================================================
# GEMMA 4 MODELS (KB-recommended settings)
# - All variants: temp 1.0, top-p 0.95, top-k 64, repeat_penalty 1.0
# - Thinking: enable via --chat-template-kwargs '{"enable_thinking":true}'
# - Multi-turn: only keep final visible answer in history (not thought blocks)
# - Context: E2B/E4B=128K, 26B-A4B/31B=256K. Start at 32K.
# - ⚠️ Do NOT use CUDA 13.2 runtime — causes poor outputs
# - Use llama-server (not llama-cli) for thinking control
# ==========================================================================
"gemma4-26b-a4b":
name: "Gemma 4 26B-A4B"
description: "Google DeepMind Gemma 4 MoE, 4B active params, 256K context. Best speed/quality tradeoff."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 64
--repeat-penalty 1.0
--chat-template-kwargs '{"enable_thinking":true}'
"gemma4-31b-dense":
name: "Gemma 4 31B Dense"
description: "Google DeepMind Gemma 4 dense 31B. Maximum quality for complex reasoning, 256K context."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 64
--repeat-penalty 1.0
--chat-template-kwargs '{"enable_thinking":true}'
# ==========================================================================
# GLM MODELS
# ==========================================================================
"glm4.7-flash":
name: "GLM 4.7 Flash UD-Q4_K_XL"
description: "THUDM GLM 4.7 Flash. Fast inference, general-purpose chat."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 40000
--flash-attn on
--temp 0.6
--top-p 0.95
"glm-steam-106b":
name: "GLM Steam 106B-A12B Q4_K_M"
description: "TheDrummer GLM Steam MoE. 12B active of 106B. Creative and RP-focused."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/RP/bartowski_TheDrummer_GLM-Steam-106B-A12B-v1-GGUF/TheDrummer_GLM-Steam-106B-A12B-v1-Q4_K_M-00001-of-00002.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 40000
--flash-attn on
--temp 0.6
--top-p 0.95
# ==========================================================================
# SKYFALL MODELS
# ==========================================================================
"skyfall-r1-31b-q6k":
name: "Skyfall R1 31B v4 Q6_K_L"
description: "TheDrummer Skyfall R1 31B v4. General-purpose reasoning."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/bartowski_TheDrummer_Skyfall-31B-v4-GGUF/TheDrummer_Skyfall-31B-v4-Q6_K_L.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 40000
--flash-attn on
"skyfall-r1-31b-v4a":
name: "Skyfall R1 31B v4a Q6_K (RP)"
description: "BeaverAI Skyfall R1 v4a variant. RP/creative-focused."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/RP/BeaverAI_Skyfall-R1-31B-v4a-GGUF/Skyfall-R1-31B-v4a-Q6_K.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 40000
--flash-attn on
# ==========================================================================
# CODER MODELS
# ==========================================================================
"qwen3-coder-next":
name: "Qwen3 Coder Next UD-Q4_K_XL"
description: "Latest Qwen3 Coder. Non-reasoning model, optimized for code gen. KB: temp 1.0, top-k 40, min-p 0.01."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 40
--min-p 0.01
--repeat-penalty 1.0
# ==========================================================================
# LARGE / SPECIAL-PURPOSE MODELS
# ==========================================================================
"kimik2-q2kxl":
name: "Kimi K2 Instruct UD-Q2_K_XL"
description: "Moonshot Kimi K2. Huge MoE model (8-shard Q2). Limited GPU layers due to size."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00001-of-00008.gguf
--port ${PORT}
--n-gpu-layers 2
--temp 0.6
--top-p 0.95
# ==========================================================================
# GRANITE MODELS (IBM)
# ==========================================================================
"granite-4-small":
name: "Granite 4.0 Small Q4_K_M"
description: "IBM Granite 4.0 Small. Deterministic utility model for structured tasks."
ttl: 0
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_granite-4.0-h-small-GGUF/granite-4.0-h-small-Q4_K_M.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 120000
--flash-attn on
--top-p 1.0
--temp 0.0
--top-k 0
"granite-4-micro":
name: "Granite 4.0 Micro Q4_K_M"
description: "IBM Granite 4.0 Micro. Ultra-lightweight for fast structured responses."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/ibm-granite_granite-4.0-micro-GGUF/granite-4.0-micro-Q4_K_M.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 0.0
--top-p 1.0
# ==========================================================================
# EMBEDDING MODELS (persistent, always loaded)
# ==========================================================================
"embeddinggemma-300M":
name: "Embedding Gemma 300M"
description: "Google Embedding Gemma for vectorization."
ttl: 0
cmd: |
/app/llama-server
--embedding
--pooling cls
--model /models/ggml-org_embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf
--port ${PORT}
--n-gpu-layers 0
--ctx-size 2048
--batch-size 1024
--no-mmap
--ubatch-size 1024
--cont-batching
--threads 24
"qwen3-embedding-0.6B":
name: "Qwen3 Embedding 0.6B"
description: "Qwen3 Embedding model for vectorization. 32K context, last-token pooling (decoder/causal LM)."
ttl: 0
cmd: |
/app/llama-server
--embeddings
--pooling last
--model /models/Qwen_Qwen3-Embedding-0.6B-GGUF/Qwen3-Embedding-0.6B-Q8_0.gguf
--port ${PORT}
--n-gpu-layers 0
--ctx-size 8192
--batch-size 8192
--ubatch-size 2048
--no-mmap
--cont-batching
--threads 24
# ==========================================================================
# RERANKING MODELS (persistent, always loaded)
# ==========================================================================
"qwen3-reranker-0.6B":
name: "Qwen3 Reranker 0.6B"
description: "Qwen3 Reranker for retrieval reranking. Causal LM scoring yes/no logits at last token. Replaces BGE v2."
ttl: 0
cmd: |
/app/llama-server
--reranking
--pooling rank
--model /models/ggml-org_Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf
--port ${PORT}
--n-gpu-layers 0
--ctx-size 8192
--batch-size 8192
--ubatch-size 2048
--cont-batching
--threads 24
# NOTE: jina-reranker-v3 removed 2025-07-22 — unused, out of rotation.
# Qwen3 Reranker handles all reranking duties.
# ============================================================================
# GROUPS
# - swap: false = models in group can coexist in memory
# - exclusive: false = group can share memory with other groups
# - persistent: true = models never unload (for utility/embedding)
# ============================================================================
groups:
"high-reasoning":
swap: false
exclusive: false
members:
- "qwen3.5-35-a3b"
- "gemma4-31b-dense"
- "nemotron-3-nano-30b"
"heavy-moe":
swap: false
exclusive: false
members:
- "qwen3.5-122b-a10b"
- "nemotron-3-super-120b"
- "kimik2-q2kxl"
- "glm-steam-106b"
"utility":
swap: false
exclusive: false
persistent: true
members:
- "embeddinggemma-300M"
- "qwen3-embedding-0.6B"
- "qwen3-reranker-0.6B"