Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.
**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).
**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.
**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.
**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.
**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.
**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
(single-output regression-style reward score, expected shape for a
reward model)
The applet outgrew "stack alongside the infra-management workspace" —
it has its own pyproject, multi-tenant deploy story, separate
release cadence, and isn't actually about managing infrastructure.
Lives at https://gitea.phasefinal.com/vh/nevermore now, with
provenance noted in its initial commit.
This commit removes:
stacks/news-digest/ (full stack tree)
playbooks/deploy-news-digest.yaml
scripts/add-digest-user.sh
The existing ana-docker deployment continues running on its baked
local/news-digest:v5 image — nothing changes for the live install
until you choose to redeploy from the new repo. Migration steps
(rename data dir, redeploy, retire old compose dir) are in
nevermore's README.
Updated:
README.md — Current stacks listing now points at the new repo
STATUS.md — milestones entry for the extraction
These four ana-docker stacks were missing from the discoverability
index — readers landing on the README couldn't find them without
walking stacks/. Found via /tend-docs.
Three scripts that surface hosts on the fleet's networks that aren't
already tracked under servers/*/. Goal: spot servers that need management
coverage (inventory, backup, monitoring) without wandering the LAN by
hand.
discover-fortigate.sh SSH to a FortiGate admin, run
`execute dhcp lease-list all`, emit TSV
(IP, MAC, hostname, vdom, source).
SSH was picked over the REST API for now
because it needs no API-token plumbing. The
parser is defensive about FortiOS output
format drift (multiple VDOM sections,
optional hostname). API variant can replace
it when the extra robustness is worth the
token setup.
discover-unifi.sh Cookie-auth REST call against a UniFi
Controller. Tries /api/auth/login (UniFi OS
/ UDM / Cloud Key Gen2+) first; falls back
to legacy /api/login for self-hosted
controllers. Output is the same TSV shape
as the FortiGate script so the two mix.
Needs UNIFI_USER / UNIFI_PASS env and jq.
discover-gaps.sh Consumes one or more TSVs from the sources
above. Builds the set of managed IPs from
servers/*/ssh-target (plus a grep of README
files for documented IPs) and prints any
discovered IPs not in that set.
Exit code is 1 if gaps found — suitable for
cron alerting.
Common pipeline:
scripts/discover-fortigate.sh ana-fw.phasefinal.com > /tmp/ana.tsv
scripts/discover-fortigate.sh nh3-gw.phasefinal.com > /tmp/nh3.tsv
UNIFI_USER=admin UNIFI_PASS=… scripts/discover-unifi.sh esh-uc.esteban.net > /tmp/esh.tsv
scripts/discover-gaps.sh /tmp/ana.tsv /tmp/nh3.tsv /tmp/esh.tsv
First-time use probably needs:
- SSH access configured to each FortiGate (admin login, key preferred)
- UniFi user with read access (the built-in API read-only role works)
- `jq` installed on the runner (for UniFi script)
nh3-dev is the author's active dev workstation at the NH3 site
(nh3-dev.phasefinal.com). 22 GB home with 12 GB of development code,
shell history, dotfiles, Claude Code memory, and scripts that weren't
being captured by any fleet backup.
Writes site-local to the Synology rest-server (10.100.50.50:8000)
matching the nh3-docker pattern.
Excludes trim ~8 GB of regenerable caches + build outputs:
- language toolchains (.cargo, .rustup, .npm, .m2, .gradle, go/pkg, etc.)
- editor caches (.vscode-server, .vscode)
- project build directories (node_modules, __pycache__, .venv, target,
dist, build, .pytest_cache, .tox, .next, .nuxt) via `**/` patterns
- trash / browser caches / steam / snap
Expected first snapshot ~14 GB, incrementals much smaller due to
content-defined dedup across dev code.
README walks through the full setup: resticprofile install, Synology-
side htpasswd addition (via DSM File Station or Container Manager since
the DSM SSH setup is tabled), repo init, profile deploy, timer enable,
first backup. Schedule is nightly 01:00 with systemd's Persistent=true
behavior so missed runs catch up at next boot (workstation is often
off/sleeping at backup time).
README.md fleet-coverage table updated to reflect 6/6 file-level
coverage (ana-docker, ana-ml2, nh3-docker, esh-docker-vm, vm-esh-nas,
nh3-dev).
Audit surfaced three DB-backed services not being dumped consistently
by the existing pre-backup.sh:
- vaultwarden — migrated to external Postgres on PFI-Postgres
(10.250.50.80); old sqlite block was dumping stale pre-migration
files. Replace with pg_dump against the live database. Requires
postgresql-client on ana-docker and VW_PG* creds in
/etc/restic/dbcreds.env.
- gitea — also on PFI-Postgres; no hook existed at all. Use
`gitea dump` for a single zip that captures DB + repos + config +
LFS + attachments consistently. No explicit creds needed; the
container reads its own GITEA__database__* env.
- openwebui — two local SQLite databases (webui.db + the ChromaDB
vector store). .backup command if sqlite3 is in the image, volume-
level fallback otherwise.
Refactor: each block now logs a WARN on failure instead of aborting the
whole script — partial dumps > no dumps when one service has an issue.
dbcreds.env.example committed as a template; real file goes to
/etc/restic/dbcreds.env root:600 on the host and is never in the repo.
Mattermost retired (user confirmed 2026-04-21); removed from the
pre-backup.sh list and flagged in README's stacks section. Mattermost
container was not running regardless; the audit surfaced that it was
already effectively gone. Compose dir on ana-docker can be removed as
separate cleanup.
Parallel to refresh-server-info.sh but pipes proxmox_inspect.sh and writes
to servers/<host>/proxmox-details.txt. Same discovery / ssh-target /
validate-only / dry-run behavior.
Fleet-wide `all` matches dir names containing `-pve` (covers *-pve and
*-pve-* so esh-pve-nas is included alongside pfi-pve / nh3-pve / esh-pve).
Explicit names are never filtered — useful for one-off PVE hosts with
non-matching names.
Validation checks the captured snapshot for: truncation, missing PVE
version line, non-Proxmox target, and surfaces backup-coverage "NO"
verdict counts so gaps show up in the validate-only output.
Initial fleet snapshot refreshed.
VM images are backed up via Proxmox vzdump, but ana-ml2 (bare metal) has
no backup at all, and file-level restic + DB dumps are still missing on
three of four Docker-host VMs. Adds an explicit coverage table to the
README backup section so the gaps don't get overlooked while planning.
Goal (per user): every Docker host + configs + every database, not just
the VM image layer.
vm-esh-nas (10.0.50.154) is a NAS-adjacent Docker VM on the esh-pve-nas
hypervisor. Runs filezilla (port 5800), dockge, beszel-agent, dozzle-agent
with /mnt/{share,music,books,media} NFS-mounted from 10.0.50.50.
Use this host when a stack needs direct NFS mounts to the ESH NAS shares.
Canonicalize dockge as stacks/dockge/ — single compose used on all five
Docker hosts with per-host DOCKGE_HOST_LABEL/DOCKGE_HOST_IP in .env so
each card on the homepage points at its own instance. Labeled
homepage.group=Service Networking.
Beszel + Dozzle agent dirs also renamed to beszel-agent-<site> /
dozzle-agent-<site> pattern across the fleet for consistency.
pfi-pve (Anaheim), nh3-pve (NH), esh-pve + esh-pve-nas (ESH home lab) now
have servers/<host>/ entries with ssh-target files, README stubs, and
first snapshots. Register all four via add-host.sh + ssh-copy-id; SSH
auth is root@<name> with key-based access.
Fix a resolver bug in the three workspace scripts: previously, when a
hostname resolved via /etc/hosts, resolve_target returned the bare name
and ssh dropped the user prefix stored in ssh-target (connecting as the
wrong user). Now ssh-target wins whenever it's present, since it can
carry user@ or port info that /etc/hosts + ssh_config cannot. Falls back
to the dir name only when no ssh-target file exists.
CLAUDE.md and README.md split the fleet table into Docker hosts vs
hypervisors with an explicit placement rule: don't deploy Docker stacks
on PVE hosts; capture host-level detail with server_inspect.sh, use
Proxmox-native tooling (qm list, pvesh, zpool) for VM/storage audits.
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
vh