Per operator call (no gitea write token on the Worldtree-team VM): a 2-min systemd
--user timer on nh3-dev polls corviduo's last-deploy.json and pings soong-dev via
althing on a NEW red deploy (green stays silent). Delivers soong-dev's red-run
visibility without a credential on corviduo. Tested (red detect+format DRY, green quiet).
Adds the rsync --link-dest hourly snapshot job (nh3-dev:~/development ->
nh3-nas, 48-snapshot retention, secrets/build-dirs excluded) that closes the
no-off-box-backup gap exposed by the 2026-07-12 working-dir clobber. Script
mirrors the live ~/.config/dev-backup/dev-backup.sh; runbook covers restore.
Auto-deploy on push to main: gitea webhook → HMAC-validated listener on irv-ml1:9008
→ git fetch/reset + docker compose up -d --build. Documents the gitea-server
ALLOWED_HOST_LIST anti-SSRF relaxation (scoped to the WG net), the irv-ml1 components
(deploy key, git-clone deploy dir preserving the proxy override/secrets, listener +
user service), and verify/debug steps.
YouTube (and a growing set of services) hard-flag datacenter IPs, bot-gating
even public content regardless of cookies/PO-tokens. Origin case: yt-voice-clipper
on irv-ml1 (Irvine colo) — every yt-dlp fetch returned LOGIN_REQUIRED. Confirmed
pure IP reputation: the same public video fetches cleanly (no cookies) once routed
through nh3-dev's residential egress (70.230.226.88).
- scripts/setup-nh3-egress-proxy.sh: idempotent dante (SOCKS5) install + config.
Internal-only ACL (10.100.0.0/16), bound to the WG interface, systemd-managed.
- docs/runbooks/nh3-egress-proxy.md: purpose, usage, security model, caveats.
Reusable fleet egress, not yt-voice-clipper-specific.
Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.
**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).
**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.
**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.
**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.
**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.
**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
(single-output regression-style reward score, expected shape for a
reward model)
New runbook captures the three-phase process:
Phase 1 — Drop --append-only via DSM Container Manager web UI
Phase 2 — sudo resticprofile forget --prune --verbose on each of
nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host)
Phase 3 — Restore --append-only via DSM
Why each phase looks the way it does, what to expect (largely no-op
runs for the first 6 months while no snapshots have aged out of the
keep window), how to verify each phase non-destructively (curl 401
on the rest-server root proves the container's up + serving), what
to do if Phase 2 fails with `repository is configured as append-only`
(skipped Phase 1 / DSM didn't apply), and the path to future
automation (find docker bin path on DSM, NOPASSWD-lock syncuser to
the specific recreate command).
Includes a "last run history" table seeded with today's first
post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the
04-25→27 CUDA stall).
Cross-referenced from docs/README.md (runbook tree), docs/
orientation.md (where-to-look table), and STATUS.md item 9 (which
now points at the runbook + records the next-round date 2026-07-27).
Bundles the post-2026-04-21 work that built out the two-layer backup
architecture (PBS for VM images + restic for file/DB), plus the cross-
site mirror and the disaster-recovery runbook.
- configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete
*_offen_backup_data exclude (offen sidecars retired fleet-wide
2026-04-23; restic now covers the equivalent scope directly).
- configs/restic/esh-vm-db/: new profile for the dedicated DB VM
(10.0.50.60), with pre-backup pg_dumpall + mongodump hooks.
- configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven)
and nh3-nas → ana-nas (05:00 daily, runs as root because DSM
rest-server-nh3 writes mode-400 files only root can read).
- docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook,
refined during the 2026-04-22 deployment with per-hypervisor
namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the
Synology syno_acl flatten step.
- docs/runbooks/disaster-recovery.md: blast-radius runbook ordered
Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty);
references incident memory + recovery-step playbooks per consumer.
VMIDs aren't globally unique across PVE hosts (esh-pve and sfsrv-ana
both have VM 100, etc.). Without namespaces, per-hypervisor backups
collide under the same /vm/<vmid>/ path in the shared datastore.
Changes:
- New Phase 1.4b: create one namespace per hypervisor up front
(pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana).
- Phase 2.1 storage-entry template now lists Namespace as a required
field, set to the hypervisor's own name.
- Critical-note explaining the collision risk so future deployers
don't skip this step.
ACL grants remain at the datastore level; they apply across all child
namespaces so the existing fleet-vzdump token continues to work. Sync
job (Phase 6) preserves namespace tree to PBS-NH3 automatically.
Discovered during Phase 1.3 (mount NAS datastore). The Debian NAS's
/mnt/backup is on ZFS with casesensitivity=insensitive; NFSv4 writes
fail with EACCES even for root with no_root_squash. Known-bad
combination at the ZFS-on-Linux + NFSv4 layer.
Workaround: mount the NFS share with vers=3. PBS's chunk-based
access pattern works fine over NFSv3.
Added a section 0.5 to Phase 0 documenting the issue + both fixes
(quick: use NFSv3; cleaner: create case-sensitive child dataset).
Future deployments against this NAS won't rediscover the same wall.
Current NFS exports on the Debian NAS (10.250.50.50) use root_squash,
which blocks PBS from writing its datastore metadata (chunks, locks,
GC state — all root-owned operations). Rest-server-ana worked around
this by running its container as UID 1000, but PBS's service model
doesn't accommodate that pattern cleanly.
Solution baked into Phase 0: create a dedicated NFS export for the
PBS-ANA datastore subtree, scoped to only the PBS-ANA VM's IP, with
no_root_squash. Bounded exposure (single client), kept in a separate
.exports file so Cockpit's File Sharing module doesn't clobber it.
Flag world-scoped export on /mnt/pve-VMStorage as a non-blocking
hygiene item for a later Cockpit pass.
End-to-end runbook for standing up Proxmox Backup Server across the
fleet. Path A architecture: single primary at ANA, one-way sync to NH3
for disaster recovery. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve,
esh-pve-nas, sfsrv-ana) migrate from local-dump vzdump to PBS-ANA.
Key decisions captured in the runbook:
- PBS in a Debian VM (not LXC) for clean capability model.
- PBS-ANA on pfi-pve, datastore via NFS from 10.250.50.50 —
separates backup data from hypervisor boot disk.
- PBS-NH3 on nh3-pve with local storage (independent failure
domain from ANA).
- Dedicated fleet-vzdump API token; read-only sync token for
PBS-NH3's pull job.
- sfsrv-ana specifically goes from zero backup coverage to full
vzdump coverage in Phase 3.
9 phases, each self-contained with a done-state and rollback
posture. User can stop between phases without leaving the fleet in a
bad state.
STATUS.md: added item 6b tracking this deployment. Original item 6
(cross-site rsync) now scoped to restic-only since PBS handles the
VM-image cross-site redundancy directly.
vh