ssh exiting 0 is not proof the capture is usable — the inspect script can emit
nothing and both refresh scripts would mv that over a good system-details.txt and
report 'ok (0 bytes)'. Every reader tests the snapshot with -s, so the writer was
producing an artifact its own readers call invalid: a guard whose test disagrees
with its writer's contract has quietly stopped guarding. Prompted by
brokkr-smithy-dev hitting the same shape from the other side (a -s test against a
sentinel written with touch, a precondition that could never pass).
- empty capture -> refused, previous snapshot kept, host counted as failed (exit 1)
- capture under 1/4 of the previous -> promoted but flagged, since a host can
legitimately shed services and the script should not guess
- header + CLAUDE.md contract lines corrected to say what is actually guaranteed
- verified red (empty inspect -> FAIL, snapshot intact, rc=1) then green (real host
-> ok 6727 bytes)
The 2026-09-06 headscale cutover retired irv-ml1's wg0 tunnel IP 10.100.79.3
(now 10.6.110.50). Repointed all LIVE canonical refs to the DNS NAME so the next
move can't re-break them: homepage.href/siteMonitor labels across 25 stack
composes, load-bearing env defaults (asset-engine INFERENCE_HOST, open-webui
AUDIO_TTS_OPENAI_API_BASE_URL, skaldsong SKALDSONG_TTS_BASE_URL, zonos-gateway
ZONOS_URL, dia), homepage services.yaml manual cards (Voice Design Studio,
IRV-ML1), and servers/irv-ml1/ssh-target. Updated the stale 'WG tunnel' comment
to the mesh reality.
Left as-is: README curl-examples and .env.example comments (docs), and historical
mentions in CLAUDE.md/persistent-memory. NOTE: applying the label repoints to the
RUNNING irv-ml1 containers needs a recreate per service (labels read at creation);
deployed .env values are separate from these canonical defaults.
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
pfi-gx10 moved from desk Wi-Fi (10.100.10.226, VLAN 10) to the rack on
10.100.50.60 (nh3-servers, VLAN 50), reachable as pfi-gx10.nh3.internal.
The address is a DHCP RESERVATION on the UDM, not a host static. Operator
ruling during the move, and the better design: a host-side static works
until the box moves and is then a stale netplan file on a machine whose
address you no longer know. The pre-written playbook wrote a host static;
it is kept for its safety ordering and annotated as retired.
The port arrived on the native VLAN, not the server VLAN, so switch port 22
was repointed first. port_overrides is a whole-array PUT — two unrelated
overrides on ports 21 and 23 were read, preserved and written back, with the
original array backed up to a file before the change.
Wi-Fi stayed up as the escape hatch until the wired path was proven from
outside, and was downed last as its own step. The step worth keeping: while
Wi-Fi was up, traffic to nh3-dev still preferred wlP9s9 because that
interface sits directly on the userland subnet, so reachability proved
nothing about the wired path. `ping -I enP7s7` across the VLAN boundary is
what actually settled it before the hatch came down.
Target settled: nh3-servers VLAN 50, static 10.100.50.60. Clear of the four
existing statics and below the .150 DHCP pool where fleet statics live.
The playbook never leaves itself one path back. Wi-Fi stays up throughout while
the wired interface is configured beside it; the new address is verified from
outside before anything is torn down, and Wi-Fi teardown is explicitly a
separate later change. A botched netplan therefore costs a retry over Wi-Fi
rather than a trip to the rack — which is what substitutes for 'netplan try',
whose interactive rollback needs a TTY that elway cannot provide.
Two preconditions are asserted as steps rather than assumed: the interface must
have carrier (writing a static config for a dead NIC and reporting success is
the failure this avoids), and its MAC must match, since interface names can
renumber across kernels but MACs do not.
Requires nothing from the operator beyond racking the box. The wired NIC has a
distinct MAC from the Wi-Fi one, so the post-move address and switch port are
both discoverable from the UDM rather than needing to be relayed.
NVIDIA GB10, aarch64, 121 GB unified, sm_121. Ships booting to graphical.target
with GDM and GNOME Remote Desktop running.
playbooks/gx10-headless.yaml sets multi-user.target, stops gnome-remote-desktop,
masks the sleep/suspend/hibernate targets, makes logind ignore lid and idle, and
adds sshd keepalives so a stalled link does not kill a long-running job.
Two things learned the hard way and recorded in the playbook:
- gdm is a STATIC unit on Ubuntu, pulled in by display-manager.service and never
'enabled'. A guard of always skips, and a verify
written the same way passes while the desktop is still running. Both now test
is-active. The first run reported six green verifies having not stopped gdm.
- elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as
the connecting user and must carry their own sudo; connect as infra-ops.
The playbook refuses to stop the display manager while a seat session is held,
overridable with --var force_dm_stop=true. Networking is deliberately out of
scope: the box is on a desk on Wi-Fi with a temporary DHCP lease and no ethernet
carrier, and belongs to the rack-install change.
Benchmarked selene against gen on selene's own job: 24 designed judge items
with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a
neutral JSON prompt and Selene's native Atla template. 288 calls, all free
local.
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
native Atla selene 21/24 (88%) gen 22/24 (92%)
gen won on both templates and selene's BEST sat below gen's WORST. Selene was
given its own fine-tuned template as a fairness check; it gained one point,
not the three it needed.
Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a
winner on every equivalent pair. For eval work that is the case that matters
most. gen returned tie correctly on the JSON template. Selene also compressed
the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only
win was ~3x latency, unexercised at ~60 calls/day with zero queueing.
TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately:
- chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says
consumers bind the capability, not a concrete model. Sampler profile copied
from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served
config matches the benchmarked condition.
- selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A
served-name is a contract about what the model IS; answering it with a
different model hides a material change behind a stable string. Operator
ruling: "never repoint a named model at a different model's endpoint --
that is intentionally misleading." Verified: the gateway now returns
HTTP 400 "Invalid model name" for it.
Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that
had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role
moved onto an existing seat rather than allocating anything new.
Canonical litellm config synced from the host; ana-ml2 README and
recommended-model-settings updated. compose.yaml kept for reference, not
deployed.
nevermore-claude handed hrafn to infra-ops for uptime ownership. Intake:
verified health independently, brought it into the inventory, and fixed one
security defect found during the check.
- canonicalize stacks/hrafn/ (compose.yaml + .env.example + README)
- list hrafn in the ana-docker running-stacks table
- README records the load-bearing bits: shm_size 1gb is required for
Chromium, playwright and the base image are version-coupled, SSRF denies
private targets by default, and restart:unless-stopped does NOT act on
healthcheck failure
Security fix applied on the host, not in this tree: /opt/docker/compose/
hrafn/.env was 0644 with a live 57-char bearer token in it, readable by
every local account (verified by reading it as `nobody`). Tightened to 0600;
owner unchanged so the deploy path still works, confirmed via `docker
compose config` as lkraven. Container untouched and still healthy.
Known gaps recorded rather than silently accepted: no CI deploy, and the
image builds from an rsync'd working tree living in the compose directory,
so the running image has no commit provenance.
The filezilla container on vm-esh-nas shipped with no restart policy. It
was SIGTERMed during the 2026-08-18 host shutdown and never came back,
staying down four days while dockge/dozzle-agent/beszel-agent all
recovered on their own.
- add `restart: unless-stopped` (applied on the host, container recreated
and verified serving 200 on :5800 with all four /NAS mounts visible)
- promote the previously un-canonicalized compose into stacks/filezilla/
so the policy is version-controlled rather than host-only
- record the missing-restart-policy failure mode in the host README
Records the deployment plus the finding that outlives it: no_speech_prob
alone does not carry a hallucination threshold on this stack. Silence and
room tone both hallucinated 'Thank you.' at no_speech_prob under 0.11,
where a conventional >0.6 gate passes them through; avg_logprob separates
~6x better. Any future consumer asking for a Whisper hallucination gate
should be pointed at the composite, not the single field.
Also banks the digest-pin rationale (the VAD flag is an upstream-declared
unstable variable name, so a floating tag can move gate semantics
silently), the two deployment gotchas, the docker-vs-native GPU index
inversion on this host, and Eyra's future diarization VRAM claim on the
same A6000.
Adds speaches to the irv-ml1 stack table alongside parakeet, with the
reason both exist.
esh-pve hard-froze at 03:34 on 2026-08-19 and stayed frozen ~4.5 hours
until a manual power cycle. No panic, no OOM, no MCE — the journal stops
mid-operation. The whole ESH site lost DNS with it, because esh-userland
(VLAN 10, the PVC SSID and wired userland LAN) was handed exactly one
resolver: 10.0.50.45, AdGuard on esh-docker-vm, on a different VLAN, with
no secondary. Internet and routing were healthy throughout.
Two fixes.
1. DNS: 10.0.10.1 (the gateway, verified resolving) added as secondary on
esh-userland via the UDM Classic API. Note this is degradation cover,
not clean failover — clients that query resolvers in parallel will
bypass AdGuard for a share of lookups.
2. Watchdog: softdog -> iTCO_wdt under systemd (RuntimeWatchdogSec=60),
watchdog-mux masked. The box looked watchdog-protected and was not: a
software watchdog cannot fire when the kernel it lives in is wedged,
and watchdog-mux only pets the device while an HA client is connected,
which never happens on a cluster with no HA resources. Firmware does
not block the TCO timer here, checked before committing to it.
Also pins VM 102 off (onboot: 0). It starts with full GPU passthrough and
vfio-pci enabling that device is the last thing the kernel logged, 39
minutes before the freeze. The other suspect is the kernel itself: the
host ran 4.5 months on 6.8.12-16, took 6.8.12-42 in an apt batch on
08-18, and died 20 hours into the first boot on it. 6.8.12-16 is still
installed as the rollback.
The playbook is idempotent — a second run skips all six steps and passes
all six verifies. The watchdog is confirmed armed (identity=iTCO_wdt,
state=active, held by PID 1) but has NOT been observed firing; proving
that needs a deliberate wedge.
Memory also corrects two wrong mid-incident calls: the mgmt VLAN is
routed over the site tunnel and is not firewalled off — both symptoms
were the dead host generating ICMP unreachables.
Two findings from chasing node `pve` showing dark in the UI.
esh-pve and esh-pve-nas are a 2-node cluster (esh-pve-cluster, expected
votes 2, quorum 2, no qdevice). This was undocumented, and last night's
migration rebooted one of the members without accounting for it. Nothing
broke -- quorum is intact and both nodes report the same ring id -- but
that was luck. Rebooting either node drops the survivor below quorum and
makes its /etc/pve read-only until the partner returns. It matters for
the pending confirmation reboot and the 225-package upgrade, both of
which take a node down.
The dark tile itself was NOT last night's doing. pvestatd SEGV'd on
2026-05-28 and had been dead 82 days; the journal has nothing between
that crash and the restart today. It is only the reporting daemon, so
the node stayed quorate and healthy with all services active and all
three guests running the whole time -- the UI simply had nothing telling
it the node was alive. Fourth SEGV in that unit's history, so treat a
recurrence as expected and consider a watchdog: nothing alerts on it,
and the sole symptom is cosmetic enough to go unnoticed for months.
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is
out of the runtime I/O path, so a bus reset can no longer drop root from
under a running hypervisor. All five guests healthy, three pools ONLINE,
system running, ext4 pve-root intact and unmounted as the rollback with
its own kernel and initrd. zfs-import-cache is now the active import
path -- the all-three-pools cachefile fix doing its job.
The window cost an unplanned outage, and the cause was this repo's own
tooling rather than the migration.
The staging chroot ran `mount --rbind /dev` and /sys with no
--make-rslave. On systemd `/` has shared propagation, so the cutover's
`umount -R` propagated back into the live host and removed the real
/sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind
could not create a session: ping fine, TCP fine, SSH authentication
succeeded, resident daemons kept serving -- and every new exec hung,
including /sbin/reboot, so the reboot never ran at all.
It impersonates failing root-disk I/O almost perfectly, and I called it
as the DOM dying. That was wrong. dmesg had the answer throughout: the
DOM attached cleanly with no errors, and the last log timestamp was
12114881s -- 140 days -- meaning this was still the original boot. A
down-detector had also never reported the host down, which I read as a
fast reboot rather than as no reboot.
Fixes and guards:
- --make-rslave after every rbind, plus a guard that refuses to proceed
while any chroot bind still reports shared propagation.
- Confirm a reboot by observing the host DOWN, not by watching for it to
come back. Those two states are indistinguishable otherwise.
- Blast radius now measured from the server: `ss` inside CT 103 found
five NFS clients, not the two documented. The new one that mattered is
esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on
purpose and it came through read-write.
- grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV
which GRUB reads but cannot write, so next_entry survived the boot
that consumed it. Steady state is saved_entry=pve-zfs-root with no
next_entry. There is no auto-fallback on this host and no IPMI.
Recovery needed no console: an idempotent cgroup2/devpts/shm remount
landed in the brief windows where exec succeeded. No data was lost, and
neither the DOM nor any pool was ever at risk.
Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.
Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.
Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.
Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:
- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
ext4 cannot shrink; freeing space from root needs a rescue boot, which
costs the one-reboot property. Space came from swap (768M -> 256M).
- The runbook's `zpool set cachefile=... nvme` would have broken the
NAS. Populating a cachefile flips the host from import-by-scan to
import-by-cache, so a one-pool cache leaves ssd and tank unimported --
and CT 103 esh-nas has twelve bind mounts spanning all three pools.
Set on all three instead, verified in the resulting cache.
- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
and rpool comes from grub-probe --target=fs_label, which returns empty
because GRUB's ZFS reader cannot open a pool with encryption,
large_dnode and zstd_compress -- the same feature set that forced /boot
to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.
The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.
PVE root on esh-pve-nas is a USB Disk-on-Module: 6 GB ext4 with the host's only
ESP. A DOM is SLC/pSLC so wear is not the driver -- the problems are that it is
on the USB bus (a reset drops root under a running hypervisor), has no headroom,
and is unmirrored while 928 GB of mirrored NVMe sits 96% empty.
Runbook targets a fresh PVE install to ZFS RAID1 across both NVMes. In-place
conversion is unsupported, and adding an ESP to the existing NVMes is impossible
-- both are whole-disk ZFS members with 1.7 MiB free and proxmox-boot-tool
manages nothing today.
The headline risk is not on the host being rebuilt: CT 103 esh-nas IS the NAS
at 10.0.50.50, and both esh-docker-vm and esh-pve mount it hard. Taking this box
down stalls esh-pve's storage layer and wedges esh-docker-vm into the D-state
whose only remedy is a host reboot -- the incident shape already on record.
Quiescing those clients is step one of the window, and the README now warns
against casual reboots.
Config snapshot captured off-box to nh3-dev (0600) with /etc/pve, network and
fstab config plus zpool/zfs/disk-by-id/guest state; the newest on-disk copy
before this was June 2024.
Two user-systemd services stood up on nh3-dev this session weren't in the
"What runs here" list:
- The Booth (:8090, booth.service) — eshpfi services/booth ephemeral media board.
- jackdaw-compose (:8787, jackdaw-compose.service) — JackDAW /compose AI backend
hosted for jackdaw-dev, Origin-gated, fronted by the :4500 bench proxy.
Keyless entries (the jackdaw unit's key stays server-side, uncommitted).
The README's running-stacks table had drifted well behind reality (still listed
llama-swap + only the embed/rerank/reward trio). Regenerated system-details.txt
and rewrote the stacks + GPU-allocation sections from a live docker ps +
nvidia-smi (2026-07-22):
- GPU 0 (hot): vllm-aeon-gen (qwen3.6-35b-a3b-heretic, NVFP4) + vllm-charrp-
reasoning-nvfp4 (char-rp-reasoning, NVFP4), ~42+45 GB.
- GPU 1 (on-demand): granite-4.1-8b, selene-1-mini-8b, Skywork reward,
Qwen3 embed/rerank, and the Magidonia-24B char-RP GGUF (llama-charrp), ~91 GB.
- Recorded the dormant on-disk stacks and llama-swap's retirement.
NH3 manager/external-dev box at 10.100.50.42 (Debian 13 VM on nh3-pve),
successor to the retired nh3-ansible. infra-ops identity here is sudo-LESS
by operator decision (2026-06-17): key-only, no NOPASSWD, not in docker
group — user-level management only. Adds servers/nh3-extdev/{README,
ssh-target,system-details.txt}, the CLAUDE.md inventory row, and a local
ssh alias (nh3-extdev -> infra-ops@10.100.50.42, infra-ops key). Login +
sudo-less posture verified.
Append 10.100.10.50:8391 to BIFROST_CLIENT_ALLOWED_HOSTS on the personal
Worldtree (.env) so the consumer may bind the memory provider at session-create
(affect :8390 was already listed; the url-guard 422s un-allowlisted endpoints).
Idempotent elway playbook; surgical worldtree-api recreate that auto-derives the
image pin from the matrix sibling to avoid the stale-:latest crash-block footgun.
Repoint servers/corviduo-dev/ssh-target to infra-ops (operator granted durable
NOPASSWD admin on corviduo-dev 2026-06-15).
ana-ml2 was upgraded 2026-06 from dual RTX 6000 Ada (46GB, cc 8.9) to
dual RTX PRO 6000 Blackwell Max-Q (96GB, cc 12.0 / sm_120). Update the
stale hardware facts across the workspace:
- CLAUDE.md servers table row
- servers/ana-ml2/README.md hardware spec (+ refreshed system-details.txt)
- stacks/vllm compose + .env.example FP8/KV comments (Ada cc 8.9 -> Blackwell cc 12.0)
- stacks/llama-swap config VRAM-budget comment (48GB -> 96GB, GPU-0 pin)
Also corrects the adjacent stale 'Phi-4-mini' comment in the granite
service block (the service has been Granite 4.1 8B since 34a43a0).
Doc/comment-only; no runtime change.
- migration-plan.md: workstation-only migration of nh3-dev's dev env to
the M2 Ultra Mac Pro Rack (on-subnet, sidecars stay on Linux).
- nh3-dev README: drop volva from live sidecars + add retired note.
volva.service/heid.service removed by infra-ops at heid's request
(re-architected daemon->CC-orchestrator; volva.service was crash-looping).
Reflects the shared install: browsers in /opt/ms-playwright (root-owned,
world-readable) + PLAYWRIGHT_BROWSERS_PATH wired globally via
/etc/environment + profile.d, so any user/project resolves the shared
binary without a per-home playwright install. Includes the add-a-browser
command for future bumps.
Adds the NH3 dev box (10.100.10.50) as a managed host: README, first
system-details snapshot, ssh-target. Adds it to the fleet bootstrap's
Tier 1 and the CLAUDE.md servers table. infra-ops identity bootstrapped
there (operator-run) — NOPASSWD sudo + key, so root-level infra work on
the box (it runs the egress proxy, ttyd seat, mead-hall, volva, and dev
checkouts) no longer needs a per-task password.
First use: installed Playwright headless-Chromium system deps + binary
for bloom_music's OSMD browser-geometry test harness; headless launch +
real SVG geometry verified.
worldtree-dev sent infra-ops a manual deploy request for v0.30.6 to
personal:8081; operator confirmed all three corviduo-dev Worldtree
deployments go through the Worldtree team's CI/CD, never a manual
infra-ops deploy. The old 'personal — manually managed' line was wrong
and misleading. docker-as-root pattern is emergency-ops (key restore)
only.
Made the host-stacks bind-mount path configurable via
DOCKGE_HOST_STACKS_ROOT (default /opt/docker, unchanged for the
existing five hosts). Override on corviduo-dev to /home/vh/docker
because that host's /opt/ is owned by deploy:deploy (Worldtree team)
and vh lacks passwordless sudo for the fleet-standard path — same
reasoning as the beszel + dozzle agent placement earlier today.
Deployed to corviduo-dev. Reachable at http://10.250.50.152:5001
(first probe 200 — Docker's port-mapping route through iptables
worked without firewall changes, unlike beszel's network_mode: host).
Scoped to PFI-managed stacks only (/home/vh/docker/compose/) — does
NOT see /opt/worldtree*/ deployments. Keeps the management boundary
clean: dockge can restart/recreate PFI's beszel+dozzle+itself but
not the Worldtree-team-owned containers.
Operator-approved fleet monitoring extension. Both agents up + healthy:
- beszel-agent on host port 45876 (KEY-mode, hub at ana-docker:8090
SSH-polls inbound; seeded with hub's ed25519 pubkey).
- dozzle-agent on host port 7007 (mTLS auto-generated; hub at
ana-docker:8088 connects inbound).
Compose lives at /home/vh/docker/compose/{beszel,dozzle-agent}/ rather
than the fleet-standard /opt/docker/compose/{...}/ because corviduo-dev's
/opt/ is owned by deploy:deploy (Worldtree team) and vh lacks
passwordless sudo to create the fleet path. Functionally identical;
documented in servers/corviduo-dev/README.md so future infra-ops
sessions find them.
Created an empty traefik-net external docker network on corviduo-dev
as a side effect of dozzle-agent's compose (which declares it external).
Future PFI services landing here can reuse it.
Dozzle hub on ana-docker had 10.250.50.152:7007 appended to
DOZZLE_REMOTE_AGENT (a host-side change to a non-tracked .env, not
canonical). Beszel hub still needs corviduo-dev added via the UI's
"Add System" action — one-time operator step, flagged in the corviduo
README.
Previously a partial list "from vzdump logs + servers/ dirs". Replaced
with the full output of `qm list` on pfi-pve (2026-05-14), cross-
referenced against `servers/`. VMs that have a server dir are
annotated with the path; ones that don't are flagged "not yet in
`servers/` inventory" so the gap is discoverable.
Resolved:
- VM 100 = pbs-ana (was "VM (TBD)" in prior version)
- VM 106 = corviduo-dev (added 2026-05-12; was missing from the list)
Surfaced (new findings, not yet inventoried):
- VM 101 — PFI-ANA-DC (Active Directory domain controller)
- VM 103 — PFI-SlaveBot (purpose unclear from name alone)
- VM 104 — PFI-Mongo (MongoDB host, separate from pfi-postgres)
The three new-to-inventory VMs deserve `servers/<name>/` directories
with READMEs, but that's a follow-up — creating new inventory entries
is out of tend-docs scope.
Section claimed gitea + paperless-ng Postgres passwords were "currently
in use" with trivial values and that rotation was pending. Per
STATUS.md: "Rotate exposed secrets — done 2026-04-23. All six rotated:
vaultwarden/gitea/paperless-ng Postgres passwords (hardcoded
compose.yaml literals moved to gitignored .env files in the process)..."
Rotation happened; literals are no longer in compose.yaml; passwords are
no longer trivial. Removing the section rather than amending — once
fixed, there's no value in carrying a "we have weak passwords"
section that lies about the current state.
Surfaced by /tend-docs audit 2026-05-14.
VM 105's annotation said "PGDATA on NFS from ana-nas" — postgres
migrated off NFS to local VM disk on 2026-04-23 per STATUS.md ("DB data
on local disk, not NFS. pfi-postgres migrated 2026-04-23"). Updated to
reflect current reality with the migration date as the rot detector.
Still-TBD note for VM 33: `pbs-ana` is listed as "VM (TBD)" — was
deployed and has its own server dir, but its VM ID isn't recorded
here. `qm list | grep pbs` on pfi-pve would resolve, but my SSH to
pfi-pve as lkraven is currently password-required so I can't pull
this myself. Flag for next pfi-pve console session.
Surfaced by /tend-docs audit 2026-05-14.
Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.
**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).
**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.
**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.
**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.
**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.
**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
(single-output regression-style reward score, expected shape for a
reward model)
Captures the access path + emergency-ops runbook surfaced during the
2026-05-12 demo outage (Z_AI_API_KEY / runtime validator / :latest tag
drift cascade).
- servers/corviduo-dev/{README.md,ssh-target,system-details.txt}
- CLAUDE.md servers table + placement-rules note
The "docker-as-root sudo bypass" pattern (vh's docker-group membership
→ effective root for any bind-mount-able operation) is the canonical
fix path when /opt/worldtree*/.env edits are needed and the deploy
user's sudo is out of reach. The README documents the pattern with
the appropriate "docker-group ≈ sudo" security warning so future
emergency-ops sessions don't have to re-derive it.
Ownership stance matches SF client hosts: PFI hosts + provides
emergency-ops backstop, Worldtree-team owns OS config / deploys /
backup decisions. Coordinate data-affecting work with the architect.
Three changes prepping infra for asset_engine's orchestration feature
(SSH-driven bring-up / bring-down of irv-ml1 inference services with
per-device VRAM gating, contract in vh/asset-engine commit 5a36f8c):
1. asset-engine compose + .env.example + playbook gain a read-only
bind-mount for /app/runtime/ssh — the dedicated ed25519 keypair
(generated on ana-docker, not in the repo) plus a pinned known_hosts
for irv-ml1's host fingerprint. Env vars SSH_KEY_PATH and
SSH_KNOWN_HOSTS are exposed for the app to consume.
2. docs/asset-engine/services.yaml gains a `lifecycle: { stack, vram_gb,
gpu_device_id }` block on each of 12 orchestratable irv-ml1 services
(kokoro, chatterbox, index-tts, qwen3-tts, cosyvoice, fish-s2,
kyutai-tts, vibevoice, voxtral, parakeet, stable-audio-open, ace-step).
VRAM numbers are estimates from model footprint at fp16 — tune from
real nvidia-smi measurements once the gate is live. comfyui and
kokoro-captioned are deliberately excluded (variable-VRAM and
shared-container respectively).
3. servers/irv-ml1/README.md docker-stacks table now lists all 13
inference stacks (was only dockge + agents + comfyui) with port +
GPU pinning columns.
Pubkey deployed to ~lkraven/.ssh/authorized_keys on irv-ml1;
end-to-end SSH from ana-docker → irv-ml1 verified with strict
host-key checking.
Bundles the inventory expansion since 2026-04-22:
- New host dirs (READMEs + ssh-target where dir name doesn't resolve):
ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3.
- New PFI VM snapshots (registered + key-installed 2026-04-23):
ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl,
pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox).
- servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when
the host was brought into the fleet) + ssh-target (10.100.79.3 over
the WG tunnel — name doesn't DNS-resolve from this workstation).
- servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC
IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware =
sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50,
distinct from the Dell R750xs iDRAC).
- configs/homepage/docker.yaml: irv-ml1-docker provider added so
homepage auto-discovers irv-ml1's stacks over the WG tunnel.
- docs/orientation.md: narrative fleet overview written for fresh
Claude sessions — sites, backup architecture, governing principles,
gotchas, where-to-look guide. Pointed at from CLAUDE.md.
New stack mirroring PFI convention (stacks/comfyui/) using
mmartial/comfyui-nvidia-docker:ubuntu24_cuda12.8-20260312. Both GPUs
exposed, pinned to CUDA 12.8 to match the host's 570.x driver and the
native cuda-toolkit already in place.
Layout — single tree under /worktank/comfyui/ (462G dedicated, 1%
used pre-deploy):
- basedir/ → /basedir user state (models, workflows, custom_nodes,
input, output); owned 1000:1000 so external
tools can edit workflow JSON directly.
- run/ → /comfy/mnt ComfyUI source + venv + pip cache (~7.8G
after bootstrap). Bind mount instead of
named volume — the image refuses to chown
mounted paths at startup, so keeping this
lkraven-owned avoids the sudo dance.
servers/irv-ml1/README.md refreshed: Docker upgraded to 29.4.1 with
traefik-net in place; dockge + beszel-agent + dozzle-agent already
present; /storetank dropped 92% → 64%; restic coverage to
rest-server-nh3 is operational (not "currently none" as prior text).
Initial framing was wrong. PFI runs these under a managed-hosting
agreement: SSH, OS ops, backups are all PFI's responsibility. Hardware
and data belong to the client.
Changes:
- ssh-target files added for sfsrv-ana (root@10.250.250.115 — same
pattern as other PVE nodes) and sf-ana-container
(lkraven@10.250.150.100 guess, adjust if different user).
- sf-r630 still lacks an ssh-target — the OS-side LAN IP isn't in
FortiGate DHCP (static config somewhere). Will fill in once
identified; README flags that gap.
- READMEs rewritten: dropped "tenant-scoped" / "not SSH-managed"
language, added "client context" section that explains the
managed-hosting relationship. Backup coverage now listed as
planned rather than blocked on tenant coordination.
- CLAUDE.md fleet table: SF rows re-labeled "SureFire client
(PFI-managed)". Placement-rules section updated to note that
SF hosts are first-class PFI-ops targets, just client-owned.
- Memory (project_surefire_tenant.md) rewritten to reflect
managed-services reality + hosts-file entries needed for name
resolution since these aren't in PFI DNS.
Six PFI VMs/LXCs previously known only via proxmox_inspect.sh —
covered by vzdump but not in servers/, so operational context
(roles, backup posture, ssh target) was missing:
pfi-ana-webhost (VMID 110) — web workload
ana-filebot (LXC 112) — file-task automation
pfi-pteradactyl (VMID 107) — Pterodactyl game panel
pfi-tacticalrmm (VMID 111) — TacticalRMM remote-management
pfi-postgres (VMID 105) — shared Postgres (vaultwarden/gitea/
paperless backends)
ana-wg (LXC 113) — WireGuard VPN gateway
Plus three SureFire tenant hosts at the Anaheim colo:
sfsrv-ana — tenant Proxmox hypervisor (10.250.250.115:8006)
sf-ana-container — container workload on that Proxmox
sf-r630 — physical R630 (iDRAC 10.250.250.110 for PFI-side
hardware mgmt; OS is tenant-scoped)
Each server dir has README + ssh-target where applicable. SureFire
entries explicitly document tenancy scope: PFI provides hosting,
SureFire owns the OS; management actions need tenant coordination.
SureFire hosts have no ssh-target by default.
Homepage Infra - ANA gains two new cards:
- SFsrv-ANA (https://10.250.250.115:8006, si-proxmox icon)
- SF-R630-iDRAC (https://10.250.250.110, si-dell icon)
PFI-ANA-ML2 BMC gained an href since it has a usable web UI.
CLAUDE.md fleet table extended with all 9 new rows. Placement-rules
section notes the SureFire tenant boundary.
Memory: new project_surefire_tenant.md so future sessions know sf-*
hosts are tenant-scoped by default.
Parallel to refresh-server-info.sh but pipes proxmox_inspect.sh and writes
to servers/<host>/proxmox-details.txt. Same discovery / ssh-target /
validate-only / dry-run behavior.
Fleet-wide `all` matches dir names containing `-pve` (covers *-pve and
*-pve-* so esh-pve-nas is included alongside pfi-pve / nh3-pve / esh-pve).
Explicit names are never filtered — useful for one-off PVE hosts with
non-matching names.
Validation checks the captured snapshot for: truncation, missing PVE
version line, non-Proxmox target, and surfaces backup-coverage "NO"
verdict counts so gaps show up in the validate-only output.
Initial fleet snapshot refreshed.
Closes the last open backup-coverage gap identified in the 2026-04-20
audit. All guests on all four hypervisors are now covered by vzdump
jobs (pfi-pve 11/11, nh3-pve 5/5, esh-pve 3/3, esh-pve-nas 5/5).
Read-only companion to server_inspect.sh. Runs over ssh via stdin the
same way:
ssh root@pfi-pve 'bash -s' < scripts/proxmox_inspect.sh \\
> servers/pfi-pve/proxmox-details.txt
Captures what the generic inspect misses on PVE nodes:
- Cluster + node resources (pvesh get /cluster/resources)
- VM list + per-vmid config highlights (qm list, qm config)
- LXC list + per-ctid config highlights (pct list, pct config)
- Storage pools (pvesm status, zpool, storage.cfg, NFS mounts)
- Backup jobs table parsed from /etc/pve/jobs.cfg (no jq needed)
- Per-guest backup coverage verdict: YES/NO with reason
- Most-recent actual backup per VMID on each local dump storage
- PBS targets, replication jobs, listening ports, subscription status
Coverage analysis surfaces the real gap: on pfi-pve, VMIDs 106, 107,
109, 112, 113 have no scheduled backup; VMID 106 (PFI-Tailscale) has
not been backed up since 2024-05-31. nh3-pve uses "all" job and covers
everything. esh-pve + esh-pve-nas share the tank-vmbu dump storage via
NFS; coverage splits cleanly across the two nodes.
First snapshots committed for each PVE host.
Retire the linuxserver.io calibre + calibre-web pair. CWA
(crocodilestick/calibre-web-automated) is an all-in-one: combined web UI
with Calibre backend, auto-ingest via drop folder, scheduled metadata
maintenance — single container on :8083.
- Reuses the existing library at /mnt/books/calibre/calibre_library
(NFS) and the existing calibre-web config at /opt/docker/conf/calibre-web,
so user accounts and the book catalogue carry over.
- Requires NETWORK_SHARE_MODE=true (library is on NFS; SQLite WAL would
corrupt otherwise).
- New drop-zone at /mnt/books/calibre/ingest; files placed there are
ingested into the library and then removed.
- Loses the LSIO desktop Calibre GUI (ports 8082/8181/8281) — run
Calibre locally if you need the desktop UI.
Old compose dirs (/opt/docker/compose/{calibre,calibre-web}) removed on
esh-docker-vm; old images can be pruned at leisure.
vm-esh-nas (10.0.50.154) is a NAS-adjacent Docker VM on the esh-pve-nas
hypervisor. Runs filezilla (port 5800), dockge, beszel-agent, dozzle-agent
with /mnt/{share,music,books,media} NFS-mounted from 10.0.50.50.
Use this host when a stack needs direct NFS mounts to the ESH NAS shares.
Canonicalize dockge as stacks/dockge/ — single compose used on all five
Docker hosts with per-host DOCKGE_HOST_LABEL/DOCKGE_HOST_IP in .env so
each card on the homepage points at its own instance. Labeled
homepage.group=Service Networking.
Beszel + Dozzle agent dirs also renamed to beszel-agent-<site> /
dozzle-agent-<site> pattern across the fleet for consistency.
vh