Operator confirmed 2026-09-19 that fv-ml1 and the R420 running OPNsense are the
only loads on a dedicated 20 A circuit.
The governing number is 1920 W, not 2400: a GPU inference host running for hours
is a continuous load, so NEC's 80% rule applies. Worst case lands at ~1625 W
with the current caps -- about 85% of budget.
Measured via the BMC rather than assumed: 390 W instantaneous, 461 W max over a
2423 s sample, with GPUs at idle, giving a ~313 W non-GPU baseline.
Compare the GPU caps against the 300 W stock TGP, NOT the 325 W firmware
ceiling. The operator corrected this: 275 W across four cards saves 100 W, not
the 200 W you get by measuring against a number nobody would ever run at. Stock
300 W would put the circuit near 90%, which is not illegal but leaves nothing
for a heavier R420, PSU efficiency, or a warm day. Keep the caps.
The coupling matters more than the trip. OPNsense IS the Fountain Valley edge
and shares the breaker with the thing most likely to trip it, so an overload
takes the router with it and removes the remote path needed to diagnose or
power-cycle anything. fv-ml1's four PSUs do not help -- PSU redundancy protects
against a PSU dying, not against the circuit going away.
Three things are explicitly NOT measured and the file says so: fv-ml1 under real
4-GPU load, whether the BMC reports AC input or DC output, and the R420's actual
draw. Treat 1625 W as a floor.
Also corrects the hardware section, which claimed 2x GPUs. nvidia-smi reports
four.
Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.
Gate design was pre-registered before any generation existed (0bb4938). Three arms, 60
held-out beats, 4 seeds, 240 generations per arm.
A. VOICE PASS 6.4x +0.413 delta_cb, pairwise floor 0.064 -- and it clears the OLD
all-arms floor (0.113) too, so this verdict does not lean on the
rule change. Closes 73.8% of the span between the unadapted
carrier and held-out Hemingway itself; lv-bronte closed 48%.
B. NOT COPIED see below
C. NO DAMAGE PASS ran-on +0.08, on-beat -0.14, both inside a 0.217 floor
AXIS B: THE NEGATIVE CONTROL WAS THE WRONG ONE, AND FIXING IT MADE THE RESULT WORSE, NOT
BETTER. memorization_check.py uses the base-unadapted arm as its control. Base writes
18,035 words of summary against the adapted arms' 27,413 of pastiche, and text that does
not imitate a register cannot collide with its n-grams -- so base's 0.00 measures "different
register", not "did not memorise". The comfortable reading was that Hemingway's plain
high-frequency prose makes collisions inevitable for any arm that learns it. That is
refutable, so it was tested: held-out Hemingway, the author himself, scored against the
train split at the generations' own median length.
HELD-OUT HEMINGWAY (never trained) 370 chunks 0.01 hit-rate mean-longest 0.1 max 10
base-unadapted 240 gens 0.00 0.0 0
ckpt850 (shipped) 240 gens 0.07 0.6 9
positive control (train vs train) 160
The hypothesis is false: the adapter reproduces train n-grams ~7x more often than the
author reproduces himself. That is real and is on the record. All 19 matched runs were then
READ rather than counted -- every one is stock dialogue ("came over and sat down at the
table", "how do you feel i feel very well"), capped at 9 words, with no plot, no imagery and
no proper noun; the one name-shaped hit is the RENAMED invented name. Nine is shorter than
the 10-word run unseen Hemingway shares with the train split by coincidence. Elevated rate,
zero protectable content. Hemingway is in copyright; lv-yarros is the in-line precedent,
also in copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s.
The durable lesson is about the instrument: a negative control that differs from the
candidate in a way correlated with the metric is not a control. memorization_selfsim.py and
memorization_dump_matches.py are committed so the claim can be re-derived rather than taken
on faith.
SHIPPED ckpt850, NOT the loss minimum at step 1750. The two are indistinguishable on voice
-- 0.072 apart against a 0.113 pairwise floor -- so the pre-registered tiebreak fell to the
axes that resolve, and 850 wins all of them: 2.3x tighter seed spread (0.050 vs 0.113),
lower memorisation, less ran-on, half an epoch less overfit. ckpt1750's spread is one seed
(0.491, 0.449, 0.468, then 0.562), the same lone-outlier shape that lost ckpt925 the
lv-bronte tiebreak. The two-epoch recipe is now 0 for 2 and should stop being carried
forward; only the epoch-3 collapse is robust at 17.4x jitter.
servers/fv-ml1/ssh-target was a bare IP, so deploy-stack.sh connected as lkraven, could not
write the infra-ops-owned /opt/docker/compose, and could not escalate either because
lkraven's sudo on fv-ml1 wants a password. Now infra-ops@10.251.50.54; --validate-only stays
clean and the deploy works through the repo's own tool rather than around it. Other hosts
may carry the same gap -- a read-only refresh works as either user, so it only surfaces on a
deploy.
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".
vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.
Committed utilization per card is the number that governs, not free bytes:
GPU 0 0.40 + 0.48 = 0.88 ~13 GB spare <- moved here
GPU 1 0.52+0.24+0.10+0.055+0.03+0.03 = 0.975 ~4.3 GB
GPU 2 0.96 ~1.8 GB
GPU 3 - kept empty as reserve
GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.
Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.
Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card
and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry
the vLLM seats at 84-95.5 GB of 96.
Changes:
- compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used
`count: all`, which would have handed a 0.6B ASR seat all four cards);
join traefik-net; port 8300; homepage href to the live FV address.
- .env.example: default to the v3 int8 model (25 European languages, 464 MiB)
rather than English-only v2; models to /tank/parakeet/models.
- app.py: warm the recognizer at startup before uvicorn accepts traffic.
The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes
lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at
45.1s on a second container) against ~0.50s warm. A 45s first request is
indistinguishable from a hang and LiteLLM's default timeout abandons it long
before it returns. Decoding 1s of silence at load moves the cost inside the
healthcheck's 300s start_period; first real request after restart is now 0.65s.
Verification, because "provider=cuda" in the log is only an echo of the env var:
ORT falls back to CPU silently and still returns correct text, so the service
being up and the transcript being right establishes nothing. The discriminator is
a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS
sentence transcribes near-exactly (positive), 3s of digital silence returns
empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread
0.47-0.65s, single-stream, one clip: a smoke measurement with its harness
stated, not a benchmark.
Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1`
(OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's
Postgres store where the ext-tts family already lives — no gateway restart, and
config.yaml is consequently not a complete picture of what the gateway serves.
Both verified end to end.
The aliases use a raw IP deliberately: ana-docker resolves no .internal names at
all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a
hand-pinned extra_hosts entry. A second hosts entry would mean recreating the
container and bouncing the gateway for every consumer.
Also records the svos_miranda plugin validation pass and its structural findings,
and notes that the irv-ml1 parakeet is still running — there are two now, and
retiring the old one is the operator's call.
nh3-dev sits on the NH3 LAN and reaches every site through its own default
gateway; RouteAll was already false, so it never used the tunnel for routing.
Membership bought a 100.64.0.4 address nothing referenced -- grep across the
repo and ~/development found only docs and memory hits.
It also cost something concrete. A host running Tailscale installs
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP, and because the fleet's
subnet routers preserve source rather than masquerading RFC1918, a mesh client's
packet reached nh3-dev's ens18 still sourced 100.64.x and was dropped silently.
That is why nh3-dev.nh3.internal failed from the mesh while every NH3 host that
does not run Tailscale worked, and it needed a -d 10.100.10.50/32 -j MASQUERADE
exception on nh3-scale to paper over. Retiring the membership removed the
anti-spoof rule, so the exception went with it -- mesh-exit-masq.sh is back to
the two rules it had before yesterday.
Verified after: nh3-dev reachable at 10.100.10.50 from ESH, Anaheim, FV, Irvine
and NH3, and reaching all four sites plus the internet itself. fv-ml1
unaffected.
The mesh is now six nodes and every one has a job: three site routers,
vb-gateway, irv-ml1 (Irvine's own router, no separate scale node), and the
operator's MacBook Air. Nothing is enrolled just in case.
Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's
mesh membership was redundant for routing and its only remaining value was as a
second way in. Keeping it enrolled bought a standing second door; joining on
demand buys the same recovery path without one.
normal tailscaled stopped + disabled; fleet reached via the gateway SNAT
fault nh3-dev / nh3-docker unreachable while the WAN is up
action start tailscaled + tailscale up -> reachable at its 100.64.x address
fv-ml1 is now off the mesh and its node record deleted. Verified it still
reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then
the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10),
answered ping and ssh from nh3-dev, and was closed again cleanly.
No auto-leave, deliberately: once open the door stays open until a human runs
systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps,
and a flapping recovery path is down exactly when someone finally looks. It also
skips entirely when already on the mesh, which is what makes it idempotent after
firing.
The question exposed a hole worth more than the redesign. The stored rejoin key
was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass
credential that dies in four days and fails silently at the only moment it
matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires
2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host.
That also closes the standing self-join risk rather than trading it: the two
stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live
reusable key -- purpose-built, on a host we control -- instead of two orphans
nobody owned.
Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14
tailscale up --accept-routes on this box accepted its OWN subnet from the
gateway and black-holed it. That happened with a human watching; here it runs
unattended, during an incident, on a box already in trouble.
FV could not reach any site but Anaheim. The cause was a single outbound-NAT
rule on the FV gateway, added 2026-09-13 and scoped to Anaheim only --
docs/runbooks/fv-to-ana-nat.md says so in as many words: "Other remote sites
remain outside this fix's scope." Three mirrors added, same interface and
source, only the destination differing: 10.100.0.0/16, 10.0.0.0/16 and
10.6.110.0/24.
After: fv-ml1 reaches NH3, ESH, Anaheim, Irvine, the mesh and the internet.
Regression sweep clean across nh3-dev, nh3-docker and esh-docker-vm.
The runbook now records what the failure looks like, because it presents as a
routing or Tailscale fault and is neither. fv-ml1 reached mesh addresses
perfectly and LAN addresses not at all; the FV firewall log showed the outbound
passing with src=10.251.50.54 and no reply returning; temporary counting rules
proved nh3-scale received 5 packets and sent 4 replies; both peers' AllowedIPs
were correct. The discriminator that settles it is that every other site pair
works -- nh3-docker to esh/ana/FV and esh-docker-vm to FV all succeed -- so a
general subnet-to-subnet limitation is ruled out and only outbound SNAT is
left.
Also reverts the remote-site MASQUERADE rules added to nh3-scale earlier on the
asymmetric-return theory. They fired but were not the fix, so they are removed
rather than left to accumulate as NAT that achieves nothing.
Applied via source_nat/add_rule + apply with a pre-change config backup taken
first. Source scope is still fv-ml1's /32, so a second FV host will hit this
again -- flagged in the runbook.
WATCHDOG (done, proven). fv-mesh-watchdog probes two independent anchors every
minute and, after 5 consecutive failures, puts Tailscale back to known-good:
accept-routes off, re-up against headscale with a stored key. It touches
nothing else — a watchdog with a wide remit is a second way to lose the box.
Two anchors that cannot share a failure mode: a plain-internet one and a
mesh-only one. If BOTH fail the site uplink is down, Tailscale cannot fix that,
and it deliberately does nothing — thrashing tailscaled during an ISP outage
turns a wait into an incident. Disable file at /etc/fv-watchdog.disable for
planned work.
Proven by positive control, not assumed: counter incremented 1..4 without
acting, fired the restore at 5 (tailscale up ran, tailscaled restarted), and
reset to 0 once the real anchor returned. fv-ml1 stayed reachable throughout.
This exists because a on fv-ml1 black-holed it
from its own LAN earlier the same day: it accepted 10.251.0.0/16 from the
gateway — its OWN subnet — and routed the local network through the tunnel.
FV CROSS-SITE ROUTING (partial). Two changes landed, the path is still broken:
1. acceptSubnetRoutes 0 -> 1 on the FV gateway's tailscale plugin, via
settings/set + service/reconfigure (the documented apply, not a reboot).
The GATEWAY now has 10.0/16, 10.100/16 and 10.250/16 in its routing table
and reaches NH3 and ESH itself. It could not before.
2. Remote-site MASQUERADE rules on nh3-scale. The existing jump matched only
-s 100.64.0.0/10, so traffic from another site's LAN never entered
MESH-EXIT and kept its original source; an NH3 host then replied via its
own LAN router instead of back through nh3-scale, making the path
asymmetric. The rule is confirmed firing (counter increments on FV
traffic) but does not complete the path.
Still failing: fv-ml1 -> NH3/ESH LAN addresses. Mesh addresses work perfectly
from fv-ml1 (100.64.0.1, 100.64.0.4), Anaheim works over the metro link, and
the FV firewall log shows the outbound passing on tailscale0 with
src=10.251.50.54 and no reply ever returning. The remaining gap is forwarded
FV-LAN traffic specifically, not the gateway's own.
Full regression sweep clean: nh3-dev, ana-docker and esh-docker-vm all reach
all four sites plus the internet.
One rule on nh3-scale (CT 107): -d 10.100.10.50/32 -j MASQUERADE, above the
RFC1918 RETURNs in /usr/local/sbin/mesh-exit-masq.sh, so it survives a reboot
rather than living only in the running ruleset.
Cause. A host that runs Tailscale installs -A ts-input -s 100.64.0.0/10
! -i tailscale0 -j DROP. The fleet's subnet routers run NoSNAT: true with
RFC1918 explicitly exempted from masquerade — deliberate source preservation,
and a departure from Tailscale's own --snat-subnet-routes=true default — so a
mesh client's packet reached nh3-dev's ens18 still sourced 100.64.x and died
at the anti-spoof rule. Every NH3 host that does not run Tailscale was
unaffected, which is why this read as a DNS or routing fault rather than a
policy one. Masquerading just this destination makes it behave like every
other host and leaves source preservation absolute elsewhere.
Verified before and after against 13 targets from nh3-dev and 9 from the
MacBook Air, and again after restarting the service so the chain was rebuilt
from the script rather than from the manual insert. nh3-dev.nh3.internal now
resolves and connects from the mesh, ssh and the Booth port included, with no
script changes anywhere.
Records the failed approach prominently, because it is the attractive one:
advertising 10.100.10.50/32 from nh3-dev itself black-holed it from ESH,
Anaheim, FV and Irvine. ip rule there puts lookup 52 at priority 5270 ahead of
main at 32766, and becoming a subnet router let table 52 capture cross-site
traffic the node has no accepted route for. Its own LAN and the internet kept
working throughout, so a single-host check confirms a break it cannot see.
svos-dev split their Bifrost wall's HS256 signing key off the Hermes Bearer
(svos main 7165272), so nh3-dev/hermes/api-server-key is free to rotate again.
The previous note said do-not-rotate and would have made a future session
refuse a legitimate rotation on stale grounds.
Not rotating now: the key was minted today, is vaulted, and has never been
exposed — rotation is a hygiene action with a trigger, and none applies. What
changed is the capability, which is what the record needs to reflect.
Also records two things for when the svos_miranda plugin arrives: it will
reference the dispatch key rather than the Bearer (expected, not a defect), and
its tools array is legitimately seven or eight entries because repo_read is
conditional on a config block SVOS owns. A third number is a real fault.
Operator ruling: root:docker, not a personal username and not a new admin
account. lkraven is one of three names he uses, so baking it into shared
infrastructure guarantees a stale owner later; a dedicated deploy account
buys nothing the existing docker group doesn't, since that group already
exists on every host holding exactly lkraven + infra-ops.
Applied to nh3-dev, nh3-docker, esh-docker-vm, irv-ml1, ana-docker. All
five now 2775 root:docker on /opt/docker and /opt/docker/compose. Clears
the 0777 on nh3-docker and ana-docker. 55 stack .env files normalized to
root:docker 0640, tightening 43 world-readable ones and opening 31 that
were readable by only one of the two deploy identities. No containers
bounced — inode metadata only, and .env is read at compose up.
Deliberately not a recursive chmod. Three acme.json files and an ssh
private key are mode 0600 and traefik/ssh refuse to start if that widens,
which would have been a delayed failure surfacing at the next restart
rather than now. Protection is both mode-based (0600/0400 untouched) and
name-based (acme.json, *.key, *.pem, *.pfx, id_*); modes are symbolic so
the 53 executable files in these trees keep their exec bit.
Two defects found and fixed mid-rollout. The name list was initially
reported but not enforced, so a .key already at 0644 on esh-docker-vm was
widened to 0664 — reverted, and the list is now enforced in the chgrp and
widening steps. And the exec-bit verify asserted every .sh is executable,
which was never true and false-FAILED irv-ml1; it now compares the
executable-file count against a recorded baseline.
The restart loop runs as the deploy identity, not root, and a stack .env is
allowed to be root-owned 0600. compose bails on the unreadable file before
doing anything, so the stack was reported FAILED while restart=unless-stopped
had already brought it back healthy — a false failure, which is worse than a
quiet one because it trains readers to skim the failure lines.
Retry under sudo -n before calling it a failure, and print compose's own
output either way. Verified on nh3-dev against beszel: plain attempt rc=1
'open /opt/docker/compose/beszel/.env: permission denied', sudo retry rc=0
'Container beszel-agent Started', container back to healthy. The happy path
is unchanged — the sudo attempt only fires after a failure.
Also record that tts-dev migrated talk from ~/talk into
/opt/docker/compose/talk, which removes the one stack on this host that was
invisible to anything walking that path.
Operator cleared the swap and ruled out a docker-compose v1 shim. Ran
playbooks/upgrade-docker-ce.yaml: docker.io 20.10.24 -> docker-ce 29.8.0,
docker-compose 1.29.2 -> compose plugin v5.5.1, containerd 1.6.20 ->
containerd.io 2.3.5, buildx v0.37.1 added. 12 changed, 0 failed, verify
4/4. talk and beszel-agent back healthy on their restart policies.
The pre-state was worse than 'old': there was no cli-plugins directory, so
'docker compose' was not a command and exited 0 on a help blurb — a silent
no-op that reads as a successful deploy.
Records two things the run surfaced. vastblue-u5-pg and its anonymous
volume were removed when the old daemon stopped; the playbook has no rm,
prune or purge and five other containers survived, so the cause is almost
certainly --rm, unprovable now that the record is gone. It was measured
beforehand as zero user tables in every database, so nothing was lost. And
the playbook's restart loop runs as infra-ops and cannot read a root-owned
0600 stack .env, so it false-FAILs that stack.
Also notes that nh3-dev is the only host where /opt/docker/compose is
root-owned; the other four are lkraven. Created /opt/docker/compose/talk
as lkraven so tts-dev can move talk out of ~/talk. Normalising the parent
is left to the operator.
Operator ruled the repoint; Miranda moves off the paid z.ai Coding Plan
onto free local compute. model.default gen-large, provider custom,
base_url http://10.250.50.70:4000/v1. Verified by a real turn rather than
by config: hermes status reports gen-large / Custom endpoint and a
completion through /v1/chat/completions returns 660 tokens. The
openrouter/nous credit warnings cleared with it.
Records the landmine found on the way: CUSTOM_API_KEY and
HERMES_CUSTOM_API_KEY are inert for bare provider: custom — they bind only
a named custom_providers entry through its key_env. Without model.api_key
the request ships the placeholder no-key-required and LiteLLM 401s inside
the response body while hermes status still reports a healthy gen-large /
Custom endpoint, so status alone cannot verify this change.
Also notes that nh3-dev/hermes/api-server-key must not be rotated until
SVOS splits its HS256 signing key off the shared value.
Set platform_toolsets[api_server] = [] in ~/.hermes/config.yaml. Measured
back on /v1/toolsets: 28 rows before and after, enabled 14 -> 0, registered
tools 40 -> 0. The row count surviving matters — SVOS's _hermes_roster
derives its required-config line from the full report, so narrowing the
surface does not blind the diagnostic. terminal, code_execution, file,
browser and computer_use are now unregistered rather than merely
unreachable.
Separately, record an unresolved finding raised by svos-dev: model.default
names anthropic/claude-opus-4.6 and model.base_url names openrouter, but
provider: auto with only a zai credential resolves Miranda to GLM-5.3 on
the paid z.ai Coding Plan. Nothing under ~/.hermes references the LiteLLM
gateway. The operator believes Miranda runs on gen-large, which does exist
on LiteLLM. Not repointed — the premise is his to confirm.
Stood up hermes-gateway.service (user systemd, vendor 'hermes gateway
install') for SVOS/Miranda, which cut over from Worldtree to Hermes on
2026-09-11 and aborts at boot without it. Bearer key vaulted as
nh3-dev/hermes/api-server-key.
Notes the wide default tool surface: with no roster config the gateway
registers all 28 toolsets on the api_server platform, terminal and
code_execution included. Loopback bind plus the key are the only
containment until SVOS installs its svos_miranda plugin and the
platform_toolsets / disabled_toolsets roster is applied.
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16,
and serving inference again. This lands the repo half of that.
Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq
reservation so the address the runbook, DNS and LiteLLM all assume is the
address it actually has. Its headscale node is renamed too.
The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the
reason this diff touches current-state files and not the record. Dated
persistent-memory entries, archival-memory and incident notes still say
ana-ml2 in 31 and 62 places respectively, because that is what the box was
when those things happened. Rewriting them would make the history lie.
LiteLLM was the load-bearing piece and needed more than the api_base sed the
runbook describes. Twenty api_base entries repointed, but a grep-and-verify
pass also caught a LIVE pass_through_endpoints target for the scalar-judge
reward route still on the old address -- an api_base-only substitution would
have left it dead. Four prose references describing current state were
repointed as well; one historical note recording where a hand-test was run
is deliberately left pointing at 10.250.50.54.
Two facts in the server tables were wrong and are corrected here. The site is
Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell
Max-Q, not two -- verified by nvidia-smi -L and independently by PCI
enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196,
which changes what fits on it.
DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback
approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all
three resolvers. The BMC record carries a warning that its 802.1q VLAN tag
must stay disabled -- it shipped tagging VLAN 250 into an untagged port,
which made it invisible to every network-side diagnostic and is the reason
it appeared dead through several cable changes.
Verified end to end: summarizer and sec both answer through the Anaheim
gateway across the mesh to FV seats on different ports.
ssh exiting 0 is not proof the capture is usable — the inspect script can emit
nothing and both refresh scripts would mv that over a good system-details.txt and
report 'ok (0 bytes)'. Every reader tests the snapshot with -s, so the writer was
producing an artifact its own readers call invalid: a guard whose test disagrees
with its writer's contract has quietly stopped guarding. Prompted by
brokkr-smithy-dev hitting the same shape from the other side (a -s test against a
sentinel written with touch, a precondition that could never pass).
- empty capture -> refused, previous snapshot kept, host counted as failed (exit 1)
- capture under 1/4 of the previous -> promoted but flagged, since a host can
legitimately shed services and the script should not guess
- header + CLAUDE.md contract lines corrected to say what is actually guaranteed
- verified red (empty inspect -> FAIL, snapshot intact, rc=1) then green (real host
-> ok 6727 bytes)
The 2026-09-06 headscale cutover retired irv-ml1's wg0 tunnel IP 10.100.79.3
(now 10.6.110.50). Repointed all LIVE canonical refs to the DNS NAME so the next
move can't re-break them: homepage.href/siteMonitor labels across 25 stack
composes, load-bearing env defaults (asset-engine INFERENCE_HOST, open-webui
AUDIO_TTS_OPENAI_API_BASE_URL, skaldsong SKALDSONG_TTS_BASE_URL, zonos-gateway
ZONOS_URL, dia), homepage services.yaml manual cards (Voice Design Studio,
IRV-ML1), and servers/irv-ml1/ssh-target. Updated the stale 'WG tunnel' comment
to the mesh reality.
Left as-is: README curl-examples and .env.example comments (docs), and historical
mentions in CLAUDE.md/persistent-memory. NOTE: applying the label repoints to the
RUNNING irv-ml1 containers needs a recreate per service (labels read at creation);
deployed .env values are separate from these canonical defaults.
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
pfi-gx10 moved from desk Wi-Fi (10.100.10.226, VLAN 10) to the rack on
10.100.50.60 (nh3-servers, VLAN 50), reachable as pfi-gx10.nh3.internal.
The address is a DHCP RESERVATION on the UDM, not a host static. Operator
ruling during the move, and the better design: a host-side static works
until the box moves and is then a stale netplan file on a machine whose
address you no longer know. The pre-written playbook wrote a host static;
it is kept for its safety ordering and annotated as retired.
The port arrived on the native VLAN, not the server VLAN, so switch port 22
was repointed first. port_overrides is a whole-array PUT — two unrelated
overrides on ports 21 and 23 were read, preserved and written back, with the
original array backed up to a file before the change.
Wi-Fi stayed up as the escape hatch until the wired path was proven from
outside, and was downed last as its own step. The step worth keeping: while
Wi-Fi was up, traffic to nh3-dev still preferred wlP9s9 because that
interface sits directly on the userland subnet, so reachability proved
nothing about the wired path. `ping -I enP7s7` across the VLAN boundary is
what actually settled it before the hatch came down.
Target settled: nh3-servers VLAN 50, static 10.100.50.60. Clear of the four
existing statics and below the .150 DHCP pool where fleet statics live.
The playbook never leaves itself one path back. Wi-Fi stays up throughout while
the wired interface is configured beside it; the new address is verified from
outside before anything is torn down, and Wi-Fi teardown is explicitly a
separate later change. A botched netplan therefore costs a retry over Wi-Fi
rather than a trip to the rack — which is what substitutes for 'netplan try',
whose interactive rollback needs a TTY that elway cannot provide.
Two preconditions are asserted as steps rather than assumed: the interface must
have carrier (writing a static config for a dead NIC and reporting success is
the failure this avoids), and its MAC must match, since interface names can
renumber across kernels but MACs do not.
Requires nothing from the operator beyond racking the box. The wired NIC has a
distinct MAC from the Wi-Fi one, so the post-move address and switch port are
both discoverable from the UDM rather than needing to be relayed.
NVIDIA GB10, aarch64, 121 GB unified, sm_121. Ships booting to graphical.target
with GDM and GNOME Remote Desktop running.
playbooks/gx10-headless.yaml sets multi-user.target, stops gnome-remote-desktop,
masks the sleep/suspend/hibernate targets, makes logind ignore lid and idle, and
adds sshd keepalives so a stalled link does not kill a long-running job.
Two things learned the hard way and recorded in the playbook:
- gdm is a STATIC unit on Ubuntu, pulled in by display-manager.service and never
'enabled'. A guard of always skips, and a verify
written the same way passes while the desktop is still running. Both now test
is-active. The first run reported six green verifies having not stopped gdm.
- elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as
the connecting user and must carry their own sudo; connect as infra-ops.
The playbook refuses to stop the display manager while a seat session is held,
overridable with --var force_dm_stop=true. Networking is deliberately out of
scope: the box is on a desk on Wi-Fi with a temporary DHCP lease and no ethernet
carrier, and belongs to the rack-install change.
Benchmarked selene against gen on selene's own job: 24 designed judge items
with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a
neutral JSON prompt and Selene's native Atla template. 288 calls, all free
local.
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
native Atla selene 21/24 (88%) gen 22/24 (92%)
gen won on both templates and selene's BEST sat below gen's WORST. Selene was
given its own fine-tuned template as a fairness check; it gained one point,
not the three it needed.
Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a
winner on every equivalent pair. For eval work that is the case that matters
most. gen returned tie correctly on the JSON template. Selene also compressed
the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only
win was ~3x latency, unexercised at ~60 calls/day with zero queueing.
TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately:
- chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says
consumers bind the capability, not a concrete model. Sampler profile copied
from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served
config matches the benchmarked condition.
- selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A
served-name is a contract about what the model IS; answering it with a
different model hides a material change behind a stable string. Operator
ruling: "never repoint a named model at a different model's endpoint --
that is intentionally misleading." Verified: the gateway now returns
HTTP 400 "Invalid model name" for it.
Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that
had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role
moved onto an existing seat rather than allocating anything new.
Canonical litellm config synced from the host; ana-ml2 README and
recommended-model-settings updated. compose.yaml kept for reference, not
deployed.
nevermore-claude handed hrafn to infra-ops for uptime ownership. Intake:
verified health independently, brought it into the inventory, and fixed one
security defect found during the check.
- canonicalize stacks/hrafn/ (compose.yaml + .env.example + README)
- list hrafn in the ana-docker running-stacks table
- README records the load-bearing bits: shm_size 1gb is required for
Chromium, playwright and the base image are version-coupled, SSRF denies
private targets by default, and restart:unless-stopped does NOT act on
healthcheck failure
Security fix applied on the host, not in this tree: /opt/docker/compose/
hrafn/.env was 0644 with a live 57-char bearer token in it, readable by
every local account (verified by reading it as `nobody`). Tightened to 0600;
owner unchanged so the deploy path still works, confirmed via `docker
compose config` as lkraven. Container untouched and still healthy.
Known gaps recorded rather than silently accepted: no CI deploy, and the
image builds from an rsync'd working tree living in the compose directory,
so the running image has no commit provenance.
The filezilla container on vm-esh-nas shipped with no restart policy. It
was SIGTERMed during the 2026-08-18 host shutdown and never came back,
staying down four days while dockge/dozzle-agent/beszel-agent all
recovered on their own.
- add `restart: unless-stopped` (applied on the host, container recreated
and verified serving 200 on :5800 with all four /NAS mounts visible)
- promote the previously un-canonicalized compose into stacks/filezilla/
so the policy is version-controlled rather than host-only
- record the missing-restart-policy failure mode in the host README
Records the deployment plus the finding that outlives it: no_speech_prob
alone does not carry a hallucination threshold on this stack. Silence and
room tone both hallucinated 'Thank you.' at no_speech_prob under 0.11,
where a conventional >0.6 gate passes them through; avg_logprob separates
~6x better. Any future consumer asking for a Whisper hallucination gate
should be pointed at the composite, not the single field.
Also banks the digest-pin rationale (the VAD flag is an upstream-declared
unstable variable name, so a floating tag can move gate semantics
silently), the two deployment gotchas, the docker-vs-native GPU index
inversion on this host, and Eyra's future diarization VRAM claim on the
same A6000.
Adds speaches to the irv-ml1 stack table alongside parakeet, with the
reason both exist.
esh-pve hard-froze at 03:34 on 2026-08-19 and stayed frozen ~4.5 hours
until a manual power cycle. No panic, no OOM, no MCE — the journal stops
mid-operation. The whole ESH site lost DNS with it, because esh-userland
(VLAN 10, the PVC SSID and wired userland LAN) was handed exactly one
resolver: 10.0.50.45, AdGuard on esh-docker-vm, on a different VLAN, with
no secondary. Internet and routing were healthy throughout.
Two fixes.
1. DNS: 10.0.10.1 (the gateway, verified resolving) added as secondary on
esh-userland via the UDM Classic API. Note this is degradation cover,
not clean failover — clients that query resolvers in parallel will
bypass AdGuard for a share of lookups.
2. Watchdog: softdog -> iTCO_wdt under systemd (RuntimeWatchdogSec=60),
watchdog-mux masked. The box looked watchdog-protected and was not: a
software watchdog cannot fire when the kernel it lives in is wedged,
and watchdog-mux only pets the device while an HA client is connected,
which never happens on a cluster with no HA resources. Firmware does
not block the TCO timer here, checked before committing to it.
Also pins VM 102 off (onboot: 0). It starts with full GPU passthrough and
vfio-pci enabling that device is the last thing the kernel logged, 39
minutes before the freeze. The other suspect is the kernel itself: the
host ran 4.5 months on 6.8.12-16, took 6.8.12-42 in an apt batch on
08-18, and died 20 hours into the first boot on it. 6.8.12-16 is still
installed as the rollback.
The playbook is idempotent — a second run skips all six steps and passes
all six verifies. The watchdog is confirmed armed (identity=iTCO_wdt,
state=active, held by PID 1) but has NOT been observed firing; proving
that needs a deliberate wedge.
Memory also corrects two wrong mid-incident calls: the mgmt VLAN is
routed over the site tunnel and is not firewalled off — both symptoms
were the dead host generating ICMP unreachables.
Two findings from chasing node `pve` showing dark in the UI.
esh-pve and esh-pve-nas are a 2-node cluster (esh-pve-cluster, expected
votes 2, quorum 2, no qdevice). This was undocumented, and last night's
migration rebooted one of the members without accounting for it. Nothing
broke -- quorum is intact and both nodes report the same ring id -- but
that was luck. Rebooting either node drops the survivor below quorum and
makes its /etc/pve read-only until the partner returns. It matters for
the pending confirmation reboot and the 225-package upgrade, both of
which take a node down.
The dark tile itself was NOT last night's doing. pvestatd SEGV'd on
2026-05-28 and had been dead 82 days; the journal has nothing between
that crash and the restart today. It is only the reporting daemon, so
the node stayed quorate and healthy with all services active and all
three guests running the whole time -- the UI simply had nothing telling
it the node was alive. Fourth SEGV in that unit's history, so treat a
recurrence as expected and consider a watchdog: nothing alerts on it,
and the sole symptom is cosmetic enough to go unnoticed for months.
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is
out of the runtime I/O path, so a bus reset can no longer drop root from
under a running hypervisor. All five guests healthy, three pools ONLINE,
system running, ext4 pve-root intact and unmounted as the rollback with
its own kernel and initrd. zfs-import-cache is now the active import
path -- the all-three-pools cachefile fix doing its job.
The window cost an unplanned outage, and the cause was this repo's own
tooling rather than the migration.
The staging chroot ran `mount --rbind /dev` and /sys with no
--make-rslave. On systemd `/` has shared propagation, so the cutover's
`umount -R` propagated back into the live host and removed the real
/sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind
could not create a session: ping fine, TCP fine, SSH authentication
succeeded, resident daemons kept serving -- and every new exec hung,
including /sbin/reboot, so the reboot never ran at all.
It impersonates failing root-disk I/O almost perfectly, and I called it
as the DOM dying. That was wrong. dmesg had the answer throughout: the
DOM attached cleanly with no errors, and the last log timestamp was
12114881s -- 140 days -- meaning this was still the original boot. A
down-detector had also never reported the host down, which I read as a
fast reboot rather than as no reboot.
Fixes and guards:
- --make-rslave after every rbind, plus a guard that refuses to proceed
while any chroot bind still reports shared propagation.
- Confirm a reboot by observing the host DOWN, not by watching for it to
come back. Those two states are indistinguishable otherwise.
- Blast radius now measured from the server: `ss` inside CT 103 found
five NFS clients, not the two documented. The new one that mattered is
esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on
purpose and it came through read-write.
- grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV
which GRUB reads but cannot write, so next_entry survived the boot
that consumed it. Steady state is saved_entry=pve-zfs-root with no
next_entry. There is no auto-fallback on this host and no IPMI.
Recovery needed no console: an idempotent cgroup2/devpts/shm remount
landed in the brief windows where exec succeeded. No data was lost, and
neither the DOM nor any pool was ever at risk.
Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.
Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.
Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.
Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:
- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
ext4 cannot shrink; freeing space from root needs a rescue boot, which
costs the one-reboot property. Space came from swap (768M -> 256M).
- The runbook's `zpool set cachefile=... nvme` would have broken the
NAS. Populating a cachefile flips the host from import-by-scan to
import-by-cache, so a one-pool cache leaves ssd and tank unimported --
and CT 103 esh-nas has twelve bind mounts spanning all three pools.
Set on all three instead, verified in the resulting cache.
- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
and rpool comes from grub-probe --target=fs_label, which returns empty
because GRUB's ZFS reader cannot open a pool with encryption,
large_dnode and zstd_compress -- the same feature set that forced /boot
to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.
The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.
PVE root on esh-pve-nas is a USB Disk-on-Module: 6 GB ext4 with the host's only
ESP. A DOM is SLC/pSLC so wear is not the driver -- the problems are that it is
on the USB bus (a reset drops root under a running hypervisor), has no headroom,
and is unmirrored while 928 GB of mirrored NVMe sits 96% empty.
Runbook targets a fresh PVE install to ZFS RAID1 across both NVMes. In-place
conversion is unsupported, and adding an ESP to the existing NVMes is impossible
-- both are whole-disk ZFS members with 1.7 MiB free and proxmox-boot-tool
manages nothing today.
The headline risk is not on the host being rebuilt: CT 103 esh-nas IS the NAS
at 10.0.50.50, and both esh-docker-vm and esh-pve mount it hard. Taking this box
down stalls esh-pve's storage layer and wedges esh-docker-vm into the D-state
whose only remedy is a host reboot -- the incident shape already on record.
Quiescing those clients is step one of the window, and the README now warns
against casual reboots.
Config snapshot captured off-box to nh3-dev (0600) with /etc/pve, network and
fstab config plus zpool/zfs/disk-by-id/guest state; the newest on-disk copy
before this was June 2024.
Two user-systemd services stood up on nh3-dev this session weren't in the
"What runs here" list:
- The Booth (:8090, booth.service) — eshpfi services/booth ephemeral media board.
- jackdaw-compose (:8787, jackdaw-compose.service) — JackDAW /compose AI backend
hosted for jackdaw-dev, Origin-gated, fronted by the :4500 bench proxy.
Keyless entries (the jackdaw unit's key stays server-side, uncommitted).
The README's running-stacks table had drifted well behind reality (still listed
llama-swap + only the embed/rerank/reward trio). Regenerated system-details.txt
and rewrote the stacks + GPU-allocation sections from a live docker ps +
nvidia-smi (2026-07-22):
- GPU 0 (hot): vllm-aeon-gen (qwen3.6-35b-a3b-heretic, NVFP4) + vllm-charrp-
reasoning-nvfp4 (char-rp-reasoning, NVFP4), ~42+45 GB.
- GPU 1 (on-demand): granite-4.1-8b, selene-1-mini-8b, Skywork reward,
Qwen3 embed/rerank, and the Magidonia-24B char-RP GGUF (llama-charrp), ~91 GB.
- Recorded the dormant on-disk stacks and llama-swap's retirement.
NH3 manager/external-dev box at 10.100.50.42 (Debian 13 VM on nh3-pve),
successor to the retired nh3-ansible. infra-ops identity here is sudo-LESS
by operator decision (2026-06-17): key-only, no NOPASSWD, not in docker
group — user-level management only. Adds servers/nh3-extdev/{README,
ssh-target,system-details.txt}, the CLAUDE.md inventory row, and a local
ssh alias (nh3-extdev -> infra-ops@10.100.50.42, infra-ops key). Login +
sudo-less posture verified.
Append 10.100.10.50:8391 to BIFROST_CLIENT_ALLOWED_HOSTS on the personal
Worldtree (.env) so the consumer may bind the memory provider at session-create
(affect :8390 was already listed; the url-guard 422s un-allowlisted endpoints).
Idempotent elway playbook; surgical worldtree-api recreate that auto-derives the
image pin from the matrix sibling to avoid the stale-:latest crash-block footgun.
Repoint servers/corviduo-dev/ssh-target to infra-ops (operator granted durable
NOPASSWD admin on corviduo-dev 2026-06-15).
ana-ml2 was upgraded 2026-06 from dual RTX 6000 Ada (46GB, cc 8.9) to
dual RTX PRO 6000 Blackwell Max-Q (96GB, cc 12.0 / sm_120). Update the
stale hardware facts across the workspace:
- CLAUDE.md servers table row
- servers/ana-ml2/README.md hardware spec (+ refreshed system-details.txt)
- stacks/vllm compose + .env.example FP8/KV comments (Ada cc 8.9 -> Blackwell cc 12.0)
- stacks/llama-swap config VRAM-budget comment (48GB -> 96GB, GPU-0 pin)
Also corrects the adjacent stale 'Phi-4-mini' comment in the granite
service block (the service has been Granite 4.1 8B since 34a43a0).
Doc/comment-only; no runtime change.
- migration-plan.md: workstation-only migration of nh3-dev's dev env to
the M2 Ultra Mac Pro Rack (on-subnet, sidecars stay on Linux).
- nh3-dev README: drop volva from live sidecars + add retired note.
volva.service/heid.service removed by infra-ops at heid's request
(re-architected daemon->CC-orchestrator; volva.service was crash-looping).
Reflects the shared install: browsers in /opt/ms-playwright (root-owned,
world-readable) + PLAYWRIGHT_BROWSERS_PATH wired globally via
/etc/environment + profile.d, so any user/project resolves the shared
binary without a per-home playwright install. Includes the add-a-browser
command for future bumps.
Adds the NH3 dev box (10.100.10.50) as a managed host: README, first
system-details snapshot, ssh-target. Adds it to the fleet bootstrap's
Tier 1 and the CLAUDE.md servers table. infra-ops identity bootstrapped
there (operator-run) — NOPASSWD sudo + key, so root-level infra work on
the box (it runs the egress proxy, ttyd seat, mead-hall, volva, and dev
checkouts) no longer needs a per-task password.
First use: installed Playwright headless-Chromium system deps + binary
for bloom_music's OSMD browser-geometry test harness; headless launch +
real SVG geometry verified.
worldtree-dev sent infra-ops a manual deploy request for v0.30.6 to
personal:8081; operator confirmed all three corviduo-dev Worldtree
deployments go through the Worldtree team's CI/CD, never a manual
infra-ops deploy. The old 'personal — manually managed' line was wrong
and misleading. docker-as-root pattern is emergency-ops (key restore)
only.
Made the host-stacks bind-mount path configurable via
DOCKGE_HOST_STACKS_ROOT (default /opt/docker, unchanged for the
existing five hosts). Override on corviduo-dev to /home/vh/docker
because that host's /opt/ is owned by deploy:deploy (Worldtree team)
and vh lacks passwordless sudo for the fleet-standard path — same
reasoning as the beszel + dozzle agent placement earlier today.
Deployed to corviduo-dev. Reachable at http://10.250.50.152:5001
(first probe 200 — Docker's port-mapping route through iptables
worked without firewall changes, unlike beszel's network_mode: host).
Scoped to PFI-managed stacks only (/home/vh/docker/compose/) — does
NOT see /opt/worldtree*/ deployments. Keeps the management boundary
clean: dockge can restart/recreate PFI's beszel+dozzle+itself but
not the Worldtree-team-owned containers.
Operator-approved fleet monitoring extension. Both agents up + healthy:
- beszel-agent on host port 45876 (KEY-mode, hub at ana-docker:8090
SSH-polls inbound; seeded with hub's ed25519 pubkey).
- dozzle-agent on host port 7007 (mTLS auto-generated; hub at
ana-docker:8088 connects inbound).
Compose lives at /home/vh/docker/compose/{beszel,dozzle-agent}/ rather
than the fleet-standard /opt/docker/compose/{...}/ because corviduo-dev's
/opt/ is owned by deploy:deploy (Worldtree team) and vh lacks
passwordless sudo to create the fleet path. Functionally identical;
documented in servers/corviduo-dev/README.md so future infra-ops
sessions find them.
Created an empty traefik-net external docker network on corviduo-dev
as a side effect of dozzle-agent's compose (which declares it external).
Future PFI services landing here can reuse it.
Dozzle hub on ana-docker had 10.250.50.152:7007 appended to
DOZZLE_REMOTE_AGENT (a host-side change to a non-tracked .env, not
canonical). Beszel hub still needs corviduo-dev added via the UI's
"Add System" action — one-time operator step, flagged in the corviduo
README.
vh