MeshCentral only pushes CIRA through an on-host agent, so this mirrors its
amtmanager.js sequence directly over the LAN: trust MeshCentral's root,
add the MPS (username = 16-char meshid prefix), a periodic policy,
BIOS+OS user-initiated connections and a random environment-detection
domain. Idempotent, with a read-only state report. Enumerate uses
Enumerate+Pull (AMT 16 ignores OptimizeEnumeration; caught by a
positive control that first read 0 instances of a class that has one).
Applied to nh3-pve's AMT 2026-10-02 alongside MeshCentral mpsPass and an
ana-gw VIP/policy for 4433; the AMT does not dial out yet (static IP).
Prime grew VM 102's scsi0 from 250 to 378 GB after the root alert hit 85%
twice in 14 h. The swap partition sat right after sda1 and blocked growth,
so the playbook moves swap to a 4 GB /swapfile, deletes sda5/sda2, grows
sda1 in place (start sector unchanged) and ext4 online, and sets initramfs
RESUME=none so boots don't wait for the vanished swap. Root is 372 GB, 58%.
Prime 2026-09-30: move scriberr to GPU 3 and extend the Jev endpoint to 32k tokens.
Scriberr holds 0 VRAM idle; verified a 20-min job on GPU 3 at 5,496 MiB. With GPU 1
freed, intern-decision's measured card peak at MAX_TOKENS=32768 is 15,220 MiB against
a 15,437 MiB budget (n=3, 1 and 16 questions); 32,769 tokens is refused 422 up front.
JevBench v1.2.16 via /v1/systemone unchanged: 202/231, 0 diffs vs the bench.
intern-decision deployed 0941 PT (cap 9.0 GiB, MAX_TOKENS 7168). Live acceptance: positive
control 240/259 and Wyrd 79/84, bit-identical to the bench (0/560 rows, Δp 0); negative control
10/122/14; largest accepted requests 200 with no 503; per-process 8,812 MiB at rest and 9,866 peak;
GPU 1 Free 15,442 before and 6,581 after (lowest 5,569 under load). Latency from nh3-dev:
21 criteria 114 ms, 16 over ~3,900 tokens 238 ms. semif README banner now REPLACED with the
rollback; fv-ml1 GPU 1 note updated.
Stopped (not removed) 2026-09-30 0135 PT. Scriberr's Parakeet path hardcodes
5-minute slices; attention memory is quadratic in slice length, so a long file
needs >6 GB and hit CUDA OOM with SemIf resident (~6.7 GB free). GPU 1 now
81,806 MiB used. Durable fix (shorter scriberr slices) deferred to later.
Uses linuxserver/blender (Selkies Wayland desktop, NVENC), digest-pinned. The
container sees only GPU 3, has restart "no", and runs only while in use,
because GPU 3 is the reserve card for a full-size vLLM seat. Web desktop on
:3001 with basic auth (vault fv-ml1/blender-web-password). /work is on /tank
and is not backed up; /config lives under /opt/docker (restic).
Acceptance: Cycles finds the card on OptiX and CUDA (sm_120 kernels ship in the
build). The heavy self-test renders in 3.54 s on OptiX vs 22.71 s on CPU, a
functional check with n=1. Web auth answers 401 without credentials and with a
wrong password, and 200 with the right one.
The shim holds 10.0.50.47/24, which gives two equal connected 10.0.50.0/24 routes,
and ens18's wins. Host-to-HA traffic therefore left via the macvlan parent and was
dropped. HA lost MQTT to the broker on this host on 2026-08-19, 2026-09-21 and
2026-09-25 (the last lasted two days). This adds an ifupdown if-up.d hook that
routes 10.0.50.46/32 via macvlan-shim; /etc/network/interfaces is not edited.
Verified: the route resolves via the shim, the host pings HA, HA reaches :1883,
and HA reconnected to the broker. Diagnosis by ha-dev.
Also corrects the zigbee2mqtt acceptance note, which had wrongly reported HA as
connected.
Requested by ha-dev; approved by Prime in this session. Radio: SLZB-MR1U chip 0
(EFR32MG21, EmberZNet 8.0.2) at tcp://10.0.90.10:6638, adapter ember. A fresh
network was formed on channel 25, PAN 0xCFF4. The broker is reached as mosquitto
user zigbee2mqtt, with HA discovery on homeassistant/. The frontend on :8099 is
token-protected.
State and the network key stay host-only in /opt/docker/data/zigbee2mqtt
(root 0700, restic). The repo carries only compose, .env.example and the README.
configuration.yaml refers to the secrets as !secret.yaml, and those references
survived Z2M's v4->v5 settings migration.
services/semif-serve is a FastAPI wrapper around SemIf's direct and shared torch
scorers (SemIf-OpenJev @ 23cf1f39, MIT). Upstream ships only a batch CLI. The
wrapper loads the pinned Qwen3.5-4B (851bf6e8, BF16) once from the offline HF
cache and returns SemIf's result dicts unchanged, with an optional per-workload
temperature-calibrated view. Contract: semif-serve.contract.md. Built with a
short contract, TDD (39 tests, fake engine and fake torch, no GPU) and a heid
bug-hunt panel (pending).
On the card:
- torch 2.10.0+cu128 with sm_120 kernels, which is SemIf's own stack;
- a hard 12 GiB VRAM cap.
Two defects surfaced only on the card, and each fix is covered by a test:
- 0.1.1: an OOM raised as a chained exception kept the failed request's tensors
alive (11.9 GiB after the 503). It is now raised unchained, after gc.
- 0.1.2: a large request left 12.6 GB reserved on the shared card. After each
call, reserved memory over the baseline + 512 MiB is now released.
Acceptance against SemIf's committed torch predictions (authored144):
- 142/144 same top choice; both misses are exact bf16 ties;
- 144/144 identical prompt hashes;
- deterministic A-vs-A;
- negative control 14/144;
- shared vs direct 72/72.
21 binary criteria over one state take 159 ms. The shared-mode capacity table
under the cap is in stacks/semif/README.md.
The Dockerfile installs dependencies from a manifest with the project version
blanked, so a version bump reuses the ~4 GB torch layer. Verified: 41 s rebuild,
dependency layer CACHED.
DNS: semif.fv.internal. Token: vault semif/api-token.
Prime bootstrapped infra-ops on vm-esh-nas with playbooks/bootstrap-infra-ops-user.yaml.
It got the fleet-pinned uid/gid 850, NOPASSWD sudo with log_output, the docker
group and a 0700 home. That let playbooks/restic-repository-file.yaml migrate
the last restic host: the live profile matched the repo's pre-change sha,
its units no longer carry the URL, its secrets are vaulted, and the live and repo
profiles now match (a5ea75ea). All eight restic hosts are clean.
The staged helper script is gone, both from Prime's home on the host and from the
repo. The docs that described vm-esh-nas as lkraven-only are updated.
Prime removed the second instance after the v0.1.3 bench. esh-ml1 handles a face
in ~48 ms, sits in the house next to the cameras, and holds the verified
backup. fv-ml1's gallery was empty (0 identities). The container, gallery volume,
image, compose dir (with its .env), backup dir and build sources are removed from
fv-ml1. GPU_ID / CARD_SUFFIX stay in the compose for any future second host.
Prime asked for augaman on fv-ml1's utility card, beside vllm-coder. Mirror
augaman-dev's f77164f compose, which parameterises the GPU reservation (GPU_ID,
default 0) and the Homepage card name (CARD_SUFFIX). esh-ml1's resolved config is
unchanged: same config hash, no recreate.
On fv-ml1: augaman:0.1.2 built on-box from the tag, GPU_ID=1, healthy on CUDA
at 1264 MiB, and pytest -m gpu tests/vision passes 3/3 on the Blackwell. It has
its own gallery and no gallery backup, so it is fixtures-only. The host's raw
restic copy of /var/lib/docker/volumes is not a consistent SQLite backup.
docs/pfi/augaman-speed-bench/ holds the harness (augaman-dev's recipe plus a
no-face control frame and a face-count check on every response), the raw rows
and the summary. Server-side, one face:
- esh-ml1 GPU 144 ms
- fv-ml1 GPU 75 ms
- fv-ml1 CPU on 6 cores 152 ms
- esh-ml1 CPU 888 ms
It agrees with augaman-dev's independent esh-ml1 measurement once each
harness's floor is subtracted. This is the before for v0.1.3's detector fix.
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only
through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the
same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned,
sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily
0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC,
so both schedules name the zone explicitly.
pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure,
including a stopped container, aborts the run. Tested with a stub docker that
exits 1: the run returned 1, and neither the snapshot count nor last-success
moved. The restore was verified at identity level against augaman-dev's
public-domain canary (snapshot fd3061a1: the restored copy's digest over
identities and samples matches the live gallery). That meets the operator gate
for real enrollments.
The repository URL is read through repository-file rather than restic.env.
resticprofile schedule copies env-file values into world-readable systemd
units, which publishes the rest-server password on the env-file hosts
(observed on esh-docker-vm). This is recorded in the backups runbook under
Known gaps, and the playbook verifies no generated unit contains the URL.
esh-ml1 is added to the freshness check's expected ana-side repos and to the
runbook tables.
augaman moves to v0.1.2 (dependency layer keyed on the lock without the
project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the
card, and the canary survived the container recreate.
Mirror pfi/augaman deploy/compose.yaml as stacks/augaman, with an .env.example and
a README carrying the biometric backup gate. The image is built on esh-ml1 from a
git archive of the release tag, because the box holds no gitea credentials.
Serving on CUDA and visible in nvidia-smi. The gallery backup is not wired yet
(esh-ml1 has no restic), so only public-domain fixtures may be enrolled.
The on-box gpu test fails its batch-vs-single tolerance 3/3; reported to
augaman-dev, who owns the contract.
For ha-dev (operator-approved 2026-09-26). CT 111 on esh-pve at 10.0.90.20:
Matter/Thread IPv6 (Echo ULA + RA route-information) is link-only, so the
server sits on esh-iot and HA reaches it over routed IPv4 ws :5580.
- playbooks/esh-matter-lxc.yaml: kernel RA (accept_ra=1,
rt_info_max_plen=64), forwarding off, Docker ip-forward/iptables off;
nftables admits 5580 from HA 10.0.50.46 only and SSH from mgmt ranges;
the CT is added to esh-pve's vzdump job (fabric credentials).
- stacks/matter-server: ghcr.io/matter-js/matterjs-server:1.4.0 (digest),
host networking, /data on the CT.
- Acceptance: fdad:: SLAAC, ping6 thermostat, 2 Thread RIO routes learned, ws
server_info from inside the HA container; 5580 refused from 10.0.50.45,
nh3-dev and a temporary VLAN 90 netns vantage.
- qwen3-embedding and reranker: second deployment on nh3-ml1 (config);
reranker-a3-bge-v2-m3: second DB deployment via /model/new.
- router_settings.enable_weighted_failover: true. Only the three
multi-deployment groups are affected.
- Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd;
with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage;
nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
For brokkr's dataset foundry (operator-approved 2026-09-26, relayed).
- stacks/lfm-vl-seat: llama.cpp server-cuda b11176 (digest-pinned), Q5_K_M +
mmproj Q8_0, :8030; gateway alias lfm25-vl-3b (LiteLLM restarted, 36 s).
Positive control exact; null control shows it describes a missing image.
- stacks/vibevoice-asr-seat: audio.cpp v0.8.2-audio8-perf-hotfix (the GGUF's
own runtime, not vibevoice.cpp) on cuda 12.8 runtime + libgomp + libsoxr,
sha256-pinned; :8031 direct. LibriSpeech WER 3/69, RTF 0.07-0.14; ~31 s
cold first request.
Same model revision (df3ce67, blob sha matches fv-ml1), vLLM v0.24.0 and flags;
0.33 of the Ada (~5.4 GB, fv-ml1's absolute budget). Healthy, KV 79,808 tokens.
Parity (teacher-forced true code, 40 FIM prompts, 2,215 tokens, cache-salted):
each host is bit-exact with itself; cross-host |dlogprob| median 0.049, top-1
agreement 0.966 (Blackwell vs Ada + fp8 KV); against ground truth no quality
difference (mean logprob diff +0.008 +/- 0.019 SE). Speed on-box: ~5x slower
(64-tok FIM p50 ~1.0 s vs ~0.2 s; 63 vs 338 tok/s). Gateway coder-fast left on
fv-ml1 pending Prime's call.
- AMT re-IP'd over WS-Man to static 10.100.250.61/24 (gw/DNS .1): AMT keeps
its old DHCP lease across a VLAN move, so it was unreachable until changed
(reached via a temporary on-link /32 on vmbr0 + ssh tunnel, since removed).
- KVM redirection enabled, redirection listener on, IPS_OptInService
OptInRequired=0; all read back. Password vaulted as nh3-pve/amt-admin.
- scripts/amt-wsman.py: stdlib WS-Man get/put/invoke client (digest auth,
AMT 16 legacy-renegotiation TLS), password from $AMT_PW.
- PFI-UDMSE port 6 override: native nh3-mgmt, tagged VLANs blocked (was
forward all / native default). Reservation nh3-pve-amt -> 10.100.250.61.
- nh3-pve: arp_ignore=8 / arp_announce=2 on enp88s0 (now on vmbr0's untagged
L2) so the host never answers ARP for 10.100.250.60 with the AMT port's MAC.
- DNS nh3-pve-amt.nh3.internal -> 10.100.250.61.
- Homepage: NH3-PVE-AMT card under Infra - NH3 (no siteMonitor/ping: AMT drops
ICMP and its legacy-renegotiation TLS fails Homepage's fetch).
- AMT keeps its old 10.100.0.151 lease until rebind/expiry (~1920-2224 PT
2026-09-26); it does not re-DHCP on a VLAN change or link drop (measured).
AMT had no IP because Linux had enp88s0 admin-down (it left vmbr0 on 09-25 and
had no auto stanza); igc powers off the I226-LM PHY when down, so AMT had no
link. With the port up (no IP, unbridged) AMT took a lease within minutes.
- nh3-pve: `auto enp88s0` (manual, unbridged) and sysctl.d disable_ipv6 on it;
backup /root/interfaces.bak-20260925-amt; runtime already matched, no reload.
- UDM: reservation nh3-pve-amt -> 10.100.0.151 (untagged nh3-default; the
cable is on an nh3-sw1 trunk, not the UDM).
- DNS nh3-pve-amt.nh3.internal. AMT 16.1.25 answers TLS :16993 and :664.
nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.
- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
airflow on the next visit.
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled.
- nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml.
- nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4)
deployed with HOST_NAME/HOST_IP labels.
- Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min
0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control
0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed
identical within rep spread.
- gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix#7006).
With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl
reopen; applied on nh3-pve (one package).
- pve-nvidia-host.yaml: document that the headers meta drags in the newest
kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot).
- Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage
nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal.
- nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels,
AMT cabled but unreachable on the network.
Gateway routing to nh3-ml1 is not changed.
- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step,
nouveau blacklist + guarded unload (refuses if nouveau bound a device)
- esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined),
rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs
- embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1
nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a
card in the x16 slot its root port takes bus 01 and every NIC moves down a
bus (measured on esh-pve), so predictable names change (enp2s0f0np0 ->
enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files
now pin all NICs by MAC, baked into every initramfs and synced to the ESP;
udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves
vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot
loop the STP-less bridge.
Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console
OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS
shares (it died on the last NH3 cold start); post-boot checklist in
persistent-memory.
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
services — the sole backends behind the gateway, so the one exception to
"seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
automounted NAS share); revived by hand, fix still open.
- Audit: Skywork-Reward-V2-Llama-3.1-8B is still #1 of 188 on AllenAI's
RewardBench 2 per-sample results; no Skywork V3; the -40M sibling is
vendor-marked experimental. Our AWQ W4A16 quant: 0.847 vs published bf16
0.860 on a 150-prompt sample (within +/-2.9 pt SE), 96.2% pairwise
agreement. Double BOS from vLLM on pre-templated text costs a further
~2.7 pts; callers must send add_special_tokens=false.
- No working consumer: 0 requests since 2026-09-13; Worldtree Domari points
at a dead IP with a non-vLLM schema (reported to worldtree-dev).
- Move: sha256-identical model copy; vLLM v0.24.0 on esh-ml1 :8003 at 0.55
util (KV 1.30x of a 16k request). Parity vs fv-ml1: 149/150 verdicts,
99.8% pairwise signs, raw |delta| median 0.049.
- Gateway /scalar-judge passthrough -> 10.0.50.80:8003; fv-ml1 vllm-reward
removed (~10.2 GB freed on GPU 1). stacks/vllm now holds only vllm-coder.
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.
- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
8001/8013, fail-closed truncation (--auto-truncate false; embed
--max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
`texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
CLAUDE.md gains the TEI convention.
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.
esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.
stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.
litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.
Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
After the 2026-09-24 NH3 outage pfi-gx10 stayed off. It has no BMC, and the
ASUS firmware defaults to staying off after AC loss. With Prime's approval,
bytes 3-4 of the UEFI Setup variable were changed from 0x1d,0x00 to
0x1c,0x01, a community recipe that matches factory power-on units. The
readback shows exactly those two bytes changed. The original 85-byte raw
variable (attr header included) is committed here and also kept on the box.
The README records that an OS shutdown is not a test of this setting. It
stays off by design, and two such "tests" were invalid. The only valid test
is pulling and restoring AC, which Prime will do on 2026-09-25. It also
records the revert procedure.
After the 2026-09-24 NH3 power outage every NFS line from nh3-nas failed at
boot: nh3-docker's /mnt/compose and /mnt/backup, and nh3-dev's /mnt/backup.
The NAS is the slowest box to serve NFS, and the plain fstab mounts stayed
down until remounted by hand. nh3-dev's /mnt/smithy already used
x-systemd.automount, and the same boot shows it self-healing on its next
access. playbooks/nh3-nfs-automount.yaml brings the other lines to that
shape (_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30).
Hard-mount semantics are unchanged; only WHEN the mount happens moves.
Applied and read back on nh3-docker and nh3-dev: every nh3-nas line carries
automount, each automount unit is active, and each share is NFS-mounted after
access. A second run on nh3-docker is a no-op. The fstab check judges only
errors the rewrite ADDS, because nh3-dev's installer cdrom line has a
pre-existing verify error. The first run caught that and aborted without
writing.
servers/nh3-pve/README.md now lists each guest's expected state after a
power loss: nh3-laser is on-demand and stays off (Prime), pbs-nh3 now has
onboot=1 (it had none and stayed down), and pfi-gx10 is bare metal that
does not auto-power-on.
A deliberate `systemctl --user restart hermes-gateway` paged infra-ops as a
FAILED unit (msg 3716) while the unit was already back up. Cause is a Hermes
v0.21.1 race: the planned-stop marker watcher runs the shutdown handler before
systemd's SIGTERM, consumes the marker, and the SIGTERM re-runs the handler,
which then classifies the stop as unexpected and exits 1.
Corrects the README claim that OnFailure never fires on a deliberate restart:
that holds only when the main process exits with a success status. Measured on a
throwaway unit (3/3 paged without SuccessExitStatus, 0/3 with it, and crash
restarts are unaffected), and a survey of every stop on nh3-dev since 09-15
found hermes-gateway to be the only unit that does this.
The host-side fix is drop-in hermes-gateway.service.d/
20-planned-stop-exit1-is-clean.conf (SuccessExitStatus=1). No alarm coverage
is lost: Restart=always ignores the classification, and StartLimitIntervalSec=0
means the unit can never reach `failed` from a start failure.
TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
2 add route failures; first was: permission denied
adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.
Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.
Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.
⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.
Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.
Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.
README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.
Recorded with it, because these cost hours tonight and will cost them again:
- Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
inspection breaks Tailscale's relay and control channels (41 cert
warnings/week, 4 control-plane episodes in 14 days). Usually invisible
because direct peer paths carry the data. No fix on our side.
- Diagnose reachability with `tailscale ping`, NOT the status output:
headscale said "online" and status said "active, 19.7 GB" while nothing
on the host answered. Both are last-known state; only a round trip is
liveness.
- The ~26 docker bridges make tailscaled report captive portals.
Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.
vh