From 0b8632a7ed2f5bad388dd86b93f2f15fac745664 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 27 Sep 2026 12:50:05 -0700 Subject: [PATCH] fix(esh-docker-vm): /32 route to Home Assistant over macvlan-shim The shim holds 10.0.50.47/24, which gives two equal connected 10.0.50.0/24 routes, and ens18's wins. Host-to-HA traffic therefore left via the macvlan parent and was dropped. HA lost MQTT to the broker on this host on 2026-08-19, 2026-09-21 and 2026-09-25 (the last lasted two days). This adds an ifupdown if-up.d hook that routes 10.0.50.46/32 via macvlan-shim; /etc/network/interfaces is not edited. Verified: the route resolves via the shim, the host pings HA, HA reaches :1883, and HA reconnected to the broker. Diagnosis by ha-dev. Also corrects the zigbee2mqtt acceptance note, which had wrongly reported HA as connected. --- persistent-memory.md | 7 ++++- .../esh-docker-vm-macvlan-shim-route.yaml | 29 +++++++++++++++++++ .../files/esh-docker-vm-macvlan-shim-route.sh | 12 ++++++++ servers/esh-docker-vm/README.md | 7 +++++ stacks/zigbee2mqtt/README.md | 7 +++-- 5 files changed, 59 insertions(+), 3 deletions(-) create mode 100644 playbooks/esh-docker-vm-macvlan-shim-route.yaml create mode 100644 playbooks/files/esh-docker-vm-macvlan-shim-route.sh diff --git a/persistent-memory.md b/persistent-memory.md index f1b597e..5f69b7f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -181,7 +181,11 @@ _As of 2026-09-27 ~0900 PT._ key are in `/opt/docker/data/zigbee2mqtt` (root 0700, restic via /opt/docker, never in git). Vault: `esh-docker-vm/zigbee2mqtt-{network-key,frontend-token,mqtt-password}`. Broker user `zigbee2mqtt` added to mosquitto (passwd backup `.bak-20260927-z2m`). -- **Next is ha-dev's:** confirm the bridge device in HA, delete the ZHA entry, and pair the Aqara T1. +- **ha-dev deleted the ZHA entry**; pairing the Aqara T1 is theirs. ⚠ The "bridge in HA" acceptance first + FAILED because HA had had no MQTT since 2026-09-25 06:13. Cause: the esh-docker-vm macvlan shim had no + host route to HA (two equal /24s, ens18 wins). Fixed at 1249 with a /32 via the shim (if-up.d hook, + `playbooks/esh-docker-vm-macvlan-shim-route.yaml`), and HA reconnected. My first report called HA + "connected" from a client count that included my own probe; `$SYS` counts are not identities. - ⚠ The first credential-mint attempt was blocked by the auto-mode classifier on a peer-relayed approval. It went ahead only on Prime's own go-ahead in this session. Treat that as correct. → `stacks/zigbee2mqtt/README.md` @@ -240,6 +244,7 @@ _As of 2026-09-27 ~0900 PT._ - `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call. - `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md` +- `[2026-09-27]` **esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August).** → `servers/esh-docker-vm/README.md` - `[2026-09-27]` **Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only.** Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → `stacks/zigbee2mqtt/README.md` - `[2026-09-27]` **semif-serve 0.1.4: object states ending in `)`, `;` or `}` no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart.** Prime ruled; heid bug hunt folded. → `stacks/semif/README.md` - `[2026-09-27]` **SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win.** Build nothing (Prime). Henge 88 carries it. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md` diff --git a/playbooks/esh-docker-vm-macvlan-shim-route.yaml b/playbooks/esh-docker-vm-macvlan-shim-route.yaml new file mode 100644 index 0000000..69c9b23 --- /dev/null +++ b/playbooks/esh-docker-vm-macvlan-shim-route.yaml @@ -0,0 +1,29 @@ +# esh-docker-vm: persist + apply a /32 host route to Home Assistant (10.0.50.46) over macvlan-shim, +# as an ifupdown if-up.d hook (the file carries the why). Rerunnable; `ip route replace` is idempotent. +# /etc/network/interfaces is deliberately NOT edited. +# scripts/elway infra-ops@10.0.50.45 --playbook playbooks/esh-docker-vm-macvlan-shim-route.yaml +steps: + - name: Install the if-up.d hook + upload: + src: playbooks/files/esh-docker-vm-macvlan-shim-route.sh + dest: /etc/network/if-up.d/macvlan-shim-routes + mode: "0755" + sudo: true + + - name: Apply the route now (same command the hook runs at ifup) + shell: IFACE=macvlan-shim /etc/network/if-up.d/macvlan-shim-routes + sudo: true + changed_when: "false" + +verify: + - name: The host routes 10.0.50.46 over the shim + shell: ip route get 10.0.50.46 | grep -q 'dev macvlan-shim' + changed_when: "false" + + - name: The host reaches Home Assistant + shell: ping -c2 -W2 10.0.50.46 >/dev/null + changed_when: "false" + + - name: Home Assistant reaches the broker on this host (TCP 1883) + shell: docker exec homeassistant python3 -c "import socket; socket.create_connection(('10.0.50.45', 1883), 3).close()" + changed_when: "false" diff --git a/playbooks/files/esh-docker-vm-macvlan-shim-route.sh b/playbooks/files/esh-docker-vm-macvlan-shim-route.sh new file mode 100644 index 0000000..61b4cc3 --- /dev/null +++ b/playbooks/files/esh-docker-vm-macvlan-shim-route.sh @@ -0,0 +1,12 @@ +#!/bin/sh +# esh-docker-vm: route the host's traffic for macvlan children over the shim (ifupdown if-up.d hook). +# +# Home Assistant sits on a macvlan (10.0.50.46, parent ens18). A macvlan parent cannot talk to its +# own children, so the host reaches HA only through macvlan-shim (10.0.50.47, a macvlan sibling). +# The shim holds a /24, which gives TWO equal connected 10.0.50.0/24 routes; ens18's is listed first +# and wins, so host->HA traffic left via ens18 and was dropped. HA lost MQTT (the broker is on +# this host) on 2026-08-19, 2026-09-21 and 2026-09-25, the last for two days. A /32 via the shim is +# more specific than either /24, so the choice no longer depends on route order. +# Installed by playbooks/esh-docker-vm-macvlan-shim-route.yaml. Add a line per macvlan child. +[ "$IFACE" = "macvlan-shim" ] || exit 0 +ip route replace 10.0.50.46/32 dev macvlan-shim # homeassistant diff --git a/servers/esh-docker-vm/README.md b/servers/esh-docker-vm/README.md index 20dbd49..816da7b 100644 --- a/servers/esh-docker-vm/README.md +++ b/servers/esh-docker-vm/README.md @@ -68,6 +68,13 @@ Like `nh3-docker`, this host runs **Dozzle** and **Beszel** agents that report b ## Notes - **Macvlan for Home Assistant** — the HA container gets its own LAN IP (`10.0.50.46`) via a macvlan network on `ens18`, avoiding NAT so multicast/mDNS for HA discovery works cleanly. +- ⚠ **The host reaches HA only through `macvlan-shim` (10.0.50.47), and only because of a /32 route** + (`/etc/network/if-up.d/macvlan-shim-routes`, from `playbooks/esh-docker-vm-macvlan-shim-route.yaml`, + 2026-09-27). The shim holds a /24, so the host has TWO equal 10.0.50.0/24 routes, and ens18's wins. + Without the /32, host→HA traffic leaves via ens18 and macvlan drops it (a parent cannot talk to its + children). HA then loses MQTT, because the broker is on this host: 2026-08-19, 2026-09-21, and + 2026-09-25 for two days. **A new macvlan child needs its own line in that hook.** + Check: `ip route get 10.0.50.46` must say `dev macvlan-shim`. - **External Postgres** — Paperless-ngx connects to a DB running elsewhere (`10.0.50.60:5432`), not a sidecar. Paperless creds in that compose file currently look like defaults; rotate before exposing. - **Volume backups already in place** — `paperless-ngx` and `pgadmin` include `offen/docker-volume-backup:latest` sidecars that tar named volumes to `/mnt/backup/docker/esh-vm-docker//`. When the fleet-wide restic plan lands, decide whether to subsume these or leave the per-stack sidecars alone. diff --git a/stacks/zigbee2mqtt/README.md b/stacks/zigbee2mqtt/README.md index 2f0b19c..523e4d2 100644 --- a/stacks/zigbee2mqtt/README.md +++ b/stacks/zigbee2mqtt/README.md @@ -47,8 +47,11 @@ device must be re-paired. It must never be in git. That is why this repo holds o is kept as `configuration_backup_v4.yaml`. - Frontend: HTTP 200. The websocket accepts the vault token, and closes with `4401 Unauthorized` on a wrong token or no token. -- Broker: the new user is accepted, a wrong password is refused, and the reload disconnected no - one. Afterwards 3 clients were connected (HA, the `denspots` Tasmota, Z2M). The Tasmotas +- Broker: the new user is accepted, and a wrong password is refused. ⚠ **Correction (1250):** I first + reported HA's MQTT client as connected. It was not. HA had lost the broker at 2026-09-25 06:13, + because host→HA traffic left via ens18 instead of the macvlan shim (ha-dev diagnosed it). The + third "client" I counted was my own probe. Fixed by `playbooks/esh-docker-vm-macvlan-shim-route.yaml`, + a /32 route to 10.0.50.46 over macvlan-shim. HA reconnected at 1249. The Tasmotas `consoletree`, `fireplacetree` and `mantlelights` show LWT `Offline`, but they have not connected at any point in the broker's current log (back to 2026-09-09), so that predates this change.