vh
cd941f7bf9
Pre-clear snapshot. Rewrote Current state / in-flight to the post-rebalance topology (cyberprev is the sec seat, gen-small stood up, char/coder retuned, all generative seats native 262K in-band, catalog+bench shipped) and added a Recent decisions index entry + detail file 2026-09-14-fv-seat-rebalance-gen-small.md. Also commits two previously-untracked durable artifacts (fv-to-ana-nat runbook + its memory detail) so memory does not lag the work. Archival: file is over the ~300-line soft cap (377) but 0 entries archived — the 4 age-eligible (>14d) entries all carry open deferred-work pointers (guarded), and all other entries are <14 days old. Reported per the archival discipline.
3.3 KiB
FV to Anaheim: scoped Tailscale NAT repair
Applied 2026-09-13 ~06:22Z after operator approval with explicit instruction not to strand the subnet. Gateway: OPNsense at 10.251.50.1, current hostname vb-gateway.internal; tailscale0 address 100.64.0.8, assigned interface opt6/MESH.
Persistent configuration
Firewall / NAT / Outbound: Hybrid mode, retaining every automatic WAN rule. One manual IPv4 rule:
- Interface: MESH (opt6 / tailscale0)
- Source: 10.251.50.54/32 (fv-ml1 only)
- Destination: 10.250.0.0/16 (Anaheim only)
- Translation: interface address (100.64.0.8 currently)
- Description:
fv-ml1 to ANA via Tailscale SNAT
No routes, access/filter rules, WAN settings, subnet advertisements, host Tailscale settings, or other FV source addresses were changed. Other remote sites remain outside this fix's scope. Translation means Anaheim services may see the gateway identity rather than fv-ml1's original LAN address.
Evidence and verification
Before: requests reached ANA with source 10.251.50.54; ANA sent SYN-ACKs then received resets attributed to that address, while FV host packet captures showed neither incoming SYN-ACKs nor outgoing resets. Gateway itself got hub HTTP200. Rules contained WAN NAT only, none on tailscale0. FV->ANA SSH, Beszel, Restic and ICMP failed; ANA->FV SSH and ping worked.
Temporary /32 host-to-hub NAT immediately fixed HTTP and restored Beszel. Only NAT rules were loaded for this first test. Persistent /32-to-ANA-/16 rule then saved using OPNsense write_config, followed by configctl filter reload. Both stages had independent 240-second rollback watchdogs. Before disarming, verified all preexisting NAT lines retained and pfctl -sr byte-identical.
After: FV->hub HTTP200, FV->ANA PostgreSQL TCP5432 reachable, FV internet HTTPS200, ANA->FV SSH reachable, gateway management intact. Beszel 18/18 up. Agent logs show WebSocket connected and intentional SSH listener shutdown; 45876 refusing after WebSocket connection is NOT a loss of host connectivity. A single BMC ping to 10.251.50.50 did not answer; no prior BMC baseline was taken, so BMC health is not claimed. The NAT rule excludes BMC source traffic.
Backup and rollback
Root-only directory on gateway: /root/fv-nat-repair-20260913/ contains
config.xml.before (secret-bearing, never commit), nat.before, filter.before,
nat-config.before.json, nat.after, nat-config.php and verified marker.
Automatic rollback disarmed only after verification; backup retained.
Manual rollback, after checking for intervening NAT edits:
php /root/fv-nat-repair-20260913/nat-config.php rollback
configctl filter reload
This restores the saved NAT config section, not unrelated configuration.
It will remove subsequent NAT edits too: inspect before using later.
Emergency runtime NAT-only restore: pfctl -N -f /root/fv-nat-repair-20260913/nat.before.
Never use pfctl -F all or flush all states for this repair.
Gateway SSH key auth is unavailable; use vaulted fv-gateway/root-password or approved infra-ops credentials, without logging secrets. This session used /tmp/fv-elway.py (FreeBSD sh + legacy SCP + existing SSH control socket).
References: OPNsense hybrid NAT, Tailscale FreeBSD userspace routing.