memory: snapshot — fv-ml1 rebalance (cyberprev→sec, gen-small A3B, all seats native 262K)

Pre-clear snapshot. Rewrote Current state / in-flight to the post-rebalance topology
(cyberprev is the sec seat, gen-small stood up, char/coder retuned, all generative
seats native 262K in-band, catalog+bench shipped) and added a Recent decisions index
entry + detail file 2026-09-14-fv-seat-rebalance-gen-small.md.

Also commits two previously-untracked durable artifacts (fv-to-ana-nat runbook + its
memory detail) so memory does not lag the work.

Archival: file is over the ~300-line soft cap (377) but 0 entries archived — the 4
age-eligible (>14d) entries all carry open deferred-work pointers (guarded), and all
other entries are <14 days old. Reported per the archival discipline.
This commit is contained in:
vh
2026-09-14 11:00:32 -07:00
parent dfa91a8eaf
commit cd941f7bf9
4 changed files with 174 additions and 70 deletions
+68
View File
@@ -0,0 +1,68 @@
# FV to Anaheim: scoped Tailscale NAT repair
Applied 2026-09-13 ~06:22Z after operator approval with explicit instruction
not to strand the subnet. Gateway: OPNsense at 10.251.50.1, current hostname
vb-gateway.internal; tailscale0 address 100.64.0.8, assigned interface opt6/MESH.
## Persistent configuration
Firewall / NAT / Outbound: **Hybrid** mode, retaining every automatic WAN rule.
One manual IPv4 rule:
- Interface: MESH (opt6 / tailscale0)
- Source: 10.251.50.54/32 (fv-ml1 only)
- Destination: 10.250.0.0/16 (Anaheim only)
- Translation: interface address (100.64.0.8 currently)
- Description: `fv-ml1 to ANA via Tailscale SNAT`
No routes, access/filter rules, WAN settings, subnet advertisements, host
Tailscale settings, or other FV source addresses were changed. Other remote
sites remain outside this fix's scope. Translation means Anaheim services may
see the gateway identity rather than fv-ml1's original LAN address.
## Evidence and verification
Before: requests reached ANA with source 10.251.50.54; ANA sent SYN-ACKs then
received resets attributed to that address, while FV host packet captures
showed neither incoming SYN-ACKs nor outgoing resets. Gateway itself got hub
HTTP200. Rules contained WAN NAT only, none on tailscale0. FV->ANA SSH, Beszel,
Restic and ICMP failed; ANA->FV SSH and ping worked.
Temporary /32 host-to-hub NAT immediately fixed HTTP and restored Beszel.
Only NAT rules were loaded for this first test. Persistent /32-to-ANA-/16 rule
then saved using OPNsense write_config, followed by configctl filter reload.
Both stages had independent 240-second rollback watchdogs. Before disarming,
verified all preexisting NAT lines retained and pfctl -sr byte-identical.
After: FV->hub HTTP200, FV->ANA PostgreSQL TCP5432 reachable, FV internet
HTTPS200, ANA->FV SSH reachable, gateway management intact. Beszel **18/18 up**.
Agent logs show WebSocket connected and intentional SSH listener shutdown;
45876 refusing after WebSocket connection is NOT a loss of host connectivity.
A single BMC ping to 10.251.50.50 did not answer; no prior BMC baseline was
taken, so BMC health is not claimed. The NAT rule excludes BMC source traffic.
## Backup and rollback
Root-only directory on gateway: `/root/fv-nat-repair-20260913/` contains
config.xml.before (secret-bearing, never commit), nat.before, filter.before,
nat-config.before.json, nat.after, nat-config.php and verified marker.
Automatic rollback disarmed only after verification; backup retained.
Manual rollback, after checking for intervening NAT edits:
```sh
php /root/fv-nat-repair-20260913/nat-config.php rollback
configctl filter reload
```
This restores the saved NAT config section, not unrelated configuration.
It will remove subsequent NAT edits too: inspect before using later.
Emergency runtime NAT-only restore: `pfctl -N -f /root/fv-nat-repair-20260913/nat.before`.
Never use pfctl -F all or flush all states for this repair.
Gateway SSH key auth is unavailable; use vaulted fv-gateway/root-password or
approved infra-ops credentials, without logging secrets. This session used
/tmp/fv-elway.py (FreeBSD sh + legacy SCP + existing SSH control socket).
References: [OPNsense hybrid NAT](https://docs.opnsense.org/manual/nat.html),
[Tailscale FreeBSD userspace routing](https://tailscale.com/docs/reference/kernel-vs-userspace-routers).