memory: snapshot — fv-ml1 rebalance (cyberprev→sec, gen-small A3B, all seats native 262K)
Pre-clear snapshot. Rewrote Current state / in-flight to the post-rebalance topology (cyberprev is the sec seat, gen-small stood up, char/coder retuned, all generative seats native 262K in-band, catalog+bench shipped) and added a Recent decisions index entry + detail file 2026-09-14-fv-seat-rebalance-gen-small.md. Also commits two previously-untracked durable artifacts (fv-to-ana-nat runbook + its memory detail) so memory does not lag the work. Archival: file is over the ~300-line soft cap (377) but 0 entries archived — the 4 age-eligible (>14d) entries all carry open deferred-work pointers (guarded), and all other entries are <14 days old. Reported per the archival discipline.
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
# FV to Anaheim: scoped Tailscale NAT repair
|
||||
|
||||
Applied 2026-09-13 ~06:22Z after operator approval with explicit instruction
|
||||
not to strand the subnet. Gateway: OPNsense at 10.251.50.1, current hostname
|
||||
vb-gateway.internal; tailscale0 address 100.64.0.8, assigned interface opt6/MESH.
|
||||
|
||||
## Persistent configuration
|
||||
|
||||
Firewall / NAT / Outbound: **Hybrid** mode, retaining every automatic WAN rule.
|
||||
One manual IPv4 rule:
|
||||
|
||||
- Interface: MESH (opt6 / tailscale0)
|
||||
- Source: 10.251.50.54/32 (fv-ml1 only)
|
||||
- Destination: 10.250.0.0/16 (Anaheim only)
|
||||
- Translation: interface address (100.64.0.8 currently)
|
||||
- Description: `fv-ml1 to ANA via Tailscale SNAT`
|
||||
|
||||
No routes, access/filter rules, WAN settings, subnet advertisements, host
|
||||
Tailscale settings, or other FV source addresses were changed. Other remote
|
||||
sites remain outside this fix's scope. Translation means Anaheim services may
|
||||
see the gateway identity rather than fv-ml1's original LAN address.
|
||||
|
||||
## Evidence and verification
|
||||
|
||||
Before: requests reached ANA with source 10.251.50.54; ANA sent SYN-ACKs then
|
||||
received resets attributed to that address, while FV host packet captures
|
||||
showed neither incoming SYN-ACKs nor outgoing resets. Gateway itself got hub
|
||||
HTTP200. Rules contained WAN NAT only, none on tailscale0. FV->ANA SSH, Beszel,
|
||||
Restic and ICMP failed; ANA->FV SSH and ping worked.
|
||||
|
||||
Temporary /32 host-to-hub NAT immediately fixed HTTP and restored Beszel.
|
||||
Only NAT rules were loaded for this first test. Persistent /32-to-ANA-/16 rule
|
||||
then saved using OPNsense write_config, followed by configctl filter reload.
|
||||
Both stages had independent 240-second rollback watchdogs. Before disarming,
|
||||
verified all preexisting NAT lines retained and pfctl -sr byte-identical.
|
||||
|
||||
After: FV->hub HTTP200, FV->ANA PostgreSQL TCP5432 reachable, FV internet
|
||||
HTTPS200, ANA->FV SSH reachable, gateway management intact. Beszel **18/18 up**.
|
||||
Agent logs show WebSocket connected and intentional SSH listener shutdown;
|
||||
45876 refusing after WebSocket connection is NOT a loss of host connectivity.
|
||||
A single BMC ping to 10.251.50.50 did not answer; no prior BMC baseline was
|
||||
taken, so BMC health is not claimed. The NAT rule excludes BMC source traffic.
|
||||
|
||||
## Backup and rollback
|
||||
|
||||
Root-only directory on gateway: `/root/fv-nat-repair-20260913/` contains
|
||||
config.xml.before (secret-bearing, never commit), nat.before, filter.before,
|
||||
nat-config.before.json, nat.after, nat-config.php and verified marker.
|
||||
Automatic rollback disarmed only after verification; backup retained.
|
||||
|
||||
Manual rollback, after checking for intervening NAT edits:
|
||||
|
||||
```sh
|
||||
php /root/fv-nat-repair-20260913/nat-config.php rollback
|
||||
configctl filter reload
|
||||
```
|
||||
|
||||
This restores the saved NAT config section, not unrelated configuration.
|
||||
It will remove subsequent NAT edits too: inspect before using later.
|
||||
Emergency runtime NAT-only restore: `pfctl -N -f /root/fv-nat-repair-20260913/nat.before`.
|
||||
Never use pfctl -F all or flush all states for this repair.
|
||||
|
||||
Gateway SSH key auth is unavailable; use vaulted fv-gateway/root-password or
|
||||
approved infra-ops credentials, without logging secrets. This session used
|
||||
/tmp/fv-elway.py (FreeBSD sh + legacy SCP + existing SSH control socket).
|
||||
|
||||
References: [OPNsense hybrid NAT](https://docs.opnsense.org/manual/nat.html),
|
||||
[Tailscale FreeBSD userspace routing](https://tailscale.com/docs/reference/kernel-vs-userspace-routers).
|
||||
Reference in New Issue
Block a user