memory: snapshot — fv-ml1 rebalance (cyberprev→sec, gen-small A3B, all seats native 262K)
Pre-clear snapshot. Rewrote Current state / in-flight to the post-rebalance topology (cyberprev is the sec seat, gen-small stood up, char/coder retuned, all generative seats native 262K in-band, catalog+bench shipped) and added a Recent decisions index entry + detail file 2026-09-14-fv-seat-rebalance-gen-small.md. Also commits two previously-untracked durable artifacts (fv-to-ana-nat runbook + its memory detail) so memory does not lag the work. Archival: file is over the ~300-line soft cap (377) but 0 entries archived — the 4 age-eligible (>14d) entries all carry open deferred-work pointers (guarded), and all other entries are <14 days old. Reported per the archival discipline.
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
# FV to Anaheim: scoped Tailscale NAT repair
|
||||
|
||||
Applied 2026-09-13 ~06:22Z after operator approval with explicit instruction
|
||||
not to strand the subnet. Gateway: OPNsense at 10.251.50.1, current hostname
|
||||
vb-gateway.internal; tailscale0 address 100.64.0.8, assigned interface opt6/MESH.
|
||||
|
||||
## Persistent configuration
|
||||
|
||||
Firewall / NAT / Outbound: **Hybrid** mode, retaining every automatic WAN rule.
|
||||
One manual IPv4 rule:
|
||||
|
||||
- Interface: MESH (opt6 / tailscale0)
|
||||
- Source: 10.251.50.54/32 (fv-ml1 only)
|
||||
- Destination: 10.250.0.0/16 (Anaheim only)
|
||||
- Translation: interface address (100.64.0.8 currently)
|
||||
- Description: `fv-ml1 to ANA via Tailscale SNAT`
|
||||
|
||||
No routes, access/filter rules, WAN settings, subnet advertisements, host
|
||||
Tailscale settings, or other FV source addresses were changed. Other remote
|
||||
sites remain outside this fix's scope. Translation means Anaheim services may
|
||||
see the gateway identity rather than fv-ml1's original LAN address.
|
||||
|
||||
## Evidence and verification
|
||||
|
||||
Before: requests reached ANA with source 10.251.50.54; ANA sent SYN-ACKs then
|
||||
received resets attributed to that address, while FV host packet captures
|
||||
showed neither incoming SYN-ACKs nor outgoing resets. Gateway itself got hub
|
||||
HTTP200. Rules contained WAN NAT only, none on tailscale0. FV->ANA SSH, Beszel,
|
||||
Restic and ICMP failed; ANA->FV SSH and ping worked.
|
||||
|
||||
Temporary /32 host-to-hub NAT immediately fixed HTTP and restored Beszel.
|
||||
Only NAT rules were loaded for this first test. Persistent /32-to-ANA-/16 rule
|
||||
then saved using OPNsense write_config, followed by configctl filter reload.
|
||||
Both stages had independent 240-second rollback watchdogs. Before disarming,
|
||||
verified all preexisting NAT lines retained and pfctl -sr byte-identical.
|
||||
|
||||
After: FV->hub HTTP200, FV->ANA PostgreSQL TCP5432 reachable, FV internet
|
||||
HTTPS200, ANA->FV SSH reachable, gateway management intact. Beszel **18/18 up**.
|
||||
Agent logs show WebSocket connected and intentional SSH listener shutdown;
|
||||
45876 refusing after WebSocket connection is NOT a loss of host connectivity.
|
||||
A single BMC ping to 10.251.50.50 did not answer; no prior BMC baseline was
|
||||
taken, so BMC health is not claimed. The NAT rule excludes BMC source traffic.
|
||||
|
||||
## Backup and rollback
|
||||
|
||||
Root-only directory on gateway: `/root/fv-nat-repair-20260913/` contains
|
||||
config.xml.before (secret-bearing, never commit), nat.before, filter.before,
|
||||
nat-config.before.json, nat.after, nat-config.php and verified marker.
|
||||
Automatic rollback disarmed only after verification; backup retained.
|
||||
|
||||
Manual rollback, after checking for intervening NAT edits:
|
||||
|
||||
```sh
|
||||
php /root/fv-nat-repair-20260913/nat-config.php rollback
|
||||
configctl filter reload
|
||||
```
|
||||
|
||||
This restores the saved NAT config section, not unrelated configuration.
|
||||
It will remove subsequent NAT edits too: inspect before using later.
|
||||
Emergency runtime NAT-only restore: `pfctl -N -f /root/fv-nat-repair-20260913/nat.before`.
|
||||
Never use pfctl -F all or flush all states for this repair.
|
||||
|
||||
Gateway SSH key auth is unavailable; use vaulted fv-gateway/root-password or
|
||||
approved infra-ops credentials, without logging secrets. This session used
|
||||
/tmp/fv-elway.py (FreeBSD sh + legacy SCP + existing SSH control socket).
|
||||
|
||||
References: [OPNsense hybrid NAT](https://docs.opnsense.org/manual/nat.html),
|
||||
[Tailscale FreeBSD userspace routing](https://tailscale.com/docs/reference/kernel-vs-userspace-routers).
|
||||
@@ -0,0 +1,15 @@
|
||||
# FV→ANA NAT repaired safely
|
||||
|
||||
Operator approved narrow fix, caution not to strand subnet. OPNsense MESH/opt6
|
||||
(tailscale0 100.64.0.8) lacked outbound NAT for forwarded LAN traffic. Temporary
|
||||
fv-ml1→hub /32 NAT proved diagnosis; persisted hybrid NAT rule source
|
||||
10.251.50.54/32 destination 10.250.0.0/16 translate interface address. Existing
|
||||
WAN NAT retained, filter rules byte-identical, no routes/mesh/host changes.
|
||||
Both stages guarded by independent rollback timers, disarmed after verification.
|
||||
|
||||
Verified hub HTTP200, ANA PostgreSQL TCP, Internet HTTPS200, reverse SSH, gateway
|
||||
management; Beszel18/18 up. Agent intentionally stops SSH45876 when WebSocket
|
||||
connects. BMC ping failed with no pre-change baseline; no BMC-health claim.
|
||||
Other FV sources and other remote subnets not covered by this narrow fix.
|
||||
Backup + rollback helper on gateway /root/fv-nat-repair-20260913. Full details:
|
||||
docs/runbooks/fv-to-ana-nat.md. No commit made.
|
||||
@@ -0,0 +1,60 @@
|
||||
# 2026-09-14 — fv-ml1 seat rebalance: cyberprev→sec, gen-small stood up, all seats to native 262K
|
||||
|
||||
Second big fv-ml1 session on 2026-09-14 (after the all-night reorg + orca blocker in
|
||||
[[2026-09-14-fv-seat-reorg-and-orca-blocker]]). Everything here is committed; regenerate
|
||||
the live view with `scripts/seat-inventory.py` and the durable catalog is
|
||||
`docs/pfi/llm-seat-catalog.md`.
|
||||
|
||||
## What changed (all committed)
|
||||
|
||||
1. **cyberprev quantized and promoted to the `sec` seat.** `hotdogs/Qwen3.8-27B-abliterated-cyber-preview`
|
||||
(a cyber offensive-security **tool-calling SFT finetune** on an abliterated base — NOT
|
||||
"an abliteration"; trained on `hotdogs/cyber-sft-agent-qwen38`, 8,400 rows / 22 tools).
|
||||
⚠ The raw hotdogs export ships **broken**: 850/1199 body tensors carry a **tripled
|
||||
`language_model.` prefix** (unsloth export bug, verified in the upstream HF index) —
|
||||
the naive quant passed every structural check and served pure gibberish. Fix:
|
||||
name-repair the source against a known-good reference (exact 1199-tensor match), then
|
||||
house mixed-NVFP4 quant → `cyberprev-nvfp4-mixed-v2`. **A coherence smoke is the gate;
|
||||
structural verification cannot substitute** (see [[reference_unsloth_tripled_prefix_export_bug]]).
|
||||
Commits 1418edb/b94d173/196416f.
|
||||
2. **mog-sec RETIRED; cyberprev is now `sec`/`sec-reasoning`.** Gateway repointed (commit
|
||||
207798d). `sec` is a ROLE alias, so this is a promotion, not silent substitution.
|
||||
⚠ mog-sec was the fleet's ONLY offense+**defense**/blue-team seat — the sec role is now
|
||||
offense-only. mog-sec stack files retained for a one-command restore. Its "persona on
|
||||
stock" description was ALSO wrong — M.O.G.-SEC is itself an offense+defense SFT finetune
|
||||
(commit 3906c68). All three sec candidates are Qwen3.8-27B finetunes differing by focus.
|
||||
3. **gen-small stood up (NEW seat).** `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only`
|
||||
— already on disk at `/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4` (no download). Heretic
|
||||
v1.3.0 (MPOA), 88% fewer refusals @ 0.0015 KL, 19 MTP. GPU0 :8026, MTP k=3 **measured
|
||||
69.6% accept / 3.09 len**. 3.8 was requested but no general Qwen3.8 A3B exists → 3.6 is
|
||||
the fallback. Backs `summarizer` + `classifier`; new `classifier-large` → flash (accuracy
|
||||
tier); `summarizer-large` stays on flash. Commit 05f3d3b.
|
||||
4. **All sec/gen/char seats raised to native 262,144 with concurrency in the 1.2-2.5x band**
|
||||
(operator target). coder: 8192@13.75x → 16384@4.70x (waste reclaimed; ~4.7x overshoots
|
||||
2-3x because the 1.5B weight floor sits under the util knob's resolution — a KV-byte pin
|
||||
would fix it, deferred). char-rp bumped 1.02→1.22x (util 0.40→0.52); GPU1 hit its ceiling
|
||||
so char-rp-fast's util cap was trimmed (its KV is pinned, so it held 2.04x) — an in-GPU
|
||||
rebalance rather than a utility-seat move.
|
||||
5. **Durable docs:** `docs/pfi/llm-seat-catalog.md` (curated lineage/provenance/cards/speed)
|
||||
+ `scripts/seat-bench.py` (reproducible warm-tok/s + depth-OOM bench). Commit dfa91a8.
|
||||
|
||||
## Measured — serial depth + warm tok/s (2026-09-14, warm/conc=1/greedy/n=3)
|
||||
|
||||
| seat | GPU | ctx | conc | warm tok/s | VRAM | depth clean to | OOM |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| cyberprev (sec) | 0 | 262144 | 1.37x | 136.6 | 47.1 GiB | 259,722 tok | none |
|
||||
| gen-small | 0 | 262144 | 2.56x | 254.8 | 36.1 GiB | 254,526 tok | none |
|
||||
| gen (flash-next) | 2 | 262144 | 1.31x | 170.7 | 95.3 GiB | 254,273 tok | none |
|
||||
| char-rp | 1 | 262144 | 1.22x | 62.7 | 42.3 GiB | 254,858 tok | none |
|
||||
| char-rp-fast | 1 | 262144 | 2.04x | 225.2 | 27.0 GiB | 254,823 tok | none |
|
||||
| coder | 1 | 16384 | 4.70x | 337.3 | 5.6 GiB | 15,905 tok | none |
|
||||
|
||||
Every seat prefilled near max context with **zero OOM/CUBLAS/illegal-memory** (allocator-log
|
||||
verified). Numbers are uncontended single-stream ceilings — re-derive with `scripts/seat-bench.py`.
|
||||
|
||||
## Open / deferred
|
||||
|
||||
- **mog-sec fate** — retired; restore for blue-team coverage vs leave retired. Operator's call.
|
||||
- **coder ≤3x** — needs a `--kv-cache-memory` byte pin (compose change) vs the fragile util floor.
|
||||
- **gen-small 2.56x** — a hair over the 2.5x band top; trimmable, left as useful headroom for the bulk seat.
|
||||
- **mog-sec / sentinel-r3 stacks** retained on disk for restore; sentinel gateway aliases deleted (they 404).
|
||||
+31
-70
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user