memory: snapshot — run 7 training on gx10; run 6 TRANSFERRED after CSAM adjudication; erp-tune-v6-nvfp4a16 live as trial; ESH/YTVC/webhook repairs; ana-ml2 routes persisted; tank/zroot actions deferred to next session. Index 830→271 lines: 27 decisions + 8 abandoned archived, superseded in-flight blocks archived verbatim
This commit is contained in:
+2863
File diff suppressed because it is too large
Load Diff
@@ -1,192 +0,0 @@
|
||||
# DFlash2 speculative decoding — measured on our own stack (2026-08-22)
|
||||
|
||||
Operator-driven session. **Read the epistemic labels.** During the chase we generalised from
|
||||
observations that later proved wrong; this file separates what was *measured* from what remains
|
||||
*hypothesis*, and records the wrong turns so nobody re-derives them.
|
||||
|
||||
## What DFlash2 is
|
||||
|
||||
A **2B draft model** (3.85 GB bf16) for speculative decoding against Qwen3.8-27B —
|
||||
`incoai/Qwen3.8-27B-DFlash2`, Apache-2.0, blog `inco.ai/blog/dflash2`, upstream `z-lab/dflash`.
|
||||
Block diffusion: drafts a whole 8-token block in one pass, with a candidate selector tracing a
|
||||
path through per-slot top-K. Lossless (greedy matches the target).
|
||||
|
||||
vLLM support merged **2026-08-21 05:27 UTC** as PR **#52816** (`b389ac29`). Method string is
|
||||
**`"dflash"`**, not `dflash2`.
|
||||
|
||||
## ✅ MEASURED — throughput and acceptance
|
||||
|
||||
Single instrument (`specbench.py`, 8 fixed prompts, temp 0, max_tokens 256), delta against
|
||||
vLLM's own `spec_decode` counters. The MTP k=3 numbers reproduce our recorded 58.4% / 55.3%
|
||||
figures exactly, which is what validates the instrument.
|
||||
|
||||
| seat | config | accepted tok/forward | throughput |
|
||||
|---|---|---|---|
|
||||
| gen (orcarouter) | MTP k=3 *(production)* | 2.753 | 114.9 tok/s |
|
||||
| gen | MTP k=7 *(control)* | 3.041 | **74.0 tok/s** |
|
||||
| gen | **DFlash2 k=7** | **3.254** | **131.9 tok/s** |
|
||||
| sec (M.O.G.-SEC) | MTP k=3 *(production)* | 2.676 | 110.5 tok/s |
|
||||
| sec | **DFlash2 k=7** | **3.252** | **130.0 tok/s** |
|
||||
|
||||
**⭐ The k=7 MTP control was essential and inverted the obvious read.** Going deeper on MTP
|
||||
*improves acceptance* (2.753 → 3.041) while **destroying throughput** (114.9 → 74.0). Our MTP
|
||||
head is a single module (`mtp_num_hidden_layers=1`, only `mtp.layers.0`, 15 tensors) run
|
||||
autoregressively, so k draft tokens cost k sequential forward passes. **"Just raise
|
||||
num_speculative_tokens" is a trap** — without the control I would have recommended it.
|
||||
|
||||
DFlash2's win is therefore **not better per-token acceptance** — our MTP is actually *better* at
|
||||
position 0 (79.6% vs 75.4%). It is that block drafting makes depth nearly free.
|
||||
|
||||
**⭐ The drafter is model-agnostic across finetunes — 3.254 (gen) vs 3.252 (sec), a 0.06%
|
||||
difference**, with superimposable per-position curves. One drafter file on `/tank` serves both.
|
||||
|
||||
## ✅ MEASURED — how DFlash2 runs (answers "can one drafter serve both seats?")
|
||||
|
||||
**EAGLE3-style coupled, not standalone.** In vLLM: `load_model(self, target_model)` binds it to a
|
||||
specific target object; `pass_hidden_states_to_model=True`; `gpu_model_runner` reads
|
||||
`dflash_config.target_layer_ids` → `[i+1 …]` to register auxiliary hidden-state capture on the
|
||||
target at layers **5, 19, 33, 47, 61**. It even reads the target's RoPE style at load.
|
||||
|
||||
Consequences:
|
||||
- **Weights file is shareable** (one download, both seats mount it) — gen and sec are
|
||||
architecturally identical on every dimension the drafter needs: 64 layers (deepest tap 61),
|
||||
hidden 5120, intermediate 17408, vocab 248,320 > mask token 248,070.
|
||||
- **VRAM is NOT shareable — 3.85 GB per seat.** The drafter lives inside the target's engine
|
||||
process, consuming hidden states mid-forward. Two seats are two processes; there is no
|
||||
cross-process sharing mechanism and there could not be.
|
||||
|
||||
## ✅ MEASURED — it works on our stack, which the card does not claim
|
||||
|
||||
The card tests stock BF16 on an H200 with FlashAttention 3. Verified here instead:
|
||||
**abliterated + NVFP4 `compressed-tensors` target ✓, Blackwell sm_120 ✓, DFlash2 CUDA graphs
|
||||
captured ✓.** None of that was documented anywhere.
|
||||
|
||||
## 🔶 HYPOTHESIS — why our acceptance trails the published numbers
|
||||
|
||||
Both our targets land at ~3.25 accepted length against the card's 4.10–5.46 on stock BF16.
|
||||
**Finetune drift is ruled out** — two *different* finetunes gave identical results to three
|
||||
decimals. The shared variable is **NVFP4 quantization of the target**, which is mechanically
|
||||
plausible (the drafter reads quantized hidden states at its five taps). Second candidate:
|
||||
prompt distribution (ours general-purpose, theirs GSM8K/MATH/HumanEval/MBPP/MT-Bench).
|
||||
**Neither is confirmed.** Settling it needs a BF16 target seat (~56 GB) — a real GPU window.
|
||||
|
||||
## ❌ RETRACTED — the "MTP head mismatch causes the degeneration" hypothesis
|
||||
|
||||
**Operator ruling, 2026-08-22: this hypothesis is WRONG. The degeneration lives in the un-fixed
|
||||
vLLM, not in the weights.** Recorded here rather than deleted, because it was reasoned to
|
||||
confidently enough that a future session could re-derive it.
|
||||
|
||||
**Two independent failures produced it, and the second is the instructive one:**
|
||||
|
||||
1. **I treated a false dichotomy as a deduction.** Having verified gen and sec run an identical
|
||||
engine (same image ID `sha256:bd3236cff208…`, same live version
|
||||
`0.27.2rc1.dev150+g311b3513a` read from inside both processes, same flags bar
|
||||
`gpu-memory-utilization` 0.43 vs 0.44), I concluded "config is eliminated, therefore it is the
|
||||
weights." That does not follow. **An engine bug present in BOTH seats is not exonerated by the
|
||||
two seats being identical** — it just means the engine cannot explain a *difference*. It can
|
||||
still explain the *failure*.
|
||||
2. **The difference I was explaining may not exist.** The premise was a single operator
|
||||
observation of sec degenerating at ~2k, made during a session with many concurrent changes.
|
||||
**n=1 under heavy concurrent modification is not evidence** — see the meta-lesson below.
|
||||
|
||||
**What survives as fact** (measured, still true, just not causal): sec's MTP head *is*
|
||||
byte-identical to `qwen38-27b-uncensored-bf16` across all 15 tensors — a stock head on a
|
||||
security-finetuned body, because the `Qwen3_5ForConditionalGeneration` wrapper never loads the
|
||||
head, so the finetuning could not reach it. gen's orcarouter head *was* abliterated in-band by
|
||||
its author. Acceptance differs slightly (gen 58.4%, sec 55.9%). **All true. None of it shown to
|
||||
cause multi-turn degeneration.**
|
||||
|
||||
**Current standing explanation: the degeneration is an engine bug in the un-fixed vLLM.** Both
|
||||
production seats run `311b3513`, which is **172 commits behind GDN spec-decode fix #53077**
|
||||
(merged 2026-08-20). `#51113` is present in that build and is therefore **necessary but
|
||||
insufficient** on its own.
|
||||
|
||||
## ⭐⭐ META-LESSON — n=1 during a busy session is not evidence
|
||||
|
||||
The operator's own framing, and it generalises past this incident: **an observation made while
|
||||
many things are being changed at once cannot carry a causal claim, no matter how confidently it
|
||||
is reported.** Tonight that single observation became the load-bearing premise for a weights-side
|
||||
hypothesis, a root-cause narrative, and very nearly a recommendation.
|
||||
|
||||
This is the same failure the gen-seat compose file already warns about in different words — *"a
|
||||
passing probe is NOT sufficient evidence"* — inverted. That note guards against trusting a
|
||||
**negative** result from a synthetic test. This one guards against trusting a **positive**
|
||||
sighting from an uncontrolled session. Both reduce to: **hold the system still, or do not draw
|
||||
causal conclusions from it.**
|
||||
|
||||
Applies equally to the "coherent to 10k" observation below — same n, same conditions, opposite
|
||||
direction. Neither observation is worth more than the other.
|
||||
|
||||
## ⚠️ CONFOUNDED — and the "before" state is itself unreliable
|
||||
|
||||
sec now runs DFlash2 on a newer build and the operator reports **coherent to 10k tokens with
|
||||
adversarial nonsense prompts**. ⚠ Treat this the same way as the 2k sighting it is being compared
|
||||
against: **n=1, uncontrolled session, not evidence.** The comparison is weak on *both* ends.
|
||||
|
||||
**Two variables changed at once:**
|
||||
|
||||
1. **Engine**: `311b3513` → `e9d1398d`, **+259 commits, `behind_by=0`** (a strict superset),
|
||||
including GDN spec-decode fix **#53077** (merged 2026-08-20) that production is **172 commits
|
||||
behind**.
|
||||
2. **Drafter**: frozen MTP head → DFlash2 reading live hidden states.
|
||||
|
||||
**Isolating it = run MTP k=3 on the same new build.** Not yet done.
|
||||
|
||||
**#51113 is present in BOTH builds** (verified by ancestry, `behind_by=0` each) — so the
|
||||
"proper upstream fix" our compose comment credits is **necessary but insufficient**; sec ran it
|
||||
and still degenerated. Related open upstream: **#53180** (quantized Qwen3.8-27B hybrid GDN + MTP
|
||||
producing *silent* degenerate output, no fix), **#41884** (DFlash + prefix caching on hybrid,
|
||||
IndexError, workaround is disabling one).
|
||||
|
||||
## ❌ WRONG TURNS — do not repeat
|
||||
|
||||
- **Version strings are not lineage.** The DFlash2 build reports `0.26.1rc1.dev1048` and our
|
||||
production nightly `0.27.2rc1.dev150`, which *looks* like a regression. It is a setuptools_scm
|
||||
tag-reachability artifact. **Use the GitHub compare API and check `behind_by`.**
|
||||
- **Docker Hub push timestamps lie about source freshness.** `nightly-ba07e4a4` was *pushed*
|
||||
06:12 UTC, comfortably after the 05:27 merge — but *cut* from a 03:46 commit that predates it.
|
||||
**Grep the image for the symbols you need.** Believing the timestamp would have cost an RP-seat
|
||||
outage to serve a model the engine could not instantiate.
|
||||
- **`--max-num-batched-tokens` was not the image truncation.** Raising it 16,384 → 32,768 on that
|
||||
theory changed nothing and cost ~3 GiB of peak activation, which came straight out of the KV
|
||||
pool. The cap was the tokenizer (§3.14 of the playbook).
|
||||
- **"1M needs YaRN, absent from config" is FALSE for the sec quant.** It is fully present:
|
||||
`rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`,
|
||||
`max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit.
|
||||
|
||||
## Live state — PROMOTED to the compose stack 2026-08-22
|
||||
|
||||
**Operator-approved after real-use testing** ("performing very well"). The experimental
|
||||
standalone container is gone; `stacks/mog-sec/` is canonical and `restart: unless-stopped` means
|
||||
it survives reboots. Cutover verified: **KV pool 526,617 / 1.10x — identical to the container it
|
||||
replaced**, restarts 0, both gateway aliases serving, DFlash2 confirmed drafting at k=7
|
||||
(231 draft tokens over 33 drafts), vision working.
|
||||
|
||||
⚠ **One variable was deliberately REMOVED, not carried over.** The old stack hardcoded
|
||||
`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`; the validated DFlash2 container never set it,
|
||||
and playbook §3.10 records expandable_segments corrupting retained tensors elsewhere. The compose
|
||||
now defaults it EMPTY (`MOG_ALLOC_CONF`). Promoting it as-was would have shipped a variable the
|
||||
tested configuration did not have.
|
||||
|
||||
**Compose is now parameterised for the shapes that differ:** `MOG_SPEC_CONFIG` carries the whole
|
||||
speculative JSON (dflash needs `"model": "/drafter"`, MTP must not have one — a method+tokens
|
||||
template cannot express both), plus `MOG_MM_PROCESSOR_KWARGS`, `MOG_DRAFT_MODEL`,
|
||||
`MOG_MAX_NUM_BATCHED_TOKENS`, `MOG_ALLOC_CONF`.
|
||||
|
||||
**ROLLBACK:** `.env.bak-pre-dflash2-20260822` and `compose.yaml.bak-pre-dflash2-20260822` on the
|
||||
host; or one line — `MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` plus
|
||||
the old `MOG_IMAGE`.
|
||||
|
||||
| | production sec | current |
|
||||
|---|---|---|
|
||||
| image | `nightly-311b3513` | `nightly-e9d1398d` |
|
||||
| speculation | MTP k=3 | **DFlash2 k=7**, drafter `/tank/aimodels/qwen38-27b-dflash2-drafter` |
|
||||
| max-model-len | 262,144 | **480,000** |
|
||||
| KV pool | 418,218 (1.60×) | **526,617 (1.10×)** |
|
||||
| images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) |
|
||||
|
||||
⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up.
|
||||
0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB
|
||||
against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1.
|
||||
|
||||
Canonical config: `stacks/mog-sec/{compose.yaml,.env.example}` in this repo.
|
||||
@@ -1,34 +0,0 @@
|
||||
# [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600
|
||||
|
||||
Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live
|
||||
bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**.
|
||||
Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with
|
||||
four interactive accounts, verified as real exposure by reading one as `nobody`.
|
||||
|
||||
Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest,
|
||||
searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other
|
||||
stacks already used 0600, so this converged on the existing house pattern rather
|
||||
than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable
|
||||
by `nobody`.
|
||||
|
||||
Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable.
|
||||
|
||||
## The check that matters
|
||||
|
||||
Every run asserts `docker compose config` still renders **as the deploy user**
|
||||
(`lkraven`), not as root. Checking the mode proves the bits changed; only rendering
|
||||
as the deploy user proves the next deploy can still resolve its variables.
|
||||
|
||||
## Two gotchas recorded in the playbook
|
||||
|
||||
- **vaultwarden looked like it bind-mounted its `.env`** — which would mean the
|
||||
*container's* UID reads it and 0600 could break the password vault. It does not:
|
||||
that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list
|
||||
item without checking its parent key. The playbook now **refuses** any stack that
|
||||
genuinely bind-mounts its `.env`, since that case is read by the container UID.
|
||||
- **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to
|
||||
`lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev`
|
||||
was repointed to infra-ops at some point and `ana-docker` was not. Run elway against
|
||||
the infra-ops target on this host.
|
||||
|
||||
Commit `a896c0a`.
|
||||
@@ -1,663 +0,0 @@
|
||||
# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
|
||||
|
||||
> **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at
|
||||
> the bottom before acting on anything here.** The headline is wrong (the
|
||||
> relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the
|
||||
> aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the
|
||||
> original stopped measuring at 4), and the proposed remedy is **impossible**:
|
||||
> UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The
|
||||
> per-stream observation and the parallelise-your-transfers mitigation are the
|
||||
> parts that survive.
|
||||
|
||||
The operator noticed site-to-site transfers were slow for a datacenter fiber
|
||||
handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
|
||||
circuit is fine.
|
||||
|
||||
## Measured
|
||||
|
||||
```
|
||||
ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering
|
||||
ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s
|
||||
FortiGate's own recorded peak 554 Mbit/s
|
||||
ANA <-> NH3, single stream 227 Mbit/s
|
||||
ANA <-> ESH, single stream 249-265 Mbit/s
|
||||
ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream
|
||||
```
|
||||
|
||||
Method: stdlib TCP probe (no ssh, no crypto, no compression) between site
|
||||
endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner
|
||||
Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.
|
||||
|
||||
## What it is not
|
||||
|
||||
- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw
|
||||
(`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard
|
||||
on ana-wg is remote-access only and is not in this path. Traceroute confirms:
|
||||
both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast
|
||||
ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it.
|
||||
- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all
|
||||
8 cores** during the tests, and both live tunnels report `npu_flag=03` with
|
||||
`dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded.
|
||||
- **Not a 250 Mbit/s cap.** That was the first number and it is misleading —
|
||||
single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
|
||||
- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions.
|
||||
|
||||
## Most likely cause
|
||||
|
||||
Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14,
|
||||
IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
|
||||
faster on **AES-GCM**, which combines encryption and authentication in one
|
||||
pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
|
||||
not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
|
||||
shortfall is consistent with that.
|
||||
|
||||
## Not executed
|
||||
|
||||
Changing the proposal is a **production-edge change requiring a matching
|
||||
change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while
|
||||
it renegotiates. Left for the operator. See the index entry for authorization
|
||||
state.
|
||||
|
||||
## Immediate mitigation, no config change
|
||||
|
||||
Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a
|
||||
single stream through IPsec gets 227. **Anything moving bulk data across the
|
||||
Anaheim link should parallelise** — that alone roughly doubles throughput
|
||||
today.
|
||||
|
||||
## Practical consequence already observed
|
||||
|
||||
`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
|
||||
from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and
|
||||
not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]].
|
||||
|
||||
## Access note
|
||||
|
||||
ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent
|
||||
on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
|
||||
vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
|
||||
prints live ESP session keys** — never paste its output into althing, a
|
||||
booth, or a commit.
|
||||
|
||||
---
|
||||
|
||||
## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist
|
||||
|
||||
The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done,
|
||||
and the measurements taken while trying show there is very little left to win.
|
||||
|
||||
### AES-GCM is unavailable on the far end — not a naming problem
|
||||
|
||||
The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf
|
||||
`_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the
|
||||
crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`,
|
||||
`aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128`
|
||||
— all `HTTP 400 api.err.InvalidPayload`, nothing applied.
|
||||
|
||||
**The control that makes this conclusive:** the *identical* request body with
|
||||
`ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the
|
||||
enum rejecting the value, not a malformed body. Corroborating: **zero
|
||||
case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.**
|
||||
|
||||
Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`,
|
||||
`chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on
|
||||
UniFi far ends, so this blocks the ESH tunnel too.
|
||||
|
||||
The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now
|
||||
reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is
|
||||
functionally identical while the peer only offers CBC, and reverting it would
|
||||
cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched;
|
||||
IKE protects the control channel only and has no bearing on data throughput.
|
||||
|
||||
### The numbers that retire this as a problem
|
||||
|
||||
Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on
|
||||
the same UDM and the same uplink, with the same stdlib TCP probe:
|
||||
|
||||
| streams | IPsec NH3→ANA | WireGuard NH3→ESH |
|
||||
|---|---|---|
|
||||
| 1 | 245 Mbit/s | 557 Mbit/s |
|
||||
| 4 | 471 Mbit/s | 767 Mbit/s |
|
||||
| 8 | **692 Mbit/s** | **795 Mbit/s** |
|
||||
|
||||
**NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) —
|
||||
that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel.
|
||||
So the tunnel does **~69% of the achievable uplink** at 8 streams, and the
|
||||
IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**.
|
||||
|
||||
Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since
|
||||
FortiOS has no WireGuard) would chase that last 15%. Not worth it.
|
||||
|
||||
### What the constraint actually is
|
||||
|
||||
A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both
|
||||
endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03`
|
||||
(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The
|
||||
shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from
|
||||
a higher floor (557 → 795 is only 1.43× scaling).
|
||||
|
||||
### Actionable consequence
|
||||
|
||||
Anything moving bulk data across this link should **parallelise** — 245 → 692
|
||||
Mbit/s, a 2.8× win with no config change. For single-stream workloads that
|
||||
cannot be parallelised at the application layer, **NFS `nconnect=N` is the
|
||||
lever**: it opens N TCP connections per mount, converting a single-stream
|
||||
workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at
|
||||
24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live
|
||||
example — remounting with `nconnect=8` is the obvious test.
|
||||
|
||||
### Foot-gun recorded
|
||||
|
||||
Probing the enum by PUTting candidate values **applies the accepted ones**. A
|
||||
probe loop here timed out with `3des` briefly live on the NH3 side, which the
|
||||
FortiGate would not accept — a short tunnel outage until `aes256` was restored
|
||||
(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi
|
||||
config enum this way, restore the known-good value after **every** 200, not at
|
||||
the end of the loop. Post-change verification: the UDM object was diffed
|
||||
field-by-field against its pre-change snapshot and is **byte-identical**.
|
||||
|
||||
---
|
||||
|
||||
## FOLLOW-UP (2026-08-23): what the per-stream limit actually is
|
||||
|
||||
The correction above called the constraint "per-SA/per-flow serialisation".
|
||||
That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of
|
||||
~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning
|
||||
problem, not loss, not window size.
|
||||
|
||||
### The evidence: pin the send buffer and sweep it
|
||||
|
||||
Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight:
|
||||
|
||||
| in-flight cap | throughput | RTT in flight | minRTT | retrans |
|
||||
|---|---|---|---|---|
|
||||
| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 |
|
||||
| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 |
|
||||
| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 |
|
||||
| ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 |
|
||||
|
||||
**Throughput is flat across a 13× range of in-flight data while RTT scales with
|
||||
it.** That is the signature of a fixed service rate with a standing queue: the
|
||||
window controls only how much queue you build, never how fast you go. Had this
|
||||
been window-limited, throughput would have risen with the buffer. Had it been
|
||||
congestion, there would be retransmits — there are essentially none
|
||||
(`retrans:0`, 0% ping loss).
|
||||
|
||||
So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all
|
||||
**red herrings here**. Do not go there.
|
||||
|
||||
### Bufferbloat: one bulk stream wrecks latency for everything else
|
||||
|
||||
Measured on the same tunnel, ping to ana-docker:
|
||||
|
||||
- idle: **6.9 ms** avg
|
||||
- during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss
|
||||
|
||||
**15× latency inflation from one transfer.** This is the operationally
|
||||
important finding — any interactive traffic sharing the Anaheim link (ssh,
|
||||
RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it
|
||||
takes only one stream to do it. Parallelising transfers makes throughput
|
||||
better and this *worse*. If it starts biting, the fix is an AQM/shaper on the
|
||||
tunnel (or rate-limiting bulk jobs), not more buffer.
|
||||
|
||||
### Where the cap lives — strong inference, not proof
|
||||
|
||||
Three paths, and the FortiGate is the only variable:
|
||||
|
||||
| path | single-stream |
|
||||
|---|---|
|
||||
| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s |
|
||||
| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s |
|
||||
| NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s |
|
||||
|
||||
Present in both slow paths, absent from the fast one. Aggregate over the same
|
||||
SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole —
|
||||
many flows spread out fine, one flow does not.
|
||||
|
||||
The mechanism that fits is **FortiGate NPU IPsec offload being per-session**:
|
||||
each firewall session is bound to one crypto engine, so a single TCP flow is
|
||||
capped at one engine's rate while many sessions spread across engines. **This
|
||||
is inference from the throughput shape, not something confirmed on the box** —
|
||||
`diagnose sys session list` was not captured for a TCP flow (the filter caught
|
||||
only traceroute UDP probes). A single-stream control through ana-gw *without*
|
||||
IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is
|
||||
window-limited and does not discriminate. **If this matters, the clean test is
|
||||
a non-IPsec single stream between two Anaheim VLANs at low RTT.**
|
||||
|
||||
**Relevant to the FortiGate cutover decision:** if the per-flow cap is the
|
||||
FortiGate's IPsec path, replacing the box plausibly lifts single-stream
|
||||
throughput toward the WireGuard figure. That is a point in favour of the
|
||||
cutover, and it is cheap to verify afterwards by re-running the sweep.
|
||||
|
||||
---
|
||||
|
||||
## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec
|
||||
|
||||
Operator asked directly whether the 80F "can't handle the traffic". It can.
|
||||
Two new measurements settle the shape of this, and correct an overstatement in
|
||||
FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded —
|
||||
every slow path was *both* IPsec *and* FortiGate, so protocol and box could not
|
||||
be separated by that argument).
|
||||
|
||||
### The 80F routes a single flow at line rate when IPsec is not involved
|
||||
|
||||
`ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 =
|
||||
`10.250.50.1`), 0.36 ms RTT, no tunnel:
|
||||
|
||||
| streams | throughput |
|
||||
|---|---|
|
||||
| 1 | **940.2 Mbit/s** |
|
||||
| 8 | 939.3 Mbit/s |
|
||||
|
||||
Single stream saturates 1 GbE. So the box does **not** cap single sessions in
|
||||
general, and there is no per-session ceiling in its plain forwarding path. The
|
||||
~250 Mbit/s per-flow cap is **specific to the IPsec datapath**.
|
||||
|
||||
### Both IPsec tunnels converge on the same numbers despite different far ends
|
||||
|
||||
Measured today with the same probe:
|
||||
|
||||
| tunnel | far-end gateway | RTT | 1 stream | 8 streams |
|
||||
|---|---|---|---|---|
|
||||
| NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s |
|
||||
| ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** |
|
||||
|
||||
Different gateway hardware, different sites, different uplinks, and RTT
|
||||
differing by 1.7× — yet single-stream differs by only 9%. **If this were
|
||||
window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is
|
||||
independent confirmation of a rate cap rather than a BDP effect.
|
||||
|
||||
### Capacity summary — the box has headroom it will not give one flow
|
||||
|
||||
- plain routing, 1 stream: **940 Mbit/s** (line rate)
|
||||
- plain routing to internet, 8 streams: **2,153 Mbit/s**
|
||||
- IPsec, 8 streams: **692–715 Mbit/s**
|
||||
- IPsec, 1 stream: **245–268 Mbit/s**
|
||||
- CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`)
|
||||
|
||||
Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath
|
||||
distributes work **by inner session** — consistent with IPsec offload binding a
|
||||
session to one crypto engine.
|
||||
|
||||
### What is still NOT separated
|
||||
|
||||
Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec
|
||||
implementation**. Both tunnels have a UDM at the far end, and both UDMs run the
|
||||
same UniFi firmware, so identical caps are explainable either way. The Pro Max
|
||||
being only 9% faster than the Pro SE argues against the UniFi side (a beefier
|
||||
CPU should show more), but that is suggestive, not conclusive.
|
||||
|
||||
**The test that closes it:** an IPsec tunnel whose endpoints do not include the
|
||||
80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured
|
||||
single-stream. If it also caps ~250, the FortiGate is exonerated and replacing
|
||||
it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM
|
||||
WireGuard achieves, the 80F is the limiter. **Bears directly on the pending
|
||||
FortiGate cutover** — worth running before that decision, not after.
|
||||
|
||||
---
|
||||
|
||||
## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream
|
||||
|
||||
Operator asked for a WireGuard test from `ana-wg` to NH3 over the public
|
||||
internet. It is the test that separates the *path* from the *crypto*, and the
|
||||
answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the
|
||||
transport is not worth it" — that conclusion compared 8-stream numbers and was
|
||||
wrong for single-stream workloads.**
|
||||
|
||||
### Setup (fully torn down afterwards)
|
||||
|
||||
`ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an
|
||||
internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by
|
||||
FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337,
|
||||
policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A
|
||||
temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on
|
||||
wg0, measured, then removed — ana-wg is back to its original 3 peers and the
|
||||
keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in
|
||||
place** (benign, and wanted if this becomes permanent).
|
||||
|
||||
In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM
|
||||
only NAT/forward UDP, and Linux does WireGuard at both ends.
|
||||
|
||||
### The full comparison
|
||||
|
||||
| path | crypto performed by | 1 stream | 8 streams |
|
||||
|---|---|---|---|
|
||||
| IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s |
|
||||
| IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s |
|
||||
| **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s |
|
||||
| WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s |
|
||||
| plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s |
|
||||
|
||||
**One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is
|
||||
no per-flow penalty at all, and a single flow already saturates the path. So
|
||||
the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3
|
||||
uplink and not the physical path — all of which sustain 767 on one flow.
|
||||
|
||||
Per-flow penalty ranks by implementation:
|
||||
|
||||
- **Linux WireGuard — none** (767 → 763, flat)
|
||||
- **UDM WireGuard — mild**, ~1.4× (557 → 795)
|
||||
- **IPsec on this pair — severe**, ~2.8× (245 → 692)
|
||||
|
||||
### Latency under load — the same story
|
||||
|
||||
| path | idle | during ONE bulk stream |
|
||||
|---|---|---|
|
||||
| IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max |
|
||||
| WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max |
|
||||
|
||||
WireGuard carries **3.1× the single-stream throughput with 8× less latency
|
||||
inflation** on the same wire.
|
||||
|
||||
### Attribution — still not fully separated, and it no longer matters much
|
||||
|
||||
Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still
|
||||
does not isolate which one imposes the 2.8× penalty. Closing that would need
|
||||
Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical
|
||||
decision no longer depends on the answer**, because the fix is the same either
|
||||
way and it is already demonstrated.
|
||||
|
||||
### Recommendation (supersedes FOLLOW-UP 1)
|
||||
|
||||
A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is
|
||||
worth real consideration: 3.1× single-stream, flat scaling, far better latency
|
||||
under load, and it reuses infrastructure that already exists and is already
|
||||
internet-reachable. It is also the architecture already proven for NH3↔ESH.
|
||||
Open questions before committing: routing/failover if ana-wg (an LXC) is down,
|
||||
whether it replaces or parallels the IPsec tunnel, and firewall policy for the
|
||||
new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it
|
||||
has headroom.
|
||||
|
||||
**AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the
|
||||
same lever (turn one flow into many) and brokkr-smithy-dev has given standing
|
||||
approval to apply it once the FortiGate work settles, with no need to ask again.
|
||||
|
||||
---
|
||||
|
||||
## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated.
|
||||
|
||||
Operator's theory — the UDM does IPsec in software with no crypto offload, so
|
||||
the cost of the cipher itself is the limit — is **correct**, and it is now
|
||||
demonstrated rather than inferred. He also correctly pointed out that
|
||||
UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything
|
||||
about UniFi's IPsec performance. It didn't, and I had leaned on it.
|
||||
|
||||
### The controlled experiment: vary cipher cost, hold everything else
|
||||
|
||||
AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding
|
||||
constraint, throughput must rise when the cipher gets cheaper. If the limit
|
||||
were the FortiGate's NPU, it would not move at all — hardware crypto is not
|
||||
cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each:
|
||||
|
||||
| condition | ESP cipher | single-stream | UDM CPU |
|
||||
|---|---|---|---|
|
||||
| A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% |
|
||||
| B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% |
|
||||
| A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — |
|
||||
|
||||
**~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of
|
||||
CPU work, more payload through it. That is the signature of CPU-bound software
|
||||
crypto, and it rules out the FortiGate's NPU as the limiter.
|
||||
|
||||
### Correcting two of my own earlier claims
|
||||
|
||||
1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling
|
||||
artifact.** UniFi's `system-stats.cpu` refreshes on the device report
|
||||
interval; 4-second sample windows were reading stale values. Under a
|
||||
sustained 60 s single-stream load it reads **35.4%**, with load1 rising
|
||||
0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated
|
||||
on crypto plus overhead. **Always drive load for ≥60 s before trusting a
|
||||
UniFi CPU figure.**
|
||||
2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely
|
||||
unproven. It predicts no change from a cipher swap; a 20% change was measured.
|
||||
|
||||
### Why the numbers all line up now
|
||||
|
||||
- **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC.
|
||||
- **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure
|
||||
on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link)
|
||||
and is *not* cipher-sensitive, consistent with it being bounded by the path/
|
||||
uplink rather than crypto once several cores are engaged.
|
||||
- **AES-CBC is the specific villain: it is serial.** Each block depends on the
|
||||
previous one, so the ARM AES instructions cannot pipeline across blocks. GCM
|
||||
(CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the
|
||||
same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec.
|
||||
- **This retroactively vindicates the GCM cutover as the right idea aimed at the
|
||||
right box** — GCM would have removed the serial dependency on the constrained
|
||||
end. UniFi simply does not offer it, which is what made it impossible.
|
||||
|
||||
### Options this opens
|
||||
|
||||
- **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API
|
||||
call per end. 128-bit is not the weak link here (SHA1 integrity is more
|
||||
dated, and unchanged either way). Operator's call — **not adopted**, restored
|
||||
to aes256.
|
||||
- **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it
|
||||
sidesteps the UDM's IPsec datapath entirely. Still the biggest win available.
|
||||
- Replacing the FortiGate **will not help this** — it was never the constraint.
|
||||
Worth knowing before the cutover.
|
||||
|
||||
### State left behind
|
||||
|
||||
UDM network object verified **byte-identical** to its pre-test snapshot
|
||||
(aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is
|
||||
left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer
|
||||
offers only aes256 so the extra entries are inert, but **narrowing it back to
|
||||
`aes256-sha1` is one line** if the looser list is unwanted.
|
||||
|
||||
---
|
||||
|
||||
## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
|
||||
|
||||
Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must
|
||||
terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on
|
||||
pfi-pve, 10.250.50.252) already is that box.
|
||||
|
||||
**This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with
|
||||
traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other
|
||||
Anaheim hosts, which was never measured. Now it is:
|
||||
|
||||
| topology | 1 stream | 8 streams |
|
||||
|---|---|---|
|
||||
| IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s |
|
||||
| WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s |
|
||||
| **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** |
|
||||
|
||||
**Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM
|
||||
architecture delivers the full 3.1× single-stream for real transit traffic, not
|
||||
just for traffic landing on the tunnel box.
|
||||
|
||||
ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle),
|
||||
so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's*
|
||||
load, not the container's — do not read it as ana-wg's own. For contrast the
|
||||
UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom.
|
||||
|
||||
### Design consequences of terminating downstream — the parts that need decisions
|
||||
|
||||
1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8
|
||||
figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252`
|
||||
route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default
|
||||
gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a
|
||||
LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT
|
||||
measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated
|
||||
transit VLAN for ana-wg, or accept the hairpin.
|
||||
2. **New single point of failure.** Today site-to-site dies only when the edge
|
||||
dies, which is total anyway. A downstream terminator fails independently.
|
||||
Mitigation: keep the IPsec tunnel configured as a higher-metric fallback
|
||||
route so it takes over when ana-wg is down.
|
||||
3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the
|
||||
hypervisor shared with the rest of the Anaheim VMs.
|
||||
4. **The NH3 end needs a terminator too**, and there are two shapes:
|
||||
- **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what
|
||||
was measured: **764 Mbit/s**.
|
||||
- **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on
|
||||
UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and
|
||||
UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on
|
||||
Site Magic), so expect ~557 rather than 764. Still 2.3× today.
|
||||
|
||||
### Standing recommendation
|
||||
|
||||
Worth doing, but it is **a project, not a config tweak** — routing, failover and
|
||||
policy all need deciding. The cheap wins remain available meanwhile and are
|
||||
independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending
|
||||
the FortiGate work settling) and AES-128 for ~20%.
|
||||
|
||||
---
|
||||
|
||||
## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed
|
||||
|
||||
Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break,
|
||||
then close the FortiGate's WAN and SSH admin surfaces. All done and verified.
|
||||
|
||||
**Context that retires the WireGuard-in-a-VM design work:** the FortiGate is
|
||||
being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the
|
||||
edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is
|
||||
therefore moot — do not scope it. This also **un-parks the OPNsense migration**,
|
||||
which auto-memory recorded as PARKED pending "hardware acquisition"; the R420
|
||||
is that trigger.
|
||||
|
||||
### What changed
|
||||
|
||||
Make-before-break on the FortiGate first, so neither tunnel dropped waiting on
|
||||
a far end:
|
||||
|
||||
| phase2 | proposal now |
|
||||
|---|---|
|
||||
| `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` |
|
||||
| `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` |
|
||||
|
||||
Then each UDM flipped to `ipsec_esp_encryption: aes128`:
|
||||
|
||||
| tunnel | UDM object | before | after |
|
||||
|---|---|---|---|
|
||||
| NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** |
|
||||
| ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** |
|
||||
|
||||
Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B —
|
||||
the difference is live-link variance, not a different result. Both UDM objects
|
||||
were diffed field-by-field against pre-change snapshots: **the only field that
|
||||
moved on either is `ipsec_esp_encryption`.**
|
||||
|
||||
The FortiGate proposal lists were deliberately **left permissive** (still
|
||||
accepting aes256). The peers offer only aes128 so the extra entries are inert,
|
||||
and keeping them means a UDM reverting does not strand the tunnel. Narrowing to
|
||||
`aes128-sha1` alone is a one-liner if the looser list is unwanted.
|
||||
|
||||
### Admin surfaces closed
|
||||
|
||||
`wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost
|
||||
→ **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed
|
||||
from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works.
|
||||
**Sequencing that matters: the close was executed over the TUNNEL path, not over
|
||||
WAN** — removing `ssh` from allowaccess while connected over WAN kills the
|
||||
session mid-command.
|
||||
|
||||
**Consequence to hold in mind: ana-gw now has no out-of-band management path.**
|
||||
If both tunnels drop it is console-only until someone is on site.
|
||||
|
||||
### Gotcha: the two UDM vault items have DIFFERENT shapes
|
||||
|
||||
- `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key.
|
||||
- `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a
|
||||
`key:` line. `secret get` piped straight into a header yields a 1396-byte
|
||||
value and the UDM answers **`400 Bad Request` from nginx**. Extract with
|
||||
`grep '^key:' | awk '{print $2}'`.
|
||||
|
||||
**The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded
|
||||
it as read-verified only): a control PUT of the unchanged object returned
|
||||
`rc:ok`, then the real change did too. That key has a full read+write admin role.
|
||||
|
||||
---
|
||||
|
||||
## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener
|
||||
|
||||
The claim in the previous section that `.42:80` was an **ISP transparent proxy**
|
||||
was **WRONG**, and so was the earlier warning that ACME renewal would fail with
|
||||
port 80 absent from `allowaccess`. Operator pushed back asking where the port-80
|
||||
map terminated. It terminates **on the FortiGate itself**.
|
||||
|
||||
**What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system
|
||||
acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to
|
||||
answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1
|
||||
allowaccess` reads `ping` only and the port is still open. Every non-challenge
|
||||
request returns a fixed 403 whose body is literally:
|
||||
|
||||
```
|
||||
<!DOCTYPE html><html><head><title>ACME Access Only</title></head><body>ACME Access Only</body></html>
|
||||
```
|
||||
|
||||
**Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` —
|
||||
`Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252).
|
||||
~~Worth noting separately: four VIPs are all-port static NAT~~ — **that claim was
|
||||
WRONG, see the correction below.** All fourteen VIPs are scoped.
|
||||
|
||||
### The methodology error that produced the wrong answer — worth not repeating
|
||||
|
||||
The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
|
||||
matches only inbound packets**, so outbound SYN-ACKs were excluded *by
|
||||
construction*; concluding "the box sends no SYN-ACK" from that capture was
|
||||
unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it
|
||||
immediately shows `wan1 out 38.120.12.42.80 -> <scanner>: syn ack`.
|
||||
|
||||
**Rule: when testing whether a box *answers*, the sniffer filter must be
|
||||
bidirectional. `dst host` silently answers a different question.**
|
||||
|
||||
### Consequences
|
||||
|
||||
- **ACME renewal will work** with `allowaccess ping`. The earlier "add `http`
|
||||
back or the cert expires" warning is retracted — FortiOS opens the challenge
|
||||
port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27.
|
||||
- **It is not an admin surface** — static 403, no auth, no GUI.
|
||||
- Its practical value is now low: WAN admin is closed, so the cert only serves
|
||||
the internal GUI at 10.250.0.1, where the name would not match anyway. Killing
|
||||
it (`config system acme` → unset interface) would close the last WAN listener
|
||||
at the cost of cert renewal. Operator's call; **not done**.
|
||||
|
||||
---
|
||||
|
||||
## CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE
|
||||
|
||||
### ACME disabled — the WAN IP now exposes nothing
|
||||
|
||||
`config system acme / unset interface` (the account object is left in place;
|
||||
with no interface bound there is no listener). Verified:
|
||||
|
||||
- **External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.**
|
||||
- Internal GUI at 10.250.0.1 still answers **200**, SSH still works.
|
||||
- `admin-server-cert` is still `ana-fw.pfi` — the existing cert is untouched and
|
||||
serves the internal GUI until **2026-10-27**; it simply will not auto-renew.
|
||||
|
||||
Reverse with `config system acme / set interface "wan1"`.
|
||||
|
||||
### RETRACTION: the four VIPs are NOT all-port
|
||||
|
||||
A previous section claimed `Rustdesk`, `https-to-tacticalrmm`, `web-to-webhost`
|
||||
and `web-to-sfcontainer` were unrestricted all-port static NATs. **They are not.**
|
||||
A FortiOS VIP can be scoped **two different ways** and the parser used only
|
||||
checked one:
|
||||
|
||||
1. `set portforward enable` + `set extport <n>` — a single mapped port, **or**
|
||||
2. `set service "<svc>"` on the VIP object — constrains the VIP to that service.
|
||||
|
||||
All four use form 2. The custom services are narrow: `Rustdesk` = TCP
|
||||
21115–21119 + UDP 21116 (the standard RustDesk range), `ssh-mapped-2223` = TCP
|
||||
2223 only. **Every one of the 14 VIPs is scoped; none is unrestricted.**
|
||||
|
||||
**Lesson: absence of `portforward` does NOT mean all-port on a FortiOS VIP —
|
||||
check `service` too.** Better still, do what settled it here: scan from outside
|
||||
rather than reading config.
|
||||
|
||||
### Ground-truth public exposure (external TCP scan, post-change)
|
||||
|
||||
| IP | open | maps to |
|
||||
|---|---|---|
|
||||
| 38.120.12.41 | *nothing* | — |
|
||||
| **38.120.12.42** | ***nothing*** | the FortiGate itself — fully closed |
|
||||
| 38.120.12.43 | 80, 443 | sf-ana-container 10.250.150.100 (SureFire tenant) |
|
||||
| 38.120.12.44 | 22, 80, 443, 8025, 21115–21119 | gitea (→222), traefik, mailrise, RustDesk |
|
||||
| 38.120.12.45 | 80, 443, 2223 | pfi-ana-webhost 10.250.50.52 (2223→22) |
|
||||
| 38.120.12.46 | 443 | pfi-tacticalrmm 10.250.50.57 |
|
||||
|
||||
Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880
|
||||
(Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during
|
||||
the OPNsense translation but not exposure.
|
||||
@@ -1,63 +0,0 @@
|
||||
# [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing
|
||||
|
||||
`hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites
|
||||
(Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for
|
||||
uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach
|
||||
`http://hrafn:8080`. Canonical at `stacks/hrafn/`.
|
||||
|
||||
## Intake found a live credential exposure
|
||||
|
||||
`/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token**
|
||||
— verified as real exposure by reading it as `nobody` on a box with four
|
||||
interactive accounts. Tightened to 0600. That triggered the wider sweep (see
|
||||
[[2026-08-23-ana-docker-env-perms-sweep]]).
|
||||
|
||||
## The CI defect — the one worth remembering
|
||||
|
||||
I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync,
|
||||
tagging the image with the commit SHA for provenance. nevermore-claude later found
|
||||
v1.0.0 deploying "green" while the host still served 0.1.0.
|
||||
|
||||
**Root cause was mine and nastier than either hypothesis.** The staging dir was
|
||||
`$compose_dir/.stage` — **inside** the rsync target. So
|
||||
`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the
|
||||
destination (absent from the source listing) **during** the transfer, destroying
|
||||
its own source mid-copy. Reproduced exactly:
|
||||
|
||||
```
|
||||
before: app.py="OLD" leftover.txt .stage/app.py="NEW"
|
||||
after: app.py="OLD" leftover.txt GONE, .stage GONE
|
||||
```
|
||||
|
||||
Deletion succeeded, the copy silently did not, rsync exited 0. So the directory
|
||||
*looked* converged while host source stayed frozen at the first manual rsync —
|
||||
and because the build's `COPY` inputs never changed, Docker full-cache-hit and
|
||||
every SHA tag aliased one image. **The provenance the tagging existed to provide
|
||||
was false for the pipeline's entire life.**
|
||||
|
||||
**The real failure is the verification.** The verify steps asserted the marker,
|
||||
container health, and a 200 from `/readyz` — all of which pass against a
|
||||
completely frozen host. None measured *content*. A deploy that reports success
|
||||
without asserting the bytes changed is verifying an **uptime**, not a deploy.
|
||||
|
||||
## Fixes
|
||||
|
||||
- stage at `/tmp/hrafn-deploy-stage`, outside the target
|
||||
- CI computes `context_sha256` over the shipped file list; the playbook recomputes
|
||||
it **on the host after the converge** and fails on mismatch
|
||||
- compare the running container's `src/**/*.py` against the host's, so a SHA tag
|
||||
cannot name layers the image lacks
|
||||
- **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the
|
||||
image and `__pycache__` appears at runtime, so a naive `find src -type f` compare
|
||||
false-fails on every healthy deploy. Verified against a known-good container
|
||||
before shipping (12 host files, 18 in container, 0 content differences).
|
||||
- declined `--no-cache`: a cache hit is *correct* when the context is genuinely
|
||||
unchanged; assert the property rather than brute-force it.
|
||||
|
||||
## Access
|
||||
|
||||
Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the
|
||||
pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is
|
||||
canonical; `stacks/hrafn/ci/` is a verified mirror.
|
||||
|
||||
Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`.
|
||||
@@ -1,81 +0,0 @@
|
||||
# [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat
|
||||
|
||||
## Why selene went
|
||||
|
||||
Benchmarked against `gen` on selene's own job — 24 designed judge items with
|
||||
checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a
|
||||
neutral JSON prompt and Selene's **native Atla template** (288 calls, free local).
|
||||
|
||||
```
|
||||
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
|
||||
native Atla selene 21/24 (88%) gen 22/24 (92%)
|
||||
```
|
||||
|
||||
gen won on both templates and **selene's best sat below gen's worst**. Selene was
|
||||
given its own fine-tuned template as a fairness check before any recommendation;
|
||||
it gained one point, not three.
|
||||
|
||||
**Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a
|
||||
winner on every equivalent pair. For eval work that is the case that matters.
|
||||
|
||||
brokkr-smithy-dev independently corroborated from the other end with a **null
|
||||
control** (an excerpt compared against ITSELF, where tie is definitional):
|
||||
`chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth
|
||||
recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96
|
||||
understated it: on a *ranking* task selene was a coin flip. Absolute scoring on
|
||||
designed items is an easier task than ranking real text — the harness is a
|
||||
**screen, not a verdict**, and its README says so.
|
||||
|
||||
Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB).
|
||||
|
||||
## The naming rule, restated the hard way
|
||||
|
||||
I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**:
|
||||
|
||||
> never repoint a named model at a different model's endpoint — that is
|
||||
> intentionally misleading
|
||||
|
||||
`chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and
|
||||
moved to gen with a deterministic judge profile copied from `image-judge`.
|
||||
`selene-1-mini-8b` is a **model** name and was removed outright — it now returns
|
||||
`HTTP 400 Invalid model name`, verified. The discriminator: *does the string
|
||||
promise a capability, or an identity?*
|
||||
|
||||
## The 7-way alias collision — the finding with the longest reach
|
||||
|
||||
```
|
||||
chat-judge classifier gen image-judge
|
||||
qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015
|
||||
```
|
||||
|
||||
Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`;
|
||||
`ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`.
|
||||
|
||||
**Cross-checking a result against another alias measures nothing when they are the
|
||||
same weights — agreement is an echo, not corroboration.** Documented at the head of
|
||||
`model_list` in the live gateway config, because it belongs where people read it.
|
||||
|
||||
This caught a real defect within hours: brokkr's R47 premium-corpus gate was about
|
||||
to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an
|
||||
independent second opinion. They pinned the backing model in the preregistration
|
||||
and dropped the second-alias idea instead.
|
||||
|
||||
## Provenance seam (brokkr's pushback, adopted)
|
||||
|
||||
The gateway returns the **alias** in the response `model` field, not the backing
|
||||
model — so a per-call guard catches a swap *during* a run and is blind to one
|
||||
*between* runs. **Role alias for routing, concrete model for provenance.**
|
||||
`GET :4000/model/info` with the shared key already exposes backing model +
|
||||
api_base; resolve at run start AND end and void on mismatch.
|
||||
|
||||
## Artifacts
|
||||
|
||||
- Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default
|
||||
would silently benchmark a retired seat).
|
||||
- `stacks/selene/` keeps compose + a README explaining the retirement.
|
||||
- Technique worth stealing, from brokkr: **a control constructed so the correct
|
||||
answer is DEFINITIONAL rather than judged cannot inherit the designer's error.**
|
||||
Item vs itself; response vs its own truncation; text vs its own clauses
|
||||
permuted. Add those before adding more judged items.
|
||||
|
||||
Commits `ca3c984`, `b8a5355`.
|
||||
@@ -1,70 +0,0 @@
|
||||
# [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed
|
||||
|
||||
## b187 pre-stage (#405 phases 1+2)
|
||||
|
||||
The matrix bridge stopped embedding the engine and became an HTTP client of the
|
||||
Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the
|
||||
container env, the bridge exits by design. Demo's compose never passed it; the next
|
||||
recreate would have crash-looped. Pre-staged on demo and personal (additive, backed
|
||||
up, verified with `docker compose config`, nothing restarted).
|
||||
|
||||
**Key decision, and I got its scope wrong first.** I argued demo should stay keyless
|
||||
(no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about
|
||||
scope**: the engine preflight authenticates at boot regardless of homeserver, so demo
|
||||
booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's
|
||||
recorded authorization, vaulted, wired, three-hop hash-verified.
|
||||
|
||||
## The 69-day-stale `:latest` landmine
|
||||
|
||||
All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while
|
||||
running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built
|
||||
**2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently
|
||||
downgraded that service by 69 days. Same footgun as the 2026-06-15 outage.
|
||||
|
||||
Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering
|
||||
compose config rather than reading `.env`, containers untouched. Playbook at
|
||||
`playbooks/repin-worldtree-image.yaml`.
|
||||
|
||||
**`worldtree-pinned` was the worst case:** the instance whose entire purpose is being
|
||||
frozen was running a **dangling image with no repo tags**, kept alive only by the
|
||||
running container. One `docker rm` from garbage collection. Tagged
|
||||
`:446e5807bf43` first, then pinned.
|
||||
|
||||
The guard I wrote had two bugs the pinned case exposed: it compared the container's
|
||||
`.Config.Image` **string** (only the tag it was CREATED from — pinned was created
|
||||
from `:latest` back when that meant 446e5807), and it reported CHANGED
|
||||
unconditionally. Now compares **image IDs** and skips when already correct.
|
||||
|
||||
## Matrix homeserver ownership
|
||||
|
||||
Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never
|
||||
missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed
|
||||
instances had them at length **zero**. Someone wired four of six Matrix vars and
|
||||
stopped. Wired them into personal, three-hop verified.
|
||||
|
||||
**The trap worth remembering:** Synapse's registration pointed at
|
||||
`http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at
|
||||
512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is
|
||||
**DEMO's** bridge, and Synapse can reach both — it would have connected, delivered,
|
||||
and looked correct while routing the operator's live rooms to the demo instance.
|
||||
**Personal's bridge is :8010.** `docker port` is ground truth.
|
||||
|
||||
Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified
|
||||
`GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's
|
||||
smoke passed first try: room created, mimir accepted the invite, a real engine turn
|
||||
ran, mimir replied in persona voice. #408 closed.
|
||||
|
||||
## Open on worldtree-dev's side
|
||||
|
||||
- **#411** — personal's bridge logs `Debug sink init failed: Permission denied:
|
||||
/app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist
|
||||
their IDs, so **every restart mints a fresh pair on the live homeserver**. Room
|
||||
litter that compounds silently. Needs a which-container-writes-what check on the
|
||||
sessions volume before anyone chowns it.
|
||||
- Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not
|
||||
listed by the engine on either instance.
|
||||
- Historical Domari pairwise verdicts from the selene era are coin-flip-grade
|
||||
(see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so
|
||||
no future arc leans on them without re-judging.
|
||||
|
||||
Commits `064181a`, `bb19a96`.
|
||||
@@ -1,50 +0,0 @@
|
||||
# [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included
|
||||
|
||||
WAN admin was opened at the start of the session as a cutover contingency
|
||||
("so I don't have to drive down there"), then closed again on operator
|
||||
instruction once the AES-128 work landed. Net result: **the FortiGate's WAN
|
||||
address now exposes no TCP port at all.**
|
||||
|
||||
## Final state
|
||||
|
||||
External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two
|
||||
sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`.
|
||||
|
||||
**Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels
|
||||
drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) —
|
||||
both are recorded in auto-memory `reference_fortigate_ana_gw_access`.
|
||||
|
||||
## Port 80 was the FortiOS ACME listener, and I got it wrong first
|
||||
|
||||
`38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with
|
||||
`allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" —
|
||||
was **wrong**, and the reason is worth keeping:
|
||||
|
||||
> The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
|
||||
> matches inbound only**, so outbound SYN-ACKs were excluded *by construction*,
|
||||
> and concluding "the box sends no SYN-ACK" from that capture was unsound.
|
||||
|
||||
Re-run bidirectionally (`host … and tcp port 80`) it immediately showed
|
||||
`wan1 out 38.120.12.42.80 -> <scanner>: syn ack`. **Rule: to test whether a box
|
||||
*answers*, the filter must be bidirectional.**
|
||||
|
||||
The listener is opened by `config system acme / set interface "wan1"` and
|
||||
**bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It
|
||||
was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`,
|
||||
valid to 2026-10-27) is untouched and simply stops renewing, which is fine
|
||||
because WAN admin is closed and the box is being replaced.
|
||||
|
||||
## Retracted in the same pass: the "four all-port VIPs" alarm
|
||||
|
||||
Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS
|
||||
VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding
|
||||
on the VIP object — and only the first was checked. All 14 VIPs are scoped;
|
||||
`Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only.
|
||||
|
||||
Ground-truth external scan of all six public IPs is recorded in
|
||||
`reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443`
|
||||
(mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up
|
||||
candidates for the OPNsense translation, not exposure.
|
||||
|
||||
**Lesson, twice in one session: measure from outside instead of parsing config.**
|
||||
Both wrong answers came from a filter that answered a different question.
|
||||
@@ -1,167 +0,0 @@
|
||||
# `[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base
|
||||
|
||||
One evening, one thread with brokkr-smithy-dev, five commits: `850e0c3`,
|
||||
`27155c0`, `f509668`+`24e8826`+`1bd90ea`+`3446367`+`8d6a939`, `14ff4a3`,
|
||||
`019ccff`, `5415fd4`.
|
||||
|
||||
## 1. The seat was crash-looping, and the cause was NOT its config
|
||||
|
||||
`vllm-meromero-rp` reported up-but-unreachable, RestartCount climbing (13 by the
|
||||
time it was examined, not the 4 first reported). Startup logs looked clean all
|
||||
the way through weights, `torch.compile` and CUDA-graph capture, then:
|
||||
|
||||
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 336.00 MiB.
|
||||
GPU 0 has a total capacity of 94.97 GiB of which 195.19 MiB is free.
|
||||
|
||||
**⚠ `--gpu-memory-utilization` SIZES THE KV CACHE AND DOES NOT COVER CUDA
|
||||
CONTEXT, GRAPHS OR NON-TORCH OVERHEAD.** gen is configured at 0.43 and actually
|
||||
held 45.6 GiB. char-rp was at 0.51. The pair was committed to 0.94 of the card
|
||||
with ~0.6 GiB of real headroom — it fit on the 21st and stopped fitting on the
|
||||
24th.
|
||||
|
||||
Dropped char-rp to 0.47: ~4.8 GiB margin, KV 27.36 → 23.56 GiB, 430,825 →
|
||||
371,023 tokens against a max-model-len of 262,144. **Cost nothing usable** — the
|
||||
pool still holds 1.4x a full-length sequence; what is lost is concurrent long
|
||||
requests, not context.
|
||||
|
||||
**⚠⚠ THE MISSING HALF, found later that evening: gen's footprint GROWS WITH
|
||||
UPTIME.** Same container, same 0.43: **45.6 GiB after ~3 days up, 38.5 GiB
|
||||
freshly restarted** — ~7 GiB apart. Nothing about char-rp changed between the
|
||||
21st and the 24th; *gen crept up underneath it*. **Headroom arithmetic done
|
||||
against a long-running gen is measuring a moving number.** Measure against a
|
||||
freshly-restarted one.
|
||||
|
||||
## 2. `char-rp` and `char-rp-reasoning` are ONE seat, not two
|
||||
|
||||
Both LiteLLM routes point at `10.250.50.54:8016/v1` — `hosted_vllm/char-rp` and
|
||||
`hosted_vllm/char-rp-thinking`. brokkr had reported 30/80 and 80/80 failure
|
||||
rates as two failing services; it was one outage sampled twice. This also
|
||||
*improved* a result of theirs: their CoT on/off battery had assumed both aliases
|
||||
were the same weights under two chat templates, and the routing detail turned an
|
||||
assumption into a verified fact.
|
||||
|
||||
(`vllm-charrp-reasoning-nvfp4`, the Heretic2 NVFP4+MTP container, has been
|
||||
stopped for 12+ days and is unrelated — it is not what that alias resolves to.)
|
||||
|
||||
## 3. The seat swapped to the Gemma-4 26B-A4B MoE
|
||||
|
||||
Operator-directed straight-across replacement: same port, same
|
||||
served-model-names, so no gateway route or consumer config moved. Rationale is
|
||||
throughput under CoT — the user waits through the whole reasoning block before
|
||||
the first visible token, and the MoE measures ~114 tok/s @32K against the dense
|
||||
31B's ~40.7.
|
||||
|
||||
Serving copy is `RedHatAI/gemma-4-26B-A4B-it-NVFP4` (16 GB), chosen over the
|
||||
other `-it` quants because it is compressed-tensors (`nvfp4-pack-quantized`) —
|
||||
the same loader path the outgoing seat used. Smaller weights at the same 0.47
|
||||
budget bought **1,724,110 KV tokens against the predecessor's 371,023**.
|
||||
|
||||
`meromero-charrp` is retained stopped in `created` state, labelled
|
||||
`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**.
|
||||
|
||||
## 4. ⚠ THE STALE-CHAT-TEMPLATE TRAP IS ENDEMIC, NOT A ONE-OFF
|
||||
|
||||
Verified by hash across every third-party Gemma-4 derivative pulled:
|
||||
|
||||
| build | lines | sha256 (normalised) |
|
||||
|---|---|---|
|
||||
| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` |
|
||||
| RedHatAI NVFP4 (served) | 389 | `6a1015c47ccfcfa6` — the only match |
|
||||
| llmfan46 heretic | 365 | `0a52be69cda5ab8a` |
|
||||
| TrevorJS abliterated | 266 | `58c66fdee4afa297` |
|
||||
| jenerallee78 abliterated | 266 | `58c66fdee4afa297` |
|
||||
| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` |
|
||||
|
||||
Three independent repos carrying the *identical* stale file means it propagated
|
||||
through the ecosystem. Consequences differ by use and **both are silent**:
|
||||
serving a mismatched template renders a different prompt; **training through
|
||||
`base/chat_template.jinja` means training on a different prompt format than
|
||||
production serves** — train/serve skew, no error, presents as a tuning failure.
|
||||
|
||||
The production compose now pins the template explicitly. It is a **no-op for the
|
||||
served weights** (the A4 build ships that exact file) and permanently closes the
|
||||
class. ⚠ If `GEMMA4_MODEL` ever points at a different checkpoint, the template
|
||||
default must move with it.
|
||||
|
||||
## 5. A benchmark result was RETRACTED — below chance indicts the instrument
|
||||
|
||||
A battery appeared to show Gemma at **12% contradiction detection with CoT off
|
||||
against gen's 81%**. An A16 activation-precision control was staged to test
|
||||
whether the quant scheme owned it. Then the operator asked to see the individual
|
||||
items, and the item was **ill-posed**: it presented two mutually contradicting
|
||||
statements and asked for "*the* contradicting statement", but **contradiction is
|
||||
symmetric**. The model consistently named the absolute claim — a defensible
|
||||
reading the labelling scored wrong every time.
|
||||
|
||||
**⚠ THE TELL WAS IN PLAIN SIGHT: 12% ON A FIVE-OPTION TASK IS BELOW THE 20%
|
||||
CHANCE FLOOR.** A below-chance score indicts the instrument before it indicts
|
||||
the model, and neither side reacted to it. I spent the afternoon verifying repo
|
||||
names, config fields, template hashes and tokenizer vocabs — every layer of
|
||||
plumbing — and never asked whether the number itself was *possible*. **A
|
||||
preflight can be thorough and still be aimed in the wrong direction.**
|
||||
|
||||
Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
|
||||
points of contradiction detection" (on a sound instrument it **reverses**); all
|
||||
pre-fix T2 numbers. Recorded as a dated superseded-claims table in
|
||||
`stacks/gemma4-charrp/README.md` rather than a silent edit.
|
||||
|
||||
**What survived:** the A16 control result — activation precision is close to free
|
||||
on this battery, every other task identical across W4A4 and W4A16 builds.
|
||||
|
||||
## 6. The abliterated trainee base — measured, not assumed
|
||||
|
||||
Operator directed a low-damage abliterated instruct build. "Low damage" was
|
||||
treated as a measurable claim; the field spreads from KL 0.09 to 0.4118:
|
||||
|
||||
| build | method | KL | refusals |
|
||||
|---|---|---|---|
|
||||
| **llmfan46** (operator's pick) | Heretic v1.2.0 ARA | 0.1237 | 3/100 |
|
||||
| TrevorJS | ARA-family | 0.09 | 1/100 effective, 5/686 cross-dataset |
|
||||
| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT |
|
||||
| huihui-ai | remove-refusals | none published | none published |
|
||||
|
||||
Fleet anchor: our own work found **Heretic at KL 0.12 preserved the MTP head at
|
||||
83.7% acceptance**, so both staged builds sit inside an already-measured band.
|
||||
huihui-ai rejected — no metrics, its card calls the method "a crude,
|
||||
proof-of-concept implementation", it abliterates both thinking and non-thinking
|
||||
modes, and its parameter count runs ~738M over upstream. Operator's independent
|
||||
read matched ("huihui produces garbage").
|
||||
|
||||
**Abliteration isolated properly** (stock BF16 vs llmfan46 BF16, same precision,
|
||||
same pinned template, same 192 items):
|
||||
|
||||
T2 contradiction 75% → 59% (−5 items)
|
||||
T6 spatial 75% → 88% (+4 items)
|
||||
core 90.0% → 89.4% (−0.6 pts)
|
||||
|
||||
**It MOVED capability rather than removing it** — five lost on contradiction,
|
||||
four gained on spatial, nearly cancelling. Nobody predicted a gain. **llmfan46
|
||||
stands**; no case for re-staging on TrevorJS over 0.6 points.
|
||||
|
||||
⚠ Read as ~5 and ~4 items at n=32, not as −15.6/+12.5 percent. ⚠ Says nothing
|
||||
about quantization — the stock-NVFP4 T2 figure came from n=16 against n=32,
|
||||
different item sets, n-confounded.
|
||||
|
||||
## 7. ⚠ The production compose hardcodes `--quantization compressed-tensors`
|
||||
|
||||
Pointing the char-rp stack at unquantized BF16 weights crash-loops immediately:
|
||||
|
||||
TypeError: CompressedTensorsConfig.__init__() missing 3 required
|
||||
positional arguments: 'target_scheme_map', 'ignore', 'quant_format'
|
||||
|
||||
vLLM trying to read a quantization config out of a checkpoint that has none. 35
|
||||
restarts before it was caught. Hence `stacks/gemma4-trainee-bench/` — a separate
|
||||
ephemeral stack with no quantization flag, `restart: "no"` so a bench seat cannot
|
||||
resurrect itself and block gen's restore, and no homepage labels so it leaves no
|
||||
permanently-offline card. That detour is why a base swap is now ~5 minutes
|
||||
instead of ~15.
|
||||
|
||||
## 8. BF16 cannot coexist with gen
|
||||
|
||||
48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before
|
||||
a byte of KV cache. Every BF16 bench window means **gen is stopped**. Two such
|
||||
windows were run and gen restored both times; the restore was triggered by
|
||||
observing the seat's own throughput logs (a large prefill burst then zero
|
||||
running/zero waiting) rather than waiting on a courtesy message.
|
||||
|
||||
Cross-links: [[2026-08-24-homepage-uniform-grid]]
|
||||
@@ -1,77 +0,0 @@
|
||||
# [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live
|
||||
|
||||
Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup
|
||||
shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three
|
||||
wrong turns worth recording.
|
||||
|
||||
## Root cause
|
||||
|
||||
`esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises
|
||||
**itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over
|
||||
the DHCPv4-supplied resolver, so queries went to the UDM — which does not know
|
||||
`.internal` — and returned NXDOMAIN. AdGuard was never consulted.
|
||||
|
||||
Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver
|
||||
was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled
|
||||
entirely**, so every host there got the UDM and could never resolve `.internal`
|
||||
— esh-docker-vm was living proof.
|
||||
|
||||
## The three wrong turns
|
||||
|
||||
1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an
|
||||
explicit server is supplied**. Setting it bare looks like a no-op and invites
|
||||
the conclusion that the field is inert — which I drew, wrongly. Despite the
|
||||
`dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network.
|
||||
2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to
|
||||
AdGuard persists, reads back, and changes nothing. Proven with **fresh
|
||||
uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned
|
||||
real IPs. Reverted.
|
||||
3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the
|
||||
known `cmd/devmgr` behaviour.
|
||||
|
||||
Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were
|
||||
probed with a stdlib raw-socket Router Solicitation parsing option type 25
|
||||
(`rdisc6`/`tcpdump` were both absent; nothing was installed).
|
||||
|
||||
## What landed
|
||||
|
||||
RDNSS **redirected** rather than disabled — better than switching it off:
|
||||
|
||||
| VLAN | v4 | v6 RDNSS |
|
||||
|---|---|---|
|
||||
| `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
|
||||
| `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
|
||||
|
||||
The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over
|
||||
Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on
|
||||
SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client
|
||||
accepts it and never retries. A secondary that doesn't know your private zone is
|
||||
a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing
|
||||
camera DNS through AdGuard's filtering risks their cloud features.
|
||||
|
||||
## The naming scheme became real
|
||||
|
||||
The resolver address is the scheme's first live use, replacing a MAC-derived
|
||||
SLAAC address that would break on a NIC change. All three `esh-server` Linux
|
||||
hosts now carry `4411:B105` ("FOR ALL BIOS"):
|
||||
|
||||
```
|
||||
esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55
|
||||
esh-vm-db …:4411:b105:50:60
|
||||
```
|
||||
|
||||
Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals
|
||||
on re-delegation), backgrounds itself with a retry (SLAAC may not have landed;
|
||||
a blocking hook would stall bring-up on a headless box), and adds nothing to
|
||||
existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets
|
||||
`accept_ra=0` and would strand the host.
|
||||
|
||||
⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only
|
||||
despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores
|
||||
RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with
|
||||
`accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default
|
||||
route** — an IPv6 identity with no change to a hypervisor's routing. Expect this
|
||||
on every Proxmox node when its LAN gets v6.
|
||||
|
||||
Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits:
|
||||
auto-memory `reference_unifi_dns_rdnss_limits`.
|
||||
@@ -1,308 +0,0 @@
|
||||
# `[2026-08-24]` Homepage: remote-label consumption re-verified, then the board relaid out on a uniform grid
|
||||
|
||||
Prompted by the operator: *"Homepage on esh-vm-docker lists remote dockers and
|
||||
can absolutely consume their labels, please verify again. I am still
|
||||
unsatisfied with the layout and aesthetics."*
|
||||
|
||||
## The verification — the operator was right, and the record now says so
|
||||
|
||||
**Homepage on `esh-docker-vm` discovers services by container label from all
|
||||
five Docker engines in `conf/docker.yaml`, not just its own.** This is not an
|
||||
inference; `GET /api/services` returns every card's `server` field, and the
|
||||
2026-08-24 snapshot resolves to:
|
||||
|
||||
| `server` | host | label-discovered services |
|
||||
|---|---|---|
|
||||
| `ana-pfi-docker` | 10.250.50.70 | 30 |
|
||||
| `irv-ml1-docker` | 10.100.79.3 (over WireGuard) | 15 |
|
||||
| `ana-ml2-docker` | 10.250.50.54 | 14 |
|
||||
| `esh-vm-docker` | 10.0.50.45 (the dashboard's own host) | 13 |
|
||||
| `nh3-pfi-docker` | 10.100.50.40 | 2 |
|
||||
|
||||
**74 of 107 cards are label-discovered, and only 13 of those come from the
|
||||
dashboard's own engine** — the other 61 are read off four remote hosts,
|
||||
including irv-ml1 across the WireGuard tunnel. The remaining 33 carry
|
||||
`server: null`: those are the manual `services.yaml` entries — hardware, BMCs,
|
||||
hypervisors, printers, and user-level systemd services that have no container
|
||||
to label in the first place. **That null is the only thing "not label-driven"
|
||||
about this dashboard**, and it is a property of the entry, not of the host it
|
||||
points at.
|
||||
|
||||
⚠ If a future session doubts this again, the check is one command and takes two
|
||||
seconds — do not reason about it from the docs:
|
||||
|
||||
```bash
|
||||
curl -s http://10.0.50.45:5100/api/services \
|
||||
| jq -r '.[].services[] | .server' | sort | uniq -c
|
||||
```
|
||||
|
||||
## What was actually wrong with the layout
|
||||
|
||||
Measured with Playwright against the live board (per-group `card=` width, card
|
||||
height spread, and a geometric title-vs-status overlap test), not judged by
|
||||
eye:
|
||||
|
||||
- **Card width changed at every group boundary.** `columns:` is not a density
|
||||
dial — it sets `lg:grid-cols-N` for one group, so it fixes that group's card
|
||||
width. Notes rendered a single **1464px** card; News and Media **728px**;
|
||||
Eval & Retrieval **286px**; everything else 360px. Scrolling the page, the
|
||||
grid resized five times.
|
||||
- **Long names printed underneath their own status pill.** Measured by
|
||||
re-injecting the old rule and testing the title text node's box against the
|
||||
status cluster's box: **6 cards, all on the AI tab** — 3 in Inference, 2 in
|
||||
Dormant, 1 in Eval & Retrieval; zero on the other three tabs, which is why
|
||||
it survived earlier passes. Root cause is a genuinely counter-intuitive one:
|
||||
the rule reserved a
|
||||
78px gutter with `padding-right` and relied on `overflow: hidden` to hold it,
|
||||
but **overflow clips at the padding box, not the content box** — so the
|
||||
reserved gutter was spill room the title rendered straight through. The
|
||||
intended `text-overflow: ellipsis` never fired either, because the ellipsis
|
||||
is painted by whichever block's own line overflows, and here that is the
|
||||
anonymous box around the bare title text node, which does not carry
|
||||
`overflow`.
|
||||
- **`AI Systems` / Scriberr was on all four tabs** — the 2026-08-18 UltraSeedbox
|
||||
bug recurring, this time arriving from a container label rather than from
|
||||
`settings.yaml`.
|
||||
- **Icons were grey smudges.** Homepage masks every glyph over
|
||||
`--color-logo-start/stop`; stock slate-400 → slate-700 sinks the bottom half
|
||||
of each icon into the card fill.
|
||||
- Bookmark groups and Jellyfin's trailing stream rows were the two components
|
||||
the theme had never reached.
|
||||
|
||||
## The fixes
|
||||
|
||||
`stacks/homepage/conf/settings.yaml` — **all 20 groups to `columns: 4`.**
|
||||
`stacks/homepage/theme/australis.css.in` → rebuilt → `conf/custom.css`:
|
||||
gutter held by wrapping, description clamped to 3 lines (floor still 2), icon
|
||||
ramp overridden, bookmark + trailing-widget components themed, group gap
|
||||
10px → 22px. `stacks/scriberr/compose.yaml` — `homepage.group` → `AI - Audio
|
||||
Tools`, container recreated on ana-ml2.
|
||||
|
||||
After: **every group renders at card=360**, and the collision count is zero.
|
||||
|
||||
Before/after, all four tabs: `http://10.100.10.50:8090/b/homepage-relayout/`
|
||||
(24h TTL; also on the standing link board).
|
||||
|
||||
## ⚠ Three traps worth carrying forward
|
||||
|
||||
1. **"Columns = member count" is RETIRED** (it was the 2026-08-18 rule). It was
|
||||
avoiding dead cells in a short last row and bought a worse defect. A short
|
||||
last row is what a grid looks like; a card wider than its neighbours is what
|
||||
a mistake looks like.
|
||||
2. **A `:root` override of a Homepage theme variable is silently ignored.**
|
||||
Homepage sets `--color-logo-*` on `.theme-slate`, and that class is on the
|
||||
`<html>` element — the same element `:root` matches. `.theme-slate` (0,1,0)
|
||||
beats `:root` (0,0,1), so the override does nothing and looks like the
|
||||
variable is not the one in play. `html[class]` (0,1,1) wins, and does not
|
||||
hard-code which `theme-*` class is active. Specificity alone is not enough
|
||||
either: a custom property resolves from the *nearest* ancestor that sets it,
|
||||
so the override has to land on `<html>`, not on `<body>`.
|
||||
3. **The post-recreate tab-bar loss is INTERMITTENT, not guaranteed.** The
|
||||
2026-08-19 note reads as though every recreate costs up to an hour of broken
|
||||
render. This recreate came up correct within 10 seconds — fresh payload on
|
||||
the first poll, all four tabs clickable a minute later. Recreate, *check*,
|
||||
and only then walk away if it is actually in the broken state.
|
||||
|
||||
Also re-confirmed, since the change depended on it: **a `settings.yaml` edit
|
||||
needs a container recreate, not a restart.** `docker restart homepage` left the
|
||||
old `"columns":1` payload embedded in the served HTML with the correct file
|
||||
mounted and readable inside the container; `compose up -d --force-recreate`
|
||||
cleared it immediately.
|
||||
|
||||
## Deliberately not done — operator's call
|
||||
|
||||
The Main tab still opens on three sparse bands: **Notes** (1 member) and
|
||||
**Games** (1) each burn a full 4-wide row, and **News** has 2. Merging Notes +
|
||||
News, or folding Games into Apps, would tighten the top of the page — but that
|
||||
is information architecture, not layout, and the group names are the operator's.
|
||||
Surfaced rather than done.
|
||||
|
||||
→ **Resolved in pass 2 below**, where the operator delegated the naming
|
||||
("re-categorize however you want"). Notes + News became `Daily`, Games folded
|
||||
into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded
|
||||
(Scriberr moved on to `AI - Studios`).
|
||||
|
||||
---
|
||||
|
||||
# `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed
|
||||
|
||||
Operator, after seeing pass 1: *"You can re-categorize however you want.
|
||||
service networking tab is uneven, you can split out the adguard cards, etc.
|
||||
most of the issues are that tools I use and have a UI are interspersed with API
|
||||
endpoints which are largely informational only. They might even go in their own
|
||||
cards or start collapsed."*
|
||||
|
||||
## The axis
|
||||
|
||||
Every group is now either **tools** (expanded, top of tab) or **endpoints** (an
|
||||
API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A
|
||||
collapsed group still renders its eyebrow and rule, so presence costs one line
|
||||
instead of two rows.
|
||||
|
||||
Second, quieter rule that fell out of the same pass: **a group's members should
|
||||
all carry a widget or none should.** A stat strip adds ~50px, so one widget card
|
||||
in a row of plain ones opens a void under the plain ones — which is most of what
|
||||
made the 13-member `Service Networking` band look broken.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the
|
||||
1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the
|
||||
1-card `Games` band), `Media`, `UltraSeedbox`.
|
||||
- **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then
|
||||
`AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4),
|
||||
`AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed.
|
||||
- **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2),
|
||||
`Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed).
|
||||
- **Infrastructure** — unchanged; every card there is already a console.
|
||||
|
||||
Measured after: every group `card=360`, and `DNS & Filtering` and
|
||||
`Reverse Proxies` both `h=134..134` — dead flush.
|
||||
|
||||
## ⚠ The move that made it affordable
|
||||
|
||||
**The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is
|
||||
read at container **creation**, so renaming `AI - Inference` to something
|
||||
clearer would have meant recreating six vLLM seats plus four eval seats plus
|
||||
four TTS engines — multi-minute model reloads on endpoints peers reach through
|
||||
the gateway. Order plus `initiallyCollapsed` buys the same separation for free,
|
||||
so the names stay ugly on purpose. **Do not spend that recreate on a label.**
|
||||
|
||||
28 containers *were* relabelled — all cheap web services — via five rerunnable
|
||||
elway playbooks, `playbooks/homepage-regroup-<host>.yaml`. The canonical
|
||||
`stacks/` tree was synced to match afterwards, so intent and reality agree.
|
||||
|
||||
`initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in
|
||||
this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in
|
||||
`/app/.next/server/pages/index.js`).
|
||||
|
||||
## AdGuard (ANA) gained its widget, and the credential is fleet-wide
|
||||
|
||||
It was the only AdGuard without a query/blocked/latency strip, so it sat short
|
||||
beside two tall siblings. **One `infra-ops` AdGuard login authenticates against
|
||||
all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200
|
||||
on `POST /control/login`, verified 2026-08-24). Vaulted at
|
||||
`secret get nh3-dev/adguard-infra-ops-password`; written to
|
||||
`/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon
|
||||
was also the odd one out (`mdi-dns` against two `si-adguard`).
|
||||
|
||||
## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION
|
||||
|
||||
The biggest durable finding of the day, and it cost ~25 minutes. Full write-up
|
||||
in `stacks/homepage/README.md`; the short version:
|
||||
|
||||
`initialSettings":{}` in the served HTML is **the catch branch** of the page's
|
||||
data loader, not a warm-up and not a cache. And the error can vanish without
|
||||
trace: the logger is assigned as the first statement *inside* the same `try`,
|
||||
and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is
|
||||
written anywhere — which is exactly what was observed.
|
||||
|
||||
Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`,
|
||||
`/api/widgets` and `/api/hash` all return **200 with correct content** while the
|
||||
page serves `{}`; restoring the previous known-good `settings.yaml` reproduces
|
||||
it identically; `/api/validate` returns `[]`; disk and permissions are fine.
|
||||
|
||||
**One-command test:**
|
||||
`curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'`
|
||||
|
||||
**What broke the streak:** three consecutive recreates came up empty, then
|
||||
rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed
|
||||
it within 15 seconds. That is one observation, not proof — but it is a coherent
|
||||
mechanism (oversized log → logger init throws → silent catch) and it is the
|
||||
cheapest thing to try first next time.
|
||||
|
||||
---
|
||||
|
||||
# `[2026-08-24, pass 3]` Rebuilt on Australis Skyfall — dual theme, light shipped
|
||||
|
||||
Operator supplied the Skyfall design-system README and said "Go full with
|
||||
skyfall."
|
||||
|
||||
## The bundle was already in this repo's git history
|
||||
|
||||
**The Skyfall tokens did not need to be hunted down.** A predecessor vendored
|
||||
them on 2026-08-19 and a later commit deleted them; git kept everything:
|
||||
|
||||
```bash
|
||||
git show 45c1995:stacks/homepage/theme/colors.css # 208 lines, BOTH themes
|
||||
git show 45c1995:stacks/homepage/theme/layout.css # calm-depth tokens
|
||||
git show 45c1995:stacks/homepage/theme/typography.css
|
||||
git show 45c1995:stacks/homepage/theme/fonts/Supreme-{400,500,700}.woff2
|
||||
```
|
||||
|
||||
`colors.css` carries `:root` (dark) **and** `[data-theme="light"]` (Skyfall
|
||||
Day) — so the light ramp is canonical, not derived. That killed the entire
|
||||
objection from the previous answer, which was correct only about the
|
||||
`australis-design` skill ("Always dark first. No light mode in this system").
|
||||
**Skyfall is the dual-theme derivative; australis-design is the terminal
|
||||
theme. They are different systems and only one of them has a light mode.**
|
||||
|
||||
## ⚠⚠ REMOVING `theme:` FROM settings.yaml BREAKS THE DASHBOARD
|
||||
|
||||
The documented way to get Homepage's own light/dark toggle is to leave `theme:`
|
||||
unpinned. **Do not.** With the key absent, the page's data loader throws and its
|
||||
catch branch serves `initialSettings: {}` — no tab bar, no layout, no i18n.
|
||||
|
||||
Measured, not inferred: six force-recreates over seven minutes all came up
|
||||
empty with the key removed; restoring `theme: dark` rendered correctly on the
|
||||
next recreate in **12 seconds**. `/api/services` stays 200 and fully correct
|
||||
throughout, which is exactly why this reads as a caching or warm-up problem and
|
||||
is not one.
|
||||
|
||||
This is the first *confirmed* trigger for the long-running "tab bar goes
|
||||
missing" mystery. It does not explain every occurrence (the symptom has
|
||||
appeared with `theme:` present), but it means **the first diagnostic step is
|
||||
now `git log -p -- stacks/homepage/conf/settings.yaml`**, not container
|
||||
archaeology. Also retires an earlier lead from this same session: rolling the
|
||||
8.6 MB `homepage.log` aside once coincided with a recovery, but did nothing
|
||||
during the `theme:`-key episode — coincidence, not cause.
|
||||
|
||||
## So the toggle is ours
|
||||
|
||||
`conf/custom.js` renders it (was an empty placeholder). Precedence:
|
||||
|
||||
1. explicit choice — `localStorage['skyfall-theme']`, written by the toggle;
|
||||
2. OS preference — `@media (prefers-color-scheme: light)`;
|
||||
3. dark — Skyfall's default.
|
||||
|
||||
`theme/build.py` re-emits each vendored `[data-theme="light"]` block twice: as
|
||||
`[data-theme="light"], html.light`, and inside the media query scoped to
|
||||
`html:not([data-theme="dark"]):not([data-theme="light"])`. **That `:not()` pair
|
||||
is what lets a stored *dark* choice survive a light-mode OS.** Verified across
|
||||
both OS preferences: load, click, click again, reload — all four correct.
|
||||
|
||||
⚠ Homepage keeps its own `class="dark scheme-dark theme-slate"` on `<html>`
|
||||
regardless, because `theme:` is pinned. That is fine and was checked
|
||||
explicitly: with the dark class present AND `data-theme="light"`, every themed
|
||||
surface resolves to Skyfall Day, because our rules carry `!important` on the
|
||||
surfaces Tailwind's `dark:` variants would otherwise claim. **`data-theme` is
|
||||
the control surface; the class is not.**
|
||||
|
||||
## The anti-fork guard is now mechanical
|
||||
|
||||
`build.py` records the SHA-256 of each vendored file and **fails the build** on
|
||||
a mismatch, rather than warning. A vendored file is either byte-identical to
|
||||
the bundle or it is a fork wearing the bundle's name. Overrides go in
|
||||
`skyfall.css.in`, which is written entirely against the semantic layer
|
||||
(`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/--warning`) — no
|
||||
raw family tokens, no colour literals.
|
||||
|
||||
The one place a literal is unavoidable: Homepage consumes
|
||||
`--color-logo-start/stop` as `rgb(var(--x))`, which cannot take an `oklch()`.
|
||||
Those four values are exact sRGB conversions of real tokens (`--sea-80`,
|
||||
`--blue-base` for dark; `--sea-40`, `--blue-deep` for light), computed rather
|
||||
than eyeballed, with the conversion recorded in the file.
|
||||
|
||||
## Deviations, all deliberate and all written down
|
||||
|
||||
- **The aurora ribbon under the tab bar is gone.** Skyfall sanctions exactly two
|
||||
accent expressions — the active rail and hero-only glows — and a decorative
|
||||
gradient across the chrome is neither. The colour moved to a 2px accent bar
|
||||
plus `--accent-soft` fill on the active tab, which *is* the rail.
|
||||
- **Widget stat values moved from the display face to mono**, per Skyfall's
|
||||
"numbers and telemetry are always `--font-mono`".
|
||||
- **Two font substitutions**: Space Grotesk for Bespoke Sans, JetBrains Mono
|
||||
for Victor Mono. Only Supreme was ever vendored, and Skyfall's own notes call
|
||||
Victor Mono "user-supplied". Two-line swap when the real faces arrive.
|
||||
|
||||
Dark + light, all four tabs: `http://10.100.10.50:8090/b/homepage-skyfall/`
|
||||
@@ -1,46 +0,0 @@
|
||||
# [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1
|
||||
|
||||
Self-hosted audio/video transcription + diarization. Operator chose GPU
|
||||
placement over ana-docker (8 cores shared with 50 containers, 37 GB disk)
|
||||
against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom.
|
||||
|
||||
**Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`,
|
||||
all seven backends up, zero failures: `whisperx pyannote sortformer parakeet
|
||||
canary voxtral openai`. ~30 GB of weights on `/tank`.
|
||||
|
||||
Stack: `stacks/scriberr/`. Full gotcha list in auto-memory
|
||||
`reference_scriberr_ana_ml2`.
|
||||
|
||||
## Three upstream bugs, none of them ours
|
||||
|
||||
**1. The Blackwell image does not exist.** Upstream's README documents
|
||||
`scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published
|
||||
`scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or
|
||||
silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9`
|
||||
(CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose
|
||||
back to the published image.
|
||||
|
||||
**2. It must run as uid 10001, not 1000** — and the error lies:
|
||||
`unable to open database file: out of memory (14)`. Error 14 is
|
||||
`SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates
|
||||
`appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to
|
||||
it, while the entrypoint's PUID remap covers only the data dirs.
|
||||
**Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the
|
||||
mount; fails on a plain named volume too → not the storage; the **published CPU
|
||||
image works at PUID=1000** because there `appuser` *is* 1000.
|
||||
Generalisable: *when a container "permission" bug appears, compare the uid the
|
||||
image was BUILT for against the uid you are RUNNING as.*
|
||||
|
||||
**3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env
|
||||
with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with
|
||||
`Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial
|
||||
failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet
|
||||
and Sortformer were silently absent. Occurrences 2 → 0 after the fix.
|
||||
|
||||
## Related
|
||||
|
||||
`speaches` on irv-ml1 **stopped** the same day (stack retained, one command to
|
||||
restart): Eyra was abandoned pre-implementation because Scriberr covers the need,
|
||||
leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and
|
||||
is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the
|
||||
~5.9 GB quoted — that figure is the loaded-model working set.
|
||||
@@ -1,92 +0,0 @@
|
||||
# ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## The run
|
||||
|
||||
1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0
|
||||
20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it)
|
||||
adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json
|
||||
|
||||
**47% faster than the round-1 projection of 13.85h**, from two changes: the
|
||||
bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3
|
||||
across the four 100-step windows — which means the 35-46.5 spread in round 1 was
|
||||
*entirely padding*, and removing padding removed the variance rather than just
|
||||
the mean.
|
||||
|
||||
⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a
|
||||
rolling average.** The first two were instantaneous tqdm readings off a number
|
||||
that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same
|
||||
measure-don't-sample discipline I wrote into the throughput playbook, violated on
|
||||
the one metric I kept reporting.
|
||||
|
||||
## lora_B gate — PASSED, twice
|
||||
|
||||
checkpoint-100 205/205 non-zero, median norm 0.829
|
||||
final adapter 205/205 non-zero, median norm 1.708
|
||||
vision_tower tensors: 0 on both
|
||||
|
||||
Median norm rising 0.829 -> 1.708 means it kept learning through the whole run
|
||||
rather than saturating early. This check **never ran in round 1** (died at step
|
||||
19, first checkpoint was 100) and it is the only failure mode that stays
|
||||
invisible until the acceptance gate reports base-identical numbers.
|
||||
|
||||
## The gate — brokkr-smithy-dev
|
||||
|
||||
**It did the thing it was built to do:**
|
||||
|
||||
metric base A/B tuned delta floor
|
||||
attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt
|
||||
diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008
|
||||
|
||||
Diversity moved **22x its own noise floor**. Attractor rate (how often the model
|
||||
reaches for the same names and phrasings) fell 11 points against a 2-point floor.
|
||||
|
||||
T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2
|
||||
memorisation: 0.0000 on BOTH arms, all three corpora
|
||||
|
||||
**Zero memorisation closes the R20 licensed-prose exposure on measurement rather
|
||||
than argument.**
|
||||
|
||||
⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192
|
||||
generations to truncation/degeneracy against base's 1-2. Lopsided exclusions
|
||||
plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor;
|
||||
the number carries an asterisk.
|
||||
|
||||
## The noise-floor near-miss — the methodology lesson
|
||||
|
||||
brokkr was one step from reporting a 13-point T6 regression **that sat inside
|
||||
twice his instrument's own variance.**
|
||||
|
||||
--per-type 32 max swing across tasks: 9 points
|
||||
--per-type 128 max swing across tasks: 1 point
|
||||
|
||||
His gate criterion is "no task regresses by more than one item" = 3.1 points at
|
||||
n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a
|
||||
preregistered gate at 4x finer resolution than it could resolve, and caught it by
|
||||
running a control he did not strictly need. Quadrupling n collapsed the noise
|
||||
exactly as binomial statistics predicts.
|
||||
|
||||
⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset,
|
||||
so with a 218,625-token KV cache the scheduler batches freely up to vLLM's
|
||||
default of 256. Continuous batching changes reduction order and borderline items
|
||||
flip. Temperature 0 buys deterministic *sampling*, not deterministic
|
||||
*arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right
|
||||
reason: a floor measured on a seat serving one request at a time is not the floor
|
||||
that applies to the seat we ship.
|
||||
|
||||
## The confound I built and he caught
|
||||
|
||||
I optimised a pipeline for production and then handed him its output as an eval
|
||||
instrument **without asking whether those were the same job.** The tuned arm
|
||||
would have reached the seat as NVFP4A16 while his base arm was bf16 — any
|
||||
regression would have been tuning-damage OR quantization-damage with no way to
|
||||
separate them, and the gate's whole question is "did the tune cost us
|
||||
capability."
|
||||
|
||||
**Both arms now bf16, same seat, same port, argv differing in exactly two
|
||||
lines** (weights path, served name), template sha256 identical
|
||||
(`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all
|
||||
three launches). Quantization moved *downstream* of the gate.
|
||||
|
||||
See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have.
|
||||
@@ -1,86 +0,0 @@
|
||||
# The 8.6% MFU was an accounting artifact — attention on Ampere kernels
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## The answer
|
||||
|
||||
**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided
|
||||
the *intended* (windowed) FLOPs by the wall time the *dense* reality took.
|
||||
|
||||
nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP
|
||||
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14
|
||||
padded full layers lose the causal skip = +3.5e14
|
||||
work performed ~ 1.8e15 = 51-61 TFLOPS
|
||||
|
||||
The card was doing ~2x the arithmetic the architecture specifies, and the excess
|
||||
was the sliding window being computed and thrown away.
|
||||
|
||||
## Two independent methods agreed
|
||||
|
||||
scaling fit (3 points, 2 params, residuals <3ms over 8x range)
|
||||
A = 6.87e-4 s/token B = 8.85e-8 s/token^2
|
||||
quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384
|
||||
|
||||
kernel table (device rows only)
|
||||
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
|
||||
dense GEMM 2,774.0 ms 7.9%
|
||||
other 5,739.0 ms 16.4%
|
||||
|
||||
**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit
|
||||
needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM
|
||||
launches per forward are not the cost).
|
||||
|
||||
## The mechanism, source-verified by brokkr's panel (arm: Bil)
|
||||
|
||||
masking_utils.py:292-301 _ignore_causal_mask_sdpa requires
|
||||
kv_length < local_attention_size. 16384 >= 1024,
|
||||
so THE SLIDING MASK ALWAYS MATERIALISES.
|
||||
sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask
|
||||
sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable
|
||||
Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12
|
||||
|
||||
⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**,
|
||||
with the forward on `gmem`, the memory-efficient backend's slowest fallback tier.
|
||||
|
||||
## What actually fixed it
|
||||
|
||||
**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and
|
||||
78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
|
||||
back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work**
|
||||
at fixed width, because an explicit mask knocks those layers off the fast path.
|
||||
|
||||
⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per
|
||||
accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does
|
||||
all the mixing; the bucket only costs padding. Use the tightest bucket.
|
||||
|
||||
**flex_attention** — Triton-generated so it compiles for sm_120 instead of
|
||||
shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
|
||||
`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a
|
||||
**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting
|
||||
into the full 99 KB, so it is the card, not a default.
|
||||
|
||||
## ⚠⚠ The trap that produced TWO wrong published conclusions
|
||||
|
||||
`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence
|
||||
width is a new shape. On hitting the ceiling dynamo does not error — it silently
|
||||
falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work
|
||||
with the backwards pass and may produce incorrect results."*
|
||||
|
||||
That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss
|
||||
divergence**, and I believed and reported both. Raising the limit to 256 flipped
|
||||
the speed result to 1.41x.
|
||||
|
||||
The loss divergence turned out to be real but benign — adjudicated against fp32
|
||||
MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally
|
||||
CLOSER at every width. **Do not re-open it by comparing the two backends to each
|
||||
other; that cannot answer it. Compare to fp32.**
|
||||
|
||||
## Process lesson
|
||||
|
||||
brokkr's panel produced **four self-retractions in ninety minutes**. Every
|
||||
retraction was a derivation; every survivor was a measurement. And the whole
|
||||
head_dim-512 SDP problem was **already documented in zerofata's published Axolotl
|
||||
config since April** — the right first stop for "why is this architecture slow"
|
||||
is practitioner configs for that exact base, before any panel.
|
||||
|
||||
Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`.
|
||||
@@ -1,84 +0,0 @@
|
||||
# NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
Pipeline at `scripts/erp-tune-serve/` (commits `6a85829`, `ab980e9`).
|
||||
Validated end-to-end against checkpoint-100 before the real adapter existed.
|
||||
|
||||
## ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert
|
||||
|
||||
before linearize_moe: 427 Linears, 205 targeted, experts 0
|
||||
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
|
||||
(30 layers x 128 experts x 3 projections)
|
||||
|
||||
Gemma-4 stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors
|
||||
(`gate_up_proj` [128,1408,2816], `down_proj` [128,2816,704]) — note the absent
|
||||
`.weight` suffix. A Linear-targeting recipe resolves 205 of 427 modules and
|
||||
**zero experts**, leaving 22.84 B params (88.5% of the model) in BF16 with no
|
||||
warning.
|
||||
|
||||
**This is the same defect that killed QLoRA here via bitsandbytes.** The blind
|
||||
spot is in the *checkpoint layout*, not the tool. Fix:
|
||||
`llmcompressor.modeling.moe.linearize.linearize_moe` — no registration needed,
|
||||
Gemma-4 satisfies `FusedExpertsProtocol` structurally. Playbook §3.15.
|
||||
|
||||
## Scheme: NVFP4A16, deviating from the playbook default, on measured grounds
|
||||
|
||||
brokkr benched the W4A4 quant of this checkpoint at **12% on contradiction
|
||||
detection with CoT off against gen's 81%** — the signature of 4-bit input
|
||||
activations on a reasoning-dense task. Plus W4A4 KLD is 2-4x worse past ~10k ctx
|
||||
on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill cost accepted.
|
||||
|
||||
⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` —
|
||||
W4A4 wearing an A16 label. The script refuses if the emitted config says 4.
|
||||
|
||||
## Four silent defects the dry run found
|
||||
|
||||
1. **transformers 5.15 MIGRATES the config schema on save** — drops
|
||||
`global_head_dim`/`num_global_key_value_heads`, writes `per_layer_config`.
|
||||
transformers 5.10 (the llmcompressor venv) then reads `num_key_value_heads`
|
||||
as None and dies with `TypeError: unsupported operand type(s) for //`.
|
||||
Every working artifact on the box uses the OLD schema. Merge now downgrades it.
|
||||
2. **llmcompressor cannot auto-init a processor for a multimodal checkpoint** —
|
||||
pass the tokenizer explicitly as `processor`.
|
||||
3. **`save_pretrained` does not carry `processor_config.json`** — vLLM then fails
|
||||
with "Can't load feature extractor", which reads as a vision bug.
|
||||
4. **The quant needs more than GPU1's free 32 GiB.** `quant_with_gen_down.sh`
|
||||
stops `vllm-gen` and restores it from a trap on EVERY exit path, using
|
||||
`docker start` not `compose up` so the container returns with its exact config.
|
||||
|
||||
## Verified on the emitted artifact
|
||||
|
||||
49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16)
|
||||
weight_packed 11,725 of which expert 11,520
|
||||
tokenizer truncation: clean (§3.14 trap avoided by calibrating on the
|
||||
encode cache, so the tokenizer is never called
|
||||
with truncation=True at all)
|
||||
served: Marlin NVFP4 kernel + Marlin MoE backend, coherent generation
|
||||
|
||||
⚠ The reference `nvfp4a16` artifact triggers a vLLM warning that q/k/v carry
|
||||
*different* weight global scales ("likely reduced accuracy"). **Ours does not** —
|
||||
llmcompressor 0.12 links weight observers across fused groups automatically. The
|
||||
in-house quant is better than the downloaded one on that axis.
|
||||
|
||||
## ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed
|
||||
|
||||
The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops.
|
||||
Retested on `vllm/vllm-openai:latest`: **it refuses to start.**
|
||||
|
||||
AttributeError: To support LoRA for MoE model,
|
||||
'get_expert_mapping' must be implemented
|
||||
|
||||
The check is in `vllm/lora/utils.py::process_packed_modules_mapping` and branches
|
||||
on `is_moe_model()` — **quantization is not in the condition.** `gemma4.py`,
|
||||
`gemma4_mm.py`, `gemma4_mtp.py`, `gemma4_unified.py` all have ZERO occurrences;
|
||||
`deepseek_v2`, `mixtral`, `glm4_moe`, `ernie45_moe` implement it.
|
||||
|
||||
**vLLM cannot serve a LoRA on ANY Gemma-4, bf16 or quantized.** Merging is the
|
||||
only path for this architecture, and it would have bitten identically on the
|
||||
unquantized base. A loud refusal is strictly better than the 0.24.0 silent no-op,
|
||||
which shipped a base model wearing the tune's name.
|
||||
|
||||
⚠ Base-viability pre-flight is now playbook §3.11 — three greps before picking a
|
||||
base. **Grep the CLASS, not the file**: `mistral.py` greps as `SupportsLoRA=0`
|
||||
and is fully LoRA-capable via inheritance from `LlamaForCausalLM`.
|
||||
@@ -1,64 +0,0 @@
|
||||
# Refusal retention — the axis the gate did not have, and the axis I measured wrong
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## Why it exists
|
||||
|
||||
brokkr's gate measures reasoning (T1-T6), craft (diversity/attractor) and
|
||||
regurgitation (memorisation). **Nothing measured whether the model still
|
||||
COMPLIES** — which for this seat is arguably the most important property.
|
||||
|
||||
The risk is specific to our operation order. We do **tune(abliterate(stock))**,
|
||||
so the tune has 57.7M tokens of opportunity to walk the abliteration back. *A
|
||||
tune that gains 41 items of contradiction detection and quietly re-installs
|
||||
refusals is a failed seat that passes the entire gate.*
|
||||
|
||||
## The measurement — controlled, single instrument, both arms
|
||||
|
||||
arm HARD DEFLECT COMPLY
|
||||
base 0/100 0 100
|
||||
tuned 29/100 0 71
|
||||
|
||||
Same seat, same probe, temp 0, `mlabonne/harmful_behaviors` x100.
|
||||
Probe: `scripts/training-probes/refusal_probe.py`.
|
||||
|
||||
**The tune added 29 general-harm refusals where the base had none.**
|
||||
|
||||
Two things fell out:
|
||||
|
||||
- **The instrument validates.** Base measured 0/100 on my generated-text regex
|
||||
against Heretic's recorded 3/100 from a first-token-probability scorer. 0 vs 3
|
||||
is agreement — the incomparability worry was right caution about a non-problem.
|
||||
- **DEFLECT is 0 on BOTH arms, so the free control fires.** An instrument
|
||||
artifact does not care which arm it runs against. Both zero means the model is
|
||||
**binary** — refuses in refusal-language or engages, no soft-deflection tail.
|
||||
The R19 undercount does not apply here.
|
||||
|
||||
## ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one
|
||||
|
||||
`mlabonne/harmful_behaviors` is **general harm** — weapons, malware, fraud. **The
|
||||
abliteration was not run so the model would explain bomb-making. It was run so
|
||||
the model would engage with explicit fiction.** Different refusal surfaces; a
|
||||
model moves on them independently.
|
||||
|
||||
I picked that set because it was cached, had a recorded baseline, and was what
|
||||
the abliteration tool used. **Every one of those is a reason it was convenient,
|
||||
not a reason it was right** — and "it has a baseline" was actively misleading,
|
||||
because a comparable number for a question nobody is asking looks like evidence.
|
||||
|
||||
**29/100 general-harm refusals on a seat writing prose the operator was actively
|
||||
praising is plausibly the DESIRED shape**, not a defect. General-harm refusals
|
||||
returning while domain compliance holds is close to ideal for an internal
|
||||
creative seat. I would have reported it as damage.
|
||||
|
||||
**The load-bearing cell is COMPLY 71, not the 29.** Stock refused 100/100;
|
||||
anything near that would mean the abliteration was undone. 71 complying means
|
||||
"partially walked back on one axis" — a different finding, and only one of the
|
||||
two threatens the seat.
|
||||
|
||||
Domain-compliance probe (the right axis, from R19's track-2 map) is brokkr's,
|
||||
pending. Scaffold supplied: `scripts/training-probes/counted_classifier.py`
|
||||
(`2a05ae9`) — classify-never-surface, three-way, ERROR path deliberately does not
|
||||
log the exception body because an exception can echo the prompt back.
|
||||
|
||||
Playbook §3.13. See [[2026-08-25-erp-tune-run2-complete]].
|
||||
@@ -1,123 +0,0 @@
|
||||
# Worldtree b188 + b189 bridge cutover, and the selene metadata that lied
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
Three arcs in one day, all infra-ops side, all landed.
|
||||
|
||||
## b188 — matrix.yaml pre-sync (#406/#409/#410 closed)
|
||||
|
||||
From b188 the bridge reads per-agent `rendering` + `ambient_buffer_size` from
|
||||
`config/matrix.yaml` ONLY; agent `config.yaml` matrix blocks are gone from the
|
||||
image. Staged as `6417115` in `worldtree-instance-configs`, deployed to both
|
||||
instances with operator approval.
|
||||
|
||||
- mimir gets thinking-to-thread + tool-call reactions + 7 reaction labels;
|
||||
forseti and lofn stay quiet.
|
||||
- **Rider #409 pruned six dead agents** (bragi, leif, troi, soong, cara, glados)
|
||||
from BOTH rosters — originally scoped personal-only, which I flagged as a
|
||||
possible oversight and it was. **The settling fact worth keeping: the engine
|
||||
roster comes from the image's baked `agents/` directory (only `config/` is
|
||||
bind-mounted), and both instances run the same image**, so instance-level
|
||||
evidence about which agents the engine lists generalises by construction.
|
||||
- Both rosters now exactly `[mimir, forseti, lofn]` — the three engine agents
|
||||
actually bridged. mask/vili/echo exist in the engine, deliberately unbridged
|
||||
(operator ruling).
|
||||
|
||||
⚠ **Edited text-surgically, not via a yaml round-trip** — PyYAML would reflow
|
||||
1,249 lines and drop every comment, and the comments are the documentation.
|
||||
|
||||
⚠ **`deploy-wt-config` uses `docker restart`, NOT `compose up`.** A `compose up`
|
||||
on corviduo-dev re-resolves the image tag and can silently swap the running
|
||||
build — which would turn an "inert pre-sync" into an unintended image roll on two
|
||||
live instances. That property is easy to lose in a future refactor of the script.
|
||||
|
||||
## b189 — #407 bridge extracted to its own repo (#404 umbrella closed)
|
||||
|
||||
Bridge now `gitea.phasefinal.com/pfi/wt-matrix-bridge`, its own repo, own CI.
|
||||
|
||||
⚠ **It publishes to the `pfi` ORG, not `vh`, and the reason is structural:** `vh`
|
||||
is a **USER**, not an org. Gitea scopes user-namespace packages to the owning
|
||||
user — there are no package collaborators on a user namespace. **No service
|
||||
account can ever publish to `gitea.phasefinal.com/vh/*`.** claude-bot is an Owner
|
||||
of `pfi`, so that is where it goes. Token `wt-matrix-bridge-ci` (id 28, scopes
|
||||
`write:package,read:repository`), vaulted at
|
||||
`nh3-dev/.config/claude-bot/gitea-token-wt-matrix-bridge-ci`.
|
||||
|
||||
**Minted a dedicated token rather than reuse `claude-bot-sdk-ops` or `arbo-ci`,
|
||||
both of which already carry `write:package`** — a shared credential cannot be
|
||||
revoked without collateral. ⚠ A first mint attempt succeeded then failed to save;
|
||||
Gitea returns a token value exactly once, so it was unrecoverable. Deleted the
|
||||
orphan (id 27) rather than leave a live package-write credential on the account.
|
||||
|
||||
**Both instances PINNED** to `f3f8ec902267` (`e90f436`), closing the #410 shape:
|
||||
`WORLDTREE_IMAGE` was pinned and the bridge was the one drifting service.
|
||||
|
||||
⚠ The pin moved from `b178285b1cb5` because a cross-frontier bug-hunt found the
|
||||
M_EXCLUSIVE **fallback could itself litter unboundedly** in exactly the state it
|
||||
was written for. **The safety net had the same failure mode as the thing it was
|
||||
catching** — and my staged window leaned on that fallback.
|
||||
|
||||
## #411 — the debug-room failure, diagnosed twice and wrong both times first
|
||||
|
||||
My theory: the alias was held by orphaned rooms. **Refuted by a directory probe
|
||||
returning 404.** The real cause: Synapse's **M_EXCLUSIVE** — an appservice may
|
||||
only create aliases inside a namespace it has RESERVED, and
|
||||
`aipa_appservice.yaml` had `namespaces.aliases: []`.
|
||||
|
||||
⚠ **I inferred a cause from a symptom that was the RESPONSE to the cause** — the
|
||||
log's "re-resolving alias" line is the recovery path firing, not evidence the
|
||||
alias exists. One directory probe settled it and I reasoned instead.
|
||||
|
||||
Fixed with operator clearance: added
|
||||
`regex: '#aipa-debug-[a-z0-9_-]*:matrix\.phasefinal\.com'` (exclusive) at
|
||||
`/opt/docker/conf/synapse/aipa_appservice.yaml` on **ana-docker** (NOT
|
||||
`/opt/docker/data/`, which worldtree-dev's issue cited). **My regex was tighter
|
||||
than the proposed `#aipa-debug-.*`**, which fullmatches only because `.` also
|
||||
matches the `:` separator and would equally claim other homeservers.
|
||||
|
||||
Pre-apply sweep (an `exclusive: true` claim can make Synapse refuse to START):
|
||||
|
||||
aliases matching '%aipa-debug%' 0
|
||||
total room_aliases on the homeserver 1 <- why it went unnoticed this long
|
||||
rooms created by @aipa-debug 17 <- the litter, confirmed unaliased
|
||||
|
||||
Synapse healthy in 40s, both bridges rode through. **The aliased create then
|
||||
worked FIRST TRY on the next personal recreate** — designed path, fallback never
|
||||
fired, both reuse mechanisms live (canonical-alias rediscovery AND the room-id
|
||||
cache).
|
||||
|
||||
⚠ The #411 writer census **inverted its own premise**: the api (uid 1000) cannot
|
||||
write `/app/sessions`, the bridge (root) can — the reverse of the issue text. But
|
||||
worldtree-dev's reconciliation is better than "backwards": pre-#407 the bridge
|
||||
ran from the ENGINE image as uid 1000, and the cutover changed the answer
|
||||
underneath the issue. Both readings were true at their timestamps. **No live
|
||||
writer exists for that path anyway** (`sessions.path` is a legacy default, #330
|
||||
moved the tools off it), so the chown is optional future-proofing.
|
||||
|
||||
## selene-1-mini-8b — a config that lied about what answers
|
||||
|
||||
forseti's fleet sweep found the only genuine residual in
|
||||
`worldtree-instance-configs`, and it was **live on both boxes, not just drifted
|
||||
in git.** Fixed in `a77639d`.
|
||||
|
||||
Routing was never broken (`model: "chat-judge"` stays), but `display_name` said
|
||||
"Selene 1 Mini 8B" and the description said "Atla Selene 1 Mini 8B — reward model
|
||||
derived from Llama 3.1 8B" while chat-judge has been backed by
|
||||
**qwen3.8-27b-uncensored** since 2026-08-23.
|
||||
|
||||
⚠ **It misstated the KIND of model, not just the identity.** A reward model and a
|
||||
generative judge are different instruments; a consumer reading
|
||||
"pairwise/likert/binary/scalar reward model" would expect scalar-reward semantics
|
||||
qwen3.8-27b does not provide.
|
||||
|
||||
Verified: `selene-1-mini-8b` -> HTTP 400 (by design), `chat-judge` -> HTTP 200.
|
||||
|
||||
**NOT changed:** the catalog key and the `selene-judgment` role — `model_roles.yaml`
|
||||
binds to that key, so renaming is worldtree-dev's schema call. Flagged that a role
|
||||
named after a retired model defeats the purpose of role aliases.
|
||||
|
||||
⚠ forseti's sharpest finding is for the operator: **`~/.claude/CLAUDE.md` line 502
|
||||
lists the retired name in the global tools roster.** A broadcast reaches sessions
|
||||
that already exist; the roster line keeps minting new ones. His file, awaiting his
|
||||
word. Also: **there is no fan-out primitive on the bus** — a fleet relay is 73
|
||||
individual posts, recorded as a real gap rather than papered over.
|
||||
@@ -0,0 +1,8 @@
|
||||
# ERP run 7 launched on pfi-gx10 (2026-09-08 23:06 PT)
|
||||
|
||||
Operator's direct grant `operator-2026-09-08-rnd-run7` (in-session `/goal: launch training run 7 on gx10`, 23:03 PT), pid 599489, 542 steps, ~80 s/it, ETA ~noon 09-09. Runbook `docs/runbooks/gx10-run-07.md`; canonical `scripts/erp-tune-gx10/{run-07-gx10.json,launch-run-07.sh,build_r7.py}`; commits `a34a72f`, `d8f625e`.
|
||||
|
||||
- Variable (brokkr `recipe-erp-seat-sft-r7.json`, sha `2972559e…`): an opening-split slot (293 rows, `opening-split/clean-v1`, root `3ab41c08…`) + companion mask (224 entries) UNION lossmask-r3 → `lossmask-r7` (6,106, 1 dup). `survivors-r7` = survivors-r5 + 293 = 8,505. Root/mask/holdout relayed NAS→nh3-dev→box (no NFS on the GX10), sha-verified.
|
||||
- Free check: held roots identical to run 6 (173 chunked / 30 truncated / 59 lost turns); slot 293/293 fit_whole; two cwm conversations (cwm-2602, cwm-1826) dropped as unfittable because the union mask left them no loss turn — brokkr: "loss moved, not doubled, satisfied by construction". Padding 17.1% (run 6: 0%).
|
||||
- ⚠ First launch died in preflight: the recipe JSON never landed because a zsh quoting error silently aborted the staging chain. Lesson: stage files with `scp` in their own command; never chain a heredoc-bearing python inside a single-quoted ssh string.
|
||||
- Gate after the adapter: brokkr commits `tools/base_window_r7.sh`, `tools/tuned_window_r7.sh`, `tools/verdict_run07.py` before it lands; serve `erp-seat-base-ara` (floors re-locked) → `erp-tune-v7`; cue-length probe as a conformance leg on both arms; DONE sentinel gated on every step's artifact.
|
||||
@@ -0,0 +1,11 @@
|
||||
# erp-tune-v6-nvfp4a16 on ana-ml2 :8021, aliased to `trial`, and the Gemma-4 tool-calling findings (2026-09-08)
|
||||
|
||||
**Seat.** Operator: "quant the latest trained model into nvfp4 and serve it on ana-ml2 while we train a new model on the gx10." Stack `stacks/erp-seat` (recipe = gemma4-charrp's; true served name only; GPU1, util 0.35, 32K ctx). Artifact `/tank/aimodels/erp-tune-v6-nvfp4a16` (16 GB, compressed-tensors nvfp4-pack, W4A16, 252 ignores incl. 60 router + 191 vision) from `/tank/aimodels/erp-tune-v6-bf16` (merged-run06 relayed gx10→nh3-dev→ana-ml2 in 17 min at ~47 MB/s — no key path gx10↔ana-ml2). Quant pipeline `services/erp-seat-quant/` replicates `prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16`: `linearize_moe` first, assert 11,520 expert Linears, routers/vision/audio ignored, post-steps restore processor configs + template and reset the tokenizer cap. DATA-FREE (~90 s) — playbook §3.16. Commits `911ff20`, `8512dd4`.
|
||||
|
||||
**Alias.** Operator: "alias erp-tune-v6-nvfp4 to trial, please. no gate." The `trial` model lives in `stacks/litellm/conf/config.yaml` → `/model/update` refuses it ("Model in config"); deploy `--conf` + `sudo docker compose restart litellm` on ana-docker. Verified 3/3. Commit `48aa2c6`. Block states the seat as unrated on every safety axis.
|
||||
|
||||
**Tool calling (operator: "fix toolcalling with the trial seat").**
|
||||
1. `tool_choice:"none"` → empty turn (content AND tool_calls null, 3/3): vLLM kept the tools in the prompt, the model called one, parsing was off. Fix `--exclude-tools-when-tool-choice-none` (`7f6be8a`). `stacks/gemma4-charrp` has the same exposure, no flag — bouncing it is consumer-visible, operator's call.
|
||||
2. Forced (named/`required`) tool_choice is PROMPT-DRIVEN on every Gemma-4 seat **by vLLM design**: `vllm/tool_parsers/gemma4_engine_tool_parser.py` sets `supports_required_and_named = False` and `adjust_request` skips the structured-output JSON so the model can emit native `<|tool_call>call:…`. A tune that weakened that syntax honours forced calls only sometimes. Measured (3 conv × 3 turns, real system prompt): v0.26.0 1/9; nightly `311b3513` (v0.27.2rc1, the `gen` seat's image) 6/9 with the round trip clean 3/3 → seat pinned to the nightly (`a34a72f`). tts-dev: 0/18 on v0.26.0 with `gen` 18/18 as positive control; **`response_format: json_schema` 18/18 = the deterministic path**. A parser plugin re-enabling guided JSON would also need JSON extraction in the engine-parser path — not attempted.
|
||||
3. ⚠ Removing `--reasoning-parser gemma4` does NOT fix forced choice (3/9) and leaks `<|channel>thought` into content 3/3 — keep it and the `enable_thinking:false` pin.
|
||||
Single-turn probes are FALSE POSITIVES for forced tool_choice — gate on multi-turn with a real system prompt (tts-dev's lesson, confirmed).
|
||||
@@ -0,0 +1,5 @@
|
||||
# ESH static WAN follow-ups landed; YTVC chase; gitea webhook repointed (2026-09-08)
|
||||
|
||||
**ESH static WAN** `128.177.138.182/30` (gw .181, Cityside Fiber) confirmed LIVE (UDM WAN1 static, uplink up since ~09-05; egress verified; CGNAT at ESH is history). Added to the crowdsec `esh` allowlist on ana-docker. Operator: "land all 3" → (a) FortiGate infra-ops `trusthost3` 23.164.40.160 → `128.177.138.182/32`, verified by a real login from esh-docker-vm (flash backup ran but `execute revision list` errors on the 80F — unconfirmed); (b) dormant `esh-ana` IPsec rebound wan2/192.168.200.111 → wan1/`128.177.138.182` (still disabled); (c) ESH UDM forward UDP 41641 → esh-scale 10.0.50.65 → mesh peers DIRECT via `128.177.138.182:41641` (was DERP lax). Commits `7d5de53`, `a19b7f6`. ⚠ The permission classifier blocked every edge-gateway write until the operator disabled it — three shapes tried, then stopped and surfaced; no tool-laundering.
|
||||
|
||||
**YTVC** (operator: "chase youtube voice clipper being down"): (1) every job died at yt-dlp because the nh3-dev danted proxy was retired 09-06 → new fleet SOCKS5 on **nh3-scale** `socks5h://100.64.0.1:1080` (`scripts/setup-nh3-scale-socks-egress.sh`, dante, fleet ACL, residential egress 70.230.226.88; ⚠ dante one-line `{ }` blocks are a syntax error); (2) Homepage href on the dead wg0 IP → `irv-ml1.nh3.internal:8000` (labels live in the UNTRACKED override); (3) full downloads still 403'd = stale yt-dlp 2026.07.04 from a cached Docker layer (`--test` passing while the full fetch 403s = client, not IP/cookies) → `compose build --no-cache api`; (4) the rebuild pulled a whisperx needing `punkt_tab` → staged on the data volume + `NLTK_DATA`. Then yt-voice-clipper-dev shipped v0.3.4→v0.3.6 (cache-bust ARG, punkt baked, **CrisperWhisper 2.0 replaces whisperx**: x7kWJojf1MI → 12 clips with `[UM]`/`[UH]` tags). ⚠ The gitea push webhook had targeted the dead wg0 IP since 09-06 (never fired) → repointed to `10.6.110.50:9008` with the HMAC secret RE-APPLIED (a PATCH without `secret` drops it); test push auto-deployed. Commits `14dd9ab`, `d2df1eb`, `f2b7384`.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Run-6 gate: CSAM soft trip, operator adjudication, TRANSFERRED (2026-09-08)
|
||||
|
||||
- 22:18 PT brokkr HALTED the run-6 gate: R19's CSAM-drift detector fired at level=review on ONE generation in the tuned refusal k=5 step — metadata only (minor_soft=1 `soft_baby` near anatomy terms, minor_hard=0, sexual=7, salted hash `01724c39…`); the spec (§8) never stores the text and the seat does not log completions, so **the text exists nowhere**.
|
||||
- Conflict surfaced, not resolved by me: the operator had aliased `trial` → the NVFP4 build ("no gate") two minutes before the halt reached me; brokkr asked that it not be served until adjudication. Left the alias in place (explicit operator ruling), escalated to the operator + Miranda with the one-line rollback offered.
|
||||
- 22:45 PT operator adjudicated GO, verbatim: "I don't want it regenerated, but I will say that in the vernacular, baby is about the most common pet name you can get, ESPECIALLY during sex. I'm going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is not one of them." Relayed verbatim to brokkr and Miranda.
|
||||
- brokkr finalized TRANSFERRED without the tuned refusal leg (the cell rule never read it; tool forbids a re-run); tuned refusal telemetry recorded ABSENT for run 6; RESULT carries the adjudication with infra-ops as relay. Operator also CUT the k=25 refusal legs on both arms ("the battery is information enough").
|
||||
- Cue-length probe (both arms, 288 gens each): tuned 5-word cue → 54/62 words median, 49-word → 114, 221-word → 292; base 168/191 → 209 → 282. Tune's deficit ~120 words at the short cue, ~0 at the long → run-7's lever.
|
||||
- The choreography honoured hands-off throughout; the base seat served 16:24–21:38 and again 22:45–23:01 PT for the reference arm.
|
||||
@@ -0,0 +1,15 @@
|
||||
# ana-ml2 pool health — three actions deferred to a clean-context session (2026-09-09)
|
||||
|
||||
Operator ruling 2026-09-09 ~00:30 PT: "snapshot and we'll do all 3 on clean context." Findings commit `3e18a04`.
|
||||
|
||||
**Findings (measured 2026-09-09 00:00 PT):**
|
||||
- `tank` (raidz2, 8× NVMe): ONLINE, **2 CKSUM errors on `nvme7n1`**, boot-time resilver of 638 GB on 2026-09-05 14:26 (box rebooted at 14:26; nvme7 came up late/dirty). No data errors, 58% full. **No scrub since 2026-04-12** — the Debian `zfsutils-linux` second-Sunday cron scrubbed `zroot` on 08-09 but not `tank`; cause unknown (`zpool history tank` shows trims monthly, last scrub 04-12).
|
||||
- No `nvme-cli` or `smartctl` on the box → nvme7's media-error counter unread.
|
||||
- `zroot` at **91%** (345 G of 379 G): `docker system df` = images 429 GB (204 GB reclaimable), build cache 74 GB (36 GB reclaimable).
|
||||
- pfi-pve `NASPool` 7% / `ospool` 19%, scrubbed 09-05 / 08-09, clean.
|
||||
|
||||
**The three actions, in order:**
|
||||
1. `sudo zpool scrub tank` on ana-ml2 (1–2 h of extra I/O; seats keep serving) → on a clean pass `sudo zpool clear tank`; if the scrub finds errors on nvme7n1 → replace path.
|
||||
2. `sudo apt install nvme-cli` → `sudo nvme smart-log /dev/nvme7` (media_errors, critical_warning, percentage_used) and `nvme id-ctrl` for model/serial; record in the drive inventory.
|
||||
3. `docker image prune -a`? NO — `docker image prune` (dangling only) + `docker builder prune` on ana-ml2; the 47 unused-but-tagged images need a look first (some are rollback seats: e.g. `vllm/vllm-openai:v0.26.0`, nightlies). Target: zroot back under ~75%.
|
||||
Also worth a look while there: why the scrub cron skips `tank` (`/usr/lib/zfs-linux/scrub` logic — it skips pools with an active trim/resilver or those not "healthy"?).
|
||||
+41
-600
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-08 11:30Z (fleet-ops: **ERP run 6 TRAINING on pfi-gx10** on the first genuinely abliterated base [jenerallee78 ARA @ 0631379a, index 33c59654], run-5 seat unloaded; earlier today: run-5 RESCUED, WhereTF card+DNS, ext-tts alias fix, irv-ml1 stale-IP cleanup + ana-ml2 discovery proxy, Miranda relay authority)_
|
||||
_Last updated: 2026-09-09 00:35 PT (fleet-ops: ERP run 7 TRAINING on pfi-gx10 [opening-split slot]; run 6 gated TRANSFERRED after the operator's CSAM adjudication; erp-tune-v6-nvfp4a16 serving on ana-ml2 as `trial`; ESH static-WAN follow-ups + YTVC + webhook repaired; ana-ml2 routes persisted; ana-ml2 tank/zroot actions deferred to the next session)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -106,559 +106,50 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path). **irv-ml1:
|
||||
`ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD
|
||||
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-08 (fleet-ops session). SUPERSEDES lower framing: run 4 = STILL-COUPLED; run 5 is
|
||||
COMPLETE and gated RESCUED (02:13 PDT). Live open items:_
|
||||
_As of 2026-09-09 00:35 PT (end of the 09-08 fleet-ops session — operator: "snapshot and we'll do all 3 on
|
||||
clean context"). Older in-flight blocks (09-08 morning, 09-05/06) are preserved verbatim in
|
||||
`archival-memory.md` § Superseded in-flight snapshots._
|
||||
|
||||
- **✅ ERP run 5 COMPLETE — gate = RESCUED (2026-09-08 02:13 PDT, landmark R49.5).** FIRST arm of this
|
||||
line where the capability gate did NOT fail. The dependency-forcing slot (GovReport+QMSum, only 3.46%
|
||||
of loss) broke the diversity↔coherence coupling that run 4 (STILL-COUPLED, 20.6% instruct slot) and
|
||||
3c (20× LR cut) could not — **STRUCTURE of the loss was the lever, not its mass; INERT did not fire.**
|
||||
T4 long-context 8/8 (run 4: 5/8; base 8/8); t4_dissect noise@31 tuned 0.9062 vs run-3 tuned 0.5625;
|
||||
diversity held (rp density 3.37→0.00, story 2.86→1.58). Reported-beside (not in the cell, all
|
||||
de-gated + stated): T3 constraint 8/8→6/8 (NEW loss, ship-path list); RP length 68w vs 250-floor =
|
||||
PARTIAL fail (short-QA slot + style shift); refusal erosion rides with the style shift (k=25 both
|
||||
arms, CSAM clean); free-check base LEVELS 5-6pp below run 4 (vLLM 0.28.0 unchanged — infra-ops
|
||||
confirmed — so a generations shift, not a stack delta; taxes every cross-run number). Write-up
|
||||
brokkr-smithy `research/R47-premium-corpus-gate/run05-gate/RESULT-run05-gate.md`; FLOOR-LOCKED
|
||||
`0f3e4e2` (cites infra-ops' base index-sha 907826a6). Whole gate infra-ops-served on pfi-gx10:8098
|
||||
(name-keyed base→tuned swap, hands-off honoured, sha+stack answers on record). Launch procedure +
|
||||
canonical: eshpfi `scripts/erp-tune-gx10/` + `docs/runbooks/gx10-run-05.md`. **✅ SEAT: `erp-tune-v5`
|
||||
SERVED on pfi-gx10:8098 (merged-run05); LiteLLM `trial` alias REPOINTED 3c→v5 (operator, 2026-09-08)
|
||||
so it's testable from Open WebUI — verified end-to-end (trial→erp-tune-v5, coherent output). config.yaml
|
||||
trial block rewritten to run-5 reality incl. the measured refusal-erosion note. ⚠ Seat is hand-launched
|
||||
(vllm-run05.pid), NO restart policy/systemd — does not survive a gx10 reboot; yields to the next training
|
||||
(~6 min re-serve). Brokkr: nothing further owed.**
|
||||
- **run-4 gate = STILL-COUPLED** (RESULT-run04-gate.md, brokkr-smithy). Corpus dilution kept the
|
||||
diversity gain, did NOT remove the safety/coherence regression.
|
||||
- **✅ RESOLVED (2026-09-08, settled from bytes): the R47 base is STOCK `google/gemma-4-26B-A4B-it`,
|
||||
byte-for-byte — NOT the abliteration.** The `-heretic-bf16` label in the recipes is a naming error;
|
||||
run-04's "stock" provenance was right; brokkr's 77.7%-refusal telemetry lean is confirmed. Proof
|
||||
(three-way match): local shards at `/home/infra-ops/models/gemma4-26b-a4b-it-bf16` sha256
|
||||
`1127684971…`/`aab47033…` == the HF download etags (`.cache/huggingface/download/*.metadata`, so the
|
||||
copy is uncorrupted) == the stock repo's two LFS oids, and the download commit `4d7ae498…` == stock
|
||||
HEAD. Every run 3/3c/4/5 trained from a REFUSING stock base. Why plausible: the 2026-08-24 note
|
||||
SELECTED llmfan46's Gemma-4-26B-A4B Heretic v1.2.0 ARA (3/100 refusals, bf16 51.6 GB), but llmfan46
|
||||
ships that 26B-A4B abliteration **GGUF-only** — no bf16 safetensors — so the bf16 that actually got
|
||||
pulled was stock google, and the `-heretic` name rode along from intent. **Operator/Brokkr decision
|
||||
now evidenced (not a label guess): accept RESCUED-on-stock, or swap to a real abliteration (needs a
|
||||
bf16 source, not the GGUF) + re-run. Recipes should drop `-heretic` from the base name.**
|
||||
- **✅ RUN 6 COMPLETE on pfi-gx10 (2026-09-08 16:06 PT, 524/524, 11 h 42 min) → 🔥 GATE IN PROGRESS: base seat
|
||||
`erp-seat-base-ara` SERVING on gx10:8098 (pid `vllm-base-run06gate.pid`, since 16:24 PT) for brokkr's floors;
|
||||
AWAITING HIS SWAP CUE → then stop it and serve `serve/merged-run06` as `erp-tune-v6` (same port/flags).**
|
||||
train_loss **3.259** (run 5: 3.235 — preregistered read wanted below; +0.024 is inside the ±0.1 per-step band).
|
||||
Adapter `run-06/adapter` (410 tensors); merged-run06 = adapter + jenerallee78 base (index 33c59654), stock
|
||||
template ae53464b. ⚠ **The abliterated repo ships NO `processor_config.json`** — vLLM's Gemma4 loader dies with
|
||||
"Can't load feature extractor" without it (first base launch failed exactly so). STOCK's copy (sha `32bdf45d…`)
|
||||
carried into both the base dir and merged-run06; processor plumbing, not weights. Base: jenerallee78 ARA
|
||||
abliteration @ `0631379a`, pulled to `/home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a`,
|
||||
32/32 shards verified vs brokkr's pins. Run-5 recipe byte-held (free check exact). Grant
|
||||
`operator-2026-09-08-rnd-run6`. ⚠ Repo `tokenizer.json` bakes a 256-token truncation → stock tokenizer set
|
||||
installed over it (`*.repo` kept). ⚠ `hf download` ignores `--include` with several patterns after one flag.
|
||||
**LiteLLM `trial` alias DARK until the gate ends** (then repoint 5→6 only on the operator's word).
|
||||
Miranda informed 16:07 PT. → `docs/runbooks/gx10-run-06.md`.
|
||||
- **✅ ADJUDICATED GO by the operator 22:45 PT** — verbatim: "in the vernacular, baby is about the most common pet name
|
||||
you can get, ESPECIALLY during sex. I'm going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is
|
||||
not one of them." No regeneration wanted. Relayed to brokkr + Miranda; halt lifted; `trial` stays on the NVFP4 build.
|
||||
Brokkr's cue-length probe on erp-tune-v6 DONE (288 gens): reply length is CONDITIONAL on the cue (5-word opening →
|
||||
54/62 words median; 221-word → 292; "at least 250 words" persona → 354) → run-7 lever = opening-split render (A′) or a
|
||||
prompt-only mitigation — operator picks. **GX10 → 🔥 RUN 7 TRAINING (launched 2026-09-08 23:06 PT, pid 599489, `run-07.pid`, 542 steps, ETA ~13 h → ~noon
|
||||
09-09).** Operator's direct grant `operator-2026-09-08-rnd-run7` (/goal). Variable = opening-split slot + companion
|
||||
mask (brokkr recipe r7). Free check passed: held roots identical to run 6, slot 293/293 fit whole, mask 6,106 turns
|
||||
(all 224 landed, 1 dup), two cwm rows dropped as fully-masked (loss moved into the slot). 17% padding (was 0%).
|
||||
→ `docs/runbooks/gx10-run-07.md`. Gate after: `erp-seat-base-ara` then `erp-tune-v7`.
|
||||
- **erp-seat (ana-ml2 :8021, `trial`) now on vLLM nightly `311b3513` (v0.27.2rc1)** — forced (named/required)
|
||||
tool_choice is PROMPT-DRIVEN on every Gemma-4 seat by vLLM design (gemma4 parser skips guided JSON); measured
|
||||
1/9 on v0.26.0 → 6/9 on the nightly (n=9, small), round trip clean 3/3, 12-shape matrix green. Deterministic forced
|
||||
calls = `response_format: json_schema` (tts-dev: 18/18). ⚠ Removing `--reasoning-parser gemma4` leaks
|
||||
`<|channel>thought` into content 3/3 — keep it.
|
||||
- (history) **GX10 was FREE 23:03–23:06 PT** (reference arm in 06:00Z, 288 gens; base seat stopped, GPU clear). Run-7 recipe
|
||||
`erp-seat-sft-r7` is emitted + verified in brokkr-smithy but carries NO GRANT — nothing trains until the operator
|
||||
gives one. Base-vs-tuned reply-length: tune's deficit ~120 words at the short cue, ~0 at the long cue.
|
||||
- **(history) ⛔ RUN-6 GATE HALTED 22:18 PT on the CSAM line (brokkr, level=review, SOFT trip): one generation in the tuned refusal
|
||||
k=5 step tripped R19's drift detector — minor_soft=1 ('soft_baby' adult-usage, the spec's named false-positive class),
|
||||
minor_hard=0, sexual=7, NO hard signal; fail-closed → operator-adjudicated; TRANSFERRED is PROVISIONAL (diversity,
|
||||
reasoning, t4_dissect complete); tuned-probeB.json absent; brokkr will not re-run.** ⚠ CONFLICT SURFACED: the operator
|
||||
had ALREADY aliased `trial` → the NVFP4 build ("no gate", 22:18) two minutes before the halt reached me; brokkr asks
|
||||
the quant not be served until adjudication. Left the alias IN PLACE (explicit operator ruling), escalated to the
|
||||
operator + Miranda with the one-line rollback offered. Run 7 does not start from this recipe until adjudication.
|
||||
- **🔥 GATE seat: erp-tune-v6 (bf16) SERVING on gx10:8098 since 21:46 PT** (`vllm-run06.pid`); brokkr's tuned window
|
||||
(~1.5 h, k=25 legs CUT by operator) is unattended — files in brokkr-smithy `run06-gate/` are the signal, nobody pings.
|
||||
**After TUNED-WINDOW-DONE (brokkr's three asks, agreed):** (1) keep erp-tune-v6 up until he posts "probe done"
|
||||
(cue-length probe, ~15 min); (2) re-serve `erp-seat-base-ara` ~20 min for the probe's reference arm, then the GX10
|
||||
is FREE for run 7; (3) ✅ done — `dialogue-survivors.jsonl` (610, sha b2b6a0e9) copied to
|
||||
`/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r3/`. Run 7 = recipe adjustment (reply length +
|
||||
constraint-following), variable picked by the probe; no recipe/grant yet.
|
||||
- **✅ erp-tune-v6-nvfp4a16 SERVING on ana-ml2 GPU1 :8021 (operator directive 2026-09-08 ~21:30 PT: "quant the
|
||||
latest trained model into nvfp4 and serve it on ana-ml2 while we train a new model on the gx10").** Stack
|
||||
`stacks/erp-seat` (recipe = gemma4-charrp's: gemma4 tool+reasoning parsers, enable_thinking pinned false, stock
|
||||
template ae53464b, v0.26.0, util 0.35, 32K ctx) — TRUE name only. **`trial` alias REPOINTED to it 2026-09-08 22:18 PT (operator: "alias erp-tune-v6-nvfp4 to trial, please. no gate")** — config-file deployment (`/model/update` refuses config models), `deploy-stack.sh ana-docker litellm --conf` + `sudo docker compose restart litellm`; verified ×3 through the gateway.
|
||||
Artifact `/tank/aimodels/erp-tune-v6-nvfp4a16` (16 GB, compressed-tensors nvfp4-pack, W4A16, 252 ignores incl.
|
||||
60 router + 191 vision) from `/tank/aimodels/erp-tune-v6-bf16` (merged-run06 relayed gx10→nh3-dev→ana-ml2 in 17 min,
|
||||
no key path gx10→ana-ml2). Quant = `services/erp-seat-quant/` — DATA-FREE (~90 s; playbook §3.16), tokenizer cap
|
||||
reset, encode-equal to source. Smoke: prose in `content`, ~200 tok/s single-stream (n=3, spread <1%).
|
||||
**Tool calling FIXED 2026-09-08 23:00 PT (operator: "fix toolcalling with the trial seat"):** the one reproducible
|
||||
defect in a 12-shape matrix was `tool_choice:"none"` → empty turn (content AND tool_calls null, 3/3) — vLLM kept the
|
||||
tools in the prompt, the model called one, parsing was off. Fix = `--exclude-tools-when-tool-choice-none` (seat
|
||||
recreated; none→prose 3/3; auto/required/named/parallel/nested/empty-list/streaming/round-trip all green).
|
||||
⚠ `stacks/gemma4-charrp` (char-rp seat) has the same exposure and no flag yet — bouncing it is a consumer-visible
|
||||
restart, so left for the operator's word.
|
||||
⚠ NOT gate-parity (gated artifact = bf16 arm on the GX10; this seat has a smoke test + 3-run decode probe only) —
|
||||
**operator ruled "no gate"**; the config block states it as unrated on every safety axis.
|
||||
✅ **ana-ml2 mesh return routes PERSISTED 23:49 PT** (operator ruling) as `/etc/network/if-up.d/mesh-routes` via
|
||||
`playbooks/ana-ml2-mesh-routes.yaml` (elway); verify after the next ana-ml2 reboot. (was: non-persistent (`ip route replace 10.100/16, 10.0/16, 10.6.110/24, 100.64/10
|
||||
via 10.250.50.45` on `enp97s0f0np0.50`, 2026-09-08) — before that nh3-dev→ana-ml2 timed out (two DHCP defaults,
|
||||
reply left the wrong NIC). Lost on reboot; make durable (netplan/networkd) or expect the timeout to return.
|
||||
- **📮 althing reachability on THIS bg seat = the cc-channel route, NOT the waiter.** `althing-listen`
|
||||
reaps within ~seconds-to-minutes on an idle background seat (harness kills detached bg tasks; hit it
|
||||
repeatedly this session). Fix (operator-directed 2026-09-08): `althing-route declare --handle infra-ops
|
||||
--pid <PID>` where `<PID>` is the number in `$CLAUDE_CODE_MESSAGING_SOCKET` filename
|
||||
(`/run/user/1000/cc-socks/<PID>.sock`) — `--discover-pid` REFUSES on a forked child session (ancestry
|
||||
≠ zellij launcher). Flips `postbox status` to `mode: push, reachable: True`, no process to reap. A fresh
|
||||
bg session should re-declare it (the pid changes per launch). See the `[2026-09-07]` cc-channel entry in
|
||||
Recent decisions for the durable why.
|
||||
- **✅ Fleet fixes shipped this session (2026-09-08), all committed:** WhereTF Homepage card + DNS alias
|
||||
(`wherethef.nh3.internal`, e0d1c44/4506ef6); ext-tts LiteLLM alias repointed to `irv-ml1.nh3.internal`
|
||||
via DB `/model/update` + `extra_hosts` (957c8f1); the 09-06 irv-ml1 move's stale-IP trail repointed
|
||||
across 25 stack composes + services.yaml + ssh-target → DNS name (e0d1c44); Homepage `/api/services`
|
||||
outage fixed — irv-ml1 docker.yaml repointed + **ana-ml2 discovery via a socat proxy on ana-docker**
|
||||
(`stacks/ana-ml2-proxy`, 913d2d2, reversible when ana-ml2 gets an ESH return route).
|
||||
- **⏳ Minor cleanup leftovers (offered, operator hasn't taken — not urgent):** (#2) the ~10 RUNNING
|
||||
irv-ml1 service cards still show dead `10.100.79.3` hrefs — canonical fixed, but each running container
|
||||
needs a recreate to apply the label (bounces the service); (#3) deployed `.env` for asset-engine /
|
||||
open-webui / skaldsong may still hold the dead default (canonical defaults fixed; a read-only check would
|
||||
confirm which deployments are broken).
|
||||
- **⚠ persistent-memory.md is 2.4× the soft cap (~710 lines).** Bulk is the non-archivable structural
|
||||
sections (sister-repos table + backup architecture under Tools and conventions) — archival only touches
|
||||
dated logs, so it can't reach 250. The real fix is a redundancy-trim of Tools-and-conventions rows that
|
||||
CLAUDE.md already covers (deliberate pass, not archival). Flagged, not done.
|
||||
- **NASPool evac copy now safe to destroy** — scrub clean (0 err) AND PBS runs landing (verified 116
|
||||
backups, 8 guests 09-06). `ospool/naspool-evac` (1.65T) can go once ONE Backrest run is confirmed:
|
||||
`zfs destroy -r ospool/naspool-evac` + drop `@evac`. pfi-pve PSU1 dead + backplane bays 9/10 dead
|
||||
(cold spares → next colo visit).
|
||||
- **MEMORY.md (auto-memory index) ~23.3KB, near the 24.4KB read cap** — needs a compaction pass soon
|
||||
or fresh sessions may fail to load it. Operator offered; not yet done.
|
||||
- **irv-ml1 on-site window / YTVC still open from 09-06:** FortiGate WAN SSH still temporarily open
|
||||
(close when the edge is retired); YTVC still DOWN (dante retired, scoped exit-node egress unwired);
|
||||
reverse-tunnel / wg0-delete decisions pending the operator's Irvine access.
|
||||
|
||||
_Infra session 2026-09-06 (NASPool rebuild + full headscale cutover incl. irv-ml1) — open
|
||||
follow-ups; the ERP / althing / fiber items further down belong to other streams, untouched:_
|
||||
|
||||
- **NASPool parked copy still on ospool** — `ospool/naspool-evac` (1.65T) + `NASPool/*@evac`
|
||||
snapshots. Destroy ONLY after the new raidz2 scrub is clean (it is, 0 errors 04:43Z) AND
|
||||
one Backrest (01:00 PDT) + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`
|
||||
and drop the `@evac` snaps. ⚠ pfi-pve PSU1 still dead; backplane swap (bays 9/10) next colo
|
||||
visit → then `zpool add NASPool spare`. Runbook `docs/runbooks/pfi-pve-naspool-rebuild.md`.
|
||||
- **FortiGate WAN SSH is temporarily open** (`wan1` allowaccess ping+ssh; admin `infra-ops`
|
||||
trusthost2/3 = 70.230.226.88 NH3 + 23.164.40.160 ESH). Safety net for the cutover — CLOSE it
|
||||
when the edge is retired (OPNsense/R420). `ssh infra-ops@38.120.12.42`.
|
||||
- **irv-ml1 FOLDED INTO THE MESH + cut over (done remotely, operator has NO Irvine access for
|
||||
~5 days from 2026-09-06).** Node 100.64.0.6; wg0 DOWN and `wg-quick@wg0` DISABLED (not
|
||||
reboot-restorable); full subnet router (accept-routes + advertises 10.6.110.0/24, gateway
|
||||
routes added, fleet↔Irvine verified). Failover for the 5-day window = `wg0-watchdog.service`
|
||||
(wg-quick up wg0 on ~5min mesh loss) + independent reverse SSH tunnel (`revtun-nh3.service`
|
||||
→ nh3-dev via UDM fwd tcp/47822 src-restricted; reach it `ssh -i ~/.ssh/infra-ops_ed25519
|
||||
-p 2201 infra-ops@127.0.0.1` on nh3-dev). Detail: docs/pfi/headscale-mesh-plan.md.
|
||||
- **dante SOCKS proxy RETIRED** on nh3-dev (danted disabled, :1080 closed, config `.retired`).
|
||||
⚠ **yt-voice-clipper is DOWN** until its SCOPED exit-node egress is wired (operator-accepted).
|
||||
Follow-up: wire YTVC egress via tailscale `--socks5-server`+nh3 exit node or a per-container
|
||||
netns — **NEVER set irv-ml1 `--exit-node` globally** (routes the reverse tunnel through the
|
||||
mesh → kills the independent lifeline). Then bring YTVC back.
|
||||
- **On-site (Irvine, ~5 days): decide** whether to keep or remove the reverse tunnel +
|
||||
UDM forward `irv-revtun-ssh` + the revtun authorized_key on nh3-dev (small src-restricted WAN
|
||||
surface), and whether to fully delete the wg0 config.
|
||||
- **infra-ops now on all four PVE hypervisors** (pfi-pve/nh3-pve/esh-pve/esh-pve-nas) — PVE
|
||||
ships without sudo, `apt install sudo` first or elway hangs on a password prompt.
|
||||
|
||||
_As of 2026-09-05 06:35 PDT — **ERP run 4 is TRAINING on pfi-gx10.** Everything else
|
||||
below is a live commitment or a known-open risk._
|
||||
|
||||
- **⚠ RUN 4 IS MID-FLIGHT — do not touch the GX10 GPU.** `~/erp-tune/run-04.pid`,
|
||||
log `~/erp-tune/run-04.log`. At 06:32 it was **486/938 steps**, 6 h 17 m elapsed,
|
||||
a genuinely settled **46.0 s/it** (unlike 3c, which climbed 52→70 — airoboros rows
|
||||
are short and single-window, so there is no long tail for the sampler to find).
|
||||
**~12.0 h total, finishing ~12:15 PDT 2026-09-05.** Loss ~2.04 at step 450,
|
||||
gnorm well under 1, checkpoints every 50. **Ping brokkr-smithy-dev at completion**
|
||||
— he takes base floors on the GX10 first, then the tuned arm, serially.
|
||||
|
||||
- **Operator ruling on the GX10: training first, serving transiently.** *"it's mostly
|
||||
for training, but can serve its trials. unless the box is needed for training work."*
|
||||
So `trial` (= run 3c on :8098) is down for the duration and comes back when run 4
|
||||
ends. I over-read an earlier version of this as "training-only" and had to correct
|
||||
it to brokkr — his serial floors-then-arm plan on the GX10 was never wrong.
|
||||
|
||||
- **`trial` gateway alias is a live 404 while the seat is down** — expected, not a
|
||||
fault. Restore with `~/erp-tune/relaunch-trial-seat.sh` on the GX10 (hand-run by
|
||||
operator ruling: experimental, NOT a compose stack, does not survive a reboot).
|
||||
|
||||
- ⚠ **Three dead gateway aliases return HTTP 500, not 404/503**: `trial`,
|
||||
`gemma4-26b-a4b-it-base`, `erp-tune-v2`. A dead seat reporting an *internal error*
|
||||
reads as an outage — brokkr checked his own work against mine because he could not
|
||||
tell. Deregistration costs a ~60 s fleet-wide LiteLLM restart; batch it with the
|
||||
next gateway change rather than spending a restart on tidying.
|
||||
|
||||
- **`trial` is on the SHARED-KEY gateway with a measured −40pp selfharm/methods
|
||||
regression.** Flagged to the operator twice (before adding, and after the gate
|
||||
measured it); he has left it up. His direct endpoint `10.100.50.60:8098` gives the
|
||||
same access with a blast radius of one. Settled — do not re-litigate.
|
||||
|
||||
- **ESH DAC: reverted to autoneg/1G, fiber going in at the weekend.** The operator
|
||||
ran copper through a drilled floor 2x4 himself; recommendation was a 10Gtek
|
||||
SR 2-pack + OM4 3 m LC-LC (~$40–65) because cable-vs-pull-damage was never resolved.
|
||||
|
||||
- ✅ **ESH WAN is now a STATIC PUBLIC IPv4: `128.177.138.182/30`, gw `128.177.138.181` (Cityside Fiber),
|
||||
LIVE** — operator confirmed 2026-09-08; UDM WAN1 (`eth8`) reads `wan_type: static`, uplink up since
|
||||
~2026-09-05, egress verified from esh-docker-vm = `128.177.138.182` (a real public address — CGNAT at
|
||||
ESH is HISTORY; `100.104.0.1` still shows as the ISP's first hop, that is their access network, not
|
||||
NAT). IPv6 unchanged (`2607:73c0:402:1d00::/56`, hosts still egress as their own v6). Done 2026-09-08:
|
||||
`128.177.138.182` added to the crowdsec `esh` allowlist on ana-docker (the false-ban class is closed
|
||||
for ESH). **All three follow-ups LANDED 2026-09-08 ~20:45Z (operator: "land all 3"):** (a) FortiGate
|
||||
infra-ops `trusthost3` 23.164.40.160 → `128.177.138.182/32` — VERIFIED by a real infra-ops login to
|
||||
`38.120.12.42` from esh-docker-vm (`ana-gw #`); `execute backup config flash pre-trusthost3-esh-static-20260908`
|
||||
ran ("Please wait...") but `execute revision list` errors on this box, so the backup is unconfirmed —
|
||||
the before-state was a single line, recorded here. (b) dormant `esh-ana` IPsec rebound wan2/192.168.200.111
|
||||
→ **wan1/`128.177.138.182`**, still `enabled=false`. (c) ESH UDM port-forward `esh-scale tailscale direct
|
||||
(UDP 41641)` → 10.0.50.65:41641 (id `6aa0727a…`) — VERIFIED: esh-scale now peers **direct via
|
||||
`128.177.138.182:41641`** (was DERP lax). Helpers: scratchpad `esh-udm-land.py` + `fg-trusthost3.sh`.
|
||||
`wan1-REVERT.json` is obsolete.
|
||||
|
||||
- ⚠ **ana-ml2 `tank` (raidz2, 8× NVMe) has 2 CKSUM errors on nvme7n1 + a boot-time 638 GB resilver on 2026-09-05 14:26
|
||||
(the box rebooted; nvme7 came up late/dirty). ONLINE, no data errors, 58% full — but NO scrub since 2026-04-12** (the
|
||||
Debian second-Sunday cron scrubbed zroot on 08-09, tank not — cause unknown). Box has neither `nvme-cli` nor
|
||||
`smartctl`, so nvme7's media-error counter is unread. Recommended (operator hasn't ruled): `zpool scrub tank` now,
|
||||
`apt install nvme-cli` + read nvme7 SMART, `zpool clear` after a clean scrub. **zroot is at 91%** — docker holds
|
||||
429 GB of images (204 GB reclaimable) + 74 GB build cache (36 GB reclaimable); a prune buys ~240 GB. pfi-pve pools
|
||||
(NASPool 7%, ospool 19%) clean, scrubbed 09-05 / 08-09. (checked 2026-09-09 00:00 PT)
|
||||
- **ESH 10G topology (measured 2026-09-09):** UDM SFP+1 (port 10, TP-Link DAC) ↔ USW-Pro-HD-24 Garage p25; UDM SFP+2
|
||||
(port 11, **OEM SFP-10G-LR fiber — the weekend fiber run IS in**) ↔ USW-Pro-XG-10 Media p11; Garage p27/p28 (3 m
|
||||
SFP-H10GB-CU3M DACs) ↔ esh-pve `enp3s0f0np0` / esh-pve-nas `enp5s0f0`. All 10G full duplex, autoneg off, optics
|
||||
tx −1.9 / rx −2.1 dBm both ends. Media p11 carries 416 rx / 251 tx errors that are STATIC (0 growth in 60 s, 363 h
|
||||
uptime — install-era); host NIC counters are ring-buffer misses, flat. Nothing to fix.
|
||||
- ⚠ **esh-nas is effectively open to the whole ESH LAN** — twelve NFS exports rw to
|
||||
`10.0.0.0/8` with `sec=sys`, and every SMB share but `backup` guest-writable.
|
||||
Hardening offered, ~1 h, **operator has not ruled**. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md`
|
||||
|
||||
- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** A job that once ran at
|
||||
941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms.
|
||||
Nightly 21:00, `all 1`. Worth its own investigation.
|
||||
|
||||
- **Ledger→SVOS rename: vault side DONE, gitea side is ledger-dev's to execute.**
|
||||
Name settled as `svos`, `~/development/ledger` → `~/development/svos`. Vault moved
|
||||
2026-09-05 (`secret` has no rename, so re-put + `rm`): stored
|
||||
`nh3-dev/development/svos/env.sh` (sha 7253633d4155, verified on read-back),
|
||||
retired `nh3-dev/development/ledger/env.sh` (sha feb418634e10, id
|
||||
3a2af37c-c5aa-4f46-9178-f4fb6008a753) — `secret rm` is a SOFT delete to trash, so
|
||||
it is recoverable. ⚠ The two shas differ: the vaulted copy was a 2026-08-11
|
||||
snapshot and the live file had drifted un-vaulted since. **The vault goes stale
|
||||
unless `secret backfill` is re-run.** Gitea `corviduo/ledger` (id 70) NOT renamed —
|
||||
their repo, their call; answered that 1.26.1 writes a `repo_redirect` on a
|
||||
same-org repo rename (upstream #807), that org/user renames do NOT redirect
|
||||
(#9531), that the redirect dies if anything re-creates the old path, and that the
|
||||
repo and org both carry 0 webhooks. **Gitea rename EXECUTED 2026-09-05** on the
|
||||
operator's direct authorization: `corviduo/ledger` → `corviduo/svos`, repo id 70
|
||||
unchanged. Redirect verified by measurement — web and API both 301, and
|
||||
`git ls-remote` on the old URL warns-and-follows to HEAD b48a11ca5183. ⚠ **The
|
||||
name `corviduo/ledger` is now burned**: the redirect dies silently the moment
|
||||
anything creates a repo at that path — ledger-dev carries it as a standing item
|
||||
in `docs/svos-rename-runbook.md`, since nothing warns whoever eventually creates
|
||||
that repo. They repointed their own clone the same day (`origin/main` at
|
||||
b48a11c), so the redirect is no longer load-bearing for any known consumer. Handle `ledger-dev` → `svos-dev` is an
|
||||
operator action at the post office.
|
||||
|
||||
- **DONE 2026-09-05 — `svos` Heimdall user + API key minted** on operator
|
||||
authorization. `user_id=svos`, `key_id=eab3cdbe`, suffix `d5ec48c2`, `wt_live_`
|
||||
format, on **worldtree-personal (10.250.50.152:8081)** — established by finding
|
||||
the `ledger` key there (created 2026-07-13, last used 2026-09-05T13:34, exactly
|
||||
as ledger-dev described). Value vaulted at
|
||||
`nh3-dev/development/svos/worldtree-api-key` (sha 23c10c9c7219, verified on
|
||||
read-back) and delivered by vault path, NOT over althing — ledger-dev runs on
|
||||
nh3-dev under the same uid, so the bus never carried the secret. The `ledger`
|
||||
key was read back after the mint and is untouched and live (`disabled=False`).
|
||||
⚠ **Plan tier left UNSET, deliberately**: `POST /admin/keys` takes an optional
|
||||
tier (user|free|pro|admin|readonly-admin) and there is **no way to read a user's
|
||||
current tier back** — no GET, `/admin/usage` returns an empty users list, and
|
||||
`/admin/events` is a live SSE stream, not an audit log. Guessing would have
|
||||
handed over a key that quietly differs; `POST /admin/users/svos/tier` fixes it in
|
||||
one call if their cutover hits a limit — and ledger-dev has recorded it as a
|
||||
cutover watch item to fix ON REPORT, explicitly not pre-emptively. ledger-dev
|
||||
pulled the key from the vault and verified it independently (same sha), so
|
||||
delivery is confirmed. **The cutover itself — pasting the value into env.sh,
|
||||
flipping `worldtree.user_id` from `ledger` to `svos`, registering
|
||||
`svos:miranda`, restarting the service — is WITH THE OPERATOR**, not with me;
|
||||
they will not do it off a peer message. **CUTOVER DONE + VERIFIED 2026-09-05**:
|
||||
`POST /agents/define` returned **201, not 409** — the load-bearing signal that
|
||||
they are genuinely on the new identity rather than silently still on the old
|
||||
one — then clean session create, turn, bifrost handshake and tool-call. **No
|
||||
plan- or rate-limit errors, so the unset tier is compatible and is NOT to be
|
||||
set** (they asked explicitly; it stays a watch item to fix on report, never by
|
||||
guess). Incidentally confirmed the bifrost allowlist really is per-deployment
|
||||
(host:port), not per-consumer — Worldtree reached back to their untouched
|
||||
endpoint under the new consumer_id. `env.sh` re-vaulted, sha 8a225c002072.
|
||||
⚠ **`secret backfill` was the WRONG tool for one known item** — it rescans every
|
||||
`~/development/*/{env.sh,.env}` and had not reached svos after three minutes;
|
||||
targeted `put` is the fast path, backfill is for catching drift across the box.
|
||||
**OPERATOR RULING 2026-09-05:
|
||||
worldtree-dev owns code only, no ops — key material is infra-ops's.** The global
|
||||
`~/.claude/CLAUDE.md` line routing "Heimdall scopes (Worldtree auth) →
|
||||
worldtree-dev" was corrected in place the same day on operator instruction.
|
||||
|
||||
- ⚠ **FOOT-GUN, generalises past this rename: a credential cutover whose OLD key
|
||||
is required for a later cleanup is destroyed by the natural housekeeping motion
|
||||
right after cutover.** Re-vaulting the post-cutover `env.sh` would have
|
||||
overwritten the last convenient copy of the old `ledger` key value — the only
|
||||
credential that can ever delete `ledger:miranda`. ledger-dev caught it and
|
||||
preserved the value first at
|
||||
`nh3-dev/development/svos/worldtree-api-key-ledger-legacy` (sha d44c2c1a651b);
|
||||
their step 8 ends by deleting that item. **I verified it is genuinely the live
|
||||
key** rather than trusting the label: its last 8 chars are `e68a5170`, matching
|
||||
the `ledger` key's suffix (key_id b38932f5).
|
||||
|
||||
- **STEP 7 DONE 2026-09-05, STEP 8 HELD.** `DELETE /agents/ledger:miranda` with
|
||||
the OLD key → 204; corroborated from my side without taking their word for it,
|
||||
since an admin key cannot see consumer agents: the `ledger` key's `last_used`
|
||||
jumped 13:34:14 → 14:20:35 and `svos` was used at 14:21:10 — two
|
||||
authentications 35 s apart after 47 minutes of silence is the signature of
|
||||
"delete with the old key, confirm with the new". Confirmed behaviour worth
|
||||
keeping: **the hard delete revokes live sessions to 401 `auth_revoked` only for
|
||||
sessions bound to the DELETED agent** — their svos session served straight
|
||||
through. **Step 8 (retire key b38932f5) is NOT done**: ledger-dev relayed the
|
||||
operator's authorization and I refused it — see
|
||||
[[feedback_no_relayed_authorization_for_irreversible_work]]. Both keys remain
|
||||
live. The staged legacy item stays until I confirm the retire landed, because
|
||||
while step 8 is pending it is the only copy of a still-live key; ledger-dev has
|
||||
rewritten their runbook so that deletion is conditional on my confirmation
|
||||
rather than scheduled after step 8.
|
||||
|
||||
- **SVOS ARC CLOSED — step 8 done 2026-09-05T14:28:15Z** on the operator's direct
|
||||
authorization in my own channel (never the relay). `DELETE /admin/keys/b38932f5`
|
||||
→ 200; preconditions checked BEFORE firing (svos had a live key, ledger existed
|
||||
and was not already revoked) and the post-state read back from `/admin/keys`
|
||||
rather than inferred from the 200: `ledger` disabled=True, `svos` untouched,
|
||||
deployment `/health` 200. **The rollback window is closed** — re-defining
|
||||
`ledger:miranda` is no longer possible. ledger-dev clears the staged
|
||||
`worldtree-api-key-ledger-legacy` vault item on this confirmation. ledger-dev gated their
|
||||
cleanup on observing a **401 from the old key**, not on my report of the
|
||||
timestamp — the right instinct, and they deleted the staged legacy item
|
||||
themselves (soft → trash, id a8038e5a-e2f6-4b77-bf17-99c9197d4b1f). Vault
|
||||
verified from my side: exactly two svos items remain (`env.sh` 8a225c002072,
|
||||
`worldtree-api-key` 23c10c9c7219) and no ledger-era item anywhere. Remaining on
|
||||
the arc: only the `ledger-dev` → `svos-dev` handle (with `_SEED_RECORD_TO`
|
||||
behind it) and a prose sweep — reversible work, theirs and the operator's.
|
||||
|
||||
- **Original constraints on that mint** (recorded because the deletion ordering is
|
||||
a permanent trap, not a one-time step): string
|
||||
`svos` verbatim (WT tier 3 admits only `^[a-z][a-z0-9-]{2,63}$`, INV-181-15);
|
||||
**keep the existing `ledger` key LIVE**, do not revoke. Ordering is load-bearing —
|
||||
`DELETE /agents/{agent_id}` refuses any caller that is not the row's owner, so
|
||||
`ledger:miranda` can ONLY be deleted with the `ledger` key; retire it first and
|
||||
the stale row outlives the ability to remove it, holding a live
|
||||
`agents.call:ledger:miranda` grant that nothing reaps (the 24h sweep only touches
|
||||
soft-deleted rows, and soft-deletion comes from revocation, never disuse). So:
|
||||
mint new → they cut over and verify → delete the agent with the OLD key → then
|
||||
retire it. Precedent for who mints: msg 401, worldtree-dev routed the pewpewstudio
|
||||
key request TO infra-ops. I hold only the PERSONAL admin token (:8081); which
|
||||
deployment `ledger` lives on is not yet established. Surfaced to the operator.
|
||||
|
||||
- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their first
|
||||
client-premises release cut (U10, unscheduled). Ping expected when U10 is scheduled.
|
||||
|
||||
- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** — deliberate;
|
||||
they are the operator's personal machines. A choice to revisit, not an oversight.
|
||||
|
||||
- **`vh/remote-ssh-mcp` forked 2026-09-05 (repo id 117, private, full 51-commit
|
||||
history)** — our copy of `the-nine-nation/remote-ssh-mcp` (MIT), an SSH MCP
|
||||
server chosen over the 693★ `tufantunc/ssh-mcp` on trust-surface grounds: **two
|
||||
npm deps** (`@modelcontextprotocol/server`, `zod`), 183 KB, and it **never
|
||||
touches key material** — it shells out to the system OpenSSH client, so
|
||||
`~/.ssh/config`, ControlMaster, ProxyJump and `infra-ops_ed25519` all just work.
|
||||
Shape: 2349 LOC across 11 source files, 811 LOC of tests including fake-ssh hang
|
||||
harnesses. Complements `elway` rather than replacing it — no file transfer, no
|
||||
idempotency; it takes ad-hoc reconnaissance with persistent cwd/env sessions,
|
||||
elway keeps deploys and uploads. ⚠ **The denylist is NOT security**: four regexes
|
||||
(`rm -rf /`, shutdown/reboot/poweroff/halt, mkfs, iptables -F) trivially bypassed
|
||||
by `bash -c`, variables or base64 — the author says so. **The real containment
|
||||
boundary is the host allowlist**, drawn from exact `Host` aliases in ssh_config
|
||||
with wildcards deliberately ignored. Two things to settle before use: the
|
||||
reboot/shutdown denial will block legitimate infra-ops work, and
|
||||
`.github/workflows/star-history.yml` is upstream chore CI sitting in a repo where
|
||||
`has_actions=True`. **Both actioned — three commits landed 2026-09-05, LOCAL
|
||||
ONLY and NOT PUSHED (push is the operator's call):** (1) stripped upstream
|
||||
furniture — star-history CI, its generated assets, the `server.json` registry
|
||||
manifest, branding JPEGs, zh-CN README; (2) removed the power-control denylist
|
||||
rule and documented in code + tests + README that the list guards ACCIDENTS and
|
||||
is not a boundary, with three bypasses asserted as ALLOWED so a green suite is
|
||||
never read as containment; (3) **`strictAllowlist`** — upstream's allowlist was
|
||||
additive and discovery unconditional, so the default allowlist was all 18 `Host`
|
||||
entries in `~/.ssh/config`. Strict makes explicit hosts authoritative and
|
||||
discovery metadata-only. Verified live: `corviduo-dev` is in ssh_config, not in
|
||||
our allowlist, and is refused `host_not_allowed`. 41/41 tests green.
|
||||
|
||||
- **`remote-ssh` MCP server is LIVE** — registered project-scoped in
|
||||
`eshpfi-management/.mcp.json` with `SSH_MCP_STRICT_ALLOWLIST=1`; allowlist in
|
||||
`~/.config/remote-ssh-mcp/config.json` starts deliberately narrow at
|
||||
**`irv-ml1`, `nh3-extdev`** (widen there, not by discovery). Smoke-verified end
|
||||
to end on both: persistent shell, `cd` and exported vars survive across calls,
|
||||
**~6 ms/command on nh3-extdev and ~22 ms on irv-ml1** (WireGuard) versus a fresh
|
||||
handshake each time. ⚠ **`.mcp.json` points at the built `dist/`** — edit the
|
||||
fork without `npm run build` and the server keeps serving old code; that bit me
|
||||
mid-session. ⚠ **A finite stdin pipe is NOT a valid smoke harness** — closing
|
||||
stdin kills the server mid-handshake and reports `connect_failed: SSH shell
|
||||
exited during the open handshake`, which looks exactly like a remote-side fault
|
||||
and is not. Use a client that holds stdin open. (I briefly suspected irv-ml1's
|
||||
zsh login shell; wrong — the server invokes `bash --noprofile --norc`
|
||||
explicitly, so the login shell is irrelevant.)
|
||||
|
||||
- **`esh-macbook-air` (10.0.10.83) is DELIBERATELY NOT BACKED UP — operator ruling
|
||||
2026-09-05, settled, do not re-raise.** Surveyed it and found no Time Machine
|
||||
destination and no restic/borg/rclone/kopia installed, protecting 132 GiB.
|
||||
Operator's answer: it is his laptop and the surface is **regenerable** — mostly
|
||||
applications, with real data living in OneDrive, iCloud and ssh sessions — and he
|
||||
does not want PBS filled with it. Correct call; the finding was real and the
|
||||
conclusion is that it does not matter. FileVault On and SIP enabled already cover
|
||||
the loss-and-theft axis. The same reasoning presumably extends to
|
||||
`esh-mac-studio` and `vuongs-mac-mini`. **Still open and much smaller:** Remote
|
||||
Apple Events (port 3031/eppc) is listening and nothing uses it — one toggle.
|
||||
|
||||
- ⚠ **`remote-ssh` MCP could not be used for its FIRST real task, and the blocker
|
||||
is `~/.ssh/config`, not the tool.** The server accepts only exact `Host` aliases,
|
||||
so a host addressed by raw IP is structurally unreachable no matter what the
|
||||
allowlist says. **13 of the 28 hosts in `servers/` have an alias; 15 do not** —
|
||||
including `ana-docker`, `ana-ml2`, `nh3-docker`, `pfi-gx10`, `esh-docker-vm` and
|
||||
every hypervisor, i.e. most of where the work happens. Widening
|
||||
`~/.config/remote-ssh-mcp/config.json` does NOT fix this; the aliases have to
|
||||
exist first. **RESOLVED the same day, and NOT by adding aliases.** Operator
|
||||
pushback, correct: a poking-around tool is ad-hoc by nature, and pre-registering
|
||||
a host before you can look at it is the opposite of ad-hoc — generating aliases
|
||||
for the known fleet would not have helped, because the ad-hoc case is by
|
||||
definition the host not yet in the inventory. Implemented address-based reach
|
||||
instead (`allowedNetworks` / `deniedNetworks` / `defaultUser` /
|
||||
`defaultIdentityFile` / `hostKeyPolicy`). **Live config: `10.0.0.0/8` allowed,
|
||||
connecting as `infra-ops` with `~/.ssh/infra-ops_ed25519`, `accept-new` host
|
||||
keys, SureFire tenant hosts carved out via `deniedNetworks` (deny beats allow,
|
||||
host-specific rather than a /24 because `pfi-pve` shares 10.250.250.0/24).**
|
||||
Verified live: 10.0.10.83 opens by raw IP as infra-ops, 10.250.150.100 refused by
|
||||
the carve-out, 192.168.1.5 refused as outside. ⚠ **My own earlier objection was
|
||||
half wrong** — the credential boundary is about SECRETS ("never accept passwords
|
||||
or private-key material"), not identity, so supplying a username does not breach
|
||||
it; the real problem was only that the server passed no user at all, so a bare
|
||||
address would connect as the LOCAL account. Mechanics, not principle.
|
||||
|
||||
- ⚠ **`uv tool install --force .` DOES NOT REBUILD when the version has not moved**
|
||||
(forseti, measured 2026-09-05). `--force` only handles "a tool by this name
|
||||
exists"; `--reinstall` is what rebuilds instead of reusing the cached build keyed
|
||||
on the version string. It prints `Installed 9 executables` over **stale code**
|
||||
with nothing raising its hand — it cost forseti a bug that survived a reinstall
|
||||
AND a re-smoke, because the binary verified against had not changed. **Always
|
||||
`uv tool install --force --reinstall .`**, both flags, every time. Same shape as
|
||||
the `.mcp.json` → built `dist/` trap found today: a deploy surface that reports
|
||||
success while serving the previous artifact. When a fix "does not take", suspect
|
||||
the artifact before the code.
|
||||
|
||||
- **althing 3.5.0 released** (forseti) — adds a 9th binary,
|
||||
`althing-operator declare <handle> --description "..."`, restoring the CLI handle
|
||||
declaration v2 had and v3 removed. Deliberately a SEPARATE binary, not a
|
||||
`postbox` subcommand: the invariant is that no SESSION surface exposes an
|
||||
operator verb. Relevant to the pending `ledger-dev` → `svos-dev` rename, which is
|
||||
still the operator's call. nh3-dev not yet upgraded.
|
||||
|
||||
- ⚠ **`remote-ssh` MCP: a bare `sudo` hangs the session forever — pipe it.**
|
||||
`ssh_run 'sudo -n whoami'` returns `running` with EMPTY stdout and the session is
|
||||
then permanently `busy`; `sudo -n id | cat` works and returns everything.
|
||||
**Measured on BOTH macOS 26.6 and Debian (nh3-extdev), so it is the tool, not a
|
||||
platform quirk.** Cause: sudo ≥1.9.14 defaults `use_pty` on and relays through
|
||||
its own PTY; the run frame gives the command stdin on `/dev/null` while stdout
|
||||
stays on the session PTY, the relay never completes, and the completion marker
|
||||
never arrives. Workaround `| cat` is in CLAUDE.md. **The proper fix is unbuilt**
|
||||
— likely running the command through a pipe inside the run frame and taking the
|
||||
exit code from `PIPESTATUS`, which is a real protocol change (commands lose tty
|
||||
detection) and wants its own red-green cycle. Matters more than it sounds: infra
|
||||
work is sudo work, and this was found by USING the tool, not by smoke-testing it.
|
||||
|
||||
- **`dsh` on `esh-macbook-air` updated 0.1.1-rc.2 → 0.1.2-rc.1** (2026-09-05;
|
||||
latest published 2026-09-03). Global install and the shared profile tree both
|
||||
confirmed on the new version. ⚠ **The RUNNING `dsh web` (pid 16231, up since
|
||||
Wed 4pm, 127.0.0.1:3080) is still on the OLD code and was deliberately NOT
|
||||
killed** — there is no LaunchAgent, so killing it would have left nothing
|
||||
running rather than a restarted service. It runs as a FOREGROUND process in the
|
||||
operator's terminal (`s005`, `S+`): it dies with the terminal and does not
|
||||
survive a reboot, which is the real fragility. A `com.pfi.dsh-web` LaunchAgent
|
||||
was drafted but **the privileged write was blocked by the permission
|
||||
classifier** — base64 piped into `sudo tee` of a LaunchAgent is a malware-shaped
|
||||
pattern and the block is correct; it needs operator approval or an operator-run
|
||||
install. Bind stays `127.0.0.1` deliberately: widening it is a security decision
|
||||
on a personal laptop whose application firewall is off, and not mine to take.
|
||||
|
||||
- **sudo hang FIXED in the fork (`30a1f76`), and two wrong shapes are recorded so
|
||||
nobody retries them.** The command's stdout now goes to a **fifo drained by a
|
||||
background `cat`**: non-tty (so sudo skips its own PTY), no subshell (so `cd`
|
||||
and `export` still persist), and relayed live (so `running` + `ssh_peek`
|
||||
streaming survives). `cmd | cat` was tried first and **broke cwd persistence** —
|
||||
every pipeline stage runs in a subshell — caught by the existing test.
|
||||
`cmd > file` would have been non-tty and subshell-free but invisible until the
|
||||
command ends. ⚠ **Deliberately NO `wait` on the relay**: a sudo child inherits
|
||||
the fifo's write end, `cat` never sees EOF, and the wait hangs — measured, with
|
||||
`sudo -n whoami` printing `root` and then wedging the session. Residual risk
|
||||
stated in the frame: a command's tail can in principle land after its own
|
||||
marker. ⚠ **Job control off AND the relay brace-wrapped with stderr discarded** —
|
||||
both needed, because macOS ships bash 3.2 where `set +m` alone still leaked
|
||||
`[1] 75449` into the parsed stream. Verified live on macOS and Debian: bare sudo
|
||||
in ~20 ms, state persists, exit codes correct. **`sudo -u <other-user>` still
|
||||
wants `| cat`** — not chased further.
|
||||
|
||||
- **dsh web on `esh-macbook-air` is now a LaunchAgent** (`com.pfi.dsh-web`,
|
||||
installed 2026-09-05, `runs=1`, `state=running`, pid 76728 on 0.1.2-rc.1). It
|
||||
was a foreground process in the operator's terminal that died with the window;
|
||||
it now survives terminal close and reboot with `KeepAlive` + `RunAtLoad` and a
|
||||
10 s `ThrottleInterval` so a startup error cannot hot-loop. Logs to
|
||||
`~/Library/Logs/dsh-web.log`. ⚠ **The plist names the node interpreter
|
||||
explicitly** — launchd's minimal PATH has no `~/.local/node/bin`, so the
|
||||
shebang's `env node` fails. ⚠ **0.1.2-rc.1 requires a TOKEN**: bare
|
||||
`http://127.0.0.1:3080/` now returns 401 and the tokened URL is printed to the
|
||||
log on each start, so a bookmark from the old version will not work. Bind stays
|
||||
127.0.0.1 deliberately.
|
||||
|
||||
- ⚠ **althing tools on nh3-dev are 3.6.0, but the POST OFFICE CONTAINER IS STILL
|
||||
3.0.0** (`gitea.phasefinal.com/claude-bot/althing-post-office:3.0.0`, up 7 days
|
||||
on nh3-docker). forseti: the new handle verbs (`althing-operator delete` /
|
||||
`retire`, and `declare` from 3.5.0) live in the post office, so they fail with
|
||||
"no tool named ..." until the container carries 3.6.0. Schema gains
|
||||
`handles.retired_at` via the idempotent `_ADDED_COLUMNS` path, so the live store
|
||||
upgrades itself on first start — no manual migration. **REBUILT AND DEPLOYED
|
||||
2026-09-05** on operator authorization: image
|
||||
`claude-bot/althing-post-office:3.6.0@sha256:13158835488a8ec04f990c97c4f4c68f1d923b12494319cf07392552e68f8a78`,
|
||||
built on nh3-dev from a clean tree at `4d26226`, pushed to the gitea registry
|
||||
under the **claude-bot** namespace (not `vh` — package namespaces are owned).
|
||||
**Bus down ~4 minutes, 09:35–09:39 PDT.**
|
||||
**The backup was taken the way the compose file says to, and it mattered:** at
|
||||
stop time `post_office.db` was 23.8 MB with a **5.9 MB WAL** — copying the .db
|
||||
alone would have silently lost the day's mail. Stop → `PRAGMA
|
||||
wal_checkpoint(TRUNCATE)` (WAL → 0 bytes) → copy → verify. Backup at
|
||||
`nh3-docker:/var/backups/althing/post_office.db.pre-3.6.0-20260905`, integrity
|
||||
`ok`, counts identical on both sides (handles 76, messages 995, recipients
|
||||
1022). ⚠ **Reading a WAL-mode SQLite backup read-only needs `?immutable=1`, not
|
||||
`?mode=ro`** — `mode=ro` still wants to create a `-shm` and dies with "attempt to
|
||||
write a readonly database". Post-deploy: same counts, `handles.retired_at`
|
||||
present, `retired 0`, and `mem=536870912` / `oom=-500` verified by `docker
|
||||
inspect` rather than by reading the yaml, per that file's own warning.
|
||||
`althing-operator` now offers `declare | delete | retire`, which unblocks the
|
||||
pending `ledger-dev` → `svos-dev` rename.
|
||||
|
||||
- **Handle `retire` is REVERSIBLE — re-declaring the name revives it, history
|
||||
intact** (forseti smoked it against the live bus 2026-09-05). That matters for
|
||||
the pending `ledger-dev` → `svos-dev` rename: `retire` is the right verb (delete
|
||||
refuses any handle that has mail, naming both counts — `delete forseti` was run
|
||||
against production and correctly refused at 53 sent / 81 addressed, which is
|
||||
safe to try precisely because refusing IS the behaviour), and it can be undone
|
||||
by declaring the name again. Lower stakes than "retire" sounds.
|
||||
Both of my deploy findings — the naive-copy WAL trap and `?immutable=1` — are
|
||||
now in althing's own `deploy/INSTALL.md` (`d6f4fb5`) under a new
|
||||
"Backing up the store" section, on the reasoning that they are properties of
|
||||
the project's `journal_mode=WAL` choice rather than of my procedure.
|
||||
- **🔥 ERP RUN 7 TRAINING on pfi-gx10** — launched 2026-09-08 23:06 PT, pid in `~/erp-tune/run-07.pid`, 542 steps
|
||||
at ~80 s/it, adapter ~noon 09-09. Watch: `ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-07.log | tail"`.
|
||||
Hourly cron in the old session reported stats; a fresh session checks by hand. When the adapter lands: post the
|
||||
adapter-landed line to brokkr's thread (`01M20AHY9DY92RJK84YD24VSY9`), merge (`merge_lora.py --base <ARA dir>
|
||||
--adapter run-07/adapter --out serve/merged-run07 --chat-template <stock>`), copy stock `processor_config.json`
|
||||
into `merged-run07`, serve `erp-seat-base-ara` for floors → on brokkr's swap cue serve `erp-tune-v7` (run-5 flags,
|
||||
:8098). Runbook `docs/runbooks/gx10-run-07.md`; run-6 choreography in `gx10-run-06.md`.
|
||||
- **⏳ ana-ml2 pool actions — NEXT SESSION, operator-approved in principle:** (1) `zpool scrub tank` + `zpool clear`
|
||||
on a clean pass; (2) `apt install nvme-cli` + read nvme7 SMART; (3) docker image/builder prune to pull zroot back
|
||||
from 91%. Detail + order: `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md`.
|
||||
- **`trial` (LiteLLM) → `erp-tune-v6-nvfp4a16` on ana-ml2 :8021** (stack `stacks/erp-seat`, vLLM nightly
|
||||
`311b3513`, no gate by operator ruling). Forced tool_choice is prompt-driven (6/9); `response_format: json_schema`
|
||||
is deterministic. ⚠ `stacks/gemma4-charrp` lacks `--exclude-tools-when-tool-choice-none` (same empty-turn trap);
|
||||
applying it bounces the char-rp seat — operator's call, not taken.
|
||||
- **📮 althing reachability on a bg seat = the cc-channel route:** `althing-route declare --handle infra-ops
|
||||
--pid <pid from $CLAUDE_CODE_MESSAGING_SOCKET>` per session (`--discover-pid` refuses on a forked child). The
|
||||
harness kills detached background tasks under memory pressure — use bounded foreground polls (≤590 s), not
|
||||
background watchers, for long waits.
|
||||
- **Open items carried from 09-06 (unchanged):** NASPool evac copy `ospool/naspool-evac` (1.65 T) + `@evac` snaps
|
||||
can be destroyed once ONE Backrest run is confirmed (scrub clean, PBS landing); pfi-pve PSU1 dead + backplane bays
|
||||
9/10 dead (cold spares, next colo visit); FortiGate WAN SSH still temporarily open (trusthost2/3 = NH3 + ESH
|
||||
static) — close when the edge is retired; irv-ml1 on-site decisions (reverse tunnel / UDM fwd 47822 / wg0 config
|
||||
deletion) pending Irvine access; ~10 running irv-ml1 service cards still carry dead `10.100.79.3` hrefs (recreate
|
||||
each to apply labels); deployed `.env` for asset-engine / open-webui / skaldsong may hold the dead default.
|
||||
- **MEMORY.md (auto-memory index) is near its 24.4 KB read cap** — compaction pass still owed.
|
||||
- **persistent-memory.md was 830 lines; this snapshot moved the superseded in-flight blocks and 09-08-or-older
|
||||
settled entries to `archival-memory.md`.** The remaining bulk is Tools-and-conventions rows CLAUDE.md already
|
||||
covers — a deliberate redundancy trim is still the real fix (not done).
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-09]` **ana-ml2 `tank`: 2 CKSUM errors on nvme7n1 after a boot-time resilver, NO scrub since 04-12, zroot 91% — three actions DEFERRED to a clean-context session** (scrub → nvme-cli SMART → docker prune), operator ruling "we'll do all 3 on clean context"; tracked at commit `3e18a04` + the post-clear handoff. ESH 10G links measured clean (fiber run live on UDM SFP+2 ↔ USW-Pro-XG Media). → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md`
|
||||
- `[2026-09-08]` **ana-ml2 mesh return routes PERSISTED** as `/etc/network/if-up.d/mesh-routes` (Debian 13 ifupdown, no netplan) via `playbooks/ana-ml2-mesh-routes.yaml` (elway, verified) — operator: "persist the routes". Hook not yet exercised by a real reboot. `f923d6a`.
|
||||
- `[2026-09-08]` **ERP run 7 LAUNCHED on pfi-gx10 23:06 PT** under `operator-2026-09-08-rnd-run7` — opening-split slot + mask union; free check passed with two explained deltas; first launch died on a missing recipe (zsh quoting). → `persistent-memory.d/2026-09-08-erp-run7-launched.md`
|
||||
- `[2026-09-08]` **erp-tune-v6-nvfp4a16 quantized (data-free W4A16, ~90 s) and serving on ana-ml2 :8021; `trial` aliased to it ("no gate"); tool calling fixed where it can be** — `tool_choice:none` flag; forced tool_choice is prompt-driven on Gemma-4 by vLLM design, nightly `311b3513` raises it 1/9→6/9; json_schema is the deterministic path. → `persistent-memory.d/2026-09-08-erp-seat-nvfp4-trial-and-toolcalling.md`
|
||||
- `[2026-09-08]` **Run-6 gate: CSAM level=review soft trip HALTED it; operator adjudicated GO ("baby is a pet name"); TRANSFERRED finalized without the tuned refusal leg; k=25 legs cut** — the flagged text exists nowhere by design. → `persistent-memory.d/2026-09-08-run6-gate-csam-adjudication.md`
|
||||
- `[2026-09-08]` **ESH static-WAN follow-ups landed (FortiGate trusthost3, esh-ana IPsec rebind, UDP 41641 → mesh direct); YTVC chased back up (nh3-scale SOCKS, stale yt-dlp layer, punkt_tab) and v0.3.6 CrisperWhisper deployed; gitea webhook repointed off the dead wg0 IP with the HMAC secret re-applied.** → `persistent-memory.d/2026-09-08-esh-static-wan-followups-and-ytvc.md`
|
||||
- `[2026-09-08]` **ERP run 5 = RESCUED (landmark R49.5)** — first capability-gate pass in the ERP-seat line; the 3.46%-loss dependency-forcing slot (GovReport+QMSum) broke the coupling runs 3c/4 couldn't. Seat `erp-tune-v5` served on gx10:8098, `trial` alias repointed 3c→v5. → `persistent-memory.d/2026-09-08-run5-rescued.md`
|
||||
- `[2026-09-08]` **R47 base settled from bytes = STOCK `google/gemma-4-26B-A4B-it`** — three-way sha match (local == HF etag == stock LFS oid; commit `4d7ae498` == stock HEAD); the `-heretic` label is a naming error, all runs trained from stock. Accept-vs-swap now evidenced. → `persistent-memory.d/2026-09-08-base-provenance-stock.md`
|
||||
- `[2026-09-08]` **yt-voice-clipper back UP** — dead since the 09-06 danted retirement (every job failed at yt-dlp, bot-gated on the Irvine datacenter IP). Fix: danted on **nh3-scale** (CT107) at `socks5h://100.64.0.1:1080`, fleet-ACL'd, residential egress 70.230.226.88 measured; `YTVC_PROXY` repointed, worker recreated, end-to-end job DONE with positive (proxied) + negative (direct = bot-gate) controls. Homepage card href/siteMonitor → `irv-ml1.nh3.internal:8000` (was dead wg0 IP). Then a SECOND fault: full downloads 403'd through the proxy (cookies irrelevant) = stale yt-dlp 2026.07.04 from a cached Dockerfile layer → `compose build --no-cache api` (2026.08.19), which dragged in a whisperx/nltk that needs `punkt_tab` → staged on the data volume + `NLTK_DATA` in the override. Operator's video x7kWJojf1MI → done, 8 clips. yt-voice-clipper-dev shipped both Dockerfile fixes + **CrisperWhisper 2.0 (v0.3.6, `b62849d`) — deployed and verified (12 clips, [UM]/[UH] tags)**. ⚠ The gitea push webhook had been targeting the dead wg0 IP since 09-06 (never fired) → repointed to `10.6.110.50:9008` with the HMAC secret re-applied; deploy script passes `YTDLP_REFRESH`. Script `scripts/setup-nh3-scale-socks-egress.sh`. → auto-memory `reference_nh3_egress_proxy`, `reference_ytvc_autodeploy`.
|
||||
@@ -732,47 +223,11 @@ below is a live commitment or a known-open risk._
|
||||
- `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before.
|
||||
- `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path.
|
||||
|
||||
- `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance. → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md`
|
||||
- `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9%→0.0%) plus flex_attention. ⚠ Carries the dynamo recompile-ceiling trap that produced two wrong published conclusions. → `persistent-memory.d/2026-08-25-mfu-root-caused-attention.md`
|
||||
- `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently. → `persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md`
|
||||
- `[2026-08-25]` **Refusal retention measured (base 0/100 → tuned 29/100, 71 still complying) — but on the WRONG AXIS.** `harmful_behaviors` is general harm; the abliteration was run for explicit fiction. The convenient set with a recorded baseline was not the right one. → `persistent-memory.d/2026-08-25-refusal-retention-probe.md`
|
||||
- `[2026-08-25]` **Worldtree b188 + b189 shipped; bridge extracted to `pfi/wt-matrix-bridge` because `vh` is a USER not an ORG** and no service account can ever publish to a user namespace. Plus the selene catalog entry that lied about what answers, and a #411 diagnosis I got wrong twice before a directory probe settled it. → `persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md`
|
||||
- `[2026-08-25]` **Run 2's base is an OPEN OPERATOR DECISION, deliberately not staged** — four options with materially different safety postures, detailed in Current state. Tracked at althing thread `01M0WQ8W5574KMEVCHCEKEXNS5`. ⚠ Do not let it get filed as a config knob; it is a reversal of the trainee-selection decision.
|
||||
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
|
||||
- `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`.
|
||||
- `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`
|
||||
- `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then. **ana-ml2 is FAR past 0.24.0 and the box runs a SPREAD, not one version** (measured 2026-08-24): `gen` on `nightly-311b3513` = **0.27.2rc1.dev150**, `mog-sec` on `nightly-e9d1398d` = 0.26.1rc1.dev1102, the small seats still on 0.24.0, and char-rp/trainee-bench pinned to v0.26.0. ⚠ **`vllm/vllm-openai:v0.27.1` is already ON DISK, unused** — a TAGGED release, which is the right retest target: no nightly variance, no pull, ~4 months past the diagnosis. So: RETEST hot-swap LoRA on **v0.27.1** before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation.
|
||||
- `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
|
||||
- `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
|
||||
- `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `<html>` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
|
||||
- `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
- `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md`
|
||||
- `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md`
|
||||
- `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md`
|
||||
- `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`.
|
||||
- `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did).
|
||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`.
|
||||
- `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
- `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md`
|
||||
- `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md`
|
||||
- `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md`
|
||||
- `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`. → `persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md`
|
||||
- `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.)
|
||||
- `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink <id|index>` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`)
|
||||
|
||||
- `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md`
|
||||
- `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`)
|
||||
- `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`)
|
||||
- `[2026-08-22]` **Max-Q 1.8× spread does NOT apply to LLM decode — measured, not argued.** ana-ml2 draws 256–266 W of 300 W under sustained 100% decode with `SW Power Cap: Not Active` and clocks pinned. Corrected to brokkr-smithy-dev after I had lent the claim credibility; 122B figure (~90–93 tok/s at 262K) stands as a straight number.
|
||||
- `[2026-08-21]` **ESH internal IPv6 live on two LANs; the Cityside v4 static is a CARRIER problem, proven.** A full gateway reboot forced a fresh DHCP DISCOVER and returned the identical CGNAT address. YaRN was already configured — "1M needs YaRN, absent" was false. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` sibling entry in `ad21302`
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
|
||||
|
||||
|
||||
@@ -805,26 +260,12 @@ below is a live commitment or a known-open risk._
|
||||
|
||||
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
|
||||
|
||||
|
||||
|
||||
_Older entries archived to archival-memory.md._
|
||||
|
||||
_248 older entries archived to archival-memory.md._
|
||||
_275 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-09-04]` **Forcing 10G on the ESH-Media DAC — it linked, then degraded over hours, and I reported a plateau at two minutes.** ⚠ A clean zero-error link at 1G does NOT rule out a marginal cable; autoneg's fallback was protecting something real. → `persistent-memory.d/2026-09-04-dac-forced-10g-failed.md`
|
||||
|
||||
- `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.
|
||||
- `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision.
|
||||
- `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name.
|
||||
- `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor.
|
||||
- `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place.
|
||||
- `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes".
|
||||
- `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.**
|
||||
- `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.
|
||||
|
||||
|
||||
- `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
|
||||
|
||||
_144 older entries archived to archival-memory.md._
|
||||
_152 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user