memory: snapshot — run 7 training on gx10; run 6 TRANSFERRED after CSAM adjudication; erp-tune-v6-nvfp4a16 live as trial; ESH/YTVC/webhook repairs; ana-ml2 routes persisted; tank/zroot actions deferred to next session. Index 830→271 lines: 27 decisions + 8 abandoned archived, superseded in-flight blocks archived verbatim
This commit is contained in:
@@ -1,192 +0,0 @@
|
||||
# DFlash2 speculative decoding — measured on our own stack (2026-08-22)
|
||||
|
||||
Operator-driven session. **Read the epistemic labels.** During the chase we generalised from
|
||||
observations that later proved wrong; this file separates what was *measured* from what remains
|
||||
*hypothesis*, and records the wrong turns so nobody re-derives them.
|
||||
|
||||
## What DFlash2 is
|
||||
|
||||
A **2B draft model** (3.85 GB bf16) for speculative decoding against Qwen3.8-27B —
|
||||
`incoai/Qwen3.8-27B-DFlash2`, Apache-2.0, blog `inco.ai/blog/dflash2`, upstream `z-lab/dflash`.
|
||||
Block diffusion: drafts a whole 8-token block in one pass, with a candidate selector tracing a
|
||||
path through per-slot top-K. Lossless (greedy matches the target).
|
||||
|
||||
vLLM support merged **2026-08-21 05:27 UTC** as PR **#52816** (`b389ac29`). Method string is
|
||||
**`"dflash"`**, not `dflash2`.
|
||||
|
||||
## ✅ MEASURED — throughput and acceptance
|
||||
|
||||
Single instrument (`specbench.py`, 8 fixed prompts, temp 0, max_tokens 256), delta against
|
||||
vLLM's own `spec_decode` counters. The MTP k=3 numbers reproduce our recorded 58.4% / 55.3%
|
||||
figures exactly, which is what validates the instrument.
|
||||
|
||||
| seat | config | accepted tok/forward | throughput |
|
||||
|---|---|---|---|
|
||||
| gen (orcarouter) | MTP k=3 *(production)* | 2.753 | 114.9 tok/s |
|
||||
| gen | MTP k=7 *(control)* | 3.041 | **74.0 tok/s** |
|
||||
| gen | **DFlash2 k=7** | **3.254** | **131.9 tok/s** |
|
||||
| sec (M.O.G.-SEC) | MTP k=3 *(production)* | 2.676 | 110.5 tok/s |
|
||||
| sec | **DFlash2 k=7** | **3.252** | **130.0 tok/s** |
|
||||
|
||||
**⭐ The k=7 MTP control was essential and inverted the obvious read.** Going deeper on MTP
|
||||
*improves acceptance* (2.753 → 3.041) while **destroying throughput** (114.9 → 74.0). Our MTP
|
||||
head is a single module (`mtp_num_hidden_layers=1`, only `mtp.layers.0`, 15 tensors) run
|
||||
autoregressively, so k draft tokens cost k sequential forward passes. **"Just raise
|
||||
num_speculative_tokens" is a trap** — without the control I would have recommended it.
|
||||
|
||||
DFlash2's win is therefore **not better per-token acceptance** — our MTP is actually *better* at
|
||||
position 0 (79.6% vs 75.4%). It is that block drafting makes depth nearly free.
|
||||
|
||||
**⭐ The drafter is model-agnostic across finetunes — 3.254 (gen) vs 3.252 (sec), a 0.06%
|
||||
difference**, with superimposable per-position curves. One drafter file on `/tank` serves both.
|
||||
|
||||
## ✅ MEASURED — how DFlash2 runs (answers "can one drafter serve both seats?")
|
||||
|
||||
**EAGLE3-style coupled, not standalone.** In vLLM: `load_model(self, target_model)` binds it to a
|
||||
specific target object; `pass_hidden_states_to_model=True`; `gpu_model_runner` reads
|
||||
`dflash_config.target_layer_ids` → `[i+1 …]` to register auxiliary hidden-state capture on the
|
||||
target at layers **5, 19, 33, 47, 61**. It even reads the target's RoPE style at load.
|
||||
|
||||
Consequences:
|
||||
- **Weights file is shareable** (one download, both seats mount it) — gen and sec are
|
||||
architecturally identical on every dimension the drafter needs: 64 layers (deepest tap 61),
|
||||
hidden 5120, intermediate 17408, vocab 248,320 > mask token 248,070.
|
||||
- **VRAM is NOT shareable — 3.85 GB per seat.** The drafter lives inside the target's engine
|
||||
process, consuming hidden states mid-forward. Two seats are two processes; there is no
|
||||
cross-process sharing mechanism and there could not be.
|
||||
|
||||
## ✅ MEASURED — it works on our stack, which the card does not claim
|
||||
|
||||
The card tests stock BF16 on an H200 with FlashAttention 3. Verified here instead:
|
||||
**abliterated + NVFP4 `compressed-tensors` target ✓, Blackwell sm_120 ✓, DFlash2 CUDA graphs
|
||||
captured ✓.** None of that was documented anywhere.
|
||||
|
||||
## 🔶 HYPOTHESIS — why our acceptance trails the published numbers
|
||||
|
||||
Both our targets land at ~3.25 accepted length against the card's 4.10–5.46 on stock BF16.
|
||||
**Finetune drift is ruled out** — two *different* finetunes gave identical results to three
|
||||
decimals. The shared variable is **NVFP4 quantization of the target**, which is mechanically
|
||||
plausible (the drafter reads quantized hidden states at its five taps). Second candidate:
|
||||
prompt distribution (ours general-purpose, theirs GSM8K/MATH/HumanEval/MBPP/MT-Bench).
|
||||
**Neither is confirmed.** Settling it needs a BF16 target seat (~56 GB) — a real GPU window.
|
||||
|
||||
## ❌ RETRACTED — the "MTP head mismatch causes the degeneration" hypothesis
|
||||
|
||||
**Operator ruling, 2026-08-22: this hypothesis is WRONG. The degeneration lives in the un-fixed
|
||||
vLLM, not in the weights.** Recorded here rather than deleted, because it was reasoned to
|
||||
confidently enough that a future session could re-derive it.
|
||||
|
||||
**Two independent failures produced it, and the second is the instructive one:**
|
||||
|
||||
1. **I treated a false dichotomy as a deduction.** Having verified gen and sec run an identical
|
||||
engine (same image ID `sha256:bd3236cff208…`, same live version
|
||||
`0.27.2rc1.dev150+g311b3513a` read from inside both processes, same flags bar
|
||||
`gpu-memory-utilization` 0.43 vs 0.44), I concluded "config is eliminated, therefore it is the
|
||||
weights." That does not follow. **An engine bug present in BOTH seats is not exonerated by the
|
||||
two seats being identical** — it just means the engine cannot explain a *difference*. It can
|
||||
still explain the *failure*.
|
||||
2. **The difference I was explaining may not exist.** The premise was a single operator
|
||||
observation of sec degenerating at ~2k, made during a session with many concurrent changes.
|
||||
**n=1 under heavy concurrent modification is not evidence** — see the meta-lesson below.
|
||||
|
||||
**What survives as fact** (measured, still true, just not causal): sec's MTP head *is*
|
||||
byte-identical to `qwen38-27b-uncensored-bf16` across all 15 tensors — a stock head on a
|
||||
security-finetuned body, because the `Qwen3_5ForConditionalGeneration` wrapper never loads the
|
||||
head, so the finetuning could not reach it. gen's orcarouter head *was* abliterated in-band by
|
||||
its author. Acceptance differs slightly (gen 58.4%, sec 55.9%). **All true. None of it shown to
|
||||
cause multi-turn degeneration.**
|
||||
|
||||
**Current standing explanation: the degeneration is an engine bug in the un-fixed vLLM.** Both
|
||||
production seats run `311b3513`, which is **172 commits behind GDN spec-decode fix #53077**
|
||||
(merged 2026-08-20). `#51113` is present in that build and is therefore **necessary but
|
||||
insufficient** on its own.
|
||||
|
||||
## ⭐⭐ META-LESSON — n=1 during a busy session is not evidence
|
||||
|
||||
The operator's own framing, and it generalises past this incident: **an observation made while
|
||||
many things are being changed at once cannot carry a causal claim, no matter how confidently it
|
||||
is reported.** Tonight that single observation became the load-bearing premise for a weights-side
|
||||
hypothesis, a root-cause narrative, and very nearly a recommendation.
|
||||
|
||||
This is the same failure the gen-seat compose file already warns about in different words — *"a
|
||||
passing probe is NOT sufficient evidence"* — inverted. That note guards against trusting a
|
||||
**negative** result from a synthetic test. This one guards against trusting a **positive**
|
||||
sighting from an uncontrolled session. Both reduce to: **hold the system still, or do not draw
|
||||
causal conclusions from it.**
|
||||
|
||||
Applies equally to the "coherent to 10k" observation below — same n, same conditions, opposite
|
||||
direction. Neither observation is worth more than the other.
|
||||
|
||||
## ⚠️ CONFOUNDED — and the "before" state is itself unreliable
|
||||
|
||||
sec now runs DFlash2 on a newer build and the operator reports **coherent to 10k tokens with
|
||||
adversarial nonsense prompts**. ⚠ Treat this the same way as the 2k sighting it is being compared
|
||||
against: **n=1, uncontrolled session, not evidence.** The comparison is weak on *both* ends.
|
||||
|
||||
**Two variables changed at once:**
|
||||
|
||||
1. **Engine**: `311b3513` → `e9d1398d`, **+259 commits, `behind_by=0`** (a strict superset),
|
||||
including GDN spec-decode fix **#53077** (merged 2026-08-20) that production is **172 commits
|
||||
behind**.
|
||||
2. **Drafter**: frozen MTP head → DFlash2 reading live hidden states.
|
||||
|
||||
**Isolating it = run MTP k=3 on the same new build.** Not yet done.
|
||||
|
||||
**#51113 is present in BOTH builds** (verified by ancestry, `behind_by=0` each) — so the
|
||||
"proper upstream fix" our compose comment credits is **necessary but insufficient**; sec ran it
|
||||
and still degenerated. Related open upstream: **#53180** (quantized Qwen3.8-27B hybrid GDN + MTP
|
||||
producing *silent* degenerate output, no fix), **#41884** (DFlash + prefix caching on hybrid,
|
||||
IndexError, workaround is disabling one).
|
||||
|
||||
## ❌ WRONG TURNS — do not repeat
|
||||
|
||||
- **Version strings are not lineage.** The DFlash2 build reports `0.26.1rc1.dev1048` and our
|
||||
production nightly `0.27.2rc1.dev150`, which *looks* like a regression. It is a setuptools_scm
|
||||
tag-reachability artifact. **Use the GitHub compare API and check `behind_by`.**
|
||||
- **Docker Hub push timestamps lie about source freshness.** `nightly-ba07e4a4` was *pushed*
|
||||
06:12 UTC, comfortably after the 05:27 merge — but *cut* from a 03:46 commit that predates it.
|
||||
**Grep the image for the symbols you need.** Believing the timestamp would have cost an RP-seat
|
||||
outage to serve a model the engine could not instantiate.
|
||||
- **`--max-num-batched-tokens` was not the image truncation.** Raising it 16,384 → 32,768 on that
|
||||
theory changed nothing and cost ~3 GiB of peak activation, which came straight out of the KV
|
||||
pool. The cap was the tokenizer (§3.14 of the playbook).
|
||||
- **"1M needs YaRN, absent from config" is FALSE for the sec quant.** It is fully present:
|
||||
`rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`,
|
||||
`max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit.
|
||||
|
||||
## Live state — PROMOTED to the compose stack 2026-08-22
|
||||
|
||||
**Operator-approved after real-use testing** ("performing very well"). The experimental
|
||||
standalone container is gone; `stacks/mog-sec/` is canonical and `restart: unless-stopped` means
|
||||
it survives reboots. Cutover verified: **KV pool 526,617 / 1.10x — identical to the container it
|
||||
replaced**, restarts 0, both gateway aliases serving, DFlash2 confirmed drafting at k=7
|
||||
(231 draft tokens over 33 drafts), vision working.
|
||||
|
||||
⚠ **One variable was deliberately REMOVED, not carried over.** The old stack hardcoded
|
||||
`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`; the validated DFlash2 container never set it,
|
||||
and playbook §3.10 records expandable_segments corrupting retained tensors elsewhere. The compose
|
||||
now defaults it EMPTY (`MOG_ALLOC_CONF`). Promoting it as-was would have shipped a variable the
|
||||
tested configuration did not have.
|
||||
|
||||
**Compose is now parameterised for the shapes that differ:** `MOG_SPEC_CONFIG` carries the whole
|
||||
speculative JSON (dflash needs `"model": "/drafter"`, MTP must not have one — a method+tokens
|
||||
template cannot express both), plus `MOG_MM_PROCESSOR_KWARGS`, `MOG_DRAFT_MODEL`,
|
||||
`MOG_MAX_NUM_BATCHED_TOKENS`, `MOG_ALLOC_CONF`.
|
||||
|
||||
**ROLLBACK:** `.env.bak-pre-dflash2-20260822` and `compose.yaml.bak-pre-dflash2-20260822` on the
|
||||
host; or one line — `MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` plus
|
||||
the old `MOG_IMAGE`.
|
||||
|
||||
| | production sec | current |
|
||||
|---|---|---|
|
||||
| image | `nightly-311b3513` | `nightly-e9d1398d` |
|
||||
| speculation | MTP k=3 | **DFlash2 k=7**, drafter `/tank/aimodels/qwen38-27b-dflash2-drafter` |
|
||||
| max-model-len | 262,144 | **480,000** |
|
||||
| KV pool | 418,218 (1.60×) | **526,617 (1.10×)** |
|
||||
| images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) |
|
||||
|
||||
⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up.
|
||||
0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB
|
||||
against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1.
|
||||
|
||||
Canonical config: `stacks/mog-sec/{compose.yaml,.env.example}` in this repo.
|
||||
@@ -1,34 +0,0 @@
|
||||
# [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600
|
||||
|
||||
Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live
|
||||
bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**.
|
||||
Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with
|
||||
four interactive accounts, verified as real exposure by reading one as `nobody`.
|
||||
|
||||
Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest,
|
||||
searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other
|
||||
stacks already used 0600, so this converged on the existing house pattern rather
|
||||
than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable
|
||||
by `nobody`.
|
||||
|
||||
Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable.
|
||||
|
||||
## The check that matters
|
||||
|
||||
Every run asserts `docker compose config` still renders **as the deploy user**
|
||||
(`lkraven`), not as root. Checking the mode proves the bits changed; only rendering
|
||||
as the deploy user proves the next deploy can still resolve its variables.
|
||||
|
||||
## Two gotchas recorded in the playbook
|
||||
|
||||
- **vaultwarden looked like it bind-mounted its `.env`** — which would mean the
|
||||
*container's* UID reads it and 0600 could break the password vault. It does not:
|
||||
that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list
|
||||
item without checking its parent key. The playbook now **refuses** any stack that
|
||||
genuinely bind-mounts its `.env`, since that case is read by the container UID.
|
||||
- **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to
|
||||
`lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev`
|
||||
was repointed to infra-ops at some point and `ana-docker` was not. Run elway against
|
||||
the infra-ops target on this host.
|
||||
|
||||
Commit `a896c0a`.
|
||||
@@ -1,663 +0,0 @@
|
||||
# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
|
||||
|
||||
> **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at
|
||||
> the bottom before acting on anything here.** The headline is wrong (the
|
||||
> relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the
|
||||
> aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the
|
||||
> original stopped measuring at 4), and the proposed remedy is **impossible**:
|
||||
> UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The
|
||||
> per-stream observation and the parallelise-your-transfers mitigation are the
|
||||
> parts that survive.
|
||||
|
||||
The operator noticed site-to-site transfers were slow for a datacenter fiber
|
||||
handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
|
||||
circuit is fine.
|
||||
|
||||
## Measured
|
||||
|
||||
```
|
||||
ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering
|
||||
ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s
|
||||
FortiGate's own recorded peak 554 Mbit/s
|
||||
ANA <-> NH3, single stream 227 Mbit/s
|
||||
ANA <-> ESH, single stream 249-265 Mbit/s
|
||||
ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream
|
||||
```
|
||||
|
||||
Method: stdlib TCP probe (no ssh, no crypto, no compression) between site
|
||||
endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner
|
||||
Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.
|
||||
|
||||
## What it is not
|
||||
|
||||
- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw
|
||||
(`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard
|
||||
on ana-wg is remote-access only and is not in this path. Traceroute confirms:
|
||||
both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast
|
||||
ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it.
|
||||
- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all
|
||||
8 cores** during the tests, and both live tunnels report `npu_flag=03` with
|
||||
`dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded.
|
||||
- **Not a 250 Mbit/s cap.** That was the first number and it is misleading —
|
||||
single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
|
||||
- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions.
|
||||
|
||||
## Most likely cause
|
||||
|
||||
Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14,
|
||||
IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
|
||||
faster on **AES-GCM**, which combines encryption and authentication in one
|
||||
pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
|
||||
not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
|
||||
shortfall is consistent with that.
|
||||
|
||||
## Not executed
|
||||
|
||||
Changing the proposal is a **production-edge change requiring a matching
|
||||
change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while
|
||||
it renegotiates. Left for the operator. See the index entry for authorization
|
||||
state.
|
||||
|
||||
## Immediate mitigation, no config change
|
||||
|
||||
Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a
|
||||
single stream through IPsec gets 227. **Anything moving bulk data across the
|
||||
Anaheim link should parallelise** — that alone roughly doubles throughput
|
||||
today.
|
||||
|
||||
## Practical consequence already observed
|
||||
|
||||
`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
|
||||
from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and
|
||||
not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]].
|
||||
|
||||
## Access note
|
||||
|
||||
ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent
|
||||
on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
|
||||
vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
|
||||
prints live ESP session keys** — never paste its output into althing, a
|
||||
booth, or a commit.
|
||||
|
||||
---
|
||||
|
||||
## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist
|
||||
|
||||
The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done,
|
||||
and the measurements taken while trying show there is very little left to win.
|
||||
|
||||
### AES-GCM is unavailable on the far end — not a naming problem
|
||||
|
||||
The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf
|
||||
`_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the
|
||||
crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`,
|
||||
`aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128`
|
||||
— all `HTTP 400 api.err.InvalidPayload`, nothing applied.
|
||||
|
||||
**The control that makes this conclusive:** the *identical* request body with
|
||||
`ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the
|
||||
enum rejecting the value, not a malformed body. Corroborating: **zero
|
||||
case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.**
|
||||
|
||||
Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`,
|
||||
`chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on
|
||||
UniFi far ends, so this blocks the ESH tunnel too.
|
||||
|
||||
The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now
|
||||
reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is
|
||||
functionally identical while the peer only offers CBC, and reverting it would
|
||||
cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched;
|
||||
IKE protects the control channel only and has no bearing on data throughput.
|
||||
|
||||
### The numbers that retire this as a problem
|
||||
|
||||
Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on
|
||||
the same UDM and the same uplink, with the same stdlib TCP probe:
|
||||
|
||||
| streams | IPsec NH3→ANA | WireGuard NH3→ESH |
|
||||
|---|---|---|
|
||||
| 1 | 245 Mbit/s | 557 Mbit/s |
|
||||
| 4 | 471 Mbit/s | 767 Mbit/s |
|
||||
| 8 | **692 Mbit/s** | **795 Mbit/s** |
|
||||
|
||||
**NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) —
|
||||
that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel.
|
||||
So the tunnel does **~69% of the achievable uplink** at 8 streams, and the
|
||||
IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**.
|
||||
|
||||
Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since
|
||||
FortiOS has no WireGuard) would chase that last 15%. Not worth it.
|
||||
|
||||
### What the constraint actually is
|
||||
|
||||
A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both
|
||||
endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03`
|
||||
(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The
|
||||
shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from
|
||||
a higher floor (557 → 795 is only 1.43× scaling).
|
||||
|
||||
### Actionable consequence
|
||||
|
||||
Anything moving bulk data across this link should **parallelise** — 245 → 692
|
||||
Mbit/s, a 2.8× win with no config change. For single-stream workloads that
|
||||
cannot be parallelised at the application layer, **NFS `nconnect=N` is the
|
||||
lever**: it opens N TCP connections per mount, converting a single-stream
|
||||
workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at
|
||||
24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live
|
||||
example — remounting with `nconnect=8` is the obvious test.
|
||||
|
||||
### Foot-gun recorded
|
||||
|
||||
Probing the enum by PUTting candidate values **applies the accepted ones**. A
|
||||
probe loop here timed out with `3des` briefly live on the NH3 side, which the
|
||||
FortiGate would not accept — a short tunnel outage until `aes256` was restored
|
||||
(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi
|
||||
config enum this way, restore the known-good value after **every** 200, not at
|
||||
the end of the loop. Post-change verification: the UDM object was diffed
|
||||
field-by-field against its pre-change snapshot and is **byte-identical**.
|
||||
|
||||
---
|
||||
|
||||
## FOLLOW-UP (2026-08-23): what the per-stream limit actually is
|
||||
|
||||
The correction above called the constraint "per-SA/per-flow serialisation".
|
||||
That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of
|
||||
~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning
|
||||
problem, not loss, not window size.
|
||||
|
||||
### The evidence: pin the send buffer and sweep it
|
||||
|
||||
Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight:
|
||||
|
||||
| in-flight cap | throughput | RTT in flight | minRTT | retrans |
|
||||
|---|---|---|---|---|
|
||||
| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 |
|
||||
| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 |
|
||||
| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 |
|
||||
| ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 |
|
||||
|
||||
**Throughput is flat across a 13× range of in-flight data while RTT scales with
|
||||
it.** That is the signature of a fixed service rate with a standing queue: the
|
||||
window controls only how much queue you build, never how fast you go. Had this
|
||||
been window-limited, throughput would have risen with the buffer. Had it been
|
||||
congestion, there would be retransmits — there are essentially none
|
||||
(`retrans:0`, 0% ping loss).
|
||||
|
||||
So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all
|
||||
**red herrings here**. Do not go there.
|
||||
|
||||
### Bufferbloat: one bulk stream wrecks latency for everything else
|
||||
|
||||
Measured on the same tunnel, ping to ana-docker:
|
||||
|
||||
- idle: **6.9 ms** avg
|
||||
- during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss
|
||||
|
||||
**15× latency inflation from one transfer.** This is the operationally
|
||||
important finding — any interactive traffic sharing the Anaheim link (ssh,
|
||||
RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it
|
||||
takes only one stream to do it. Parallelising transfers makes throughput
|
||||
better and this *worse*. If it starts biting, the fix is an AQM/shaper on the
|
||||
tunnel (or rate-limiting bulk jobs), not more buffer.
|
||||
|
||||
### Where the cap lives — strong inference, not proof
|
||||
|
||||
Three paths, and the FortiGate is the only variable:
|
||||
|
||||
| path | single-stream |
|
||||
|---|---|
|
||||
| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s |
|
||||
| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s |
|
||||
| NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s |
|
||||
|
||||
Present in both slow paths, absent from the fast one. Aggregate over the same
|
||||
SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole —
|
||||
many flows spread out fine, one flow does not.
|
||||
|
||||
The mechanism that fits is **FortiGate NPU IPsec offload being per-session**:
|
||||
each firewall session is bound to one crypto engine, so a single TCP flow is
|
||||
capped at one engine's rate while many sessions spread across engines. **This
|
||||
is inference from the throughput shape, not something confirmed on the box** —
|
||||
`diagnose sys session list` was not captured for a TCP flow (the filter caught
|
||||
only traceroute UDP probes). A single-stream control through ana-gw *without*
|
||||
IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is
|
||||
window-limited and does not discriminate. **If this matters, the clean test is
|
||||
a non-IPsec single stream between two Anaheim VLANs at low RTT.**
|
||||
|
||||
**Relevant to the FortiGate cutover decision:** if the per-flow cap is the
|
||||
FortiGate's IPsec path, replacing the box plausibly lifts single-stream
|
||||
throughput toward the WireGuard figure. That is a point in favour of the
|
||||
cutover, and it is cheap to verify afterwards by re-running the sweep.
|
||||
|
||||
---
|
||||
|
||||
## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec
|
||||
|
||||
Operator asked directly whether the 80F "can't handle the traffic". It can.
|
||||
Two new measurements settle the shape of this, and correct an overstatement in
|
||||
FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded —
|
||||
every slow path was *both* IPsec *and* FortiGate, so protocol and box could not
|
||||
be separated by that argument).
|
||||
|
||||
### The 80F routes a single flow at line rate when IPsec is not involved
|
||||
|
||||
`ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 =
|
||||
`10.250.50.1`), 0.36 ms RTT, no tunnel:
|
||||
|
||||
| streams | throughput |
|
||||
|---|---|
|
||||
| 1 | **940.2 Mbit/s** |
|
||||
| 8 | 939.3 Mbit/s |
|
||||
|
||||
Single stream saturates 1 GbE. So the box does **not** cap single sessions in
|
||||
general, and there is no per-session ceiling in its plain forwarding path. The
|
||||
~250 Mbit/s per-flow cap is **specific to the IPsec datapath**.
|
||||
|
||||
### Both IPsec tunnels converge on the same numbers despite different far ends
|
||||
|
||||
Measured today with the same probe:
|
||||
|
||||
| tunnel | far-end gateway | RTT | 1 stream | 8 streams |
|
||||
|---|---|---|---|---|
|
||||
| NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s |
|
||||
| ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** |
|
||||
|
||||
Different gateway hardware, different sites, different uplinks, and RTT
|
||||
differing by 1.7× — yet single-stream differs by only 9%. **If this were
|
||||
window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is
|
||||
independent confirmation of a rate cap rather than a BDP effect.
|
||||
|
||||
### Capacity summary — the box has headroom it will not give one flow
|
||||
|
||||
- plain routing, 1 stream: **940 Mbit/s** (line rate)
|
||||
- plain routing to internet, 8 streams: **2,153 Mbit/s**
|
||||
- IPsec, 8 streams: **692–715 Mbit/s**
|
||||
- IPsec, 1 stream: **245–268 Mbit/s**
|
||||
- CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`)
|
||||
|
||||
Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath
|
||||
distributes work **by inner session** — consistent with IPsec offload binding a
|
||||
session to one crypto engine.
|
||||
|
||||
### What is still NOT separated
|
||||
|
||||
Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec
|
||||
implementation**. Both tunnels have a UDM at the far end, and both UDMs run the
|
||||
same UniFi firmware, so identical caps are explainable either way. The Pro Max
|
||||
being only 9% faster than the Pro SE argues against the UniFi side (a beefier
|
||||
CPU should show more), but that is suggestive, not conclusive.
|
||||
|
||||
**The test that closes it:** an IPsec tunnel whose endpoints do not include the
|
||||
80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured
|
||||
single-stream. If it also caps ~250, the FortiGate is exonerated and replacing
|
||||
it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM
|
||||
WireGuard achieves, the 80F is the limiter. **Bears directly on the pending
|
||||
FortiGate cutover** — worth running before that decision, not after.
|
||||
|
||||
---
|
||||
|
||||
## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream
|
||||
|
||||
Operator asked for a WireGuard test from `ana-wg` to NH3 over the public
|
||||
internet. It is the test that separates the *path* from the *crypto*, and the
|
||||
answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the
|
||||
transport is not worth it" — that conclusion compared 8-stream numbers and was
|
||||
wrong for single-stream workloads.**
|
||||
|
||||
### Setup (fully torn down afterwards)
|
||||
|
||||
`ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an
|
||||
internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by
|
||||
FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337,
|
||||
policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A
|
||||
temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on
|
||||
wg0, measured, then removed — ana-wg is back to its original 3 peers and the
|
||||
keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in
|
||||
place** (benign, and wanted if this becomes permanent).
|
||||
|
||||
In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM
|
||||
only NAT/forward UDP, and Linux does WireGuard at both ends.
|
||||
|
||||
### The full comparison
|
||||
|
||||
| path | crypto performed by | 1 stream | 8 streams |
|
||||
|---|---|---|---|
|
||||
| IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s |
|
||||
| IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s |
|
||||
| **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s |
|
||||
| WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s |
|
||||
| plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s |
|
||||
|
||||
**One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is
|
||||
no per-flow penalty at all, and a single flow already saturates the path. So
|
||||
the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3
|
||||
uplink and not the physical path — all of which sustain 767 on one flow.
|
||||
|
||||
Per-flow penalty ranks by implementation:
|
||||
|
||||
- **Linux WireGuard — none** (767 → 763, flat)
|
||||
- **UDM WireGuard — mild**, ~1.4× (557 → 795)
|
||||
- **IPsec on this pair — severe**, ~2.8× (245 → 692)
|
||||
|
||||
### Latency under load — the same story
|
||||
|
||||
| path | idle | during ONE bulk stream |
|
||||
|---|---|---|
|
||||
| IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max |
|
||||
| WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max |
|
||||
|
||||
WireGuard carries **3.1× the single-stream throughput with 8× less latency
|
||||
inflation** on the same wire.
|
||||
|
||||
### Attribution — still not fully separated, and it no longer matters much
|
||||
|
||||
Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still
|
||||
does not isolate which one imposes the 2.8× penalty. Closing that would need
|
||||
Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical
|
||||
decision no longer depends on the answer**, because the fix is the same either
|
||||
way and it is already demonstrated.
|
||||
|
||||
### Recommendation (supersedes FOLLOW-UP 1)
|
||||
|
||||
A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is
|
||||
worth real consideration: 3.1× single-stream, flat scaling, far better latency
|
||||
under load, and it reuses infrastructure that already exists and is already
|
||||
internet-reachable. It is also the architecture already proven for NH3↔ESH.
|
||||
Open questions before committing: routing/failover if ana-wg (an LXC) is down,
|
||||
whether it replaces or parallels the IPsec tunnel, and firewall policy for the
|
||||
new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it
|
||||
has headroom.
|
||||
|
||||
**AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the
|
||||
same lever (turn one flow into many) and brokkr-smithy-dev has given standing
|
||||
approval to apply it once the FortiGate work settles, with no need to ask again.
|
||||
|
||||
---
|
||||
|
||||
## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated.
|
||||
|
||||
Operator's theory — the UDM does IPsec in software with no crypto offload, so
|
||||
the cost of the cipher itself is the limit — is **correct**, and it is now
|
||||
demonstrated rather than inferred. He also correctly pointed out that
|
||||
UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything
|
||||
about UniFi's IPsec performance. It didn't, and I had leaned on it.
|
||||
|
||||
### The controlled experiment: vary cipher cost, hold everything else
|
||||
|
||||
AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding
|
||||
constraint, throughput must rise when the cipher gets cheaper. If the limit
|
||||
were the FortiGate's NPU, it would not move at all — hardware crypto is not
|
||||
cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each:
|
||||
|
||||
| condition | ESP cipher | single-stream | UDM CPU |
|
||||
|---|---|---|---|
|
||||
| A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% |
|
||||
| B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% |
|
||||
| A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — |
|
||||
|
||||
**~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of
|
||||
CPU work, more payload through it. That is the signature of CPU-bound software
|
||||
crypto, and it rules out the FortiGate's NPU as the limiter.
|
||||
|
||||
### Correcting two of my own earlier claims
|
||||
|
||||
1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling
|
||||
artifact.** UniFi's `system-stats.cpu` refreshes on the device report
|
||||
interval; 4-second sample windows were reading stale values. Under a
|
||||
sustained 60 s single-stream load it reads **35.4%**, with load1 rising
|
||||
0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated
|
||||
on crypto plus overhead. **Always drive load for ≥60 s before trusting a
|
||||
UniFi CPU figure.**
|
||||
2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely
|
||||
unproven. It predicts no change from a cipher swap; a 20% change was measured.
|
||||
|
||||
### Why the numbers all line up now
|
||||
|
||||
- **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC.
|
||||
- **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure
|
||||
on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link)
|
||||
and is *not* cipher-sensitive, consistent with it being bounded by the path/
|
||||
uplink rather than crypto once several cores are engaged.
|
||||
- **AES-CBC is the specific villain: it is serial.** Each block depends on the
|
||||
previous one, so the ARM AES instructions cannot pipeline across blocks. GCM
|
||||
(CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the
|
||||
same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec.
|
||||
- **This retroactively vindicates the GCM cutover as the right idea aimed at the
|
||||
right box** — GCM would have removed the serial dependency on the constrained
|
||||
end. UniFi simply does not offer it, which is what made it impossible.
|
||||
|
||||
### Options this opens
|
||||
|
||||
- **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API
|
||||
call per end. 128-bit is not the weak link here (SHA1 integrity is more
|
||||
dated, and unchanged either way). Operator's call — **not adopted**, restored
|
||||
to aes256.
|
||||
- **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it
|
||||
sidesteps the UDM's IPsec datapath entirely. Still the biggest win available.
|
||||
- Replacing the FortiGate **will not help this** — it was never the constraint.
|
||||
Worth knowing before the cutover.
|
||||
|
||||
### State left behind
|
||||
|
||||
UDM network object verified **byte-identical** to its pre-test snapshot
|
||||
(aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is
|
||||
left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer
|
||||
offers only aes256 so the extra entries are inert, but **narrowing it back to
|
||||
`aes256-sha1` is one line** if the looser list is unwanted.
|
||||
|
||||
---
|
||||
|
||||
## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
|
||||
|
||||
Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must
|
||||
terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on
|
||||
pfi-pve, 10.250.50.252) already is that box.
|
||||
|
||||
**This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with
|
||||
traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other
|
||||
Anaheim hosts, which was never measured. Now it is:
|
||||
|
||||
| topology | 1 stream | 8 streams |
|
||||
|---|---|---|
|
||||
| IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s |
|
||||
| WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s |
|
||||
| **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** |
|
||||
|
||||
**Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM
|
||||
architecture delivers the full 3.1× single-stream for real transit traffic, not
|
||||
just for traffic landing on the tunnel box.
|
||||
|
||||
ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle),
|
||||
so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's*
|
||||
load, not the container's — do not read it as ana-wg's own. For contrast the
|
||||
UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom.
|
||||
|
||||
### Design consequences of terminating downstream — the parts that need decisions
|
||||
|
||||
1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8
|
||||
figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252`
|
||||
route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default
|
||||
gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a
|
||||
LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT
|
||||
measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated
|
||||
transit VLAN for ana-wg, or accept the hairpin.
|
||||
2. **New single point of failure.** Today site-to-site dies only when the edge
|
||||
dies, which is total anyway. A downstream terminator fails independently.
|
||||
Mitigation: keep the IPsec tunnel configured as a higher-metric fallback
|
||||
route so it takes over when ana-wg is down.
|
||||
3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the
|
||||
hypervisor shared with the rest of the Anaheim VMs.
|
||||
4. **The NH3 end needs a terminator too**, and there are two shapes:
|
||||
- **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what
|
||||
was measured: **764 Mbit/s**.
|
||||
- **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on
|
||||
UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and
|
||||
UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on
|
||||
Site Magic), so expect ~557 rather than 764. Still 2.3× today.
|
||||
|
||||
### Standing recommendation
|
||||
|
||||
Worth doing, but it is **a project, not a config tweak** — routing, failover and
|
||||
policy all need deciding. The cheap wins remain available meanwhile and are
|
||||
independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending
|
||||
the FortiGate work settling) and AES-128 for ~20%.
|
||||
|
||||
---
|
||||
|
||||
## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed
|
||||
|
||||
Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break,
|
||||
then close the FortiGate's WAN and SSH admin surfaces. All done and verified.
|
||||
|
||||
**Context that retires the WireGuard-in-a-VM design work:** the FortiGate is
|
||||
being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the
|
||||
edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is
|
||||
therefore moot — do not scope it. This also **un-parks the OPNsense migration**,
|
||||
which auto-memory recorded as PARKED pending "hardware acquisition"; the R420
|
||||
is that trigger.
|
||||
|
||||
### What changed
|
||||
|
||||
Make-before-break on the FortiGate first, so neither tunnel dropped waiting on
|
||||
a far end:
|
||||
|
||||
| phase2 | proposal now |
|
||||
|---|---|
|
||||
| `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` |
|
||||
| `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` |
|
||||
|
||||
Then each UDM flipped to `ipsec_esp_encryption: aes128`:
|
||||
|
||||
| tunnel | UDM object | before | after |
|
||||
|---|---|---|---|
|
||||
| NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** |
|
||||
| ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** |
|
||||
|
||||
Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B —
|
||||
the difference is live-link variance, not a different result. Both UDM objects
|
||||
were diffed field-by-field against pre-change snapshots: **the only field that
|
||||
moved on either is `ipsec_esp_encryption`.**
|
||||
|
||||
The FortiGate proposal lists were deliberately **left permissive** (still
|
||||
accepting aes256). The peers offer only aes128 so the extra entries are inert,
|
||||
and keeping them means a UDM reverting does not strand the tunnel. Narrowing to
|
||||
`aes128-sha1` alone is a one-liner if the looser list is unwanted.
|
||||
|
||||
### Admin surfaces closed
|
||||
|
||||
`wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost
|
||||
→ **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed
|
||||
from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works.
|
||||
**Sequencing that matters: the close was executed over the TUNNEL path, not over
|
||||
WAN** — removing `ssh` from allowaccess while connected over WAN kills the
|
||||
session mid-command.
|
||||
|
||||
**Consequence to hold in mind: ana-gw now has no out-of-band management path.**
|
||||
If both tunnels drop it is console-only until someone is on site.
|
||||
|
||||
### Gotcha: the two UDM vault items have DIFFERENT shapes
|
||||
|
||||
- `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key.
|
||||
- `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a
|
||||
`key:` line. `secret get` piped straight into a header yields a 1396-byte
|
||||
value and the UDM answers **`400 Bad Request` from nginx**. Extract with
|
||||
`grep '^key:' | awk '{print $2}'`.
|
||||
|
||||
**The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded
|
||||
it as read-verified only): a control PUT of the unchanged object returned
|
||||
`rc:ok`, then the real change did too. That key has a full read+write admin role.
|
||||
|
||||
---
|
||||
|
||||
## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener
|
||||
|
||||
The claim in the previous section that `.42:80` was an **ISP transparent proxy**
|
||||
was **WRONG**, and so was the earlier warning that ACME renewal would fail with
|
||||
port 80 absent from `allowaccess`. Operator pushed back asking where the port-80
|
||||
map terminated. It terminates **on the FortiGate itself**.
|
||||
|
||||
**What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system
|
||||
acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to
|
||||
answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1
|
||||
allowaccess` reads `ping` only and the port is still open. Every non-challenge
|
||||
request returns a fixed 403 whose body is literally:
|
||||
|
||||
```
|
||||
<!DOCTYPE html><html><head><title>ACME Access Only</title></head><body>ACME Access Only</body></html>
|
||||
```
|
||||
|
||||
**Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` —
|
||||
`Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252).
|
||||
~~Worth noting separately: four VIPs are all-port static NAT~~ — **that claim was
|
||||
WRONG, see the correction below.** All fourteen VIPs are scoped.
|
||||
|
||||
### The methodology error that produced the wrong answer — worth not repeating
|
||||
|
||||
The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
|
||||
matches only inbound packets**, so outbound SYN-ACKs were excluded *by
|
||||
construction*; concluding "the box sends no SYN-ACK" from that capture was
|
||||
unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it
|
||||
immediately shows `wan1 out 38.120.12.42.80 -> <scanner>: syn ack`.
|
||||
|
||||
**Rule: when testing whether a box *answers*, the sniffer filter must be
|
||||
bidirectional. `dst host` silently answers a different question.**
|
||||
|
||||
### Consequences
|
||||
|
||||
- **ACME renewal will work** with `allowaccess ping`. The earlier "add `http`
|
||||
back or the cert expires" warning is retracted — FortiOS opens the challenge
|
||||
port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27.
|
||||
- **It is not an admin surface** — static 403, no auth, no GUI.
|
||||
- Its practical value is now low: WAN admin is closed, so the cert only serves
|
||||
the internal GUI at 10.250.0.1, where the name would not match anyway. Killing
|
||||
it (`config system acme` → unset interface) would close the last WAN listener
|
||||
at the cost of cert renewal. Operator's call; **not done**.
|
||||
|
||||
---
|
||||
|
||||
## CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE
|
||||
|
||||
### ACME disabled — the WAN IP now exposes nothing
|
||||
|
||||
`config system acme / unset interface` (the account object is left in place;
|
||||
with no interface bound there is no listener). Verified:
|
||||
|
||||
- **External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.**
|
||||
- Internal GUI at 10.250.0.1 still answers **200**, SSH still works.
|
||||
- `admin-server-cert` is still `ana-fw.pfi` — the existing cert is untouched and
|
||||
serves the internal GUI until **2026-10-27**; it simply will not auto-renew.
|
||||
|
||||
Reverse with `config system acme / set interface "wan1"`.
|
||||
|
||||
### RETRACTION: the four VIPs are NOT all-port
|
||||
|
||||
A previous section claimed `Rustdesk`, `https-to-tacticalrmm`, `web-to-webhost`
|
||||
and `web-to-sfcontainer` were unrestricted all-port static NATs. **They are not.**
|
||||
A FortiOS VIP can be scoped **two different ways** and the parser used only
|
||||
checked one:
|
||||
|
||||
1. `set portforward enable` + `set extport <n>` — a single mapped port, **or**
|
||||
2. `set service "<svc>"` on the VIP object — constrains the VIP to that service.
|
||||
|
||||
All four use form 2. The custom services are narrow: `Rustdesk` = TCP
|
||||
21115–21119 + UDP 21116 (the standard RustDesk range), `ssh-mapped-2223` = TCP
|
||||
2223 only. **Every one of the 14 VIPs is scoped; none is unrestricted.**
|
||||
|
||||
**Lesson: absence of `portforward` does NOT mean all-port on a FortiOS VIP —
|
||||
check `service` too.** Better still, do what settled it here: scan from outside
|
||||
rather than reading config.
|
||||
|
||||
### Ground-truth public exposure (external TCP scan, post-change)
|
||||
|
||||
| IP | open | maps to |
|
||||
|---|---|---|
|
||||
| 38.120.12.41 | *nothing* | — |
|
||||
| **38.120.12.42** | ***nothing*** | the FortiGate itself — fully closed |
|
||||
| 38.120.12.43 | 80, 443 | sf-ana-container 10.250.150.100 (SureFire tenant) |
|
||||
| 38.120.12.44 | 22, 80, 443, 8025, 21115–21119 | gitea (→222), traefik, mailrise, RustDesk |
|
||||
| 38.120.12.45 | 80, 443, 2223 | pfi-ana-webhost 10.250.50.52 (2223→22) |
|
||||
| 38.120.12.46 | 443 | pfi-tacticalrmm 10.250.50.57 |
|
||||
|
||||
Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880
|
||||
(Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during
|
||||
the OPNsense translation but not exposure.
|
||||
@@ -1,63 +0,0 @@
|
||||
# [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing
|
||||
|
||||
`hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites
|
||||
(Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for
|
||||
uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach
|
||||
`http://hrafn:8080`. Canonical at `stacks/hrafn/`.
|
||||
|
||||
## Intake found a live credential exposure
|
||||
|
||||
`/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token**
|
||||
— verified as real exposure by reading it as `nobody` on a box with four
|
||||
interactive accounts. Tightened to 0600. That triggered the wider sweep (see
|
||||
[[2026-08-23-ana-docker-env-perms-sweep]]).
|
||||
|
||||
## The CI defect — the one worth remembering
|
||||
|
||||
I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync,
|
||||
tagging the image with the commit SHA for provenance. nevermore-claude later found
|
||||
v1.0.0 deploying "green" while the host still served 0.1.0.
|
||||
|
||||
**Root cause was mine and nastier than either hypothesis.** The staging dir was
|
||||
`$compose_dir/.stage` — **inside** the rsync target. So
|
||||
`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the
|
||||
destination (absent from the source listing) **during** the transfer, destroying
|
||||
its own source mid-copy. Reproduced exactly:
|
||||
|
||||
```
|
||||
before: app.py="OLD" leftover.txt .stage/app.py="NEW"
|
||||
after: app.py="OLD" leftover.txt GONE, .stage GONE
|
||||
```
|
||||
|
||||
Deletion succeeded, the copy silently did not, rsync exited 0. So the directory
|
||||
*looked* converged while host source stayed frozen at the first manual rsync —
|
||||
and because the build's `COPY` inputs never changed, Docker full-cache-hit and
|
||||
every SHA tag aliased one image. **The provenance the tagging existed to provide
|
||||
was false for the pipeline's entire life.**
|
||||
|
||||
**The real failure is the verification.** The verify steps asserted the marker,
|
||||
container health, and a 200 from `/readyz` — all of which pass against a
|
||||
completely frozen host. None measured *content*. A deploy that reports success
|
||||
without asserting the bytes changed is verifying an **uptime**, not a deploy.
|
||||
|
||||
## Fixes
|
||||
|
||||
- stage at `/tmp/hrafn-deploy-stage`, outside the target
|
||||
- CI computes `context_sha256` over the shipped file list; the playbook recomputes
|
||||
it **on the host after the converge** and fails on mismatch
|
||||
- compare the running container's `src/**/*.py` against the host's, so a SHA tag
|
||||
cannot name layers the image lacks
|
||||
- **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the
|
||||
image and `__pycache__` appears at runtime, so a naive `find src -type f` compare
|
||||
false-fails on every healthy deploy. Verified against a known-good container
|
||||
before shipping (12 host files, 18 in container, 0 content differences).
|
||||
- declined `--no-cache`: a cache hit is *correct* when the context is genuinely
|
||||
unchanged; assert the property rather than brute-force it.
|
||||
|
||||
## Access
|
||||
|
||||
Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the
|
||||
pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is
|
||||
canonical; `stacks/hrafn/ci/` is a verified mirror.
|
||||
|
||||
Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`.
|
||||
@@ -1,81 +0,0 @@
|
||||
# [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat
|
||||
|
||||
## Why selene went
|
||||
|
||||
Benchmarked against `gen` on selene's own job — 24 designed judge items with
|
||||
checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a
|
||||
neutral JSON prompt and Selene's **native Atla template** (288 calls, free local).
|
||||
|
||||
```
|
||||
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
|
||||
native Atla selene 21/24 (88%) gen 22/24 (92%)
|
||||
```
|
||||
|
||||
gen won on both templates and **selene's best sat below gen's worst**. Selene was
|
||||
given its own fine-tuned template as a fairness check before any recommendation;
|
||||
it gained one point, not three.
|
||||
|
||||
**Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a
|
||||
winner on every equivalent pair. For eval work that is the case that matters.
|
||||
|
||||
brokkr-smithy-dev independently corroborated from the other end with a **null
|
||||
control** (an excerpt compared against ITSELF, where tie is definitional):
|
||||
`chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth
|
||||
recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96
|
||||
understated it: on a *ranking* task selene was a coin flip. Absolute scoring on
|
||||
designed items is an easier task than ranking real text — the harness is a
|
||||
**screen, not a verdict**, and its README says so.
|
||||
|
||||
Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB).
|
||||
|
||||
## The naming rule, restated the hard way
|
||||
|
||||
I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**:
|
||||
|
||||
> never repoint a named model at a different model's endpoint — that is
|
||||
> intentionally misleading
|
||||
|
||||
`chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and
|
||||
moved to gen with a deterministic judge profile copied from `image-judge`.
|
||||
`selene-1-mini-8b` is a **model** name and was removed outright — it now returns
|
||||
`HTTP 400 Invalid model name`, verified. The discriminator: *does the string
|
||||
promise a capability, or an identity?*
|
||||
|
||||
## The 7-way alias collision — the finding with the longest reach
|
||||
|
||||
```
|
||||
chat-judge classifier gen image-judge
|
||||
qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015
|
||||
```
|
||||
|
||||
Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`;
|
||||
`ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`.
|
||||
|
||||
**Cross-checking a result against another alias measures nothing when they are the
|
||||
same weights — agreement is an echo, not corroboration.** Documented at the head of
|
||||
`model_list` in the live gateway config, because it belongs where people read it.
|
||||
|
||||
This caught a real defect within hours: brokkr's R47 premium-corpus gate was about
|
||||
to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an
|
||||
independent second opinion. They pinned the backing model in the preregistration
|
||||
and dropped the second-alias idea instead.
|
||||
|
||||
## Provenance seam (brokkr's pushback, adopted)
|
||||
|
||||
The gateway returns the **alias** in the response `model` field, not the backing
|
||||
model — so a per-call guard catches a swap *during* a run and is blind to one
|
||||
*between* runs. **Role alias for routing, concrete model for provenance.**
|
||||
`GET :4000/model/info` with the shared key already exposes backing model +
|
||||
api_base; resolve at run start AND end and void on mismatch.
|
||||
|
||||
## Artifacts
|
||||
|
||||
- Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default
|
||||
would silently benchmark a retired seat).
|
||||
- `stacks/selene/` keeps compose + a README explaining the retirement.
|
||||
- Technique worth stealing, from brokkr: **a control constructed so the correct
|
||||
answer is DEFINITIONAL rather than judged cannot inherit the designer's error.**
|
||||
Item vs itself; response vs its own truncation; text vs its own clauses
|
||||
permuted. Add those before adding more judged items.
|
||||
|
||||
Commits `ca3c984`, `b8a5355`.
|
||||
@@ -1,70 +0,0 @@
|
||||
# [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed
|
||||
|
||||
## b187 pre-stage (#405 phases 1+2)
|
||||
|
||||
The matrix bridge stopped embedding the engine and became an HTTP client of the
|
||||
Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the
|
||||
container env, the bridge exits by design. Demo's compose never passed it; the next
|
||||
recreate would have crash-looped. Pre-staged on demo and personal (additive, backed
|
||||
up, verified with `docker compose config`, nothing restarted).
|
||||
|
||||
**Key decision, and I got its scope wrong first.** I argued demo should stay keyless
|
||||
(no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about
|
||||
scope**: the engine preflight authenticates at boot regardless of homeserver, so demo
|
||||
booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's
|
||||
recorded authorization, vaulted, wired, three-hop hash-verified.
|
||||
|
||||
## The 69-day-stale `:latest` landmine
|
||||
|
||||
All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while
|
||||
running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built
|
||||
**2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently
|
||||
downgraded that service by 69 days. Same footgun as the 2026-06-15 outage.
|
||||
|
||||
Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering
|
||||
compose config rather than reading `.env`, containers untouched. Playbook at
|
||||
`playbooks/repin-worldtree-image.yaml`.
|
||||
|
||||
**`worldtree-pinned` was the worst case:** the instance whose entire purpose is being
|
||||
frozen was running a **dangling image with no repo tags**, kept alive only by the
|
||||
running container. One `docker rm` from garbage collection. Tagged
|
||||
`:446e5807bf43` first, then pinned.
|
||||
|
||||
The guard I wrote had two bugs the pinned case exposed: it compared the container's
|
||||
`.Config.Image` **string** (only the tag it was CREATED from — pinned was created
|
||||
from `:latest` back when that meant 446e5807), and it reported CHANGED
|
||||
unconditionally. Now compares **image IDs** and skips when already correct.
|
||||
|
||||
## Matrix homeserver ownership
|
||||
|
||||
Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never
|
||||
missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed
|
||||
instances had them at length **zero**. Someone wired four of six Matrix vars and
|
||||
stopped. Wired them into personal, three-hop verified.
|
||||
|
||||
**The trap worth remembering:** Synapse's registration pointed at
|
||||
`http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at
|
||||
512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is
|
||||
**DEMO's** bridge, and Synapse can reach both — it would have connected, delivered,
|
||||
and looked correct while routing the operator's live rooms to the demo instance.
|
||||
**Personal's bridge is :8010.** `docker port` is ground truth.
|
||||
|
||||
Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified
|
||||
`GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's
|
||||
smoke passed first try: room created, mimir accepted the invite, a real engine turn
|
||||
ran, mimir replied in persona voice. #408 closed.
|
||||
|
||||
## Open on worldtree-dev's side
|
||||
|
||||
- **#411** — personal's bridge logs `Debug sink init failed: Permission denied:
|
||||
/app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist
|
||||
their IDs, so **every restart mints a fresh pair on the live homeserver**. Room
|
||||
litter that compounds silently. Needs a which-container-writes-what check on the
|
||||
sessions volume before anyone chowns it.
|
||||
- Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not
|
||||
listed by the engine on either instance.
|
||||
- Historical Domari pairwise verdicts from the selene era are coin-flip-grade
|
||||
(see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so
|
||||
no future arc leans on them without re-judging.
|
||||
|
||||
Commits `064181a`, `bb19a96`.
|
||||
@@ -1,50 +0,0 @@
|
||||
# [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included
|
||||
|
||||
WAN admin was opened at the start of the session as a cutover contingency
|
||||
("so I don't have to drive down there"), then closed again on operator
|
||||
instruction once the AES-128 work landed. Net result: **the FortiGate's WAN
|
||||
address now exposes no TCP port at all.**
|
||||
|
||||
## Final state
|
||||
|
||||
External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two
|
||||
sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`.
|
||||
|
||||
**Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels
|
||||
drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) —
|
||||
both are recorded in auto-memory `reference_fortigate_ana_gw_access`.
|
||||
|
||||
## Port 80 was the FortiOS ACME listener, and I got it wrong first
|
||||
|
||||
`38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with
|
||||
`allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" —
|
||||
was **wrong**, and the reason is worth keeping:
|
||||
|
||||
> The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
|
||||
> matches inbound only**, so outbound SYN-ACKs were excluded *by construction*,
|
||||
> and concluding "the box sends no SYN-ACK" from that capture was unsound.
|
||||
|
||||
Re-run bidirectionally (`host … and tcp port 80`) it immediately showed
|
||||
`wan1 out 38.120.12.42.80 -> <scanner>: syn ack`. **Rule: to test whether a box
|
||||
*answers*, the filter must be bidirectional.**
|
||||
|
||||
The listener is opened by `config system acme / set interface "wan1"` and
|
||||
**bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It
|
||||
was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`,
|
||||
valid to 2026-10-27) is untouched and simply stops renewing, which is fine
|
||||
because WAN admin is closed and the box is being replaced.
|
||||
|
||||
## Retracted in the same pass: the "four all-port VIPs" alarm
|
||||
|
||||
Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS
|
||||
VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding
|
||||
on the VIP object — and only the first was checked. All 14 VIPs are scoped;
|
||||
`Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only.
|
||||
|
||||
Ground-truth external scan of all six public IPs is recorded in
|
||||
`reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443`
|
||||
(mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up
|
||||
candidates for the OPNsense translation, not exposure.
|
||||
|
||||
**Lesson, twice in one session: measure from outside instead of parsing config.**
|
||||
Both wrong answers came from a filter that answered a different question.
|
||||
@@ -1,167 +0,0 @@
|
||||
# `[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base
|
||||
|
||||
One evening, one thread with brokkr-smithy-dev, five commits: `850e0c3`,
|
||||
`27155c0`, `f509668`+`24e8826`+`1bd90ea`+`3446367`+`8d6a939`, `14ff4a3`,
|
||||
`019ccff`, `5415fd4`.
|
||||
|
||||
## 1. The seat was crash-looping, and the cause was NOT its config
|
||||
|
||||
`vllm-meromero-rp` reported up-but-unreachable, RestartCount climbing (13 by the
|
||||
time it was examined, not the 4 first reported). Startup logs looked clean all
|
||||
the way through weights, `torch.compile` and CUDA-graph capture, then:
|
||||
|
||||
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 336.00 MiB.
|
||||
GPU 0 has a total capacity of 94.97 GiB of which 195.19 MiB is free.
|
||||
|
||||
**⚠ `--gpu-memory-utilization` SIZES THE KV CACHE AND DOES NOT COVER CUDA
|
||||
CONTEXT, GRAPHS OR NON-TORCH OVERHEAD.** gen is configured at 0.43 and actually
|
||||
held 45.6 GiB. char-rp was at 0.51. The pair was committed to 0.94 of the card
|
||||
with ~0.6 GiB of real headroom — it fit on the 21st and stopped fitting on the
|
||||
24th.
|
||||
|
||||
Dropped char-rp to 0.47: ~4.8 GiB margin, KV 27.36 → 23.56 GiB, 430,825 →
|
||||
371,023 tokens against a max-model-len of 262,144. **Cost nothing usable** — the
|
||||
pool still holds 1.4x a full-length sequence; what is lost is concurrent long
|
||||
requests, not context.
|
||||
|
||||
**⚠⚠ THE MISSING HALF, found later that evening: gen's footprint GROWS WITH
|
||||
UPTIME.** Same container, same 0.43: **45.6 GiB after ~3 days up, 38.5 GiB
|
||||
freshly restarted** — ~7 GiB apart. Nothing about char-rp changed between the
|
||||
21st and the 24th; *gen crept up underneath it*. **Headroom arithmetic done
|
||||
against a long-running gen is measuring a moving number.** Measure against a
|
||||
freshly-restarted one.
|
||||
|
||||
## 2. `char-rp` and `char-rp-reasoning` are ONE seat, not two
|
||||
|
||||
Both LiteLLM routes point at `10.250.50.54:8016/v1` — `hosted_vllm/char-rp` and
|
||||
`hosted_vllm/char-rp-thinking`. brokkr had reported 30/80 and 80/80 failure
|
||||
rates as two failing services; it was one outage sampled twice. This also
|
||||
*improved* a result of theirs: their CoT on/off battery had assumed both aliases
|
||||
were the same weights under two chat templates, and the routing detail turned an
|
||||
assumption into a verified fact.
|
||||
|
||||
(`vllm-charrp-reasoning-nvfp4`, the Heretic2 NVFP4+MTP container, has been
|
||||
stopped for 12+ days and is unrelated — it is not what that alias resolves to.)
|
||||
|
||||
## 3. The seat swapped to the Gemma-4 26B-A4B MoE
|
||||
|
||||
Operator-directed straight-across replacement: same port, same
|
||||
served-model-names, so no gateway route or consumer config moved. Rationale is
|
||||
throughput under CoT — the user waits through the whole reasoning block before
|
||||
the first visible token, and the MoE measures ~114 tok/s @32K against the dense
|
||||
31B's ~40.7.
|
||||
|
||||
Serving copy is `RedHatAI/gemma-4-26B-A4B-it-NVFP4` (16 GB), chosen over the
|
||||
other `-it` quants because it is compressed-tensors (`nvfp4-pack-quantized`) —
|
||||
the same loader path the outgoing seat used. Smaller weights at the same 0.47
|
||||
budget bought **1,724,110 KV tokens against the predecessor's 371,023**.
|
||||
|
||||
`meromero-charrp` is retained stopped in `created` state, labelled
|
||||
`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**.
|
||||
|
||||
## 4. ⚠ THE STALE-CHAT-TEMPLATE TRAP IS ENDEMIC, NOT A ONE-OFF
|
||||
|
||||
Verified by hash across every third-party Gemma-4 derivative pulled:
|
||||
|
||||
| build | lines | sha256 (normalised) |
|
||||
|---|---|---|
|
||||
| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` |
|
||||
| RedHatAI NVFP4 (served) | 389 | `6a1015c47ccfcfa6` — the only match |
|
||||
| llmfan46 heretic | 365 | `0a52be69cda5ab8a` |
|
||||
| TrevorJS abliterated | 266 | `58c66fdee4afa297` |
|
||||
| jenerallee78 abliterated | 266 | `58c66fdee4afa297` |
|
||||
| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` |
|
||||
|
||||
Three independent repos carrying the *identical* stale file means it propagated
|
||||
through the ecosystem. Consequences differ by use and **both are silent**:
|
||||
serving a mismatched template renders a different prompt; **training through
|
||||
`base/chat_template.jinja` means training on a different prompt format than
|
||||
production serves** — train/serve skew, no error, presents as a tuning failure.
|
||||
|
||||
The production compose now pins the template explicitly. It is a **no-op for the
|
||||
served weights** (the A4 build ships that exact file) and permanently closes the
|
||||
class. ⚠ If `GEMMA4_MODEL` ever points at a different checkpoint, the template
|
||||
default must move with it.
|
||||
|
||||
## 5. A benchmark result was RETRACTED — below chance indicts the instrument
|
||||
|
||||
A battery appeared to show Gemma at **12% contradiction detection with CoT off
|
||||
against gen's 81%**. An A16 activation-precision control was staged to test
|
||||
whether the quant scheme owned it. Then the operator asked to see the individual
|
||||
items, and the item was **ill-posed**: it presented two mutually contradicting
|
||||
statements and asked for "*the* contradicting statement", but **contradiction is
|
||||
symmetric**. The model consistently named the absolute claim — a defensible
|
||||
reading the labelling scored wrong every time.
|
||||
|
||||
**⚠ THE TELL WAS IN PLAIN SIGHT: 12% ON A FIVE-OPTION TASK IS BELOW THE 20%
|
||||
CHANCE FLOOR.** A below-chance score indicts the instrument before it indicts
|
||||
the model, and neither side reacted to it. I spent the afternoon verifying repo
|
||||
names, config fields, template hashes and tokenizer vocabs — every layer of
|
||||
plumbing — and never asked whether the number itself was *possible*. **A
|
||||
preflight can be thorough and still be aimed in the wrong direction.**
|
||||
|
||||
Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
|
||||
points of contradiction detection" (on a sound instrument it **reverses**); all
|
||||
pre-fix T2 numbers. Recorded as a dated superseded-claims table in
|
||||
`stacks/gemma4-charrp/README.md` rather than a silent edit.
|
||||
|
||||
**What survived:** the A16 control result — activation precision is close to free
|
||||
on this battery, every other task identical across W4A4 and W4A16 builds.
|
||||
|
||||
## 6. The abliterated trainee base — measured, not assumed
|
||||
|
||||
Operator directed a low-damage abliterated instruct build. "Low damage" was
|
||||
treated as a measurable claim; the field spreads from KL 0.09 to 0.4118:
|
||||
|
||||
| build | method | KL | refusals |
|
||||
|---|---|---|---|
|
||||
| **llmfan46** (operator's pick) | Heretic v1.2.0 ARA | 0.1237 | 3/100 |
|
||||
| TrevorJS | ARA-family | 0.09 | 1/100 effective, 5/686 cross-dataset |
|
||||
| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT |
|
||||
| huihui-ai | remove-refusals | none published | none published |
|
||||
|
||||
Fleet anchor: our own work found **Heretic at KL 0.12 preserved the MTP head at
|
||||
83.7% acceptance**, so both staged builds sit inside an already-measured band.
|
||||
huihui-ai rejected — no metrics, its card calls the method "a crude,
|
||||
proof-of-concept implementation", it abliterates both thinking and non-thinking
|
||||
modes, and its parameter count runs ~738M over upstream. Operator's independent
|
||||
read matched ("huihui produces garbage").
|
||||
|
||||
**Abliteration isolated properly** (stock BF16 vs llmfan46 BF16, same precision,
|
||||
same pinned template, same 192 items):
|
||||
|
||||
T2 contradiction 75% → 59% (−5 items)
|
||||
T6 spatial 75% → 88% (+4 items)
|
||||
core 90.0% → 89.4% (−0.6 pts)
|
||||
|
||||
**It MOVED capability rather than removing it** — five lost on contradiction,
|
||||
four gained on spatial, nearly cancelling. Nobody predicted a gain. **llmfan46
|
||||
stands**; no case for re-staging on TrevorJS over 0.6 points.
|
||||
|
||||
⚠ Read as ~5 and ~4 items at n=32, not as −15.6/+12.5 percent. ⚠ Says nothing
|
||||
about quantization — the stock-NVFP4 T2 figure came from n=16 against n=32,
|
||||
different item sets, n-confounded.
|
||||
|
||||
## 7. ⚠ The production compose hardcodes `--quantization compressed-tensors`
|
||||
|
||||
Pointing the char-rp stack at unquantized BF16 weights crash-loops immediately:
|
||||
|
||||
TypeError: CompressedTensorsConfig.__init__() missing 3 required
|
||||
positional arguments: 'target_scheme_map', 'ignore', 'quant_format'
|
||||
|
||||
vLLM trying to read a quantization config out of a checkpoint that has none. 35
|
||||
restarts before it was caught. Hence `stacks/gemma4-trainee-bench/` — a separate
|
||||
ephemeral stack with no quantization flag, `restart: "no"` so a bench seat cannot
|
||||
resurrect itself and block gen's restore, and no homepage labels so it leaves no
|
||||
permanently-offline card. That detour is why a base swap is now ~5 minutes
|
||||
instead of ~15.
|
||||
|
||||
## 8. BF16 cannot coexist with gen
|
||||
|
||||
48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before
|
||||
a byte of KV cache. Every BF16 bench window means **gen is stopped**. Two such
|
||||
windows were run and gen restored both times; the restore was triggered by
|
||||
observing the seat's own throughput logs (a large prefill burst then zero
|
||||
running/zero waiting) rather than waiting on a courtesy message.
|
||||
|
||||
Cross-links: [[2026-08-24-homepage-uniform-grid]]
|
||||
@@ -1,77 +0,0 @@
|
||||
# [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live
|
||||
|
||||
Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup
|
||||
shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three
|
||||
wrong turns worth recording.
|
||||
|
||||
## Root cause
|
||||
|
||||
`esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises
|
||||
**itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over
|
||||
the DHCPv4-supplied resolver, so queries went to the UDM — which does not know
|
||||
`.internal` — and returned NXDOMAIN. AdGuard was never consulted.
|
||||
|
||||
Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver
|
||||
was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled
|
||||
entirely**, so every host there got the UDM and could never resolve `.internal`
|
||||
— esh-docker-vm was living proof.
|
||||
|
||||
## The three wrong turns
|
||||
|
||||
1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an
|
||||
explicit server is supplied**. Setting it bare looks like a no-op and invites
|
||||
the conclusion that the field is inert — which I drew, wrongly. Despite the
|
||||
`dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network.
|
||||
2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to
|
||||
AdGuard persists, reads back, and changes nothing. Proven with **fresh
|
||||
uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned
|
||||
real IPs. Reverted.
|
||||
3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the
|
||||
known `cmd/devmgr` behaviour.
|
||||
|
||||
Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were
|
||||
probed with a stdlib raw-socket Router Solicitation parsing option type 25
|
||||
(`rdisc6`/`tcpdump` were both absent; nothing was installed).
|
||||
|
||||
## What landed
|
||||
|
||||
RDNSS **redirected** rather than disabled — better than switching it off:
|
||||
|
||||
| VLAN | v4 | v6 RDNSS |
|
||||
|---|---|---|
|
||||
| `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
|
||||
| `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
|
||||
|
||||
The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over
|
||||
Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on
|
||||
SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client
|
||||
accepts it and never retries. A secondary that doesn't know your private zone is
|
||||
a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing
|
||||
camera DNS through AdGuard's filtering risks their cloud features.
|
||||
|
||||
## The naming scheme became real
|
||||
|
||||
The resolver address is the scheme's first live use, replacing a MAC-derived
|
||||
SLAAC address that would break on a NIC change. All three `esh-server` Linux
|
||||
hosts now carry `4411:B105` ("FOR ALL BIOS"):
|
||||
|
||||
```
|
||||
esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55
|
||||
esh-vm-db …:4411:b105:50:60
|
||||
```
|
||||
|
||||
Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals
|
||||
on re-delegation), backgrounds itself with a retry (SLAAC may not have landed;
|
||||
a blocking hook would stall bring-up on a headless box), and adds nothing to
|
||||
existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets
|
||||
`accept_ra=0` and would strand the host.
|
||||
|
||||
⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only
|
||||
despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores
|
||||
RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with
|
||||
`accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default
|
||||
route** — an IPv6 identity with no change to a hypervisor's routing. Expect this
|
||||
on every Proxmox node when its LAN gets v6.
|
||||
|
||||
Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits:
|
||||
auto-memory `reference_unifi_dns_rdnss_limits`.
|
||||
@@ -1,308 +0,0 @@
|
||||
# `[2026-08-24]` Homepage: remote-label consumption re-verified, then the board relaid out on a uniform grid
|
||||
|
||||
Prompted by the operator: *"Homepage on esh-vm-docker lists remote dockers and
|
||||
can absolutely consume their labels, please verify again. I am still
|
||||
unsatisfied with the layout and aesthetics."*
|
||||
|
||||
## The verification — the operator was right, and the record now says so
|
||||
|
||||
**Homepage on `esh-docker-vm` discovers services by container label from all
|
||||
five Docker engines in `conf/docker.yaml`, not just its own.** This is not an
|
||||
inference; `GET /api/services` returns every card's `server` field, and the
|
||||
2026-08-24 snapshot resolves to:
|
||||
|
||||
| `server` | host | label-discovered services |
|
||||
|---|---|---|
|
||||
| `ana-pfi-docker` | 10.250.50.70 | 30 |
|
||||
| `irv-ml1-docker` | 10.100.79.3 (over WireGuard) | 15 |
|
||||
| `ana-ml2-docker` | 10.250.50.54 | 14 |
|
||||
| `esh-vm-docker` | 10.0.50.45 (the dashboard's own host) | 13 |
|
||||
| `nh3-pfi-docker` | 10.100.50.40 | 2 |
|
||||
|
||||
**74 of 107 cards are label-discovered, and only 13 of those come from the
|
||||
dashboard's own engine** — the other 61 are read off four remote hosts,
|
||||
including irv-ml1 across the WireGuard tunnel. The remaining 33 carry
|
||||
`server: null`: those are the manual `services.yaml` entries — hardware, BMCs,
|
||||
hypervisors, printers, and user-level systemd services that have no container
|
||||
to label in the first place. **That null is the only thing "not label-driven"
|
||||
about this dashboard**, and it is a property of the entry, not of the host it
|
||||
points at.
|
||||
|
||||
⚠ If a future session doubts this again, the check is one command and takes two
|
||||
seconds — do not reason about it from the docs:
|
||||
|
||||
```bash
|
||||
curl -s http://10.0.50.45:5100/api/services \
|
||||
| jq -r '.[].services[] | .server' | sort | uniq -c
|
||||
```
|
||||
|
||||
## What was actually wrong with the layout
|
||||
|
||||
Measured with Playwright against the live board (per-group `card=` width, card
|
||||
height spread, and a geometric title-vs-status overlap test), not judged by
|
||||
eye:
|
||||
|
||||
- **Card width changed at every group boundary.** `columns:` is not a density
|
||||
dial — it sets `lg:grid-cols-N` for one group, so it fixes that group's card
|
||||
width. Notes rendered a single **1464px** card; News and Media **728px**;
|
||||
Eval & Retrieval **286px**; everything else 360px. Scrolling the page, the
|
||||
grid resized five times.
|
||||
- **Long names printed underneath their own status pill.** Measured by
|
||||
re-injecting the old rule and testing the title text node's box against the
|
||||
status cluster's box: **6 cards, all on the AI tab** — 3 in Inference, 2 in
|
||||
Dormant, 1 in Eval & Retrieval; zero on the other three tabs, which is why
|
||||
it survived earlier passes. Root cause is a genuinely counter-intuitive one:
|
||||
the rule reserved a
|
||||
78px gutter with `padding-right` and relied on `overflow: hidden` to hold it,
|
||||
but **overflow clips at the padding box, not the content box** — so the
|
||||
reserved gutter was spill room the title rendered straight through. The
|
||||
intended `text-overflow: ellipsis` never fired either, because the ellipsis
|
||||
is painted by whichever block's own line overflows, and here that is the
|
||||
anonymous box around the bare title text node, which does not carry
|
||||
`overflow`.
|
||||
- **`AI Systems` / Scriberr was on all four tabs** — the 2026-08-18 UltraSeedbox
|
||||
bug recurring, this time arriving from a container label rather than from
|
||||
`settings.yaml`.
|
||||
- **Icons were grey smudges.** Homepage masks every glyph over
|
||||
`--color-logo-start/stop`; stock slate-400 → slate-700 sinks the bottom half
|
||||
of each icon into the card fill.
|
||||
- Bookmark groups and Jellyfin's trailing stream rows were the two components
|
||||
the theme had never reached.
|
||||
|
||||
## The fixes
|
||||
|
||||
`stacks/homepage/conf/settings.yaml` — **all 20 groups to `columns: 4`.**
|
||||
`stacks/homepage/theme/australis.css.in` → rebuilt → `conf/custom.css`:
|
||||
gutter held by wrapping, description clamped to 3 lines (floor still 2), icon
|
||||
ramp overridden, bookmark + trailing-widget components themed, group gap
|
||||
10px → 22px. `stacks/scriberr/compose.yaml` — `homepage.group` → `AI - Audio
|
||||
Tools`, container recreated on ana-ml2.
|
||||
|
||||
After: **every group renders at card=360**, and the collision count is zero.
|
||||
|
||||
Before/after, all four tabs: `http://10.100.10.50:8090/b/homepage-relayout/`
|
||||
(24h TTL; also on the standing link board).
|
||||
|
||||
## ⚠ Three traps worth carrying forward
|
||||
|
||||
1. **"Columns = member count" is RETIRED** (it was the 2026-08-18 rule). It was
|
||||
avoiding dead cells in a short last row and bought a worse defect. A short
|
||||
last row is what a grid looks like; a card wider than its neighbours is what
|
||||
a mistake looks like.
|
||||
2. **A `:root` override of a Homepage theme variable is silently ignored.**
|
||||
Homepage sets `--color-logo-*` on `.theme-slate`, and that class is on the
|
||||
`<html>` element — the same element `:root` matches. `.theme-slate` (0,1,0)
|
||||
beats `:root` (0,0,1), so the override does nothing and looks like the
|
||||
variable is not the one in play. `html[class]` (0,1,1) wins, and does not
|
||||
hard-code which `theme-*` class is active. Specificity alone is not enough
|
||||
either: a custom property resolves from the *nearest* ancestor that sets it,
|
||||
so the override has to land on `<html>`, not on `<body>`.
|
||||
3. **The post-recreate tab-bar loss is INTERMITTENT, not guaranteed.** The
|
||||
2026-08-19 note reads as though every recreate costs up to an hour of broken
|
||||
render. This recreate came up correct within 10 seconds — fresh payload on
|
||||
the first poll, all four tabs clickable a minute later. Recreate, *check*,
|
||||
and only then walk away if it is actually in the broken state.
|
||||
|
||||
Also re-confirmed, since the change depended on it: **a `settings.yaml` edit
|
||||
needs a container recreate, not a restart.** `docker restart homepage` left the
|
||||
old `"columns":1` payload embedded in the served HTML with the correct file
|
||||
mounted and readable inside the container; `compose up -d --force-recreate`
|
||||
cleared it immediately.
|
||||
|
||||
## Deliberately not done — operator's call
|
||||
|
||||
The Main tab still opens on three sparse bands: **Notes** (1 member) and
|
||||
**Games** (1) each burn a full 4-wide row, and **News** has 2. Merging Notes +
|
||||
News, or folding Games into Apps, would tighten the top of the page — but that
|
||||
is information architecture, not layout, and the group names are the operator's.
|
||||
Surfaced rather than done.
|
||||
|
||||
→ **Resolved in pass 2 below**, where the operator delegated the naming
|
||||
("re-categorize however you want"). Notes + News became `Daily`, Games folded
|
||||
into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded
|
||||
(Scriberr moved on to `AI - Studios`).
|
||||
|
||||
---
|
||||
|
||||
# `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed
|
||||
|
||||
Operator, after seeing pass 1: *"You can re-categorize however you want.
|
||||
service networking tab is uneven, you can split out the adguard cards, etc.
|
||||
most of the issues are that tools I use and have a UI are interspersed with API
|
||||
endpoints which are largely informational only. They might even go in their own
|
||||
cards or start collapsed."*
|
||||
|
||||
## The axis
|
||||
|
||||
Every group is now either **tools** (expanded, top of tab) or **endpoints** (an
|
||||
API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A
|
||||
collapsed group still renders its eyebrow and rule, so presence costs one line
|
||||
instead of two rows.
|
||||
|
||||
Second, quieter rule that fell out of the same pass: **a group's members should
|
||||
all carry a widget or none should.** A stat strip adds ~50px, so one widget card
|
||||
in a row of plain ones opens a void under the plain ones — which is most of what
|
||||
made the 13-member `Service Networking` band look broken.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the
|
||||
1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the
|
||||
1-card `Games` band), `Media`, `UltraSeedbox`.
|
||||
- **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then
|
||||
`AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4),
|
||||
`AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed.
|
||||
- **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2),
|
||||
`Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed).
|
||||
- **Infrastructure** — unchanged; every card there is already a console.
|
||||
|
||||
Measured after: every group `card=360`, and `DNS & Filtering` and
|
||||
`Reverse Proxies` both `h=134..134` — dead flush.
|
||||
|
||||
## ⚠ The move that made it affordable
|
||||
|
||||
**The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is
|
||||
read at container **creation**, so renaming `AI - Inference` to something
|
||||
clearer would have meant recreating six vLLM seats plus four eval seats plus
|
||||
four TTS engines — multi-minute model reloads on endpoints peers reach through
|
||||
the gateway. Order plus `initiallyCollapsed` buys the same separation for free,
|
||||
so the names stay ugly on purpose. **Do not spend that recreate on a label.**
|
||||
|
||||
28 containers *were* relabelled — all cheap web services — via five rerunnable
|
||||
elway playbooks, `playbooks/homepage-regroup-<host>.yaml`. The canonical
|
||||
`stacks/` tree was synced to match afterwards, so intent and reality agree.
|
||||
|
||||
`initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in
|
||||
this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in
|
||||
`/app/.next/server/pages/index.js`).
|
||||
|
||||
## AdGuard (ANA) gained its widget, and the credential is fleet-wide
|
||||
|
||||
It was the only AdGuard without a query/blocked/latency strip, so it sat short
|
||||
beside two tall siblings. **One `infra-ops` AdGuard login authenticates against
|
||||
all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200
|
||||
on `POST /control/login`, verified 2026-08-24). Vaulted at
|
||||
`secret get nh3-dev/adguard-infra-ops-password`; written to
|
||||
`/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon
|
||||
was also the odd one out (`mdi-dns` against two `si-adguard`).
|
||||
|
||||
## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION
|
||||
|
||||
The biggest durable finding of the day, and it cost ~25 minutes. Full write-up
|
||||
in `stacks/homepage/README.md`; the short version:
|
||||
|
||||
`initialSettings":{}` in the served HTML is **the catch branch** of the page's
|
||||
data loader, not a warm-up and not a cache. And the error can vanish without
|
||||
trace: the logger is assigned as the first statement *inside* the same `try`,
|
||||
and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is
|
||||
written anywhere — which is exactly what was observed.
|
||||
|
||||
Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`,
|
||||
`/api/widgets` and `/api/hash` all return **200 with correct content** while the
|
||||
page serves `{}`; restoring the previous known-good `settings.yaml` reproduces
|
||||
it identically; `/api/validate` returns `[]`; disk and permissions are fine.
|
||||
|
||||
**One-command test:**
|
||||
`curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'`
|
||||
|
||||
**What broke the streak:** three consecutive recreates came up empty, then
|
||||
rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed
|
||||
it within 15 seconds. That is one observation, not proof — but it is a coherent
|
||||
mechanism (oversized log → logger init throws → silent catch) and it is the
|
||||
cheapest thing to try first next time.
|
||||
|
||||
---
|
||||
|
||||
# `[2026-08-24, pass 3]` Rebuilt on Australis Skyfall — dual theme, light shipped
|
||||
|
||||
Operator supplied the Skyfall design-system README and said "Go full with
|
||||
skyfall."
|
||||
|
||||
## The bundle was already in this repo's git history
|
||||
|
||||
**The Skyfall tokens did not need to be hunted down.** A predecessor vendored
|
||||
them on 2026-08-19 and a later commit deleted them; git kept everything:
|
||||
|
||||
```bash
|
||||
git show 45c1995:stacks/homepage/theme/colors.css # 208 lines, BOTH themes
|
||||
git show 45c1995:stacks/homepage/theme/layout.css # calm-depth tokens
|
||||
git show 45c1995:stacks/homepage/theme/typography.css
|
||||
git show 45c1995:stacks/homepage/theme/fonts/Supreme-{400,500,700}.woff2
|
||||
```
|
||||
|
||||
`colors.css` carries `:root` (dark) **and** `[data-theme="light"]` (Skyfall
|
||||
Day) — so the light ramp is canonical, not derived. That killed the entire
|
||||
objection from the previous answer, which was correct only about the
|
||||
`australis-design` skill ("Always dark first. No light mode in this system").
|
||||
**Skyfall is the dual-theme derivative; australis-design is the terminal
|
||||
theme. They are different systems and only one of them has a light mode.**
|
||||
|
||||
## ⚠⚠ REMOVING `theme:` FROM settings.yaml BREAKS THE DASHBOARD
|
||||
|
||||
The documented way to get Homepage's own light/dark toggle is to leave `theme:`
|
||||
unpinned. **Do not.** With the key absent, the page's data loader throws and its
|
||||
catch branch serves `initialSettings: {}` — no tab bar, no layout, no i18n.
|
||||
|
||||
Measured, not inferred: six force-recreates over seven minutes all came up
|
||||
empty with the key removed; restoring `theme: dark` rendered correctly on the
|
||||
next recreate in **12 seconds**. `/api/services` stays 200 and fully correct
|
||||
throughout, which is exactly why this reads as a caching or warm-up problem and
|
||||
is not one.
|
||||
|
||||
This is the first *confirmed* trigger for the long-running "tab bar goes
|
||||
missing" mystery. It does not explain every occurrence (the symptom has
|
||||
appeared with `theme:` present), but it means **the first diagnostic step is
|
||||
now `git log -p -- stacks/homepage/conf/settings.yaml`**, not container
|
||||
archaeology. Also retires an earlier lead from this same session: rolling the
|
||||
8.6 MB `homepage.log` aside once coincided with a recovery, but did nothing
|
||||
during the `theme:`-key episode — coincidence, not cause.
|
||||
|
||||
## So the toggle is ours
|
||||
|
||||
`conf/custom.js` renders it (was an empty placeholder). Precedence:
|
||||
|
||||
1. explicit choice — `localStorage['skyfall-theme']`, written by the toggle;
|
||||
2. OS preference — `@media (prefers-color-scheme: light)`;
|
||||
3. dark — Skyfall's default.
|
||||
|
||||
`theme/build.py` re-emits each vendored `[data-theme="light"]` block twice: as
|
||||
`[data-theme="light"], html.light`, and inside the media query scoped to
|
||||
`html:not([data-theme="dark"]):not([data-theme="light"])`. **That `:not()` pair
|
||||
is what lets a stored *dark* choice survive a light-mode OS.** Verified across
|
||||
both OS preferences: load, click, click again, reload — all four correct.
|
||||
|
||||
⚠ Homepage keeps its own `class="dark scheme-dark theme-slate"` on `<html>`
|
||||
regardless, because `theme:` is pinned. That is fine and was checked
|
||||
explicitly: with the dark class present AND `data-theme="light"`, every themed
|
||||
surface resolves to Skyfall Day, because our rules carry `!important` on the
|
||||
surfaces Tailwind's `dark:` variants would otherwise claim. **`data-theme` is
|
||||
the control surface; the class is not.**
|
||||
|
||||
## The anti-fork guard is now mechanical
|
||||
|
||||
`build.py` records the SHA-256 of each vendored file and **fails the build** on
|
||||
a mismatch, rather than warning. A vendored file is either byte-identical to
|
||||
the bundle or it is a fork wearing the bundle's name. Overrides go in
|
||||
`skyfall.css.in`, which is written entirely against the semantic layer
|
||||
(`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/--warning`) — no
|
||||
raw family tokens, no colour literals.
|
||||
|
||||
The one place a literal is unavoidable: Homepage consumes
|
||||
`--color-logo-start/stop` as `rgb(var(--x))`, which cannot take an `oklch()`.
|
||||
Those four values are exact sRGB conversions of real tokens (`--sea-80`,
|
||||
`--blue-base` for dark; `--sea-40`, `--blue-deep` for light), computed rather
|
||||
than eyeballed, with the conversion recorded in the file.
|
||||
|
||||
## Deviations, all deliberate and all written down
|
||||
|
||||
- **The aurora ribbon under the tab bar is gone.** Skyfall sanctions exactly two
|
||||
accent expressions — the active rail and hero-only glows — and a decorative
|
||||
gradient across the chrome is neither. The colour moved to a 2px accent bar
|
||||
plus `--accent-soft` fill on the active tab, which *is* the rail.
|
||||
- **Widget stat values moved from the display face to mono**, per Skyfall's
|
||||
"numbers and telemetry are always `--font-mono`".
|
||||
- **Two font substitutions**: Space Grotesk for Bespoke Sans, JetBrains Mono
|
||||
for Victor Mono. Only Supreme was ever vendored, and Skyfall's own notes call
|
||||
Victor Mono "user-supplied". Two-line swap when the real faces arrive.
|
||||
|
||||
Dark + light, all four tabs: `http://10.100.10.50:8090/b/homepage-skyfall/`
|
||||
@@ -1,46 +0,0 @@
|
||||
# [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1
|
||||
|
||||
Self-hosted audio/video transcription + diarization. Operator chose GPU
|
||||
placement over ana-docker (8 cores shared with 50 containers, 37 GB disk)
|
||||
against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom.
|
||||
|
||||
**Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`,
|
||||
all seven backends up, zero failures: `whisperx pyannote sortformer parakeet
|
||||
canary voxtral openai`. ~30 GB of weights on `/tank`.
|
||||
|
||||
Stack: `stacks/scriberr/`. Full gotcha list in auto-memory
|
||||
`reference_scriberr_ana_ml2`.
|
||||
|
||||
## Three upstream bugs, none of them ours
|
||||
|
||||
**1. The Blackwell image does not exist.** Upstream's README documents
|
||||
`scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published
|
||||
`scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or
|
||||
silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9`
|
||||
(CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose
|
||||
back to the published image.
|
||||
|
||||
**2. It must run as uid 10001, not 1000** — and the error lies:
|
||||
`unable to open database file: out of memory (14)`. Error 14 is
|
||||
`SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates
|
||||
`appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to
|
||||
it, while the entrypoint's PUID remap covers only the data dirs.
|
||||
**Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the
|
||||
mount; fails on a plain named volume too → not the storage; the **published CPU
|
||||
image works at PUID=1000** because there `appuser` *is* 1000.
|
||||
Generalisable: *when a container "permission" bug appears, compare the uid the
|
||||
image was BUILT for against the uid you are RUNNING as.*
|
||||
|
||||
**3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env
|
||||
with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with
|
||||
`Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial
|
||||
failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet
|
||||
and Sortformer were silently absent. Occurrences 2 → 0 after the fix.
|
||||
|
||||
## Related
|
||||
|
||||
`speaches` on irv-ml1 **stopped** the same day (stack retained, one command to
|
||||
restart): Eyra was abandoned pre-implementation because Scriberr covers the need,
|
||||
leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and
|
||||
is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the
|
||||
~5.9 GB quoted — that figure is the loaded-model working set.
|
||||
@@ -1,92 +0,0 @@
|
||||
# ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## The run
|
||||
|
||||
1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0
|
||||
20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it)
|
||||
adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json
|
||||
|
||||
**47% faster than the round-1 projection of 13.85h**, from two changes: the
|
||||
bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3
|
||||
across the four 100-step windows — which means the 35-46.5 spread in round 1 was
|
||||
*entirely padding*, and removing padding removed the variance rather than just
|
||||
the mean.
|
||||
|
||||
⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a
|
||||
rolling average.** The first two were instantaneous tqdm readings off a number
|
||||
that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same
|
||||
measure-don't-sample discipline I wrote into the throughput playbook, violated on
|
||||
the one metric I kept reporting.
|
||||
|
||||
## lora_B gate — PASSED, twice
|
||||
|
||||
checkpoint-100 205/205 non-zero, median norm 0.829
|
||||
final adapter 205/205 non-zero, median norm 1.708
|
||||
vision_tower tensors: 0 on both
|
||||
|
||||
Median norm rising 0.829 -> 1.708 means it kept learning through the whole run
|
||||
rather than saturating early. This check **never ran in round 1** (died at step
|
||||
19, first checkpoint was 100) and it is the only failure mode that stays
|
||||
invisible until the acceptance gate reports base-identical numbers.
|
||||
|
||||
## The gate — brokkr-smithy-dev
|
||||
|
||||
**It did the thing it was built to do:**
|
||||
|
||||
metric base A/B tuned delta floor
|
||||
attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt
|
||||
diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008
|
||||
|
||||
Diversity moved **22x its own noise floor**. Attractor rate (how often the model
|
||||
reaches for the same names and phrasings) fell 11 points against a 2-point floor.
|
||||
|
||||
T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2
|
||||
memorisation: 0.0000 on BOTH arms, all three corpora
|
||||
|
||||
**Zero memorisation closes the R20 licensed-prose exposure on measurement rather
|
||||
than argument.**
|
||||
|
||||
⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192
|
||||
generations to truncation/degeneracy against base's 1-2. Lopsided exclusions
|
||||
plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor;
|
||||
the number carries an asterisk.
|
||||
|
||||
## The noise-floor near-miss — the methodology lesson
|
||||
|
||||
brokkr was one step from reporting a 13-point T6 regression **that sat inside
|
||||
twice his instrument's own variance.**
|
||||
|
||||
--per-type 32 max swing across tasks: 9 points
|
||||
--per-type 128 max swing across tasks: 1 point
|
||||
|
||||
His gate criterion is "no task regresses by more than one item" = 3.1 points at
|
||||
n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a
|
||||
preregistered gate at 4x finer resolution than it could resolve, and caught it by
|
||||
running a control he did not strictly need. Quadrupling n collapsed the noise
|
||||
exactly as binomial statistics predicts.
|
||||
|
||||
⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset,
|
||||
so with a 218,625-token KV cache the scheduler batches freely up to vLLM's
|
||||
default of 256. Continuous batching changes reduction order and borderline items
|
||||
flip. Temperature 0 buys deterministic *sampling*, not deterministic
|
||||
*arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right
|
||||
reason: a floor measured on a seat serving one request at a time is not the floor
|
||||
that applies to the seat we ship.
|
||||
|
||||
## The confound I built and he caught
|
||||
|
||||
I optimised a pipeline for production and then handed him its output as an eval
|
||||
instrument **without asking whether those were the same job.** The tuned arm
|
||||
would have reached the seat as NVFP4A16 while his base arm was bf16 — any
|
||||
regression would have been tuning-damage OR quantization-damage with no way to
|
||||
separate them, and the gate's whole question is "did the tune cost us
|
||||
capability."
|
||||
|
||||
**Both arms now bf16, same seat, same port, argv differing in exactly two
|
||||
lines** (weights path, served name), template sha256 identical
|
||||
(`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all
|
||||
three launches). Quantization moved *downstream* of the gate.
|
||||
|
||||
See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have.
|
||||
@@ -1,86 +0,0 @@
|
||||
# The 8.6% MFU was an accounting artifact — attention on Ampere kernels
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## The answer
|
||||
|
||||
**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided
|
||||
the *intended* (windowed) FLOPs by the wall time the *dense* reality took.
|
||||
|
||||
nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP
|
||||
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14
|
||||
padded full layers lose the causal skip = +3.5e14
|
||||
work performed ~ 1.8e15 = 51-61 TFLOPS
|
||||
|
||||
The card was doing ~2x the arithmetic the architecture specifies, and the excess
|
||||
was the sliding window being computed and thrown away.
|
||||
|
||||
## Two independent methods agreed
|
||||
|
||||
scaling fit (3 points, 2 params, residuals <3ms over 8x range)
|
||||
A = 6.87e-4 s/token B = 8.85e-8 s/token^2
|
||||
quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384
|
||||
|
||||
kernel table (device rows only)
|
||||
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
|
||||
dense GEMM 2,774.0 ms 7.9%
|
||||
other 5,739.0 ms 16.4%
|
||||
|
||||
**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit
|
||||
needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM
|
||||
launches per forward are not the cost).
|
||||
|
||||
## The mechanism, source-verified by brokkr's panel (arm: Bil)
|
||||
|
||||
masking_utils.py:292-301 _ignore_causal_mask_sdpa requires
|
||||
kv_length < local_attention_size. 16384 >= 1024,
|
||||
so THE SLIDING MASK ALWAYS MATERIALISES.
|
||||
sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask
|
||||
sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable
|
||||
Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12
|
||||
|
||||
⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**,
|
||||
with the forward on `gmem`, the memory-efficient backend's slowest fallback tier.
|
||||
|
||||
## What actually fixed it
|
||||
|
||||
**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and
|
||||
78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
|
||||
back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work**
|
||||
at fixed width, because an explicit mask knocks those layers off the fast path.
|
||||
|
||||
⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per
|
||||
accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does
|
||||
all the mixing; the bucket only costs padding. Use the tightest bucket.
|
||||
|
||||
**flex_attention** — Triton-generated so it compiles for sm_120 instead of
|
||||
shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
|
||||
`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a
|
||||
**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting
|
||||
into the full 99 KB, so it is the card, not a default.
|
||||
|
||||
## ⚠⚠ The trap that produced TWO wrong published conclusions
|
||||
|
||||
`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence
|
||||
width is a new shape. On hitting the ceiling dynamo does not error — it silently
|
||||
falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work
|
||||
with the backwards pass and may produce incorrect results."*
|
||||
|
||||
That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss
|
||||
divergence**, and I believed and reported both. Raising the limit to 256 flipped
|
||||
the speed result to 1.41x.
|
||||
|
||||
The loss divergence turned out to be real but benign — adjudicated against fp32
|
||||
MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally
|
||||
CLOSER at every width. **Do not re-open it by comparing the two backends to each
|
||||
other; that cannot answer it. Compare to fp32.**
|
||||
|
||||
## Process lesson
|
||||
|
||||
brokkr's panel produced **four self-retractions in ninety minutes**. Every
|
||||
retraction was a derivation; every survivor was a measurement. And the whole
|
||||
head_dim-512 SDP problem was **already documented in zerofata's published Axolotl
|
||||
config since April** — the right first stop for "why is this architecture slow"
|
||||
is practitioner configs for that exact base, before any panel.
|
||||
|
||||
Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`.
|
||||
@@ -1,84 +0,0 @@
|
||||
# NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
Pipeline at `scripts/erp-tune-serve/` (commits `6a85829`, `ab980e9`).
|
||||
Validated end-to-end against checkpoint-100 before the real adapter existed.
|
||||
|
||||
## ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert
|
||||
|
||||
before linearize_moe: 427 Linears, 205 targeted, experts 0
|
||||
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
|
||||
(30 layers x 128 experts x 3 projections)
|
||||
|
||||
Gemma-4 stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors
|
||||
(`gate_up_proj` [128,1408,2816], `down_proj` [128,2816,704]) — note the absent
|
||||
`.weight` suffix. A Linear-targeting recipe resolves 205 of 427 modules and
|
||||
**zero experts**, leaving 22.84 B params (88.5% of the model) in BF16 with no
|
||||
warning.
|
||||
|
||||
**This is the same defect that killed QLoRA here via bitsandbytes.** The blind
|
||||
spot is in the *checkpoint layout*, not the tool. Fix:
|
||||
`llmcompressor.modeling.moe.linearize.linearize_moe` — no registration needed,
|
||||
Gemma-4 satisfies `FusedExpertsProtocol` structurally. Playbook §3.15.
|
||||
|
||||
## Scheme: NVFP4A16, deviating from the playbook default, on measured grounds
|
||||
|
||||
brokkr benched the W4A4 quant of this checkpoint at **12% on contradiction
|
||||
detection with CoT off against gen's 81%** — the signature of 4-bit input
|
||||
activations on a reasoning-dense task. Plus W4A4 KLD is 2-4x worse past ~10k ctx
|
||||
on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill cost accepted.
|
||||
|
||||
⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` —
|
||||
W4A4 wearing an A16 label. The script refuses if the emitted config says 4.
|
||||
|
||||
## Four silent defects the dry run found
|
||||
|
||||
1. **transformers 5.15 MIGRATES the config schema on save** — drops
|
||||
`global_head_dim`/`num_global_key_value_heads`, writes `per_layer_config`.
|
||||
transformers 5.10 (the llmcompressor venv) then reads `num_key_value_heads`
|
||||
as None and dies with `TypeError: unsupported operand type(s) for //`.
|
||||
Every working artifact on the box uses the OLD schema. Merge now downgrades it.
|
||||
2. **llmcompressor cannot auto-init a processor for a multimodal checkpoint** —
|
||||
pass the tokenizer explicitly as `processor`.
|
||||
3. **`save_pretrained` does not carry `processor_config.json`** — vLLM then fails
|
||||
with "Can't load feature extractor", which reads as a vision bug.
|
||||
4. **The quant needs more than GPU1's free 32 GiB.** `quant_with_gen_down.sh`
|
||||
stops `vllm-gen` and restores it from a trap on EVERY exit path, using
|
||||
`docker start` not `compose up` so the container returns with its exact config.
|
||||
|
||||
## Verified on the emitted artifact
|
||||
|
||||
49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16)
|
||||
weight_packed 11,725 of which expert 11,520
|
||||
tokenizer truncation: clean (§3.14 trap avoided by calibrating on the
|
||||
encode cache, so the tokenizer is never called
|
||||
with truncation=True at all)
|
||||
served: Marlin NVFP4 kernel + Marlin MoE backend, coherent generation
|
||||
|
||||
⚠ The reference `nvfp4a16` artifact triggers a vLLM warning that q/k/v carry
|
||||
*different* weight global scales ("likely reduced accuracy"). **Ours does not** —
|
||||
llmcompressor 0.12 links weight observers across fused groups automatically. The
|
||||
in-house quant is better than the downloaded one on that axis.
|
||||
|
||||
## ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed
|
||||
|
||||
The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops.
|
||||
Retested on `vllm/vllm-openai:latest`: **it refuses to start.**
|
||||
|
||||
AttributeError: To support LoRA for MoE model,
|
||||
'get_expert_mapping' must be implemented
|
||||
|
||||
The check is in `vllm/lora/utils.py::process_packed_modules_mapping` and branches
|
||||
on `is_moe_model()` — **quantization is not in the condition.** `gemma4.py`,
|
||||
`gemma4_mm.py`, `gemma4_mtp.py`, `gemma4_unified.py` all have ZERO occurrences;
|
||||
`deepseek_v2`, `mixtral`, `glm4_moe`, `ernie45_moe` implement it.
|
||||
|
||||
**vLLM cannot serve a LoRA on ANY Gemma-4, bf16 or quantized.** Merging is the
|
||||
only path for this architecture, and it would have bitten identically on the
|
||||
unquantized base. A loud refusal is strictly better than the 0.24.0 silent no-op,
|
||||
which shipped a base model wearing the tune's name.
|
||||
|
||||
⚠ Base-viability pre-flight is now playbook §3.11 — three greps before picking a
|
||||
base. **Grep the CLASS, not the file**: `mistral.py` greps as `SupportsLoRA=0`
|
||||
and is fully LoRA-capable via inheritance from `LlamaForCausalLM`.
|
||||
@@ -1,64 +0,0 @@
|
||||
# Refusal retention — the axis the gate did not have, and the axis I measured wrong
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## Why it exists
|
||||
|
||||
brokkr's gate measures reasoning (T1-T6), craft (diversity/attractor) and
|
||||
regurgitation (memorisation). **Nothing measured whether the model still
|
||||
COMPLIES** — which for this seat is arguably the most important property.
|
||||
|
||||
The risk is specific to our operation order. We do **tune(abliterate(stock))**,
|
||||
so the tune has 57.7M tokens of opportunity to walk the abliteration back. *A
|
||||
tune that gains 41 items of contradiction detection and quietly re-installs
|
||||
refusals is a failed seat that passes the entire gate.*
|
||||
|
||||
## The measurement — controlled, single instrument, both arms
|
||||
|
||||
arm HARD DEFLECT COMPLY
|
||||
base 0/100 0 100
|
||||
tuned 29/100 0 71
|
||||
|
||||
Same seat, same probe, temp 0, `mlabonne/harmful_behaviors` x100.
|
||||
Probe: `scripts/training-probes/refusal_probe.py`.
|
||||
|
||||
**The tune added 29 general-harm refusals where the base had none.**
|
||||
|
||||
Two things fell out:
|
||||
|
||||
- **The instrument validates.** Base measured 0/100 on my generated-text regex
|
||||
against Heretic's recorded 3/100 from a first-token-probability scorer. 0 vs 3
|
||||
is agreement — the incomparability worry was right caution about a non-problem.
|
||||
- **DEFLECT is 0 on BOTH arms, so the free control fires.** An instrument
|
||||
artifact does not care which arm it runs against. Both zero means the model is
|
||||
**binary** — refuses in refusal-language or engages, no soft-deflection tail.
|
||||
The R19 undercount does not apply here.
|
||||
|
||||
## ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one
|
||||
|
||||
`mlabonne/harmful_behaviors` is **general harm** — weapons, malware, fraud. **The
|
||||
abliteration was not run so the model would explain bomb-making. It was run so
|
||||
the model would engage with explicit fiction.** Different refusal surfaces; a
|
||||
model moves on them independently.
|
||||
|
||||
I picked that set because it was cached, had a recorded baseline, and was what
|
||||
the abliteration tool used. **Every one of those is a reason it was convenient,
|
||||
not a reason it was right** — and "it has a baseline" was actively misleading,
|
||||
because a comparable number for a question nobody is asking looks like evidence.
|
||||
|
||||
**29/100 general-harm refusals on a seat writing prose the operator was actively
|
||||
praising is plausibly the DESIRED shape**, not a defect. General-harm refusals
|
||||
returning while domain compliance holds is close to ideal for an internal
|
||||
creative seat. I would have reported it as damage.
|
||||
|
||||
**The load-bearing cell is COMPLY 71, not the 29.** Stock refused 100/100;
|
||||
anything near that would mean the abliteration was undone. 71 complying means
|
||||
"partially walked back on one axis" — a different finding, and only one of the
|
||||
two threatens the seat.
|
||||
|
||||
Domain-compliance probe (the right axis, from R19's track-2 map) is brokkr's,
|
||||
pending. Scaffold supplied: `scripts/training-probes/counted_classifier.py`
|
||||
(`2a05ae9`) — classify-never-surface, three-way, ERROR path deliberately does not
|
||||
log the exception body because an exception can echo the prompt back.
|
||||
|
||||
Playbook §3.13. See [[2026-08-25-erp-tune-run2-complete]].
|
||||
@@ -1,123 +0,0 @@
|
||||
# Worldtree b188 + b189 bridge cutover, and the selene metadata that lied
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
Three arcs in one day, all infra-ops side, all landed.
|
||||
|
||||
## b188 — matrix.yaml pre-sync (#406/#409/#410 closed)
|
||||
|
||||
From b188 the bridge reads per-agent `rendering` + `ambient_buffer_size` from
|
||||
`config/matrix.yaml` ONLY; agent `config.yaml` matrix blocks are gone from the
|
||||
image. Staged as `6417115` in `worldtree-instance-configs`, deployed to both
|
||||
instances with operator approval.
|
||||
|
||||
- mimir gets thinking-to-thread + tool-call reactions + 7 reaction labels;
|
||||
forseti and lofn stay quiet.
|
||||
- **Rider #409 pruned six dead agents** (bragi, leif, troi, soong, cara, glados)
|
||||
from BOTH rosters — originally scoped personal-only, which I flagged as a
|
||||
possible oversight and it was. **The settling fact worth keeping: the engine
|
||||
roster comes from the image's baked `agents/` directory (only `config/` is
|
||||
bind-mounted), and both instances run the same image**, so instance-level
|
||||
evidence about which agents the engine lists generalises by construction.
|
||||
- Both rosters now exactly `[mimir, forseti, lofn]` — the three engine agents
|
||||
actually bridged. mask/vili/echo exist in the engine, deliberately unbridged
|
||||
(operator ruling).
|
||||
|
||||
⚠ **Edited text-surgically, not via a yaml round-trip** — PyYAML would reflow
|
||||
1,249 lines and drop every comment, and the comments are the documentation.
|
||||
|
||||
⚠ **`deploy-wt-config` uses `docker restart`, NOT `compose up`.** A `compose up`
|
||||
on corviduo-dev re-resolves the image tag and can silently swap the running
|
||||
build — which would turn an "inert pre-sync" into an unintended image roll on two
|
||||
live instances. That property is easy to lose in a future refactor of the script.
|
||||
|
||||
## b189 — #407 bridge extracted to its own repo (#404 umbrella closed)
|
||||
|
||||
Bridge now `gitea.phasefinal.com/pfi/wt-matrix-bridge`, its own repo, own CI.
|
||||
|
||||
⚠ **It publishes to the `pfi` ORG, not `vh`, and the reason is structural:** `vh`
|
||||
is a **USER**, not an org. Gitea scopes user-namespace packages to the owning
|
||||
user — there are no package collaborators on a user namespace. **No service
|
||||
account can ever publish to `gitea.phasefinal.com/vh/*`.** claude-bot is an Owner
|
||||
of `pfi`, so that is where it goes. Token `wt-matrix-bridge-ci` (id 28, scopes
|
||||
`write:package,read:repository`), vaulted at
|
||||
`nh3-dev/.config/claude-bot/gitea-token-wt-matrix-bridge-ci`.
|
||||
|
||||
**Minted a dedicated token rather than reuse `claude-bot-sdk-ops` or `arbo-ci`,
|
||||
both of which already carry `write:package`** — a shared credential cannot be
|
||||
revoked without collateral. ⚠ A first mint attempt succeeded then failed to save;
|
||||
Gitea returns a token value exactly once, so it was unrecoverable. Deleted the
|
||||
orphan (id 27) rather than leave a live package-write credential on the account.
|
||||
|
||||
**Both instances PINNED** to `f3f8ec902267` (`e90f436`), closing the #410 shape:
|
||||
`WORLDTREE_IMAGE` was pinned and the bridge was the one drifting service.
|
||||
|
||||
⚠ The pin moved from `b178285b1cb5` because a cross-frontier bug-hunt found the
|
||||
M_EXCLUSIVE **fallback could itself litter unboundedly** in exactly the state it
|
||||
was written for. **The safety net had the same failure mode as the thing it was
|
||||
catching** — and my staged window leaned on that fallback.
|
||||
|
||||
## #411 — the debug-room failure, diagnosed twice and wrong both times first
|
||||
|
||||
My theory: the alias was held by orphaned rooms. **Refuted by a directory probe
|
||||
returning 404.** The real cause: Synapse's **M_EXCLUSIVE** — an appservice may
|
||||
only create aliases inside a namespace it has RESERVED, and
|
||||
`aipa_appservice.yaml` had `namespaces.aliases: []`.
|
||||
|
||||
⚠ **I inferred a cause from a symptom that was the RESPONSE to the cause** — the
|
||||
log's "re-resolving alias" line is the recovery path firing, not evidence the
|
||||
alias exists. One directory probe settled it and I reasoned instead.
|
||||
|
||||
Fixed with operator clearance: added
|
||||
`regex: '#aipa-debug-[a-z0-9_-]*:matrix\.phasefinal\.com'` (exclusive) at
|
||||
`/opt/docker/conf/synapse/aipa_appservice.yaml` on **ana-docker** (NOT
|
||||
`/opt/docker/data/`, which worldtree-dev's issue cited). **My regex was tighter
|
||||
than the proposed `#aipa-debug-.*`**, which fullmatches only because `.` also
|
||||
matches the `:` separator and would equally claim other homeservers.
|
||||
|
||||
Pre-apply sweep (an `exclusive: true` claim can make Synapse refuse to START):
|
||||
|
||||
aliases matching '%aipa-debug%' 0
|
||||
total room_aliases on the homeserver 1 <- why it went unnoticed this long
|
||||
rooms created by @aipa-debug 17 <- the litter, confirmed unaliased
|
||||
|
||||
Synapse healthy in 40s, both bridges rode through. **The aliased create then
|
||||
worked FIRST TRY on the next personal recreate** — designed path, fallback never
|
||||
fired, both reuse mechanisms live (canonical-alias rediscovery AND the room-id
|
||||
cache).
|
||||
|
||||
⚠ The #411 writer census **inverted its own premise**: the api (uid 1000) cannot
|
||||
write `/app/sessions`, the bridge (root) can — the reverse of the issue text. But
|
||||
worldtree-dev's reconciliation is better than "backwards": pre-#407 the bridge
|
||||
ran from the ENGINE image as uid 1000, and the cutover changed the answer
|
||||
underneath the issue. Both readings were true at their timestamps. **No live
|
||||
writer exists for that path anyway** (`sessions.path` is a legacy default, #330
|
||||
moved the tools off it), so the chown is optional future-proofing.
|
||||
|
||||
## selene-1-mini-8b — a config that lied about what answers
|
||||
|
||||
forseti's fleet sweep found the only genuine residual in
|
||||
`worldtree-instance-configs`, and it was **live on both boxes, not just drifted
|
||||
in git.** Fixed in `a77639d`.
|
||||
|
||||
Routing was never broken (`model: "chat-judge"` stays), but `display_name` said
|
||||
"Selene 1 Mini 8B" and the description said "Atla Selene 1 Mini 8B — reward model
|
||||
derived from Llama 3.1 8B" while chat-judge has been backed by
|
||||
**qwen3.8-27b-uncensored** since 2026-08-23.
|
||||
|
||||
⚠ **It misstated the KIND of model, not just the identity.** A reward model and a
|
||||
generative judge are different instruments; a consumer reading
|
||||
"pairwise/likert/binary/scalar reward model" would expect scalar-reward semantics
|
||||
qwen3.8-27b does not provide.
|
||||
|
||||
Verified: `selene-1-mini-8b` -> HTTP 400 (by design), `chat-judge` -> HTTP 200.
|
||||
|
||||
**NOT changed:** the catalog key and the `selene-judgment` role — `model_roles.yaml`
|
||||
binds to that key, so renaming is worldtree-dev's schema call. Flagged that a role
|
||||
named after a retired model defeats the purpose of role aliases.
|
||||
|
||||
⚠ forseti's sharpest finding is for the operator: **`~/.claude/CLAUDE.md` line 502
|
||||
lists the retired name in the global tools roster.** A broadcast reaches sessions
|
||||
that already exist; the roster line keeps minting new ones. His file, awaiting his
|
||||
word. Also: **there is no fan-out primitive on the bus** — a fleet relay is 73
|
||||
individual posts, recorded as a real gap rather than papered over.
|
||||
@@ -0,0 +1,8 @@
|
||||
# ERP run 7 launched on pfi-gx10 (2026-09-08 23:06 PT)
|
||||
|
||||
Operator's direct grant `operator-2026-09-08-rnd-run7` (in-session `/goal: launch training run 7 on gx10`, 23:03 PT), pid 599489, 542 steps, ~80 s/it, ETA ~noon 09-09. Runbook `docs/runbooks/gx10-run-07.md`; canonical `scripts/erp-tune-gx10/{run-07-gx10.json,launch-run-07.sh,build_r7.py}`; commits `a34a72f`, `d8f625e`.
|
||||
|
||||
- Variable (brokkr `recipe-erp-seat-sft-r7.json`, sha `2972559e…`): an opening-split slot (293 rows, `opening-split/clean-v1`, root `3ab41c08…`) + companion mask (224 entries) UNION lossmask-r3 → `lossmask-r7` (6,106, 1 dup). `survivors-r7` = survivors-r5 + 293 = 8,505. Root/mask/holdout relayed NAS→nh3-dev→box (no NFS on the GX10), sha-verified.
|
||||
- Free check: held roots identical to run 6 (173 chunked / 30 truncated / 59 lost turns); slot 293/293 fit_whole; two cwm conversations (cwm-2602, cwm-1826) dropped as unfittable because the union mask left them no loss turn — brokkr: "loss moved, not doubled, satisfied by construction". Padding 17.1% (run 6: 0%).
|
||||
- ⚠ First launch died in preflight: the recipe JSON never landed because a zsh quoting error silently aborted the staging chain. Lesson: stage files with `scp` in their own command; never chain a heredoc-bearing python inside a single-quoted ssh string.
|
||||
- Gate after the adapter: brokkr commits `tools/base_window_r7.sh`, `tools/tuned_window_r7.sh`, `tools/verdict_run07.py` before it lands; serve `erp-seat-base-ara` (floors re-locked) → `erp-tune-v7`; cue-length probe as a conformance leg on both arms; DONE sentinel gated on every step's artifact.
|
||||
@@ -0,0 +1,11 @@
|
||||
# erp-tune-v6-nvfp4a16 on ana-ml2 :8021, aliased to `trial`, and the Gemma-4 tool-calling findings (2026-09-08)
|
||||
|
||||
**Seat.** Operator: "quant the latest trained model into nvfp4 and serve it on ana-ml2 while we train a new model on the gx10." Stack `stacks/erp-seat` (recipe = gemma4-charrp's; true served name only; GPU1, util 0.35, 32K ctx). Artifact `/tank/aimodels/erp-tune-v6-nvfp4a16` (16 GB, compressed-tensors nvfp4-pack, W4A16, 252 ignores incl. 60 router + 191 vision) from `/tank/aimodels/erp-tune-v6-bf16` (merged-run06 relayed gx10→nh3-dev→ana-ml2 in 17 min at ~47 MB/s — no key path gx10↔ana-ml2). Quant pipeline `services/erp-seat-quant/` replicates `prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16`: `linearize_moe` first, assert 11,520 expert Linears, routers/vision/audio ignored, post-steps restore processor configs + template and reset the tokenizer cap. DATA-FREE (~90 s) — playbook §3.16. Commits `911ff20`, `8512dd4`.
|
||||
|
||||
**Alias.** Operator: "alias erp-tune-v6-nvfp4 to trial, please. no gate." The `trial` model lives in `stacks/litellm/conf/config.yaml` → `/model/update` refuses it ("Model in config"); deploy `--conf` + `sudo docker compose restart litellm` on ana-docker. Verified 3/3. Commit `48aa2c6`. Block states the seat as unrated on every safety axis.
|
||||
|
||||
**Tool calling (operator: "fix toolcalling with the trial seat").**
|
||||
1. `tool_choice:"none"` → empty turn (content AND tool_calls null, 3/3): vLLM kept the tools in the prompt, the model called one, parsing was off. Fix `--exclude-tools-when-tool-choice-none` (`7f6be8a`). `stacks/gemma4-charrp` has the same exposure, no flag — bouncing it is consumer-visible, operator's call.
|
||||
2. Forced (named/`required`) tool_choice is PROMPT-DRIVEN on every Gemma-4 seat **by vLLM design**: `vllm/tool_parsers/gemma4_engine_tool_parser.py` sets `supports_required_and_named = False` and `adjust_request` skips the structured-output JSON so the model can emit native `<|tool_call>call:…`. A tune that weakened that syntax honours forced calls only sometimes. Measured (3 conv × 3 turns, real system prompt): v0.26.0 1/9; nightly `311b3513` (v0.27.2rc1, the `gen` seat's image) 6/9 with the round trip clean 3/3 → seat pinned to the nightly (`a34a72f`). tts-dev: 0/18 on v0.26.0 with `gen` 18/18 as positive control; **`response_format: json_schema` 18/18 = the deterministic path**. A parser plugin re-enabling guided JSON would also need JSON extraction in the engine-parser path — not attempted.
|
||||
3. ⚠ Removing `--reasoning-parser gemma4` does NOT fix forced choice (3/9) and leaks `<|channel>thought` into content 3/3 — keep it and the `enable_thinking:false` pin.
|
||||
Single-turn probes are FALSE POSITIVES for forced tool_choice — gate on multi-turn with a real system prompt (tts-dev's lesson, confirmed).
|
||||
@@ -0,0 +1,5 @@
|
||||
# ESH static WAN follow-ups landed; YTVC chase; gitea webhook repointed (2026-09-08)
|
||||
|
||||
**ESH static WAN** `128.177.138.182/30` (gw .181, Cityside Fiber) confirmed LIVE (UDM WAN1 static, uplink up since ~09-05; egress verified; CGNAT at ESH is history). Added to the crowdsec `esh` allowlist on ana-docker. Operator: "land all 3" → (a) FortiGate infra-ops `trusthost3` 23.164.40.160 → `128.177.138.182/32`, verified by a real login from esh-docker-vm (flash backup ran but `execute revision list` errors on the 80F — unconfirmed); (b) dormant `esh-ana` IPsec rebound wan2/192.168.200.111 → wan1/`128.177.138.182` (still disabled); (c) ESH UDM forward UDP 41641 → esh-scale 10.0.50.65 → mesh peers DIRECT via `128.177.138.182:41641` (was DERP lax). Commits `7d5de53`, `a19b7f6`. ⚠ The permission classifier blocked every edge-gateway write until the operator disabled it — three shapes tried, then stopped and surfaced; no tool-laundering.
|
||||
|
||||
**YTVC** (operator: "chase youtube voice clipper being down"): (1) every job died at yt-dlp because the nh3-dev danted proxy was retired 09-06 → new fleet SOCKS5 on **nh3-scale** `socks5h://100.64.0.1:1080` (`scripts/setup-nh3-scale-socks-egress.sh`, dante, fleet ACL, residential egress 70.230.226.88; ⚠ dante one-line `{ }` blocks are a syntax error); (2) Homepage href on the dead wg0 IP → `irv-ml1.nh3.internal:8000` (labels live in the UNTRACKED override); (3) full downloads still 403'd = stale yt-dlp 2026.07.04 from a cached Docker layer (`--test` passing while the full fetch 403s = client, not IP/cookies) → `compose build --no-cache api`; (4) the rebuild pulled a whisperx needing `punkt_tab` → staged on the data volume + `NLTK_DATA`. Then yt-voice-clipper-dev shipped v0.3.4→v0.3.6 (cache-bust ARG, punkt baked, **CrisperWhisper 2.0 replaces whisperx**: x7kWJojf1MI → 12 clips with `[UM]`/`[UH]` tags). ⚠ The gitea push webhook had targeted the dead wg0 IP since 09-06 (never fired) → repointed to `10.6.110.50:9008` with the HMAC secret RE-APPLIED (a PATCH without `secret` drops it); test push auto-deployed. Commits `14dd9ab`, `d2df1eb`, `f2b7384`.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Run-6 gate: CSAM soft trip, operator adjudication, TRANSFERRED (2026-09-08)
|
||||
|
||||
- 22:18 PT brokkr HALTED the run-6 gate: R19's CSAM-drift detector fired at level=review on ONE generation in the tuned refusal k=5 step — metadata only (minor_soft=1 `soft_baby` near anatomy terms, minor_hard=0, sexual=7, salted hash `01724c39…`); the spec (§8) never stores the text and the seat does not log completions, so **the text exists nowhere**.
|
||||
- Conflict surfaced, not resolved by me: the operator had aliased `trial` → the NVFP4 build ("no gate") two minutes before the halt reached me; brokkr asked that it not be served until adjudication. Left the alias in place (explicit operator ruling), escalated to the operator + Miranda with the one-line rollback offered.
|
||||
- 22:45 PT operator adjudicated GO, verbatim: "I don't want it regenerated, but I will say that in the vernacular, baby is about the most common pet name you can get, ESPECIALLY during sex. I'm going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is not one of them." Relayed verbatim to brokkr and Miranda.
|
||||
- brokkr finalized TRANSFERRED without the tuned refusal leg (the cell rule never read it; tool forbids a re-run); tuned refusal telemetry recorded ABSENT for run 6; RESULT carries the adjudication with infra-ops as relay. Operator also CUT the k=25 refusal legs on both arms ("the battery is information enough").
|
||||
- Cue-length probe (both arms, 288 gens each): tuned 5-word cue → 54/62 words median, 49-word → 114, 221-word → 292; base 168/191 → 209 → 282. Tune's deficit ~120 words at the short cue, ~0 at the long → run-7's lever.
|
||||
- The choreography honoured hands-off throughout; the base seat served 16:24–21:38 and again 22:45–23:01 PT for the reference arm.
|
||||
@@ -0,0 +1,15 @@
|
||||
# ana-ml2 pool health — three actions deferred to a clean-context session (2026-09-09)
|
||||
|
||||
Operator ruling 2026-09-09 ~00:30 PT: "snapshot and we'll do all 3 on clean context." Findings commit `3e18a04`.
|
||||
|
||||
**Findings (measured 2026-09-09 00:00 PT):**
|
||||
- `tank` (raidz2, 8× NVMe): ONLINE, **2 CKSUM errors on `nvme7n1`**, boot-time resilver of 638 GB on 2026-09-05 14:26 (box rebooted at 14:26; nvme7 came up late/dirty). No data errors, 58% full. **No scrub since 2026-04-12** — the Debian `zfsutils-linux` second-Sunday cron scrubbed `zroot` on 08-09 but not `tank`; cause unknown (`zpool history tank` shows trims monthly, last scrub 04-12).
|
||||
- No `nvme-cli` or `smartctl` on the box → nvme7's media-error counter unread.
|
||||
- `zroot` at **91%** (345 G of 379 G): `docker system df` = images 429 GB (204 GB reclaimable), build cache 74 GB (36 GB reclaimable).
|
||||
- pfi-pve `NASPool` 7% / `ospool` 19%, scrubbed 09-05 / 08-09, clean.
|
||||
|
||||
**The three actions, in order:**
|
||||
1. `sudo zpool scrub tank` on ana-ml2 (1–2 h of extra I/O; seats keep serving) → on a clean pass `sudo zpool clear tank`; if the scrub finds errors on nvme7n1 → replace path.
|
||||
2. `sudo apt install nvme-cli` → `sudo nvme smart-log /dev/nvme7` (media_errors, critical_warning, percentage_used) and `nvme id-ctrl` for model/serial; record in the drive inventory.
|
||||
3. `docker image prune -a`? NO — `docker image prune` (dangling only) + `docker builder prune` on ana-ml2; the 47 unused-but-tagged images need a look first (some are rollback seats: e.g. `vllm/vllm-openai:v0.26.0`, nightlies). Target: zroot back under ~75%.
|
||||
Also worth a look while there: why the scrub cron skips `tank` (`/usr/lib/zfs-linux/scrub` logic — it skips pools with an active trim/resilver or those not "healthy"?).
|
||||
Reference in New Issue
Block a user