memory: snapshot — Anaheim tunnel at 25% of circuit; selene retired; hrafn CI fixed
Session captured for a context reset. Six new detail entries. THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization; execution pending, untracked by operator choice. Also recorded: selene retired after losing a head-to-head on its own job with chat-judge moved to gen and the model name left to 404; the 7-alias collision on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and its CI found to have been reporting green while deploying nothing for its whole life; all three Worldtree instances de-armed from a 69-day-stale :latest and the Matrix homeserver re-plumbed to personal; every secret-bearing .env on ana-docker tightened to 0600; the pfi org closing the repo-creation half of the credential-migration directive; booth kept-board deletion and link pruning. Two entries under Tried and abandoned: the CI checkout assertion that broke the pipeline twice and was removed, and my proposal to alias a retired model name at a different model, which the operator correctly overruled. Index 271 -> 282 lines, under the 300 cap, so no archival fired.
This commit is contained in:
@@ -0,0 +1,34 @@
|
||||
# [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600
|
||||
|
||||
Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live
|
||||
bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**.
|
||||
Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with
|
||||
four interactive accounts, verified as real exposure by reading one as `nobody`.
|
||||
|
||||
Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest,
|
||||
searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other
|
||||
stacks already used 0600, so this converged on the existing house pattern rather
|
||||
than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable
|
||||
by `nobody`.
|
||||
|
||||
Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable.
|
||||
|
||||
## The check that matters
|
||||
|
||||
Every run asserts `docker compose config` still renders **as the deploy user**
|
||||
(`lkraven`), not as root. Checking the mode proves the bits changed; only rendering
|
||||
as the deploy user proves the next deploy can still resolve its variables.
|
||||
|
||||
## Two gotchas recorded in the playbook
|
||||
|
||||
- **vaultwarden looked like it bind-mounted its `.env`** — which would mean the
|
||||
*container's* UID reads it and 0600 could break the password vault. It does not:
|
||||
that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list
|
||||
item without checking its parent key. The playbook now **refuses** any stack that
|
||||
genuinely bind-mounts its `.env`, since that case is read by the container UID.
|
||||
- **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to
|
||||
`lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev`
|
||||
was repointed to infra-ops at some point and `ana-docker` was not. Run elway against
|
||||
the infra-ops target on this host.
|
||||
|
||||
Commit `a896c0a`.
|
||||
@@ -0,0 +1,71 @@
|
||||
# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
|
||||
|
||||
The operator noticed site-to-site transfers were slow for a datacenter fiber
|
||||
handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
|
||||
circuit is fine.
|
||||
|
||||
## Measured
|
||||
|
||||
```
|
||||
ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering
|
||||
ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s
|
||||
FortiGate's own recorded peak 554 Mbit/s
|
||||
ANA <-> NH3, single stream 227 Mbit/s
|
||||
ANA <-> ESH, single stream 249-265 Mbit/s
|
||||
ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream
|
||||
```
|
||||
|
||||
Method: stdlib TCP probe (no ssh, no crypto, no compression) between site
|
||||
endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner
|
||||
Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.
|
||||
|
||||
## What it is not
|
||||
|
||||
- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw
|
||||
(`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard
|
||||
on ana-wg is remote-access only and is not in this path. Traceroute confirms:
|
||||
both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast
|
||||
ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it.
|
||||
- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all
|
||||
8 cores** during the tests, and both live tunnels report `npu_flag=03` with
|
||||
`dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded.
|
||||
- **Not a 250 Mbit/s cap.** That was the first number and it is misleading —
|
||||
single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
|
||||
- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions.
|
||||
|
||||
## Most likely cause
|
||||
|
||||
Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14,
|
||||
IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
|
||||
faster on **AES-GCM**, which combines encryption and authentication in one
|
||||
pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
|
||||
not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
|
||||
shortfall is consistent with that.
|
||||
|
||||
## Not executed
|
||||
|
||||
Changing the proposal is a **production-edge change requiring a matching
|
||||
change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while
|
||||
it renegotiates. Left for the operator. See the index entry for authorization
|
||||
state.
|
||||
|
||||
## Immediate mitigation, no config change
|
||||
|
||||
Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a
|
||||
single stream through IPsec gets 227. **Anything moving bulk data across the
|
||||
Anaheim link should parallelise** — that alone roughly doubles throughput
|
||||
today.
|
||||
|
||||
## Practical consequence already observed
|
||||
|
||||
`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
|
||||
from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and
|
||||
not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]].
|
||||
|
||||
## Access note
|
||||
|
||||
ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent
|
||||
on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
|
||||
vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
|
||||
prints live ESP session keys** — never paste its output into althing, a
|
||||
booth, or a commit.
|
||||
@@ -0,0 +1,63 @@
|
||||
# [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing
|
||||
|
||||
`hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites
|
||||
(Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for
|
||||
uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach
|
||||
`http://hrafn:8080`. Canonical at `stacks/hrafn/`.
|
||||
|
||||
## Intake found a live credential exposure
|
||||
|
||||
`/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token**
|
||||
— verified as real exposure by reading it as `nobody` on a box with four
|
||||
interactive accounts. Tightened to 0600. That triggered the wider sweep (see
|
||||
[[2026-08-23-ana-docker-env-perms-sweep]]).
|
||||
|
||||
## The CI defect — the one worth remembering
|
||||
|
||||
I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync,
|
||||
tagging the image with the commit SHA for provenance. nevermore-claude later found
|
||||
v1.0.0 deploying "green" while the host still served 0.1.0.
|
||||
|
||||
**Root cause was mine and nastier than either hypothesis.** The staging dir was
|
||||
`$compose_dir/.stage` — **inside** the rsync target. So
|
||||
`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the
|
||||
destination (absent from the source listing) **during** the transfer, destroying
|
||||
its own source mid-copy. Reproduced exactly:
|
||||
|
||||
```
|
||||
before: app.py="OLD" leftover.txt .stage/app.py="NEW"
|
||||
after: app.py="OLD" leftover.txt GONE, .stage GONE
|
||||
```
|
||||
|
||||
Deletion succeeded, the copy silently did not, rsync exited 0. So the directory
|
||||
*looked* converged while host source stayed frozen at the first manual rsync —
|
||||
and because the build's `COPY` inputs never changed, Docker full-cache-hit and
|
||||
every SHA tag aliased one image. **The provenance the tagging existed to provide
|
||||
was false for the pipeline's entire life.**
|
||||
|
||||
**The real failure is the verification.** The verify steps asserted the marker,
|
||||
container health, and a 200 from `/readyz` — all of which pass against a
|
||||
completely frozen host. None measured *content*. A deploy that reports success
|
||||
without asserting the bytes changed is verifying an **uptime**, not a deploy.
|
||||
|
||||
## Fixes
|
||||
|
||||
- stage at `/tmp/hrafn-deploy-stage`, outside the target
|
||||
- CI computes `context_sha256` over the shipped file list; the playbook recomputes
|
||||
it **on the host after the converge** and fails on mismatch
|
||||
- compare the running container's `src/**/*.py` against the host's, so a SHA tag
|
||||
cannot name layers the image lacks
|
||||
- **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the
|
||||
image and `__pycache__` appears at runtime, so a naive `find src -type f` compare
|
||||
false-fails on every healthy deploy. Verified against a known-good container
|
||||
before shipping (12 host files, 18 in container, 0 content differences).
|
||||
- declined `--no-cache`: a cache hit is *correct* when the context is genuinely
|
||||
unchanged; assert the property rather than brute-force it.
|
||||
|
||||
## Access
|
||||
|
||||
Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the
|
||||
pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is
|
||||
canonical; `stacks/hrafn/ci/` is a verified mirror.
|
||||
|
||||
Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`.
|
||||
@@ -0,0 +1,81 @@
|
||||
# [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat
|
||||
|
||||
## Why selene went
|
||||
|
||||
Benchmarked against `gen` on selene's own job — 24 designed judge items with
|
||||
checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a
|
||||
neutral JSON prompt and Selene's **native Atla template** (288 calls, free local).
|
||||
|
||||
```
|
||||
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
|
||||
native Atla selene 21/24 (88%) gen 22/24 (92%)
|
||||
```
|
||||
|
||||
gen won on both templates and **selene's best sat below gen's worst**. Selene was
|
||||
given its own fine-tuned template as a fairness check before any recommendation;
|
||||
it gained one point, not three.
|
||||
|
||||
**Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a
|
||||
winner on every equivalent pair. For eval work that is the case that matters.
|
||||
|
||||
brokkr-smithy-dev independently corroborated from the other end with a **null
|
||||
control** (an excerpt compared against ITSELF, where tie is definitional):
|
||||
`chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth
|
||||
recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96
|
||||
understated it: on a *ranking* task selene was a coin flip. Absolute scoring on
|
||||
designed items is an easier task than ranking real text — the harness is a
|
||||
**screen, not a verdict**, and its README says so.
|
||||
|
||||
Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB).
|
||||
|
||||
## The naming rule, restated the hard way
|
||||
|
||||
I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**:
|
||||
|
||||
> never repoint a named model at a different model's endpoint — that is
|
||||
> intentionally misleading
|
||||
|
||||
`chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and
|
||||
moved to gen with a deterministic judge profile copied from `image-judge`.
|
||||
`selene-1-mini-8b` is a **model** name and was removed outright — it now returns
|
||||
`HTTP 400 Invalid model name`, verified. The discriminator: *does the string
|
||||
promise a capability, or an identity?*
|
||||
|
||||
## The 7-way alias collision — the finding with the longest reach
|
||||
|
||||
```
|
||||
chat-judge classifier gen image-judge
|
||||
qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015
|
||||
```
|
||||
|
||||
Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`;
|
||||
`ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`.
|
||||
|
||||
**Cross-checking a result against another alias measures nothing when they are the
|
||||
same weights — agreement is an echo, not corroboration.** Documented at the head of
|
||||
`model_list` in the live gateway config, because it belongs where people read it.
|
||||
|
||||
This caught a real defect within hours: brokkr's R47 premium-corpus gate was about
|
||||
to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an
|
||||
independent second opinion. They pinned the backing model in the preregistration
|
||||
and dropped the second-alias idea instead.
|
||||
|
||||
## Provenance seam (brokkr's pushback, adopted)
|
||||
|
||||
The gateway returns the **alias** in the response `model` field, not the backing
|
||||
model — so a per-call guard catches a swap *during* a run and is blind to one
|
||||
*between* runs. **Role alias for routing, concrete model for provenance.**
|
||||
`GET :4000/model/info` with the shared key already exposes backing model +
|
||||
api_base; resolve at run start AND end and void on mismatch.
|
||||
|
||||
## Artifacts
|
||||
|
||||
- Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default
|
||||
would silently benchmark a retired seat).
|
||||
- `stacks/selene/` keeps compose + a README explaining the retirement.
|
||||
- Technique worth stealing, from brokkr: **a control constructed so the correct
|
||||
answer is DEFINITIONAL rather than judged cannot inherit the designer's error.**
|
||||
Item vs itself; response vs its own truncation; text vs its own clauses
|
||||
permuted. Add those before adding more judged items.
|
||||
|
||||
Commits `ca3c984`, `b8a5355`.
|
||||
@@ -0,0 +1,50 @@
|
||||
# [2026-08-23] `/mnt/smithy` mounted on ana-ml2 — read-only and SOFT, deliberately not matching nh3-dev
|
||||
|
||||
brokkr-smithy-dev asked for `10.100.50.50:/volume1/smithy` on ana-ml2 to run R47's
|
||||
CPU-bound corpus pipeline on 96 idle EPYC cores instead of one nh3-dev vCPU. Granted,
|
||||
with two deliberate deviations from what was requested.
|
||||
|
||||
```
|
||||
sudo mount -t nfs4 -o ro,soft,timeo=30,retrans=3,proto=tcp,vers=4.1 \
|
||||
10.100.50.50:/volume1/smithy /mnt/smithy
|
||||
```
|
||||
|
||||
The export already permitted ana-ml2 — no DSM change needed. Write is genuinely
|
||||
refused.
|
||||
|
||||
## Why soft, not hard
|
||||
|
||||
They asked to match nh3-dev's mount, which is `hard`. **nh3-dev is same-site as the
|
||||
NAS; ana-ml2 is not** — this is cross-site NFS on the box running the fleet's
|
||||
inference seats. A hard mount turns a link blip into unkillable D-state, and this
|
||||
fleet has already lost a host that way (esh-docker-vm; only fix was a reboot). Soft
|
||||
returns EIO, the batch job fails, you rerun it. The soft-mount corruption caveat is a
|
||||
**write** hazard and this is read-only. Mirrors the existing ESH books mount.
|
||||
|
||||
## Why not in fstab
|
||||
|
||||
Manual only, matching irv-ml1's `/mnt/smithy` precedent. A cross-site NFS entry in
|
||||
fstab can hang boot on a GPU host with 71 days uptime. **Needs remounting after a
|
||||
reboot.**
|
||||
|
||||
## The performance reality, measured on the same file through the same mount
|
||||
|
||||
```
|
||||
ana-ml2 (cross-site) nh3-dev (same-site)
|
||||
sequential read 24.7 MB/s 98.3 MB/s
|
||||
small-file rate 45.3 files/s 34.6 files/s
|
||||
```
|
||||
|
||||
Two different stories, and file layout decides which you get:
|
||||
|
||||
- **Many small records -> ana-ml2 wins on BOTH axes.** That path is bound by per-file
|
||||
round-trips and NAS overhead, not bandwidth, and ana-ml2 is an idle 96-core box
|
||||
while nh3-dev is a loaded 16-vCPU VM.
|
||||
- **Bulk sequential streaming -> the link eats the win.** 4x read penalty against a
|
||||
6x CPU gain. `datasets/raw` is 126 GB, `datasets/derived` is 1.7 GB — which one the
|
||||
pipeline traverses changes the answer by two orders of magnitude. Staging a subset
|
||||
to ana-ml2 local disk (195 GB free) beats pulling it over the wire repeatedly.
|
||||
|
||||
The 24.7 MB/s is the Anaheim tunnel, not NFS and not the NAS — see
|
||||
[[2026-08-23-anaheim-ipsec-tunnel-ceiling]]. No mount tuning will move it;
|
||||
parallelism will.
|
||||
@@ -0,0 +1,70 @@
|
||||
# [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed
|
||||
|
||||
## b187 pre-stage (#405 phases 1+2)
|
||||
|
||||
The matrix bridge stopped embedding the engine and became an HTTP client of the
|
||||
Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the
|
||||
container env, the bridge exits by design. Demo's compose never passed it; the next
|
||||
recreate would have crash-looped. Pre-staged on demo and personal (additive, backed
|
||||
up, verified with `docker compose config`, nothing restarted).
|
||||
|
||||
**Key decision, and I got its scope wrong first.** I argued demo should stay keyless
|
||||
(no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about
|
||||
scope**: the engine preflight authenticates at boot regardless of homeserver, so demo
|
||||
booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's
|
||||
recorded authorization, vaulted, wired, three-hop hash-verified.
|
||||
|
||||
## The 69-day-stale `:latest` landmine
|
||||
|
||||
All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while
|
||||
running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built
|
||||
**2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently
|
||||
downgraded that service by 69 days. Same footgun as the 2026-06-15 outage.
|
||||
|
||||
Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering
|
||||
compose config rather than reading `.env`, containers untouched. Playbook at
|
||||
`playbooks/repin-worldtree-image.yaml`.
|
||||
|
||||
**`worldtree-pinned` was the worst case:** the instance whose entire purpose is being
|
||||
frozen was running a **dangling image with no repo tags**, kept alive only by the
|
||||
running container. One `docker rm` from garbage collection. Tagged
|
||||
`:446e5807bf43` first, then pinned.
|
||||
|
||||
The guard I wrote had two bugs the pinned case exposed: it compared the container's
|
||||
`.Config.Image` **string** (only the tag it was CREATED from — pinned was created
|
||||
from `:latest` back when that meant 446e5807), and it reported CHANGED
|
||||
unconditionally. Now compares **image IDs** and skips when already correct.
|
||||
|
||||
## Matrix homeserver ownership
|
||||
|
||||
Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never
|
||||
missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed
|
||||
instances had them at length **zero**. Someone wired four of six Matrix vars and
|
||||
stopped. Wired them into personal, three-hop verified.
|
||||
|
||||
**The trap worth remembering:** Synapse's registration pointed at
|
||||
`http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at
|
||||
512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is
|
||||
**DEMO's** bridge, and Synapse can reach both — it would have connected, delivered,
|
||||
and looked correct while routing the operator's live rooms to the demo instance.
|
||||
**Personal's bridge is :8010.** `docker port` is ground truth.
|
||||
|
||||
Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified
|
||||
`GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's
|
||||
smoke passed first try: room created, mimir accepted the invite, a real engine turn
|
||||
ran, mimir replied in persona voice. #408 closed.
|
||||
|
||||
## Open on worldtree-dev's side
|
||||
|
||||
- **#411** — personal's bridge logs `Debug sink init failed: Permission denied:
|
||||
/app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist
|
||||
their IDs, so **every restart mints a fresh pair on the live homeserver**. Room
|
||||
litter that compounds silently. Needs a which-container-writes-what check on the
|
||||
sessions volume before anyone chowns it.
|
||||
- Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not
|
||||
listed by the engine on either instance.
|
||||
- Historical Domari pairwise verdicts from the selene era are coin-flip-grade
|
||||
(see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so
|
||||
no future arc leans on them without re-judging.
|
||||
|
||||
Commits `064181a`, `bb19a96`.
|
||||
+20
-9
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-08-22_
|
||||
_Last updated: 2026-08-23_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -108,24 +108,32 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-08-22 — three AI seats live on ana-ml2; `sec` is the one that moved this session._
|
||||
_As of 2026-08-23 — a long multi-party ops session; the one thing waiting on the operator is the Anaheim tunnel cipher._
|
||||
|
||||
- **SEAT MAP.** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, 7 aliases, **still on the OLD nightly `311b3513` with MTP k=3**. **`char-rp`** = MeroMero-v2 dual-mode (prose + streaming CoT, one weight set, two aliases), GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC, GPU1 :8019 — **rebuilt this session, see below**.
|
||||
- **⏳ THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit.** Circuit measured at **2,153 Mbit/s**; the tunnel ceilings ~550 Mbit/s aggregate, ~230 single-stream. FortiGate CPU **idle**, IPsec **NPU-offloaded**, interface error-free — so it is not crypto exhaustion. Both tunnels negotiate **`aes256-sha1`**; **AES-GCM is the proposed change** and the operator has signalled he will authorize it. Not executed: production edge, needs a matching change at NH3 + ESH, each tunnel drops during renegotiation. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
|
||||
- **🟢 `sec` NOW RUNS DFLASH2 ON A NEWER vLLM — promoted to its compose stack after real-use testing.** `nightly-e9d1398d` (+259 commits over production, `behind_by=0`), `dflash` k=7 with the 3.85 GB drafter, **util 0.52 / max-model-len 420,000 / KV ~453k**, 2048² vision. `restart: unless-stopped`, survives reboot. Canonical in `stacks/mog-sec/` with a fully-commented `.env.example`. **ROLLBACK:** `.env.bak-pre-dflash2-20260822` on the host, or swap `MOG_SPEC_CONFIG` + `MOG_IMAGE`. ⚠ `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` was **deliberately dropped** — the validated container never had it.
|
||||
- **🟢 SEAT MAP (unchanged this session except selene).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **now 7 aliases** (see the collision note). **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019. **`selene` RETIRED** — 17.2 GiB reclaimed on GPU1 (free now ~19.4 GiB).
|
||||
|
||||
- **⚠️ THE `sec` DEGENERATION QUESTION IS OPEN AND CONFOUNDED.** It no longer degenerates, but **engine and drafter changed together**. **The isolating experiment is MTP k=3 on `e9d1398d`** — not yet run. Operator ruling: the degeneration lives in the **un-fixed vLLM**, not the weights; my MTP-head hypothesis is **retracted**. ⚠⚠ **Both the "degenerates at 2k" and "coherent to 10k" sightings are n=1 from uncontrolled sessions and are NOT evidence.** Production is **172 commits behind GDN spec-decode fix #53077**; `#51113` is present in both builds and is **necessary but insufficient**.
|
||||
- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Engine and drafter changed together; **the isolating experiment is MTP k=3 on `e9d1398d`** — still not run. Operator ruling stands: degeneration lives in the **un-fixed vLLM**, not the weights; the MTP-head hypothesis is **retracted**. Both prior sightings are n=1 and are NOT evidence. `gen` remains on the old nightly, untouched, gated on that experiment.
|
||||
|
||||
- **⏳ `gen` IS UNTOUCHED and still on the old build.** If DFlash2 + the newer engine are the answer, gen is the obvious next beneficiary — but that decision is gated on the isolating experiment above, not on sec's n=1 result.
|
||||
- **🟢 ana-ml2 now mounts `/mnt/smithy`** (nh3-nas) **ro + soft, NOT in fstab** — needs a manual remount after reboot. For brokkr's R47 CPU work. Reads 24.7 MB/s sequential vs 98.3 on nh3-dev (that gap is the tunnel above), but **45 files/s vs 34** — small-file work is genuinely faster there. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
|
||||
|
||||
- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` (SSID `PVC`) and `esh-server` from a delegated `2607:73c0:402:1d00::/56`; hosts egress over v6 as themselves, un-NATted. **v4 remains CGNAT (`100.104.3.250`) and a full gateway reboot proved the purchased static is NOT provisioned** — carrier ticket, nothing left to try locally. v6 firewall audited: default-deny inbound both versions, correct. NH3 stays v6-off deliberately (single /64 reserved for meshing). Flat-zone lateral-movement finding **parked, id 44**.
|
||||
- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` and `esh-server` from a delegated `/56`. v4 remains CGNAT and a full gateway reboot proved the purchased static is **not provisioned** — carrier ticket, nothing left locally. NH3 stays v6-off deliberately. Flat-zone lateral-movement finding **parked, id 44**.
|
||||
|
||||
- **🟢 OTHER SERVICES.** speaches ASR live irv-ml1:8204 (Eyra; loop closed). Open WebUI on esh-docker-vm:3211 (admin creds + admin-scoped API key vaulted; **Lobe retirement still the operator's call**). Waterland, Homepage/Skyfall, fleet `.internal` DNS all landed earlier and are stable.
|
||||
- **🟢 OTHER SERVICES.** `hrafn` browser-fetch adopted on ana-docker (infra-ops owns uptime; CI now genuinely deploys). speaches ASR live irv-ml1:8204. Open WebUI esh-docker-vm:3211 — **Lobe retirement still the operator's call**. Booth gained kept-board deletion + per-row link pruning. `pfi` gitea org created; claude-bot is an Owner and can create repos self-serve.
|
||||
|
||||
- **⏳ OPEN:** the MTP-k3-on-new-build isolating experiment; file the drafted upstream vLLM issue (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift (`:main` vs pinned v0.11.0); `/tank` DEGRADED **70+ days**; Brokkr duplicate `reranker-a3-bge-v2-m3` alias; **MANY commits unpushed** — push is the operator's call.
|
||||
- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; bridge/engine agent-roster drift on both WT instances; brokkr's `gen` vs trained-reward-model bake-off (theirs to initiate). **Working tree is clean and pushed** through `0ad332b`.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-08-23]` **Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit — not WireGuard, not CPU, not the fibre.** Cipher change proposed and operator-signalled; **execution pending, untracked by operator choice.** → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
- `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md`
|
||||
- `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md`
|
||||
- `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md`
|
||||
- `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`. → `persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md`
|
||||
- `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.)
|
||||
- `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink <id|index>` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`)
|
||||
|
||||
- `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md`
|
||||
- `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`)
|
||||
- `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`)
|
||||
@@ -256,6 +264,9 @@ _As of 2026-08-22 — three AI seats live on ana-ml2; `sec` is the one that move
|
||||
_209 older entries archived to archival-memory.md._
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.**
|
||||
- `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.
|
||||
|
||||
- `[2026-08-15]` **Grafted bf16 MTP loads UNINITIALIZED (0% accept) unless `re:^mtp.*` is in the quant-config `ignore`; and W4A16=Marlin (not native FP4) costs ~20% even on decode.** Cost a premature 79 GB delete of a good model (declared desync-dead off the 0%). Lessons: test MTP on bf16 FIRST, isolate before deleting; modelopt 0.43 is dependency-hell for qwen3_5 (list-vs-dict quant_cfg + transformers conflict) — use llm-compressor. Full → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
|
||||
|
||||
- `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
|
||||
|
||||
Reference in New Issue
Block a user