memory: snapshot — the ops log, and a day spent on instruments that report without looking
Archived 15 entries (Recent decisions 14, Tried and abandoned 1) oldest-first to archival-memory.md; 5 held back on the open-deferred-work guard and 164 on the 14-day guard, so the index stays over the soft cap at 477 lines. An over-cap file that keeps live decisions beats a scannable one that lost a belayed item. Four new detail files cover the day: the ops log and its four self-inflicted failure modes, the Booth's two dead controls and the four-iteration layout probe, the Gitea org grant plus the dead claude-bot token that had been misreporting permissions, and the disk triage that rescued a LoRA adapter from a directory this box sweeps at three days. lv-mccarthy's run outcome remains unverified after two days and is the first line of the in-flight section and step 1 of the handoff.
This commit is contained in:
@@ -1,41 +0,0 @@
|
||||
# `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour
|
||||
|
||||
`vllm-embed` had OOM-crashed **7 times** (`RestartCount 7`): GPU1 was at **216 MiB free** of
|
||||
97,887, and a ~3.4 GB tenant sharing a card with five other seats dies when it cannot get another
|
||||
96 MiB mid-inference. Root cause of the `qwen3-embed` 500s brokkr saw — **not** the LiteLLM
|
||||
gateway restart he attributed them to (different host, different component, 50 min earlier, and
|
||||
six of the seven crashes predate it).
|
||||
|
||||
**Operator ruling:** move `gen` to GPU0. *"we were keeping it free for training, but we don't have
|
||||
the power for sustained load."* GPU0 was never free — `mog-sec` has been there at util 0.52
|
||||
(55,126 MiB) since the August move.
|
||||
|
||||
## ⚠ THE SIZING ERROR, WHICH IS THE POINT OF THIS ENTRY
|
||||
|
||||
gen at util 0.43 wants 42,091 MiB GPU0 free 42,113 margin 22 MiB — would not start
|
||||
gen at util 0.41 wants 40,134 MiB margin 1,979 MiB — I called this safe. IT WAS NOT.
|
||||
result 1,548 MiB free, 56 FlashInfer autotuner OOM-fallbacks per 3 min
|
||||
gen at util 0.38 34,558 MiB actual 4,466-7,548 MiB free ZERO OOM events over 6 min
|
||||
|
||||
**`gpu-memory-utilization` governs vLLM's declared weights+KV budget. It does not cover what the
|
||||
process actually needs at runtime** — FlashInfer JIT autotuner workspace, CUDA graphs, expandable
|
||||
segments all allocate on top. My "1,979 MiB margin" was against the declared budget, so the seat
|
||||
came up and then silently fell back to slower kernels for want of 20 MB chunks.
|
||||
|
||||
⚠ **Same class of error as the DAC plateau the same day: checked a proxy, reported it as the
|
||||
thing.** The fix was to *measure* — count OOM-fallback events before and after — not to re-derive.
|
||||
|
||||
## Final state and what it cost
|
||||
|
||||
GPU0 mog-sec 55,126 + gen 34,558 = 89,701 / 97,887
|
||||
GPU1 50,507 / 97,887 — 46.7 GB free, was 216 MiB
|
||||
|
||||
`gen` KV cache is now **348,515 → 268,205 tokens, 1.02x concurrency at its 262,144 max-model-len**
|
||||
— it holds exactly one full-context request. Short/medium requests still batch; long-context
|
||||
throughput effectively serialises. The lever to reclaim it is `MOG_GPU_MEM_UTIL 0.52`, untouched.
|
||||
|
||||
.env.bak-preGPU0-20260904-164032 the GPU move
|
||||
.env.bak-preShrink-165133 the utilization
|
||||
|
||||
⚠ `/opt/docker/compose/*/.env` is root-owned — `docker compose restart` needs **sudo** on
|
||||
ana-docker/ana-ml2 or it fails with a bare `permission denied` on the .env.
|
||||
@@ -1,46 +0,0 @@
|
||||
# `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early
|
||||
|
||||
The ESH-Media ↔ UDM uplink is a 3 m OEM `SFP-H10GB-CU3M` twinax that negotiated **1000 Mbps**
|
||||
with **zero errors** on both ends. Diagnosis: `sfp_compliance: Unknown` — the switch reads the
|
||||
EEPROM but cannot parse the compliance codes on a third-party cable wearing Cisco coding, so
|
||||
autoneg falls back to the safe rate. Control case: a TP-Link `TL-SM5220-1M` on the adjacent UDM
|
||||
port runs 10G with the same speed_caps, exonerating the port, firmware and autoneg.
|
||||
|
||||
Operator authorised forcing it. Forced `autoneg False / 10000 / full` on the **UDM end only**;
|
||||
ESH-Media followed to 10000 unforced — proof the cable was electrically *capable*.
|
||||
|
||||
## ⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG
|
||||
|
||||
10:03 200 errors link-up burst
|
||||
10:48 221 +21 in 42 min — I reported this as "flat", it was not
|
||||
14:38 416 +195 over 4 h, plus user-visible flapping the operator felt
|
||||
|
||||
**A marginal link declares itself over hours, not minutes.** I watched a two-minute flat window
|
||||
and reported a plateau; the counter simply had not moved yet. The operator noticed the flapping
|
||||
before the soak I left running had accumulated enough to raise it. Reverted 14:38, both ends back
|
||||
to autoneg/1000, stable.
|
||||
|
||||
⚠⚠ **The correction that matters: a clean, zero-error link at 1G does NOT rule out a marginal
|
||||
cable.** It only proves the cable is clean *at 1G*. Clean at 1G and marginal at 10G is exactly
|
||||
what a cheap 3 m twinax is — which means the platform's autoneg fallback was **protecting
|
||||
something real**, not being fussy about vendor coding. The coding explains the negotiation; it
|
||||
does not explain errors once forced. I conflated the two.
|
||||
|
||||
⚠ **Do not re-force this port.** The fix is the cable.
|
||||
|
||||
## Method notes worth keeping
|
||||
|
||||
- **Force the RECOVERABLE end.** ESH-Media reaches the controller *through* this link, so a
|
||||
failed force there strands the switch behind a dead uplink — a physical visit. The UDM end is
|
||||
safe because the path to the controller (`nh3-dev → 10.100.10.1 → 10.0.0.1`, site VPN) does not
|
||||
cross it. Verified with `traceroute` **before** the change; revert payload written before the
|
||||
forward one.
|
||||
- ⚠ `port_overrides` is a **whole-array PUT** — the array was diffed to prove exactly one field
|
||||
on one port changed before sending, and read back after.
|
||||
- Blast radius named before acting: ESH-Media backhauls the operator's office switch, a WiFi AP
|
||||
and the Zigbee coordinator.
|
||||
|
||||
Resolution: the operator already owns a replacement and ran the copper himself through a drilled
|
||||
floor 2x4. Recommendation given was **fiber** (10Gtek SR 2-pack + OM4 3 m LC-LC, ~$40-65) because
|
||||
the cable-vs-pull-damage question was never resolved and a second DAC through the same hole risks
|
||||
the same fault. Runbook `docs/runbooks/esh-media-dac-10g.md`; commits `b9988a7`, `514ce7a`, `054c098`.
|
||||
@@ -1,33 +0,0 @@
|
||||
# `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN
|
||||
|
||||
`10.0.50.50` is **CT 103 on esh-pve-nas**, a hand-rolled Debian NAS (not a Synology). Created an
|
||||
SMB account for the operator's Windows AudioGridder DSP box:
|
||||
|
||||
username dsp uid 999, /usr/sbin/nologin, no home — SMB ONLY, cannot log in anywhere
|
||||
password vaulted at esh-nas/dsp-smb-password (round-trip verified by sha)
|
||||
verified authenticates, sees all 8 shares, WRITE to //share confirmed (mkdir/rmdir)
|
||||
|
||||
⚠ `smbclient` is absent from the NAS, nh3-dev and esh-docker-vm — verification ran in a throwaway
|
||||
`alpine:3.20 --network host` container with `samba-client`, leaving nothing installed.
|
||||
|
||||
Shares are registry-defined (`registry shares = Yes`), so they are invisible in `smb.conf` — use
|
||||
**`testparm -s`**, not grep, or you will conclude there are no shares.
|
||||
|
||||
## ⚠⚠ THE EXPOSURE, UNACTIONED — operator has not ruled
|
||||
|
||||
**NFS: twelve exports, `rw` to all of `10.0.0.0/8`, `sec=sys`, no authentication.**
|
||||
|
||||
/mnt/{backup,books,compose,documents,iso,media,music,share,
|
||||
pvestore,nvme-pvestore,ssd-pvestore,tank-vmbu}
|
||||
|
||||
`sec=sys` means the NAS trusts whatever uid the client claims. **Every VLAN at ESH — IoT,
|
||||
cameras, guest — can mount the NAS read-write today.** `/mnt/backup` additionally has
|
||||
`all_squash,anonuid=2000` so every client collapses to `nas_user`.
|
||||
|
||||
**SMB: every share except `backup` is `guest ok = Yes` and writable**, with
|
||||
`map to guest = Bad User` — an unknown username lands as guest with write access.
|
||||
|
||||
So the `dsp` credential is auditable and survives guest being turned off, but it is **not what is
|
||||
gating access today**. Hardening (narrow exports to `10.0.50.0/24` + named hosts, drop `guest ok`
|
||||
where unneeded) was offered and is roughly an hour; it would break anything relying on guest,
|
||||
which is why it needs the operator's say-so. **Not actioned. No tracking surface beyond this entry.**
|
||||
@@ -1,52 +0,0 @@
|
||||
# `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression
|
||||
|
||||
Run 3c is the lr **1e-05** arm against run 3's 2e-04; corpus, base, mask and every other
|
||||
hyperparameter byte-identical. It exists to answer one question: does a gentler learning rate
|
||||
keep run 3's diversity benefit while dropping its safety regression? **Answer: no, and the
|
||||
failure is the informative kind.**
|
||||
|
||||
trained 604/604 in 11:55:40 on pfi-gx10 (GB10, aarch64, sm_121), 69.8 s/it
|
||||
merged /home/infra-ops/erp-tune/serve/merged-run03c, bf16 51.6 GB
|
||||
served erp-tune-v3c on :8098, exposed as gateway alias `trial`
|
||||
|
||||
## The verdict (brokkr-smithy-dev, run03-gate/RESULT-run03c-gate.md)
|
||||
|
||||
CSAM 175 gens x 3 arms — ZERO hits, hard-line guard passed everywhere
|
||||
selfharm/methods base 10/10 refused -> 3c 0.6 = -40pp de-refusal
|
||||
meta pooled 0.0 delta — a CANCELLATION, not cleanliness: selfharm -40pp
|
||||
offset by advice/medical +40pp
|
||||
diversity attractor primary returned to BASE level — the gain is GONE
|
||||
|
||||
cross-arm run 3 (lr 2e-04) selfharm -44pp diversity: real gain
|
||||
run 3c (lr 1e-05) selfharm -40pp diversity: none
|
||||
|
||||
⚠ **The selfharm de-refusal is LR-ROBUST.** A 20x cut moved it 4pp while erasing the diversity
|
||||
benefit entirely. It comes from corpus content and imprints at even gentle exposure — it is not
|
||||
something a lower learning rate dials out. That is what the LR sweep was run to find out.
|
||||
|
||||
⚠ **A pooled preserve-list test cannot see a single-axis collapse.** The pooled operational
|
||||
delta reads 0.0 because two axes moved 40pp in opposite directions. Same structural defect that
|
||||
let run 3's gate pass — recorded as R47 §8 item 11.
|
||||
|
||||
## What the port proved about the box
|
||||
|
||||
- **The tooling loads on aarch64/sm_121.** Full training stack plus flex_attention and the
|
||||
chunked-loss path. Nothing exotic needed beyond `python3-dev`.
|
||||
- **Verification that earned its keep**: both 49 GB base shards sha256-matched ana-ml2's, and a
|
||||
full encode was run into a throwaway dir and compared byte-for-byte — 197,360,233 B, sha256
|
||||
`c08bb1fe2ecb0be3`, identical. transformers 5.15.1→5.16.1 and x86-64→aarch64 are *measured*
|
||||
inert, not assumed.
|
||||
- ⚠ **The encode-cache FILENAME differs by design** — `base_model_path` is in the cache key, so
|
||||
rehoming the base changes the key while content stays identical. Input hash, not output hash.
|
||||
Do not read it as drift; do not "fix" it by faking `/tank` on the GX10.
|
||||
|
||||
## The lora_B signal worth carrying forward
|
||||
|
||||
run 2 (lr 2e-04) min 0.6826 median 1.7212 max 3.7573
|
||||
run 3c (lr 1e-05) min 0.0480 median 0.1561 max 0.4133
|
||||
|
||||
~11x gentler across the board — exactly what a 20x LR cut should produce. A consistency check
|
||||
passing, not a red flag, but worth in hand *before* reading diversity numbers: "the tune did
|
||||
nothing" and "the tune did less on purpose" look alike in the output.
|
||||
|
||||
Runbook `docs/runbooks/gx10-run-03c.md`; commit `dae77ee`.
|
||||
@@ -1,39 +0,0 @@
|
||||
# `[2026-09-05]` A peer's "2.7x stack effect" was a coin flip — the operator caught it and the arithmetic is worth keeping
|
||||
|
||||
brokkr-smithy-dev reported that a diversity **noise floor** was 2.7x tighter on ana-ml2 than on
|
||||
the GX10 and ranked ana-ml2 the better instrument on it. I relayed it. **The operator rejected it
|
||||
on instinct** — *"that makes zero sense. except for speed, serving a model should be identical
|
||||
across servers"* — and he was substantially right.
|
||||
|
||||
ana-ml2 0.9688, 0.9574 spread 1.14pp mean 0.9631
|
||||
gx10 0.9583, 0.9895 spread 3.12pp mean 0.9739
|
||||
|
||||
between-box LEVEL difference 1.08pp
|
||||
ana-ml2's own replicate spread 1.14pp <- LARGER than the between-box gap
|
||||
pooled range ignoring box 3.21pp <- ~= the entire "GX10 floor"
|
||||
|
||||
**Each "floor" is `|block0 - block1|` from n=2.** A range over two draws is not an estimate of
|
||||
dispersion; the ratio of two such ranges is a ratio of two half-normals — a **half-Cauchy**.
|
||||
brokkr computed the tail himself on retraction: `P(ratio >= 2.7) = 1 - (2/pi)*arctan(2.7)`,
|
||||
doubled = **0.452**, confirmed by 400,000-draw simulation. He had reported a coin flip as a
|
||||
measured effect and ranked hardware on it. Retracted at `97f73dd`.
|
||||
|
||||
⚠ **The disconfirming evidence was inside his own sentence.** He wrote "the base level is nearly
|
||||
identical, it is the spread that shifts" and offered it as *reassurance*. A real stack effect
|
||||
moves the level. Level agreeing while a two-draw range differs is the signature of a noisy range
|
||||
estimator — he had the refutation in hand and read it as support.
|
||||
|
||||
⚠ **His own diagnosis of why it got through is the transferable part:** he had spent the session
|
||||
triaging *my* claims hard — reading the encode path, simulating preflight, recomputing shas from
|
||||
bytes — and this one was **his, and flattering**: it made his earlier work look prescient and
|
||||
produced a clean recommendation. **Asymmetric scepticism is one error twice, and the flattering
|
||||
direction needs the extra pass.**
|
||||
|
||||
**What survived, deliberately separated:** re-measuring the floor on whatever stack actually
|
||||
serves stays non-negotiable. The mechanism list is sound whether or not those four numbers show
|
||||
it — sm_121 vs sm_120 kernel selection, vLLM 0.28.0 against mixed 0.26/nightly, and separately
|
||||
measured batch-invariance (3.12pp at jobs=8, same order as the whole claimed effect).
|
||||
**Retracting the evidence and keeping the discipline are different acts.** Settling it properly
|
||||
wants several blocks per box and is its own probe, not a by-product of a gate.
|
||||
|
||||
See [[2026-09-05-vllm-on-sm121-and-run4]].
|
||||
@@ -1,53 +0,0 @@
|
||||
# `[2026-09-04/05]` vLLM RUNS on sm_121 — and the blocker was `ninja` not being on PATH, not the silicon
|
||||
|
||||
Long-standing open question closed: **vLLM 0.28.0 serves on the GB10 (aarch64, sm_121)** from a
|
||||
stock wheel, no source build. Model loads, `torch.compile` completes (~29 s), CUDA graphs capture.
|
||||
|
||||
⚠ **The one trap, and it looks exactly like an sm_121 kernel problem:** FlashInfer JIT-builds its
|
||||
sampling kernel at first use and needs **`ninja` on PATH**. It ships as a vLLM dependency but
|
||||
lives in the venv `bin/`, so the failure surfaces as `FileNotFoundError: 'ninja'` from deep inside
|
||||
a `profile_run` traceback. Same shape as the `python3-dev` trap that bit the training harness on
|
||||
this box, and the third present-but-not-on-PATH false-absence on this hardware (after `nvcc`).
|
||||
|
||||
Launch with **both** `~/vllm-env/bin` and `/usr/local/cuda/bin` on PATH. Harmless and expected:
|
||||
`Using default MoE config ... device_name=NVIDIA_GB10` — nobody has tuned MoE kernels for this chip.
|
||||
|
||||
⚠ **Open WebUI sends `tool_choice: "auto"` on every request**, which vLLM 400s unless launched with
|
||||
`--enable-auto-tool-choice --tool-call-parser gemma4`. Taken from the working sibling seat
|
||||
(`gemma4-charrp`), which is where the canonical flags live. **Deliberately did NOT copy
|
||||
`--reasoning-parser gemma4`** — that seat pairs it with `--default-chat-template-kwargs
|
||||
'{"enable_thinking": false}'`, and adding it alone moves output into `reasoning_content` with a
|
||||
null `content`, which OWUI renders as an empty reply. A 400 traded for a blank box.
|
||||
|
||||
## Run 4 — the corpus arm
|
||||
|
||||
Run 4 adds an **airoboros-3.2 instruct root** (7,229 rows) and displaces kvasir 38% → 18% context
|
||||
share, at run 3's lr 2e-04, everything else held byte-identical. Operator granted a run-scoped
|
||||
training-eligibility override `operator-2026-09-04-rnd-run4` (the SECOND grant; run-1's was
|
||||
one-run-scoped, a run 5 needs a third).
|
||||
|
||||
**Two peer artifacts were rejected before they cost GPU time, both by reading the harness rather
|
||||
than accepting a "confirm this":**
|
||||
|
||||
1. `sample_kind: "instruct-dialogue"` — `core.py:521-544` dispatches on the **literal string** and
|
||||
raises on anything outside `rp-dialogue`/`prose-chunk`/`actual-play-passage`. Would have
|
||||
hard-stopped the encode on the first airoboros row. brokkr had labelled it *non-blocking*.
|
||||
2. Recipe targets collapsed three dialogue roots into one descriptive label with `root_sha256:
|
||||
null` on three of four. Preflight resolves `roots_dir/<dataset_id>/clean-v1/CLEANROOT.json`
|
||||
literally and requires the sha.
|
||||
|
||||
⚠ **I did NOT fill the missing shas in myself**, though the values were known and it would have
|
||||
taken two minutes. `root_sha256` is the *author's* assertion and preflight exists to check the
|
||||
deployed root against it — supply both sides and the check verifies my copy-paste, an inert gate.
|
||||
brokkr then went further and recomputed his shas **from shard bytes** rather than reading them
|
||||
back out of the deployed CLEANROOT, which had the same defect one step removed.
|
||||
|
||||
**The kvasir prefix cut has three independent confirmations**: my build measured 1,613 samples /
|
||||
3,347,622 ctx = **47.4%** of run-3 kvasir, which reproduces the recipe's 18/38 share ratio
|
||||
(47.37%); the harness's own `[mix]` block then reported **ctx 0.1805** against the 18.0% target.
|
||||
|
||||
⚠ **`str.splitlines()` splits on U+2028**, which `json.dumps` leaves unescaped under
|
||||
`ensure_ascii=False`. The harness splitlines() in four places and four roots DO carry U+2028
|
||||
(fireball 8, c2-logs 31, kvasir 1, creative-writing 1) — but those are read by ordinary file
|
||||
iteration (safe), and the splitlines() paths touch only files written with the default
|
||||
`ensure_ascii=True`. Latent, one flag away from live, not acting. Found by brokkr the hard way.
|
||||
@@ -1,33 +0,0 @@
|
||||
# 2026-09-06 — Headscale cutover COMPLETE: all three site-pairs on the mesh
|
||||
|
||||
**DONE.** Operator disabled Site Magic in the UI; NH3↔ESH re-homed to a DIRECT mesh path (8ms, no DERP). Full 6-direction matrix OPEN. Site Magic disabled, both IPsec tunnels dormant, headscale is the sole active site-to-site transport. ana-wg WG fallback untouched. Tunnels re-enablable for backup.
|
||||
|
||||
Operator goal (/goal): replace Site Magic + IPsec with headscale, tunnels dormant as backup;
|
||||
"if paranoid, enable world-accessible SSH on the FortiGate first." Full detail + method +
|
||||
follow-ups in `docs/pfi/headscale-mesh-plan.md` § CUTOVER EXECUTED. Headlines:
|
||||
|
||||
- **colo↔NH3 and colo↔ESH IPsec = DORMANT; the mesh carries both, verified bidirectional.**
|
||||
NH3 UDM `pfi-nh3-ana` + ESH UDM `esh-ana` set enabled=false (API). Mesh /16 routes added on
|
||||
both UDMs and the FortiGate (→ ana-scale 10.250.50.45 / nh3-scale 10.100.50.46 / esh-scale
|
||||
10.0.50.65). Dependent flows OK over mesh: restic ESH→rest-server-ana, FortiGate mgmt.
|
||||
- **NH3↔ESH Site Magic NOT cut by API** — `sdwan-mesh-tunnel` = `api.err.NoEdit` (cloud
|
||||
orchestrated). Routes PRE-STAGED + shadowed; DERP path 9ms ready. **Operator disables it in
|
||||
the UniFi UI**, then the mesh takes over. Told the operator "mesh is online" → he does it.
|
||||
- **FortiGate WAN SSH safety net (TEMPORARY):** wan1 allowaccess ping+ssh; admin infra-ops
|
||||
trusthost2/3 = NH3 70.230.226.88 + ESH **128.177.138.182** (static since 09-08; was CGNAT 23.164.40.160) (not 0.0.0.0). Reach it at
|
||||
`ssh infra-ops@38.120.12.42`. Config backed up flash `pre-wan-ssh-cutover-20260906`. Remove
|
||||
when the edge (being replaced by OPNsense/R420) is retired.
|
||||
- ⚠ **Method lesson:** tunnel + mesh static route for the same /16 on one gateway = asymmetric
|
||||
drop. Disable the tunnel FIRST, then add the route. Broke colo once doing it tunnel-up; rolled
|
||||
back. See [[incident_crowdsec_cgnat_false_ban]] (same day) and the plan doc.
|
||||
- Dormancy = disabled+retained (flip UDM object back to enabled=true to restore); NO auto
|
||||
failover wired. Bonus: exit nodes → free multi-location egress proxy (parked).
|
||||
|
||||
|
||||
## Exit nodes (2026-09-06, operator-requested)
|
||||
All three routers advertise+serve exit nodes (approved). Clients pick location:
|
||||
`tailscale set --exit-node=nh3-scale|esh-scale|ana-scale`. NH3 = residential egress
|
||||
(70.230.226.88) → replaces the nh3-dev SOCKS5 proxy. Exit nodes + source preservation BOTH work via a selective-masquerade rule (NoSNAT kept true;
|
||||
`mesh-exit-masq.service` per router masquerades only internet-bound exit traffic, RETURNs fleet
|
||||
dests). Verified: colo sees real NH3 host; nh3-dev via colo exit → egress 38.120.12.42. A node advertising an exit node can't consume one — test from
|
||||
the laptop/iPad, not the routers.
|
||||
@@ -1,16 +0,0 @@
|
||||
# 2026-09-06 — Headscale overlay mesh: control plane + 3 subnet routers live, not cut over
|
||||
|
||||
Operator-directed (Headscale over NetBird; NH3 for the control plane, never the colo; 443
|
||||
direct; names nh3-headscale / nh3-scale / esh-scale / ana-scale). Full state, lessons and
|
||||
next steps in `docs/pfi/headscale-mesh-plan.md` § Status. Headline facts:
|
||||
|
||||
- `https://headscale.phasefinal.com` = CT 106 on nh3-pve (10.100.50.45), headscale v0.29.3,
|
||||
LE cert via TLS-ALPN-01, UDM forward tcp/443, DDNS timer on nh3-dev (user systemd).
|
||||
- Routers CT 107 nh3-scale / CT 108 esh-scale / CT 114 ana-scale advertise their /16s,
|
||||
approved, SNAT off, accept-routes OFF. nh3-dev enrolled as first client (100.64.0.4).
|
||||
- ⚠ Old tunnels (Site Magic, IPsec) are STILL the site-to-site path. The mesh currently
|
||||
rides inside them. Nothing has been disabled.
|
||||
- ⚠ Lesson: `--accept-routes` on a client before a return path for 100.64.0.0/10 exists
|
||||
black-holes that client's LAN (own-site /16 included). Return path first.
|
||||
- Pre-auth keys in the vault (`headscale/preauth-*-48h-20260906`, expire 09-08).
|
||||
- infra-ops user now exists on all four PVE hosts (needed `apt install sudo` first).
|
||||
@@ -0,0 +1,45 @@
|
||||
# `[2026-09-21]` The Booth gained blur and a closed keep round trip — after shipping two controls that did nothing
|
||||
|
||||
**Shipped** (2e7fd71, 271cb11, 751eecb, 07c9cb2): per-item cosmetic blur
|
||||
(`.blurred` marker, CLI `blur`/`unblur`, caption toggle, click-to-reveal, cover
|
||||
thumbs inheriting it), the ephemeral→kept `★` button closing a round trip that
|
||||
previously needed a shell, a direct `×` on kept cards, and in-booth
|
||||
keep/release with an open-redirect-safe `next`.
|
||||
|
||||
⚠ **BLUR IS NOT ACCESS CONTROL** and the code, docs and a test all say so
|
||||
deliberately. A blurred item is still served at its own URL, still in the zip.
|
||||
`test_blur_is_cosmetic_the_file_is_still_served` asserts the **200** on
|
||||
purpose: if someone later "hardens" it into a 403 that test fails, and it
|
||||
should — half-implemented access control is more dangerous than none.
|
||||
|
||||
⚠ **Two controls shipped INERT, both found by the operator, both by me reading
|
||||
templates instead of rendering them:**
|
||||
|
||||
- **The reveal button.** Its handler sat **after `{% endblock %}`**, which
|
||||
Jinja DISCARDS in a child template. The button rendered; the handler never
|
||||
reached the browser. Two commits and a README claimed click-to-reveal worked,
|
||||
and the suite passed throughout because nothing asserted against the SERVED
|
||||
page. Guards added and **confirmed to fail on reintroduction**.
|
||||
- **The kept-card `×`.** Both it and `release` were `position:absolute` on the
|
||||
same corner with independently guessed offsets; `release` is the later
|
||||
sibling so it won. Measured **30×22 px overlap on a 30 px button**, and
|
||||
`elementFromPoint` at the ×'s centre returned the release form. Unclickable
|
||||
from the moment it shipped. Replaced with one flex row positioned once.
|
||||
|
||||
⚠ **The blur feature itself was shipped twice having patched only SOME of
|
||||
booth.html's three item branches** (doc / media / other) — first the blurred
|
||||
class, then the toggle. The toggle is now ONE Jinja macro called from all three
|
||||
sites, and `test_every_item_kind_gets_exactly_one_blur_toggle` counts toggles
|
||||
against figures so a fourth branch cannot quietly skip it.
|
||||
|
||||
⭐ **`scripts/layout-probe.py`** exists because markup inspection structurally
|
||||
cannot see occlusion. It took **four iterations** to become trustworthy and the
|
||||
failures are the point: (1) `top.contains(el)` counted an ANCESTOR overlay as a
|
||||
hit — the exact case it exists to catch; (2) `elementFromPoint` is
|
||||
viewport-relative, so everything below the fold read as occluded; (3)
|
||||
`getBoundingClientRect()` on a WRAPPED INLINE element is the union of its line
|
||||
boxes, whose centre lands in the gutter, on the parent. Only the fourth version
|
||||
fires on a real overlay while staying silent on a clean page. **Both controls
|
||||
were run** — my first attempt at validating it was itself invalid.
|
||||
|
||||
See [[2026-09-21-ops-log-and-the-instruments-that-lied]].
|
||||
@@ -0,0 +1,41 @@
|
||||
# `[2026-09-21]` nh3-dev disk triage: 27 GB reclaimed, and a LoRA adapter rescued from a directory the box sweeps every 3 days
|
||||
|
||||
Beszel alerted root >85%. Reclaimed **7 GB** from regenerable caches (`uv cache
|
||||
prune`, npm, pip), then a deep dive found the real shape.
|
||||
|
||||
⚠⚠ **The headline was not disk.** `/tmp` held **19 GB** of Claude Code session
|
||||
scratchpads, and inside one of them sat the **`babyyarros` LoRA adapter** —
|
||||
252 MB, r=32/α=64 on Qwen3-4B-Instruct, with its loss series and provenance —
|
||||
**existing nowhere else**: not on `/mnt/smithy`, not under `~/development`.
|
||||
`/etc/tmpfiles.d/tmp.conf` sets `D /tmp … 3d`, an admin file from 2026-07-18
|
||||
that **overrides** the stock no-age rule, and the cleaner runs daily.
|
||||
|
||||
**Rescued** to `/mnt/smithy/adapter-rescue/babyyarros-20260921`, verified by
|
||||
content: sha256 `63fda6cc…` matching both the source and the `adapter.sha256`
|
||||
recorded at training time.
|
||||
|
||||
⚠ **Two corrections I made to myself during the dive, both worth keeping:** I
|
||||
alarmed that shutterchute's RAW deliverables were 19 hours from deletion — their
|
||||
mtimes were *that day*, a live session working. And I suspected my own `du`/
|
||||
`find` had reset the atime clock and manufactured the "0 would-remove" result;
|
||||
it had not (`relatime`, and an untouched comparison file still showed an old
|
||||
atime) — but it was right to check before trusting a number my own measurement
|
||||
could have created.
|
||||
|
||||
**Operator-authorized deletions:** 7.6 GB duplicate Qwen base shards, 12 GB
|
||||
`models-staging/retro-diffusion` (cold since 09 Aug), 7.5 GB `splat-assets`
|
||||
(cold since 15 Aug), 178 session dirs idle 7d+. **39 GB → 66 GB free, 84% →
|
||||
72%.**
|
||||
|
||||
⚠⚠ **MY PRUNE DELETED AN ACTIVE SESSION'S DIRECTORY.** `-mtime +7` on a session
|
||||
dir is an unsound liveness test: **a directory's mtime does not change when
|
||||
files are written into its subdirectories.** `dfacccde/` looked 7+ days idle
|
||||
while `dfacccde/tasks/` was being written continuously. Cost: one lost tool
|
||||
output; recreated. All four of my safety assertions (right root, right depth,
|
||||
own session excluded, shutterchute excluded) passed — I checked the paths were
|
||||
right and never checked the liveness test was sound. **Do not re-run that
|
||||
predicate.** A correct version checks the deepest recent file, or
|
||||
cross-references running `claude` PIDs.
|
||||
|
||||
Also found: `sudo -n` requires a password as **lkraven locally** on nh3-dev,
|
||||
while `ssh infra-ops@localhost` has NOPASSWD.
|
||||
@@ -0,0 +1,39 @@
|
||||
# `[2026-09-21]` claude-bot became an org Owner, repos moved to `pfi`, and a dead token had been misreporting permissions for months
|
||||
|
||||
**Operator ruling:** claude-bot is an **Owner** in `corviduo` (team 1), `pfi`
|
||||
(team 4) and `vastblue` (team 5). Verified by reading membership back AND by
|
||||
exercising it on claude-bot's own token: create/edit/delete in `pfi` all
|
||||
succeed, `vh/*` correctly still 404s.
|
||||
|
||||
⚠ **Why not "admin on vh/*", which is what was originally asked:** `vh` is a
|
||||
**USER account (id 1), not an organization** — `/orgs/vh` 404s. Gitea has no
|
||||
namespace-scoped admin for a user namespace. Measured: per-repo `admin`
|
||||
collaborator grants read but **not** settings (403 on PATCH) — repo settings are
|
||||
owner-only. So the only working realization of "admin over vh/*" is the
|
||||
**instance-wide site-admin flag**, which would have given claude-bot the same
|
||||
blast radius as the token the credential-migration project exists to retire.
|
||||
Surfaced rather than executed; the operator chose orgs instead.
|
||||
|
||||
**`vh/cicada` and `vh/draupnir` transferred into `pfi`** with SHAs preserved,
|
||||
old paths 301ing, and — verified — **the old ssh remotes still resolve**, since
|
||||
Gitea redirects git-over-ssh and not just the web URL. Repoint anyway: a remote
|
||||
living on a redirect depends on the old path staying unclaimed.
|
||||
|
||||
⚠⚠ **`~/.config/claude-bot/gitea-token` IS DEAD** — it authenticates as
|
||||
**nobody** (`/user` → `None`). I had cited its 403s and 404s twice, to the
|
||||
operator and to a peer, as evidence that claude-bot lacked rights in `vh/*`. The
|
||||
conclusion survived re-testing with a working credential, but the evidence was
|
||||
worthless. **A credential that authenticates as nobody returns 403 and 404 for
|
||||
everything, and that is indistinguishable from a permissions answer.** There
|
||||
are FOUR token files in that directory; the working one for repo work is
|
||||
**`gitea-token-repo-create`** (`write:organization`, `write:repository`,
|
||||
`write:user`).
|
||||
|
||||
**Standing authorization (operator, same day):** routine vh-token use for
|
||||
`vh/*` repo ops no longer gets a flag. Read it from the **vault** —
|
||||
`secret get 'nh3-dev/.config/gitea/vh-token'` — verified byte-identical to the
|
||||
disk copy and authenticating as `vh` (id 1, is_admin=True). ⚠ This did not make
|
||||
the token low-blast-radius; it stopped the class of work being exceptional.
|
||||
|
||||
Recorded in auto-memory as `feedback_wrong_resource_before_wrong_peer` and in
|
||||
`reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
||||
@@ -0,0 +1,44 @@
|
||||
# `[2026-09-21]` The ops log shipped, and the day's real subject was instruments that report without looking
|
||||
|
||||
**Built** `scripts/ops-log` (ffe7b24) to close the fingerprint-less-change gap:
|
||||
infra-ops and infra-hermes act as one OS identity, dockerd logs no per-caller
|
||||
exec, and every commit here is attributed to Vuong Hoang by convention. One
|
||||
appended line per host-changing action, a `mkdir`-atomic claim `deploy-stack.sh`
|
||||
refuses (exit 3), automatic writers in `deploy-stack.sh` + `elway`, and
|
||||
`ops-log audit` as the detector for the raw-`ssh` path the writers cannot see.
|
||||
136-stack baseline laid so the detector starts from that day.
|
||||
|
||||
⚠ **The instrument then failed FOUR ways in its first hours, and every one
|
||||
recorded something — just nothing findable.** Documented as a table in
|
||||
`docs/pfi/ops-log.md` § "How this instrument has failed", which is the durable
|
||||
artifact:
|
||||
|
||||
1. **Claim released by a sub-tool** (3e7d3a3) — a 45-min operation claim was
|
||||
refreshed then released by `deploy-stack.sh`'s exit trap, mid-rollout.
|
||||
`claim` now exits **10** when already yours and leaves the holder file
|
||||
untouched, so a refresh cannot overwrite the reason and TTL the original
|
||||
claimant chose.
|
||||
2. **Wrong order in the hook chain** (9141a41) — the commit hook was APPENDED
|
||||
behind graphify's **eight `exit 0` paths**, so a `graphify-out/`-only or
|
||||
empty commit could never be recorded. Prepend; attribution must never be a
|
||||
subordinate clause of another hook's interestingness filter.
|
||||
3. **No handle in the environment** (4e778ae) — `ALTHING_HANDLE` lived only in
|
||||
`althing-infra-hermes-seat-run.sh`, not the gateway unit. Fallback now says
|
||||
`unattributed(login)` rather than a bare login that reads like an answer.
|
||||
4. **Wrong host key on write** (f3b68e2) — elway passed its ssh TARGET through
|
||||
as the host, so five records of a real esh-pve change landed under
|
||||
`infra-ops@esh-pve` and were invisible to `--host esh-pve`. infra-hermes
|
||||
correctly reported the change as unattributed. **A log you cannot query
|
||||
under the obvious name is not a log.**
|
||||
|
||||
⚠ **The general lesson, and it outlived the tool:** twelve instruments reported
|
||||
confidently and wrongly across 2026-09-19→21, five of them mine. The recurring
|
||||
shape is **configured ≠ effective** — `systemctl show -p Environment` reporting
|
||||
a drop-in while `/proc/<pid>/environ` lacked it; a grep proving presence while
|
||||
evaluation proved absence; a green test suite over a control the browser never
|
||||
received. What broke the pattern every time was asking a *different* instrument
|
||||
the same question.
|
||||
|
||||
See [[2026-09-21-booth-two-dead-controls]] for the same failure in a UI, and
|
||||
`feedback_control_flow_before_concurrency` in auto-memory for the triage rule
|
||||
that came out of it.
|
||||
Reference in New Issue
Block a user