vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — the ops log, and a day spent on instruments that report without looking

Browse Source 
Archived 15 entries (Recent decisions 14, Tried and abandoned 1) oldest-first
to archival-memory.md; 5 held back on the open-deferred-work guard and 164 on
the 14-day guard, so the index stays over the soft cap at 477 lines. An
over-cap file that keeps live decisions beats a scannable one that lost a
belayed item.

Four new detail files cover the day: the ops log and its four self-inflicted
failure modes, the Booth's two dead controls and the four-iteration layout
probe, the Gitea org grant plus the dead claude-bot token that had been
misreporting permissions, and the disk triage that rescued a LoRA adapter from
a directory this box sweeps at three days.

lv-mccarthy's run outcome remains unverified after two days and is the first
line of the in-flight section and step 1 of the handoff.
This commit is contained in:
vh
2026-09-21 14:26:55 -07:00
parent 2e08edcaab
commit e52def115c
14 changed files with 572 additions and 428 deletions
Download Patch File Download Diff File Expand all files Collapse all files
archival-memory.md
+350
View File
@@ -235,6 +235,308 @@ _Archived 2026-09-16._
Operator-approved, tagged `v3.2.0` at `c4ede0f` on master. forseti authored; infra-ops
deployed. Host work, nh3-dev only.
- `[2026-09-07]` **Fleet internal TLS pattern shipped** — caddy (cloudflare-plugin build, `~/.local/bin/caddy-cf`, `fleet-tls-caddy.service`) on nh3-dev is the wildcard cert authority: publicly-trusted LE `*.nh3.phasefinal.com` via Cloudflare DNS-01, resolved internally by an AdGuard split-horizon rewrite (all 3 resolvers). `talk` self-terminates on :8092 with the trusted cert (operator's in-container-TLS ruling), renewal auto-synced by `fleet-tls-cert-check.timer`. Interstitial gone; secure-context+AudioWorklet verified via headless Chromium. Pattern + foot-guns (restart-disrupts-inflight → clients need retry; wildcard = name-only, never IP) → auto-memory `reference_fleet_internal_tls_pattern`.
_Archived 2026-09-21._
- `[2026-09-07]` **cc-channel registered for this infra-ops session's wake** — `althing-route` cc route → the CC session's `$XDG_RUNTIME_DIR/cc-socks/<pid>.sock`; herald pokes the socket directly at a turn boundary. Replaces the FIFO/poll waiter that Claude Code 2.1.257 kept killing while the seat sat in `shell`. Session-local — re-declare per session.
_Archived 2026-09-21._
- `[2026-09-07]` **irv-ml1 /mnt/smithy remount fixed post-cutover** — export allowed `10.0.0.0/8` (old wg0) but not the mesh `100.64.0.0/10` irv-ml1 now sources from → all-uid "permission denied"; added the mesh range to the nh3-nas smithy export + remounted (clientaddr now 100.64.0.6). nh3-nas admin = `infra-ops` PASSWORD auth (vault `nh3-nas/infra-ops-password`), sudo ALL, SFTP subsystem OFF. → auto-memory `reference_irv_ml1_gpu_r14` (corrected).
_Archived 2026-09-21._
- `[2026-09-07]` **irv-ml1.nh3.internal DNS repointed** to the live Irvine LAN IP `10.6.110.50` (was the dead wg0 `10.100.79.3`); CLAUDE.md fleet-row + placement-rule updated to mesh reality. commit `0336e03`.
_Archived 2026-09-21._
- `[2026-09-07]` **Subnet routers excluded from vzdump fleet-wide** (ana-scale 114/pfi-pve, nh3-scale 107/nh3-pve, esh-scale 108/esh-pve) so a hung backup can't blackhole a site; nh3-headscale (106, control plane) KEPT; ESH backup moved 02:15→03:30. Root cause of this morning's ESH outage: an overnight vzdump left CT108 (esh-scale) locked → whole site dark. → auto-memory `feedback_esh_backup_window_0330`.
_Archived 2026-09-21._
- `[2026-09-07]` **Booth link board: pin/favorite + multi-select delete + newest-first** (booth-v0.1.8, commit `76fdf45`, tag `booth-v0.1.8`) — pins in a `.pins` sidecar (content-ids), one `<form>` + `formaction` buttons so ×/★/bulk-delete all degrade with JS off.
_Archived 2026-09-21._
# 2026-09-06 — Headscale cutover COMPLETE: all three site-pairs on the mesh
**DONE.** Operator disabled Site Magic in the UI; NH3↔ESH re-homed to a DIRECT mesh path (8ms, no DERP). Full 6-direction matrix OPEN. Site Magic disabled, both IPsec tunnels dormant, headscale is the sole active site-to-site transport. ana-wg WG fallback untouched. Tunnels re-enablable for backup.
Operator goal (/goal): replace Site Magic + IPsec with headscale, tunnels dormant as backup;
"if paranoid, enable world-accessible SSH on the FortiGate first." Full detail + method +
follow-ups in `docs/pfi/headscale-mesh-plan.md` § CUTOVER EXECUTED. Headlines:
- **colo↔NH3 and colo↔ESH IPsec = DORMANT; the mesh carries both, verified bidirectional.**
NH3 UDM `pfi-nh3-ana` + ESH UDM `esh-ana` set enabled=false (API). Mesh /16 routes added on
both UDMs and the FortiGate (→ ana-scale 10.250.50.45 / nh3-scale 10.100.50.46 / esh-scale
10.0.50.65). Dependent flows OK over mesh: restic ESH→rest-server-ana, FortiGate mgmt.
- **NH3↔ESH Site Magic NOT cut by API** — `sdwan-mesh-tunnel` = `api.err.NoEdit` (cloud
orchestrated). Routes PRE-STAGED + shadowed; DERP path 9ms ready. **Operator disables it in
the UniFi UI**, then the mesh takes over. Told the operator "mesh is online" → he does it.
- **FortiGate WAN SSH safety net (TEMPORARY):** wan1 allowaccess ping+ssh; admin infra-ops
trusthost2/3 = NH3 70.230.226.88 + ESH **128.177.138.182** (static since 09-08; was CGNAT 23.164.40.160) (not 0.0.0.0). Reach it at
`ssh infra-ops@38.120.12.42`. Config backed up flash `pre-wan-ssh-cutover-20260906`. Remove
when the edge (being replaced by OPNsense/R420) is retired.
- ⚠ **Method lesson:** tunnel + mesh static route for the same /16 on one gateway = asymmetric
drop. Disable the tunnel FIRST, then add the route. Broke colo once doing it tunnel-up; rolled
back. See [[incident_crowdsec_cgnat_false_ban]] (same day) and the plan doc.
- Dormancy = disabled+retained (flip UDM object back to enabled=true to restore); NO auto
failover wired. Bonus: exit nodes → free multi-location egress proxy (parked).
## Exit nodes (2026-09-06, operator-requested)
All three routers advertise+serve exit nodes (approved). Clients pick location:
`tailscale set --exit-node=nh3-scale|esh-scale|ana-scale`. NH3 = residential egress
(70.230.226.88) → replaces the nh3-dev SOCKS5 proxy. Exit nodes + source preservation BOTH work via a selective-masquerade rule (NoSNAT kept true;
`mesh-exit-masq.service` per router masquerades only internet-bound exit traffic, RETURNs fleet
dests). Verified: colo sees real NH3 host; nh3-dev via colo exit → egress 38.120.12.42. A node advertising an exit node can't consume one — test from
the laptop/iPad, not the routers.
_Archived 2026-09-21._
# 2026-09-06 — Headscale overlay mesh: control plane + 3 subnet routers live, not cut over
Operator-directed (Headscale over NetBird; NH3 for the control plane, never the colo; 443
direct; names nh3-headscale / nh3-scale / esh-scale / ana-scale). Full state, lessons and
next steps in `docs/pfi/headscale-mesh-plan.md` § Status. Headline facts:
- `https://headscale.phasefinal.com` = CT 106 on nh3-pve (10.100.50.45), headscale v0.29.3,
LE cert via TLS-ALPN-01, UDM forward tcp/443, DDNS timer on nh3-dev (user systemd).
- Routers CT 107 nh3-scale / CT 108 esh-scale / CT 114 ana-scale advertise their /16s,
approved, SNAT off, accept-routes OFF. nh3-dev enrolled as first client (100.64.0.4).
- ⚠ Old tunnels (Site Magic, IPsec) are STILL the site-to-site path. The mesh currently
rides inside them. Nothing has been disabled.
- ⚠ Lesson: `--accept-routes` on a client before a return path for 100.64.0.0/10 exists
black-holes that client's LAN (own-site /16 included). Return path first.
- Pre-auth keys in the vault (`headscale/preauth-*-48h-20260906`, expire 09-08).
- infra-ops user now exists on all four PVE hosts (needed `apt install sudo` first).
_Archived 2026-09-21._
# `[2026-09-05]` A peer's "2.7x stack effect" was a coin flip — the operator caught it and the arithmetic is worth keeping
brokkr-smithy-dev reported that a diversity **noise floor** was 2.7x tighter on ana-ml2 than on
the GX10 and ranked ana-ml2 the better instrument on it. I relayed it. **The operator rejected it
on instinct** — *"that makes zero sense. except for speed, serving a model should be identical
across servers"* — and he was substantially right.
ana-ml2 0.9688, 0.9574 spread 1.14pp mean 0.9631
gx10 0.9583, 0.9895 spread 3.12pp mean 0.9739
between-box LEVEL difference 1.08pp
ana-ml2's own replicate spread 1.14pp <- LARGER than the between-box gap
pooled range ignoring box 3.21pp <- ~= the entire "GX10 floor"
**Each "floor" is `|block0 - block1|` from n=2.** A range over two draws is not an estimate of
dispersion; the ratio of two such ranges is a ratio of two half-normals — a **half-Cauchy**.
brokkr computed the tail himself on retraction: `P(ratio >= 2.7) = 1 - (2/pi)*arctan(2.7)`,
doubled = **0.452**, confirmed by 400,000-draw simulation. He had reported a coin flip as a
measured effect and ranked hardware on it. Retracted at `97f73dd`.
⚠ **The disconfirming evidence was inside his own sentence.** He wrote "the base level is nearly
identical, it is the spread that shifts" and offered it as *reassurance*. A real stack effect
moves the level. Level agreeing while a two-draw range differs is the signature of a noisy range
estimator — he had the refutation in hand and read it as support.
⚠ **His own diagnosis of why it got through is the transferable part:** he had spent the session
triaging *my* claims hard — reading the encode path, simulating preflight, recomputing shas from
bytes — and this one was **his, and flattering**: it made his earlier work look prescient and
produced a clean recommendation. **Asymmetric scepticism is one error twice, and the flattering
direction needs the extra pass.**
**What survived, deliberately separated:** re-measuring the floor on whatever stack actually
serves stays non-negotiable. The mechanism list is sound whether or not those four numbers show
it — sm_121 vs sm_120 kernel selection, vLLM 0.28.0 against mixed 0.26/nightly, and separately
measured batch-invariance (3.12pp at jobs=8, same order as the whole claimed effect).
**Retracting the evidence and keeping the discipline are different acts.** Settling it properly
wants several blocks per box and is its own probe, not a by-product of a gate.
See [[2026-09-05-vllm-on-sm121-and-run4]].
_Archived 2026-09-21._
# `[2026-09-04/05]` vLLM RUNS on sm_121 — and the blocker was `ninja` not being on PATH, not the silicon
Long-standing open question closed: **vLLM 0.28.0 serves on the GB10 (aarch64, sm_121)** from a
stock wheel, no source build. Model loads, `torch.compile` completes (~29 s), CUDA graphs capture.
⚠ **The one trap, and it looks exactly like an sm_121 kernel problem:** FlashInfer JIT-builds its
sampling kernel at first use and needs **`ninja` on PATH**. It ships as a vLLM dependency but
lives in the venv `bin/`, so the failure surfaces as `FileNotFoundError: 'ninja'` from deep inside
a `profile_run` traceback. Same shape as the `python3-dev` trap that bit the training harness on
this box, and the third present-but-not-on-PATH false-absence on this hardware (after `nvcc`).
Launch with **both** `~/vllm-env/bin` and `/usr/local/cuda/bin` on PATH. Harmless and expected:
`Using default MoE config ... device_name=NVIDIA_GB10` — nobody has tuned MoE kernels for this chip.
⚠ **Open WebUI sends `tool_choice: "auto"` on every request**, which vLLM 400s unless launched with
`--enable-auto-tool-choice --tool-call-parser gemma4`. Taken from the working sibling seat
(`gemma4-charrp`), which is where the canonical flags live. **Deliberately did NOT copy
`--reasoning-parser gemma4`** — that seat pairs it with `--default-chat-template-kwargs
'{"enable_thinking": false}'`, and adding it alone moves output into `reasoning_content` with a
null `content`, which OWUI renders as an empty reply. A 400 traded for a blank box.
## Run 4 — the corpus arm
Run 4 adds an **airoboros-3.2 instruct root** (7,229 rows) and displaces kvasir 38% → 18% context
share, at run 3's lr 2e-04, everything else held byte-identical. Operator granted a run-scoped
training-eligibility override `operator-2026-09-04-rnd-run4` (the SECOND grant; run-1's was
one-run-scoped, a run 5 needs a third).
**Two peer artifacts were rejected before they cost GPU time, both by reading the harness rather
than accepting a "confirm this":**
1. `sample_kind: "instruct-dialogue"` — `core.py:521-544` dispatches on the **literal string** and
raises on anything outside `rp-dialogue`/`prose-chunk`/`actual-play-passage`. Would have
hard-stopped the encode on the first airoboros row. brokkr had labelled it *non-blocking*.
2. Recipe targets collapsed three dialogue roots into one descriptive label with `root_sha256:
null` on three of four. Preflight resolves `roots_dir/<dataset_id>/clean-v1/CLEANROOT.json`
literally and requires the sha.
⚠ **I did NOT fill the missing shas in myself**, though the values were known and it would have
taken two minutes. `root_sha256` is the *author's* assertion and preflight exists to check the
deployed root against it — supply both sides and the check verifies my copy-paste, an inert gate.
brokkr then went further and recomputed his shas **from shard bytes** rather than reading them
back out of the deployed CLEANROOT, which had the same defect one step removed.
**The kvasir prefix cut has three independent confirmations**: my build measured 1,613 samples /
3,347,622 ctx = **47.4%** of run-3 kvasir, which reproduces the recipe's 18/38 share ratio
(47.37%); the harness's own `[mix]` block then reported **ctx 0.1805** against the 18.0% target.
⚠ **`str.splitlines()` splits on U+2028**, which `json.dumps` leaves unescaped under
`ensure_ascii=False`. The harness splitlines() in four places and four roots DO carry U+2028
(fireball 8, c2-logs 31, kvasir 1, creative-writing 1) — but those are read by ordinary file
iteration (safe), and the splitlines() paths touch only files written with the default
`ensure_ascii=True`. Latent, one flag away from live, not acting. Found by brokkr the hard way.
_Archived 2026-09-21._
# `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression
Run 3c is the lr **1e-05** arm against run 3's 2e-04; corpus, base, mask and every other
hyperparameter byte-identical. It exists to answer one question: does a gentler learning rate
keep run 3's diversity benefit while dropping its safety regression? **Answer: no, and the
failure is the informative kind.**
trained 604/604 in 11:55:40 on pfi-gx10 (GB10, aarch64, sm_121), 69.8 s/it
merged /home/infra-ops/erp-tune/serve/merged-run03c, bf16 51.6 GB
served erp-tune-v3c on :8098, exposed as gateway alias `trial`
## The verdict (brokkr-smithy-dev, run03-gate/RESULT-run03c-gate.md)
CSAM 175 gens x 3 arms — ZERO hits, hard-line guard passed everywhere
selfharm/methods base 10/10 refused -> 3c 0.6 = -40pp de-refusal
meta pooled 0.0 delta — a CANCELLATION, not cleanliness: selfharm -40pp
offset by advice/medical +40pp
diversity attractor primary returned to BASE level — the gain is GONE
cross-arm run 3 (lr 2e-04) selfharm -44pp diversity: real gain
run 3c (lr 1e-05) selfharm -40pp diversity: none
⚠ **The selfharm de-refusal is LR-ROBUST.** A 20x cut moved it 4pp while erasing the diversity
benefit entirely. It comes from corpus content and imprints at even gentle exposure — it is not
something a lower learning rate dials out. That is what the LR sweep was run to find out.
⚠ **A pooled preserve-list test cannot see a single-axis collapse.** The pooled operational
delta reads 0.0 because two axes moved 40pp in opposite directions. Same structural defect that
let run 3's gate pass — recorded as R47 §8 item 11.
## What the port proved about the box
- **The tooling loads on aarch64/sm_121.** Full training stack plus flex_attention and the
chunked-loss path. Nothing exotic needed beyond `python3-dev`.
- **Verification that earned its keep**: both 49 GB base shards sha256-matched ana-ml2's, and a
full encode was run into a throwaway dir and compared byte-for-byte — 197,360,233 B, sha256
`c08bb1fe2ecb0be3`, identical. transformers 5.15.1→5.16.1 and x86-64→aarch64 are *measured*
inert, not assumed.
- ⚠ **The encode-cache FILENAME differs by design** — `base_model_path` is in the cache key, so
rehoming the base changes the key while content stays identical. Input hash, not output hash.
Do not read it as drift; do not "fix" it by faking `/tank` on the GX10.
## The lora_B signal worth carrying forward
run 2 (lr 2e-04) min 0.6826 median 1.7212 max 3.7573
run 3c (lr 1e-05) min 0.0480 median 0.1561 max 0.4133
~11x gentler across the board — exactly what a 20x LR cut should produce. A consistency check
passing, not a red flag, but worth in hand *before* reading diversity numbers: "the tune did
nothing" and "the tune did less on purpose" look alike in the output.
Runbook `docs/runbooks/gx10-run-03c.md`; commit `dae77ee`.
_Archived 2026-09-21._
# `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour
`vllm-embed` had OOM-crashed **7 times** (`RestartCount 7`): GPU1 was at **216 MiB free** of
97,887, and a ~3.4 GB tenant sharing a card with five other seats dies when it cannot get another
96 MiB mid-inference. Root cause of the `qwen3-embed` 500s brokkr saw — **not** the LiteLLM
gateway restart he attributed them to (different host, different component, 50 min earlier, and
six of the seven crashes predate it).
**Operator ruling:** move `gen` to GPU0. *"we were keeping it free for training, but we don't have
the power for sustained load."* GPU0 was never free — `mog-sec` has been there at util 0.52
(55,126 MiB) since the August move.
## ⚠ THE SIZING ERROR, WHICH IS THE POINT OF THIS ENTRY
gen at util 0.43 wants 42,091 MiB GPU0 free 42,113 margin 22 MiB — would not start
gen at util 0.41 wants 40,134 MiB margin 1,979 MiB — I called this safe. IT WAS NOT.
result 1,548 MiB free, 56 FlashInfer autotuner OOM-fallbacks per 3 min
gen at util 0.38 34,558 MiB actual 4,466-7,548 MiB free ZERO OOM events over 6 min
**`gpu-memory-utilization` governs vLLM's declared weights+KV budget. It does not cover what the
process actually needs at runtime** — FlashInfer JIT autotuner workspace, CUDA graphs, expandable
segments all allocate on top. My "1,979 MiB margin" was against the declared budget, so the seat
came up and then silently fell back to slower kernels for want of 20 MB chunks.
⚠ **Same class of error as the DAC plateau the same day: checked a proxy, reported it as the
thing.** The fix was to *measure* — count OOM-fallback events before and after — not to re-derive.
## Final state and what it cost
GPU0 mog-sec 55,126 + gen 34,558 = 89,701 / 97,887
GPU1 50,507 / 97,887 — 46.7 GB free, was 216 MiB
`gen` KV cache is now **348,515 → 268,205 tokens, 1.02x concurrency at its 262,144 max-model-len**
— it holds exactly one full-context request. Short/medium requests still batch; long-context
throughput effectively serialises. The lever to reclaim it is `MOG_GPU_MEM_UTIL 0.52`, untouched.
.env.bak-preGPU0-20260904-164032 the GPU move
.env.bak-preShrink-165133 the utilization
⚠ `/opt/docker/compose/*/.env` is root-owned — `docker compose restart` needs **sudo** on
ana-docker/ana-ml2 or it fails with a bare `permission denied` on the .env.
_Archived 2026-09-21._
# `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN
`10.0.50.50` is **CT 103 on esh-pve-nas**, a hand-rolled Debian NAS (not a Synology). Created an
SMB account for the operator's Windows AudioGridder DSP box:
username dsp uid 999, /usr/sbin/nologin, no home — SMB ONLY, cannot log in anywhere
password vaulted at esh-nas/dsp-smb-password (round-trip verified by sha)
verified authenticates, sees all 8 shares, WRITE to //share confirmed (mkdir/rmdir)
⚠ `smbclient` is absent from the NAS, nh3-dev and esh-docker-vm — verification ran in a throwaway
`alpine:3.20 --network host` container with `samba-client`, leaving nothing installed.
Shares are registry-defined (`registry shares = Yes`), so they are invisible in `smb.conf` — use
**`testparm -s`**, not grep, or you will conclude there are no shares.
## ⚠⚠ THE EXPOSURE, UNACTIONED — operator has not ruled
**NFS: twelve exports, `rw` to all of `10.0.0.0/8`, `sec=sys`, no authentication.**
/mnt/{backup,books,compose,documents,iso,media,music,share,
pvestore,nvme-pvestore,ssd-pvestore,tank-vmbu}
`sec=sys` means the NAS trusts whatever uid the client claims. **Every VLAN at ESH — IoT,
cameras, guest — can mount the NAS read-write today.** `/mnt/backup` additionally has
`all_squash,anonuid=2000` so every client collapses to `nas_user`.
**SMB: every share except `backup` is `guest ok = Yes` and writable**, with
`map to guest = Bad User` — an unknown username lands as guest with write access.
So the `dsp` credential is auditable and survives guest being turned off, but it is **not what is
gating access today**. Hardening (narrow exports to `10.0.50.0/24` + named hosts, drop `guest ok`
where unneeded) was offered and is roughly an hour; it would break anything relying on guest,
which is why it needs the operator's say-so. **Not actioned. No tracking surface beyond this entry.**
_Archived 2026-09-21._
- `[2026-09-04]` **SLZB-MR1U Zigbee coordinator moved to esh-iot (VLAN 90) at `10.0.90.10`, DHCP-reserved, DNS'd, handed to ha-dev.** ⚠ Home Assistant cannot resolve `.internal` at all (Docker's 127.0.0.11 upstream excludes the fleet AdGuard) — pre-existing; ha-dev declined the fix. Runbook `docs/runbooks/slzb-mr1u-zigbee-coordinator.md`, commits `fed29be`/`0bbdaf9`.
_Archived 2026-09-21._
## What 3.2.0 changes
A Claude Code session in a zellij pane is now poked **in its own pane** instead of through a
@@ -6451,6 +6753,54 @@ _76 older entries archived to archival-memory.md._
_Archived 2026-09-09._
# `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early
The ESH-Media ↔ UDM uplink is a 3 m OEM `SFP-H10GB-CU3M` twinax that negotiated **1000 Mbps**
with **zero errors** on both ends. Diagnosis: `sfp_compliance: Unknown` — the switch reads the
EEPROM but cannot parse the compliance codes on a third-party cable wearing Cisco coding, so
autoneg falls back to the safe rate. Control case: a TP-Link `TL-SM5220-1M` on the adjacent UDM
port runs 10G with the same speed_caps, exonerating the port, firmware and autoneg.
Operator authorised forcing it. Forced `autoneg False / 10000 / full` on the **UDM end only**;
ESH-Media followed to 10000 unforced — proof the cable was electrically *capable*.
## ⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG
10:03 200 errors link-up burst
10:48 221 +21 in 42 min — I reported this as "flat", it was not
14:38 416 +195 over 4 h, plus user-visible flapping the operator felt
**A marginal link declares itself over hours, not minutes.** I watched a two-minute flat window
and reported a plateau; the counter simply had not moved yet. The operator noticed the flapping
before the soak I left running had accumulated enough to raise it. Reverted 14:38, both ends back
to autoneg/1000, stable.
⚠⚠ **The correction that matters: a clean, zero-error link at 1G does NOT rule out a marginal
cable.** It only proves the cable is clean *at 1G*. Clean at 1G and marginal at 10G is exactly
what a cheap 3 m twinax is — which means the platform's autoneg fallback was **protecting
something real**, not being fussy about vendor coding. The coding explains the negotiation; it
does not explain errors once forced. I conflated the two.
⚠ **Do not re-force this port.** The fix is the cable.
## Method notes worth keeping
- **Force the RECOVERABLE end.** ESH-Media reaches the controller *through* this link, so a
failed force there strands the switch behind a dead uplink — a physical visit. The UDM end is
safe because the path to the controller (`nh3-dev → 10.100.10.1 → 10.0.0.1`, site VPN) does not
cross it. Verified with `traceroute` **before** the change; revert payload written before the
forward one.
- ⚠ `port_overrides` is a **whole-array PUT** — the array was diffed to prove exactly one field
on one port changed before sending, and read back after.
- Blast radius named before acting: ESH-Media backhauls the operator's office switch, a WiFi AP
and the Zigbee coordinator.
Resolution: the operator already owns a replacement and ran the copper himself through a drilled
floor 2x4. Recommendation given was **fiber** (10Gtek SR 2-pack + OM4 3 m LC-LC, ~$40-65) because
the cable-vs-pull-damage question was never resolved and a second DAC through the same hole risks
the same fault. Runbook `docs/runbooks/esh-media-dac-10g.md`; commits `b9988a7`, `514ce7a`, `054c098`.
_Archived 2026-09-21._
## Archived 2026-08-02 — Recent decisions (archived)
### 2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session