memory: snapshot — 2026-10-09 SVOS LAN/voice/ac42ace/kernel 0.0.8 (fd outage fixed), U13 ingress approved+deferred, albok library, gate watch; 11 entries archived
This commit is contained in:
@@ -15007,3 +15007,288 @@ reported: ana-docker `/root/.nvm` (lkraven), irv-ml1 `uv`/`uvx`, fv-ml1
|
||||
|
||||
- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage.
|
||||
_Archived 2026-10-08._
|
||||
|
||||
## Recent decisions (archived 2026-10-09 batch)
|
||||
|
||||
# `[2026-09-25]` nh3-pve AMT LIVE: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None
|
||||
|
||||
**nh3-pve AMT LIVE: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None** (Prime). Root cause of "no IP": Linux held the I226-LM down, so igc powered off the PHY; the fix is `auto enp88s0` + IPv6 off + `arp_ignore=8`. Password vaulted `nh3-pve/amt-admin`; tool `scripts/amt-wsman.py`. `0f032fa`…`50f113c`.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
- `[2026-09-25]` **pfi-gx10 AC-restore VALIDATED by Prime's AC pull; Homepage stays on esh-docker-vm (Prime) after the mmap_lock wedge reboot.** `servers/pfi-gx10/README.md`, `servers/esh-docker-vm/README.md`.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
# `[2026-09-25]` nh3-ml1 LIVE after the NH3 site visit
|
||||
|
||||
**Prime, ~1524 PT:** "secure boot disabled, amt enabled, card plugged." This is the
|
||||
resume of `2026-09-25-nh3-ml1-standup.md`. nh3-pve came up at 1523.
|
||||
|
||||
## Post-boot checklist: all green
|
||||
|
||||
- Uplink is `enp2s0f1np1`. Guests 100, 101, 102, 105 and CTs 103, 106, 107 are up;
|
||||
104 and 108 are off as expected.
|
||||
- NFS automounts are good on nh3-docker, nh3-dev and pbs-nh3. The post office is
|
||||
push/reachable. The 14 queued althing messages were all Beszel/Kuma "UP"
|
||||
recoveries, so nothing needed acting on.
|
||||
- Secure Boot is disabled. `00:02.0` is back with `boot_vga=1` and i915 loaded.
|
||||
|
||||
## Build (elway, both playbooks idempotent)
|
||||
|
||||
1. `pve-nvidia-host.yaml`: the driver installed (DKMS, 36 s), 6/6 verify.
|
||||
2. `gpu-lxc.yaml`: CT 109 built, then **verify 3 FAILED**. `docker run` died with
|
||||
`open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission
|
||||
denied`.
|
||||
- The cause is the lxc-pve version, not runc. Both CTs run runc 1.5.1. nh3-pve
|
||||
(PVE 8.4.1) had lxc-pve **6.0.0-1**, and esh-pve (8.4.20) has **6.0.0-2**,
|
||||
whose changelog is Proxmox "fix #7006: do not restrict /proc or /sys if
|
||||
nesting is enabled".
|
||||
- **Decided autonomously:** upgrade that one package (a dry-run showed
|
||||
1 upgraded, 0 new, 0 removed) and `pct reboot 109`. GPU-in-docker then passed.
|
||||
ops-log recorded.
|
||||
- The playbook gained a first step that upgrades lxc-pve when it is below
|
||||
6.0.0-2, with a quoted name, because an unquoted `#7006` truncated it as a YAML
|
||||
comment. The re-run was 5/5 verify OK. The guard skips on esh-pve (tested).
|
||||
3. `embed-rerank` deploy. `deploy-stack.sh --yes` was **denied by the auto-mode
|
||||
classifier as a blind apply**, so the pattern now is: preview with
|
||||
`echo n | deploy-stack.sh …` (it shows the diff and aborts), then `echo y |`.
|
||||
`.env` = example + `HOST_NAME=nh3-ml1`, `HOST_IP=10.100.50.80`. Healthy in about
|
||||
120 s including the image pull; 2.7 GB VRAM.
|
||||
|
||||
## Parity and speed (vs esh-ml1)
|
||||
|
||||
Script: session scratchpad, not committed. Corpus: 1,120 repo-doc paragraphs
|
||||
plus 6 fixed texts, 2 runs per host, 50 queries.
|
||||
- Embedding cosine esh↔nh3: median 0.999998, min 0.999993. The floors are esh
|
||||
0.999995 and nh3 0.999993.
|
||||
- Negative control (different texts): median 0.50.
|
||||
- overlap@10 = 1.000 both ways. Positive control, MRL-256: 0.684.
|
||||
- Rerank, 100×20: top-1 1.00, max score diff 0.0014 (the esh-esh floor is
|
||||
0.0020). Positive control, a 4-word query: top-5 order 0.09, max diff 0.97.
|
||||
- Speed on-box, 3 interleaved reps: short embed p50 6.6–6.7 vs 7.0 ms; bulk
|
||||
104–105 vs 106–108 passages/s; rerank-20 p50 156–159 vs 152–161 ms. The same.
|
||||
|
||||
## Monitoring and wiring
|
||||
|
||||
- Beszel: agent (`hosts/nh3-ml1.yaml`, same hub key as esh-ml1), system
|
||||
`1feeeq61g4mkqre`, 5 alerts, GPU samples present.
|
||||
- ⚠ zsh gotcha: `for a in "X 1 2"; do set -- $a` does NOT word-split in zsh, so
|
||||
the first batch of alert POSTs failed silently (empty fields). Use a function
|
||||
with explicit args.
|
||||
- Kuma #29/#30 (the seed also refreshed the esh-ml1 descriptions). Homepage
|
||||
`nh3-ml1-docker` shows 2 cards with no restart. Dozzle agent added, and the hub
|
||||
`.env` backed up as `.env.bak-20260925-nh3ml1` before the edit.
|
||||
- DNS `nh3-ml1.nh3.internal` on all 3 resolvers. `~/.ssh/config` `Host nh3-ml1`;
|
||||
`servers/nh3-ml1/` has ssh-target, system-details and README.
|
||||
|
||||
## Found along the way
|
||||
|
||||
- **Kernel 6.8.12-43 came from our own playbook.** The headers step's
|
||||
`proxmox-headers-6.8` install upgraded the `proxmox-kernel-6.8` meta
|
||||
(apt history 14:19:23). The visit reboot booted it. Documented in
|
||||
`pve-nvidia-host.yaml`.
|
||||
- **btmtk Bluetooth oops on every -4x boot** (2/2 lines on each of 3 -43 boots,
|
||||
0 on the -11 boots; esh-pve -42 is the same). It taints `D`. Only the BT worker
|
||||
dies. Proposed fix: blacklist btusb on both PVE hosts. Not done.
|
||||
- **AMT is not reachable.**
|
||||
- The LM port has a 1 Gb carrier (brief unbridged `ip link up` with IPv6
|
||||
disabled, then restored; ops-log recorded).
|
||||
- No UDM client or lease for `58:47:ca:76:96:0e`.
|
||||
- A /dev/tcp sweep of 16992/16993 over 10.100.{0,10,50,250}.0/24 found nothing.
|
||||
Its positive controls, :8390 and :8006, did answer.
|
||||
- The USW Pro 24's up ports (19, 22, 23, 26) are all other devices. So the cable
|
||||
is probably on nh3-sw1 (`10.100.250.2`), where SSH times out and no creds are
|
||||
held.
|
||||
- Asked Prime.
|
||||
|
||||
## Open (Prime's)
|
||||
|
||||
- Gateway routing for `qwen3-embedding` / `reranker`. Recommendation: load-share
|
||||
(identical cards, and a second site).
|
||||
- AMT network settings and switch port. The NanoKVM stays on nh3-pve until AMT KVM
|
||||
shows the console.
|
||||
- btusb blacklist on nh3-pve and esh-pve. Low priority.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
# `[2026-09-25]` nh3-ml1 — second embed/rerank GPU LXC on nh3-pve (DEFERRED: Secure Boot)
|
||||
|
||||
**Ask (Prime, ~1410 PT):** "stand up an nh3-ml1 that takes advantage of that GPU"
|
||||
(the RTX 2000E Ada installed in nh3-pve that morning, `01:00.0`).
|
||||
|
||||
## Decisions taken on Prime's behalf (he said "clear to proceed" at ~1418)
|
||||
|
||||
- **Purpose: the second TEI embed/rerank backend.** esh-ml1's README had been
|
||||
waiting on "the second RTX 2000" for failover. No reward-seat copy is planned,
|
||||
because that seat has no working consumer.
|
||||
- **Placement: CT 109, `10.100.50.80/24`, VLAN 50, gw `.1`, rootfs on `local-zfs`,
|
||||
6 cores / 16 GB / 80 GB, startup order 30.** The address came from the UDM
|
||||
(`nh3-servers` DHCP pool `.150–.249`; in use `.40 .42 .45 .46 .50 .60 .71 .90`).
|
||||
`.80` mirrors esh-ml1. nh3-pve had 40 of 62 GB RAM free.
|
||||
- **Not backed up**, like esh-ml1. nh3-pve's vzdump job is `all 1` (exclude 107),
|
||||
so `gpu-lxc.yaml` adds the CT to that job's `exclude` list. The guard and the
|
||||
step were tested against sample configs: one positive case, plus negatives for
|
||||
already excluded, a vmid-list job, a non-vzdump `all 1` stanza and a missing
|
||||
file.
|
||||
- **Playbooks made host-generic (`bc278d4`):** `esh-pve-nvidia-host` →
|
||||
`pve-nvidia-host`, which gained headers/dkms/build-essential, a nouveau
|
||||
blacklist and an unload that refuses if nouveau has bound a device.
|
||||
`esh-ml1-lxc` → `gpu-lxc`, whose host vars have **no defaults**; elway aborts on
|
||||
an undefined `{{ var }}` (tested). Per-host `--var` lines are in its header.
|
||||
- **`stacks/embed-rerank` Homepage labels** take `HOST_NAME` / `HOST_IP`. The
|
||||
defaults render byte-identical to esh-ml1's live labels, so nothing is
|
||||
recreated there.
|
||||
|
||||
## What happened
|
||||
|
||||
1. The first `pve-nvidia-host` run died on host-key verification for
|
||||
`nh3-pve.nh3.internal`. Every `when:` got ssh rc 255 and SKIPPED, and verify
|
||||
failed. I pinned the name after confirming its ed25519 key equals the known
|
||||
IP entry.
|
||||
2. The second run installed headers 6.8.12-11 + `proxmox-headers-6.8` + dkms +
|
||||
build-essential, blacklisted and unloaded nouveau (it had failed its probe
|
||||
with `gsp ctor failed: -2` and was bound to nothing), then **the DKMS module
|
||||
was refused at load: Secure Boot is ON** (`mokutil --sb-state` enabled;
|
||||
lockdown `integrity`; the only MOK is the Proxmox Secure Boot CA). esh-pve,
|
||||
the same MS-01 and BIOS, has Secure Boot OFF. The installer rolled back: no
|
||||
dkms entry, no `nvidia-smi`. All guests were unaffected.
|
||||
3. `7ddd116` added a pre-flight that refuses when Secure Boot is on without an
|
||||
enrolled `/var/lib/dkms/mok.pub`. Tested: it refuses on nh3-pve and passes on
|
||||
esh-pve.
|
||||
|
||||
## Open calls (Prime's)
|
||||
|
||||
- **Secure Boot:** (1) turn it off in the BIOS on the NH3 visit (recommended; one
|
||||
more toggle in the menu he is already in for IGFX, and it matches esh-pve);
|
||||
(2) keep it and enroll a DKMS MOK (prep remotely, then answer MokManager at
|
||||
boot; DKMS 3.x then signs rebuilds itself); (3) a VFIO VM (works without a
|
||||
visit, but reverses his 09-24 "LXC + host driver, not VFIO" rule, and the RTX is
|
||||
currently the boot VGA). Options 1 and 2 need the console at boot, which is
|
||||
blind until IGFX is set, so the visit carries both.
|
||||
- **Gateway routing** once parity passes: recommendation load-share. The two
|
||||
cards are identical, and the old "don't load-share" rule came from pairing
|
||||
esh-ml1 with the much faster fv-ml1.
|
||||
|
||||
## Resume
|
||||
|
||||
The Current-state checklist, step 9. Nothing host-side exists yet for CT 109.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
# `[2026-09-25]` MS-01 foot-gun: a GPU in the PCIe slot renames every NIC
|
||||
|
||||
**MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). On nh3-pve that means booting with no uplink. Pinned by MAC with `.link` files before the install, and the AMT port was pulled from vmbr0. → `servers/nh3-pve/README.md`, `playbooks/nh3-pve-pin-nic-names.yaml` (`2118449`)
|
||||
_Archived 2026-10-09._
|
||||
|
||||
- `[2026-09-25]` Created empty private repo `corviduo/norn` (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the `norn-dev` handle is the operator's to declare.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
# `[2026-09-25]` Reward seat audit + move to esh-ml1; esh-ml1 monitoring wired
|
||||
|
||||
**Prime:** "Audit the reward seat to see if something has superseded what we
|
||||
deployed — then move it to esh-ml1 as well, then wire up telemetry and monitoring."
|
||||
|
||||
**Audit (full record `stacks/reward-seat/README.md`).**
|
||||
- Not superseded. Skywork-Reward-V2-Llama-3.1-8B is #1 of 188 in AllenAI's official
|
||||
RewardBench 2 per-sample results (mean over the 5 non-Ties subsets, 84.7; the
|
||||
dataset was last updated 2025-12-11). No Skywork V3 exists. The -40M sibling is
|
||||
vendor-marked experimental. Caveat: 2026 models never submitted to RewardBench 2
|
||||
would not appear.
|
||||
- Our local AWQ W4A16: 0.847 vs published bf16 0.860 on 150 prompts (±2.9 pt SE),
|
||||
96.2% pairwise agreement, Pearson 0.991.
|
||||
- ⚠ vLLM adds a SECOND BOS to already-templated text unless
|
||||
`add_special_tokens:false` is sent: −2.7 pts. That was a harness confound in the
|
||||
first measurement (0.820), caught by checking `/tokenize`.
|
||||
- **No working consumer.** 0 requests 2026-09-13 → 09-25. Worldtree Domari
|
||||
hardcodes the dead `10.250.50.54:8003` and sends a non-vLLM schema
|
||||
(`prompt`/`responses` → `rewards`), so it always fell back to Selene/chat-judge.
|
||||
worldtree-dev told (single message; their call).
|
||||
|
||||
**Move.**
|
||||
- sha256-identical model copy.
|
||||
- vLLM v0.24.0 on esh-ml1 `:8003`, util 0.55 (KV 21,328 tokens = 1.30× of a 16k
|
||||
request).
|
||||
- Parity vs fv-ml1 on 600 RewardBench 2 texts:
|
||||
- 149/150 verdicts and 99.8% pairwise signs identical.
|
||||
- Raw |Δ| median 0.049; p99 0.22 vs the self-noise p99 of 0.16–0.20.
|
||||
- Gateway `/scalar-judge` → `10.0.50.80:8003` (verified 3/3).
|
||||
- fv-ml1 `vllm-reward` removed; `stacks/vllm` now = `vllm-coder` only.
|
||||
|
||||
**Monitoring.**
|
||||
- **Beszel:** NVIDIA agent (GPU util/VRAM/power/temp), system
|
||||
`ridkfdpwfq3f730`, 5 alerts incl. Temperature >85 °C.
|
||||
- **Kuma:** #27 embed, #28 rerank (the only seats inside Kuma's lane, because they
|
||||
are the sole backends). Reward excluded (no consumer).
|
||||
- **Homepage:** dockerd tcp/2375 on 10.0.50.80, which restarted docker (all
|
||||
services healthy within ~31 s), and `esh-ml1-docker` in docker.yaml. Three cards
|
||||
appeared without a Homepage restart.
|
||||
- **Dozzle:** agent v10.4.1 pinned to the hub's version. It needed an empty
|
||||
`traefik-net` network.
|
||||
|
||||
**Found while wiring (same "rots on re-IP" class as the a3 alias):**
|
||||
- The Dozzle hub still listed ana-ml2 `10.250.50.54` and irv-ml1's retired
|
||||
`10.100.79.3`. fv-ml1 and irv-ml1 logs had been missing since their moves. Fixed;
|
||||
the hub now reports 7 clients.
|
||||
- nh3-dev's Beszel agent had been down since the 09-24 NH3 power recovery (bind of
|
||||
the automounted `/mnt/smithy` failed at boot; docker never retries a failed
|
||||
create). Started; the Beszel "up" alert reached althing, an end-to-end proof of
|
||||
the alert route. Recurs on the next cold start until fixed.
|
||||
- The nh3-docker Dozzle agent has been stopped by hand for ~5 months. Left alone.
|
||||
|
||||
⚠ While checking the Beszel credential's shape, its password was printed into this
|
||||
session's transcript. The transcript stays on nh3-dev and it was not written to any
|
||||
memory; rotation offered to Prime.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
# `[2026-09-25]` TEI adopted as the fleet embed/rerank engine; esh-ml1 is the sole backend; fv-ml1 seats retired
|
||||
|
||||
**Prime's rulings (2026-09-25):**
|
||||
- Utility seats move off fv-ml1 to a utility host like esh-ml1, accepting slower
|
||||
service; the Blackwells are kept for higher-requirement work.
|
||||
- **parakeet STAYS on fv-ml1** (real-time chat path, every ms counts).
|
||||
- **"TEI is embed/reranker server for esh-ml1 and the FLEET in general, in
|
||||
future."**
|
||||
- fv-ml1 retires both embed + rerank once esh-ml1 is up. A **second RTX 2000 is
|
||||
coming to the fleet soon**; until then embed/rerank has one backend (accepted).
|
||||
|
||||
**Evidence chain:** speed A/B vs fv-ml1 (single queries a wash, bulk 3–10×
|
||||
slower on the Ada); whole-novel test (*The Stand*, 12,814 paragraphs) through
|
||||
the gateway ~12 min one per request on either site, because the gateway is the
|
||||
bottleneck; TEI bake-off (`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`):
|
||||
matches vLLM vectors, no re-embed, slower on bulk, much lighter. Dvalin
|
||||
(Grok research peer) reviewed: agree with caveats. Two caveats were actioned:
|
||||
fail-closed truncation (TEI's default silently embedded a 16,384-token prefix of a
|
||||
40k input and returned 200) and nevermore's 0.3 rerank threshold (0 flips in 2,000
|
||||
scores; low sensitivity, since few scores sit near 0.3).
|
||||
|
||||
**Cut-over, 0820–0830 PT:**
|
||||
1. esh-ml1: vLLM pair down, TEI `tei-embed` :8001 + `tei-rerank` :8013 up (same
|
||||
ports). The old `.env` is kept as `.env.vllm-retired-2026-09-25`. The vLLM image
|
||||
(~30 GB) is kept in case the reward classifier (vLLM-only) moves here.
|
||||
2. Gateway: `qwen3-embedding` → esh-ml1 only (`hosted_vllm/`, unchanged
|
||||
address); `reranker` → **`huggingface/`** provider (hosted_vllm 422s on TEI's
|
||||
`texts` body); `order` fields removed; LiteLLM restarted (~58 s). The DB
|
||||
alias `reranker-a3-bge-v2-m3` was PATCHed to the same TEI target.
|
||||
3. Verified through the gateway against the still-running FV seats: embed
|
||||
cosine median 0.999927 / min 0.999881 (n=203); rerank top-1/top-3 29/30 (2 of
|
||||
30 lists had a duplicate-source tie; the disagreeing query wasn't logged).
|
||||
4. **No direct (non-gateway) consumers existed**: per day, fv-ml1 seat POSTs =
|
||||
gateway spend-log rows exactly (09-20→09-24). After the repoint, the only FV hits
|
||||
were my own verification calls.
|
||||
5. fv-ml1: `vllm-embed` + `vllm-rerank-a3` stopped and removed; `stacks/vllm`
|
||||
compose + `.env.example` updated (reward + coder untouched, 10–11 days up);
|
||||
GPU 1 93,086 → 86,956 MiB (~6.1 GB freed).
|
||||
|
||||
**Open:** esh-ml1 is now load-bearing with no monitoring (no Beszel, Kuma or
|
||||
Homepage). Next: "what else moves". vllm-reward (9.2 GB, vLLM-only) is the
|
||||
candidate and fits beside TEI's ~2.6 GB on 16 GB; vllm-coder (6.3 GB) is
|
||||
latency-sensitive.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
## Tried and abandoned (archived 2026-10-09 batch)
|
||||
|
||||
- `[2026-09-25]` **Concluding "not on the UDM" from a port table read 15 s after link-up** — UniFi polls (~60 s); AMT was on port 6 as Prime said. Wait a poll, and prefer a which-port-lists-the-MAC check. auto-memory `feedback_polled_stats_lag_the_event`.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
- `[2026-09-25]` **The esh-pve NVIDIA DKMS recipe on nh3-pve** — it built, then the kernel refused to load it: nh3-pve has SECURE BOOT ON (esh-pve, the same MS-01, has it off). The installer rolled back. The playbook now pre-flights `mokutil --sb-state` (`7ddd116`).
|
||||
_Archived 2026-10-09._
|
||||
|
||||
- `[2026-09-25]` **elway against an unpinned host-key name** — every `when:` hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in `when:` should fail.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-25]` MS-01 foot-gun: a GPU in the PCIe slot renames every NIC
|
||||
|
||||
**MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). On nh3-pve that means booting with no uplink. Pinned by MAC with `.link` files before the install, and the AMT port was pulled from vmbr0. → `servers/nh3-pve/README.md`, `playbooks/nh3-pve-pin-nic-names.yaml` (`2118449`)
|
||||
@@ -1,89 +0,0 @@
|
||||
# `[2026-09-25]` nh3-ml1 LIVE after the NH3 site visit
|
||||
|
||||
**Prime, ~1524 PT:** "secure boot disabled, amt enabled, card plugged." This is the
|
||||
resume of `2026-09-25-nh3-ml1-standup.md`. nh3-pve came up at 1523.
|
||||
|
||||
## Post-boot checklist: all green
|
||||
|
||||
- Uplink is `enp2s0f1np1`. Guests 100, 101, 102, 105 and CTs 103, 106, 107 are up;
|
||||
104 and 108 are off as expected.
|
||||
- NFS automounts are good on nh3-docker, nh3-dev and pbs-nh3. The post office is
|
||||
push/reachable. The 14 queued althing messages were all Beszel/Kuma "UP"
|
||||
recoveries, so nothing needed acting on.
|
||||
- Secure Boot is disabled. `00:02.0` is back with `boot_vga=1` and i915 loaded.
|
||||
|
||||
## Build (elway, both playbooks idempotent)
|
||||
|
||||
1. `pve-nvidia-host.yaml`: the driver installed (DKMS, 36 s), 6/6 verify.
|
||||
2. `gpu-lxc.yaml`: CT 109 built, then **verify 3 FAILED**. `docker run` died with
|
||||
`open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission
|
||||
denied`.
|
||||
- The cause is the lxc-pve version, not runc. Both CTs run runc 1.5.1. nh3-pve
|
||||
(PVE 8.4.1) had lxc-pve **6.0.0-1**, and esh-pve (8.4.20) has **6.0.0-2**,
|
||||
whose changelog is Proxmox "fix #7006: do not restrict /proc or /sys if
|
||||
nesting is enabled".
|
||||
- **Decided autonomously:** upgrade that one package (a dry-run showed
|
||||
1 upgraded, 0 new, 0 removed) and `pct reboot 109`. GPU-in-docker then passed.
|
||||
ops-log recorded.
|
||||
- The playbook gained a first step that upgrades lxc-pve when it is below
|
||||
6.0.0-2, with a quoted name, because an unquoted `#7006` truncated it as a YAML
|
||||
comment. The re-run was 5/5 verify OK. The guard skips on esh-pve (tested).
|
||||
3. `embed-rerank` deploy. `deploy-stack.sh --yes` was **denied by the auto-mode
|
||||
classifier as a blind apply**, so the pattern now is: preview with
|
||||
`echo n | deploy-stack.sh …` (it shows the diff and aborts), then `echo y |`.
|
||||
`.env` = example + `HOST_NAME=nh3-ml1`, `HOST_IP=10.100.50.80`. Healthy in about
|
||||
120 s including the image pull; 2.7 GB VRAM.
|
||||
|
||||
## Parity and speed (vs esh-ml1)
|
||||
|
||||
Script: session scratchpad, not committed. Corpus: 1,120 repo-doc paragraphs
|
||||
plus 6 fixed texts, 2 runs per host, 50 queries.
|
||||
- Embedding cosine esh↔nh3: median 0.999998, min 0.999993. The floors are esh
|
||||
0.999995 and nh3 0.999993.
|
||||
- Negative control (different texts): median 0.50.
|
||||
- overlap@10 = 1.000 both ways. Positive control, MRL-256: 0.684.
|
||||
- Rerank, 100×20: top-1 1.00, max score diff 0.0014 (the esh-esh floor is
|
||||
0.0020). Positive control, a 4-word query: top-5 order 0.09, max diff 0.97.
|
||||
- Speed on-box, 3 interleaved reps: short embed p50 6.6–6.7 vs 7.0 ms; bulk
|
||||
104–105 vs 106–108 passages/s; rerank-20 p50 156–159 vs 152–161 ms. The same.
|
||||
|
||||
## Monitoring and wiring
|
||||
|
||||
- Beszel: agent (`hosts/nh3-ml1.yaml`, same hub key as esh-ml1), system
|
||||
`1feeeq61g4mkqre`, 5 alerts, GPU samples present.
|
||||
- ⚠ zsh gotcha: `for a in "X 1 2"; do set -- $a` does NOT word-split in zsh, so
|
||||
the first batch of alert POSTs failed silently (empty fields). Use a function
|
||||
with explicit args.
|
||||
- Kuma #29/#30 (the seed also refreshed the esh-ml1 descriptions). Homepage
|
||||
`nh3-ml1-docker` shows 2 cards with no restart. Dozzle agent added, and the hub
|
||||
`.env` backed up as `.env.bak-20260925-nh3ml1` before the edit.
|
||||
- DNS `nh3-ml1.nh3.internal` on all 3 resolvers. `~/.ssh/config` `Host nh3-ml1`;
|
||||
`servers/nh3-ml1/` has ssh-target, system-details and README.
|
||||
|
||||
## Found along the way
|
||||
|
||||
- **Kernel 6.8.12-43 came from our own playbook.** The headers step's
|
||||
`proxmox-headers-6.8` install upgraded the `proxmox-kernel-6.8` meta
|
||||
(apt history 14:19:23). The visit reboot booted it. Documented in
|
||||
`pve-nvidia-host.yaml`.
|
||||
- **btmtk Bluetooth oops on every -4x boot** (2/2 lines on each of 3 -43 boots,
|
||||
0 on the -11 boots; esh-pve -42 is the same). It taints `D`. Only the BT worker
|
||||
dies. Proposed fix: blacklist btusb on both PVE hosts. Not done.
|
||||
- **AMT is not reachable.**
|
||||
- The LM port has a 1 Gb carrier (brief unbridged `ip link up` with IPv6
|
||||
disabled, then restored; ops-log recorded).
|
||||
- No UDM client or lease for `58:47:ca:76:96:0e`.
|
||||
- A /dev/tcp sweep of 16992/16993 over 10.100.{0,10,50,250}.0/24 found nothing.
|
||||
Its positive controls, :8390 and :8006, did answer.
|
||||
- The USW Pro 24's up ports (19, 22, 23, 26) are all other devices. So the cable
|
||||
is probably on nh3-sw1 (`10.100.250.2`), where SSH times out and no creds are
|
||||
held.
|
||||
- Asked Prime.
|
||||
|
||||
## Open (Prime's)
|
||||
|
||||
- Gateway routing for `qwen3-embedding` / `reranker`. Recommendation: load-share
|
||||
(identical cards, and a second site).
|
||||
- AMT network settings and switch port. The NanoKVM stays on nh3-pve until AMT KVM
|
||||
shows the console.
|
||||
- btusb blacklist on nh3-pve and esh-pve. Low priority.
|
||||
@@ -1,61 +0,0 @@
|
||||
# `[2026-09-25]` nh3-ml1 — second embed/rerank GPU LXC on nh3-pve (DEFERRED: Secure Boot)
|
||||
|
||||
**Ask (Prime, ~1410 PT):** "stand up an nh3-ml1 that takes advantage of that GPU"
|
||||
(the RTX 2000E Ada installed in nh3-pve that morning, `01:00.0`).
|
||||
|
||||
## Decisions taken on Prime's behalf (he said "clear to proceed" at ~1418)
|
||||
|
||||
- **Purpose: the second TEI embed/rerank backend.** esh-ml1's README had been
|
||||
waiting on "the second RTX 2000" for failover. No reward-seat copy is planned,
|
||||
because that seat has no working consumer.
|
||||
- **Placement: CT 109, `10.100.50.80/24`, VLAN 50, gw `.1`, rootfs on `local-zfs`,
|
||||
6 cores / 16 GB / 80 GB, startup order 30.** The address came from the UDM
|
||||
(`nh3-servers` DHCP pool `.150–.249`; in use `.40 .42 .45 .46 .50 .60 .71 .90`).
|
||||
`.80` mirrors esh-ml1. nh3-pve had 40 of 62 GB RAM free.
|
||||
- **Not backed up**, like esh-ml1. nh3-pve's vzdump job is `all 1` (exclude 107),
|
||||
so `gpu-lxc.yaml` adds the CT to that job's `exclude` list. The guard and the
|
||||
step were tested against sample configs: one positive case, plus negatives for
|
||||
already excluded, a vmid-list job, a non-vzdump `all 1` stanza and a missing
|
||||
file.
|
||||
- **Playbooks made host-generic (`bc278d4`):** `esh-pve-nvidia-host` →
|
||||
`pve-nvidia-host`, which gained headers/dkms/build-essential, a nouveau
|
||||
blacklist and an unload that refuses if nouveau has bound a device.
|
||||
`esh-ml1-lxc` → `gpu-lxc`, whose host vars have **no defaults**; elway aborts on
|
||||
an undefined `{{ var }}` (tested). Per-host `--var` lines are in its header.
|
||||
- **`stacks/embed-rerank` Homepage labels** take `HOST_NAME` / `HOST_IP`. The
|
||||
defaults render byte-identical to esh-ml1's live labels, so nothing is
|
||||
recreated there.
|
||||
|
||||
## What happened
|
||||
|
||||
1. The first `pve-nvidia-host` run died on host-key verification for
|
||||
`nh3-pve.nh3.internal`. Every `when:` got ssh rc 255 and SKIPPED, and verify
|
||||
failed. I pinned the name after confirming its ed25519 key equals the known
|
||||
IP entry.
|
||||
2. The second run installed headers 6.8.12-11 + `proxmox-headers-6.8` + dkms +
|
||||
build-essential, blacklisted and unloaded nouveau (it had failed its probe
|
||||
with `gsp ctor failed: -2` and was bound to nothing), then **the DKMS module
|
||||
was refused at load: Secure Boot is ON** (`mokutil --sb-state` enabled;
|
||||
lockdown `integrity`; the only MOK is the Proxmox Secure Boot CA). esh-pve,
|
||||
the same MS-01 and BIOS, has Secure Boot OFF. The installer rolled back: no
|
||||
dkms entry, no `nvidia-smi`. All guests were unaffected.
|
||||
3. `7ddd116` added a pre-flight that refuses when Secure Boot is on without an
|
||||
enrolled `/var/lib/dkms/mok.pub`. Tested: it refuses on nh3-pve and passes on
|
||||
esh-pve.
|
||||
|
||||
## Open calls (Prime's)
|
||||
|
||||
- **Secure Boot:** (1) turn it off in the BIOS on the NH3 visit (recommended; one
|
||||
more toggle in the menu he is already in for IGFX, and it matches esh-pve);
|
||||
(2) keep it and enroll a DKMS MOK (prep remotely, then answer MokManager at
|
||||
boot; DKMS 3.x then signs rebuilds itself); (3) a VFIO VM (works without a
|
||||
visit, but reverses his 09-24 "LXC + host driver, not VFIO" rule, and the RTX is
|
||||
currently the boot VGA). Options 1 and 2 need the console at boot, which is
|
||||
blind until IGFX is set, so the visit carries both.
|
||||
- **Gateway routing** once parity passes: recommendation load-share. The two
|
||||
cards are identical, and the old "don't load-share" rule came from pairing
|
||||
esh-ml1 with the much faster fv-ml1.
|
||||
|
||||
## Resume
|
||||
|
||||
The Current-state checklist, step 9. Nothing host-side exists yet for CT 109.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-25]` nh3-pve AMT LIVE: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None
|
||||
|
||||
**nh3-pve AMT LIVE: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None** (Prime). Root cause of "no IP": Linux held the I226-LM down, so igc powered off the PHY; the fix is `auto enp88s0` + IPv6 off + `arp_ignore=8`. Password vaulted `nh3-pve/amt-admin`; tool `scripts/amt-wsman.py`. `0f032fa`…`50f113c`.
|
||||
@@ -1,55 +0,0 @@
|
||||
# `[2026-09-25]` Reward seat audit + move to esh-ml1; esh-ml1 monitoring wired
|
||||
|
||||
**Prime:** "Audit the reward seat to see if something has superseded what we
|
||||
deployed — then move it to esh-ml1 as well, then wire up telemetry and monitoring."
|
||||
|
||||
**Audit (full record `stacks/reward-seat/README.md`).**
|
||||
- Not superseded. Skywork-Reward-V2-Llama-3.1-8B is #1 of 188 in AllenAI's official
|
||||
RewardBench 2 per-sample results (mean over the 5 non-Ties subsets, 84.7; the
|
||||
dataset was last updated 2025-12-11). No Skywork V3 exists. The -40M sibling is
|
||||
vendor-marked experimental. Caveat: 2026 models never submitted to RewardBench 2
|
||||
would not appear.
|
||||
- Our local AWQ W4A16: 0.847 vs published bf16 0.860 on 150 prompts (±2.9 pt SE),
|
||||
96.2% pairwise agreement, Pearson 0.991.
|
||||
- ⚠ vLLM adds a SECOND BOS to already-templated text unless
|
||||
`add_special_tokens:false` is sent: −2.7 pts. That was a harness confound in the
|
||||
first measurement (0.820), caught by checking `/tokenize`.
|
||||
- **No working consumer.** 0 requests 2026-09-13 → 09-25. Worldtree Domari
|
||||
hardcodes the dead `10.250.50.54:8003` and sends a non-vLLM schema
|
||||
(`prompt`/`responses` → `rewards`), so it always fell back to Selene/chat-judge.
|
||||
worldtree-dev told (single message; their call).
|
||||
|
||||
**Move.**
|
||||
- sha256-identical model copy.
|
||||
- vLLM v0.24.0 on esh-ml1 `:8003`, util 0.55 (KV 21,328 tokens = 1.30× of a 16k
|
||||
request).
|
||||
- Parity vs fv-ml1 on 600 RewardBench 2 texts:
|
||||
- 149/150 verdicts and 99.8% pairwise signs identical.
|
||||
- Raw |Δ| median 0.049; p99 0.22 vs the self-noise p99 of 0.16–0.20.
|
||||
- Gateway `/scalar-judge` → `10.0.50.80:8003` (verified 3/3).
|
||||
- fv-ml1 `vllm-reward` removed; `stacks/vllm` now = `vllm-coder` only.
|
||||
|
||||
**Monitoring.**
|
||||
- **Beszel:** NVIDIA agent (GPU util/VRAM/power/temp), system
|
||||
`ridkfdpwfq3f730`, 5 alerts incl. Temperature >85 °C.
|
||||
- **Kuma:** #27 embed, #28 rerank (the only seats inside Kuma's lane, because they
|
||||
are the sole backends). Reward excluded (no consumer).
|
||||
- **Homepage:** dockerd tcp/2375 on 10.0.50.80, which restarted docker (all
|
||||
services healthy within ~31 s), and `esh-ml1-docker` in docker.yaml. Three cards
|
||||
appeared without a Homepage restart.
|
||||
- **Dozzle:** agent v10.4.1 pinned to the hub's version. It needed an empty
|
||||
`traefik-net` network.
|
||||
|
||||
**Found while wiring (same "rots on re-IP" class as the a3 alias):**
|
||||
- The Dozzle hub still listed ana-ml2 `10.250.50.54` and irv-ml1's retired
|
||||
`10.100.79.3`. fv-ml1 and irv-ml1 logs had been missing since their moves. Fixed;
|
||||
the hub now reports 7 clients.
|
||||
- nh3-dev's Beszel agent had been down since the 09-24 NH3 power recovery (bind of
|
||||
the automounted `/mnt/smithy` failed at boot; docker never retries a failed
|
||||
create). Started; the Beszel "up" alert reached althing, an end-to-end proof of
|
||||
the alert route. Recurs on the next cold start until fixed.
|
||||
- The nh3-docker Dozzle agent has been stopped by hand for ~5 months. Left alone.
|
||||
|
||||
⚠ While checking the Beszel credential's shape, its password was printed into this
|
||||
session's transcript. The transcript stays on nh3-dev and it was not written to any
|
||||
memory; rotation offered to Prime.
|
||||
@@ -1,43 +0,0 @@
|
||||
# `[2026-09-25]` TEI adopted as the fleet embed/rerank engine; esh-ml1 is the sole backend; fv-ml1 seats retired
|
||||
|
||||
**Prime's rulings (2026-09-25):**
|
||||
- Utility seats move off fv-ml1 to a utility host like esh-ml1, accepting slower
|
||||
service; the Blackwells are kept for higher-requirement work.
|
||||
- **parakeet STAYS on fv-ml1** (real-time chat path, every ms counts).
|
||||
- **"TEI is embed/reranker server for esh-ml1 and the FLEET in general, in
|
||||
future."**
|
||||
- fv-ml1 retires both embed + rerank once esh-ml1 is up. A **second RTX 2000 is
|
||||
coming to the fleet soon**; until then embed/rerank has one backend (accepted).
|
||||
|
||||
**Evidence chain:** speed A/B vs fv-ml1 (single queries a wash, bulk 3–10×
|
||||
slower on the Ada); whole-novel test (*The Stand*, 12,814 paragraphs) through
|
||||
the gateway ~12 min one per request on either site, because the gateway is the
|
||||
bottleneck; TEI bake-off (`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`):
|
||||
matches vLLM vectors, no re-embed, slower on bulk, much lighter. Dvalin
|
||||
(Grok research peer) reviewed: agree with caveats. Two caveats were actioned:
|
||||
fail-closed truncation (TEI's default silently embedded a 16,384-token prefix of a
|
||||
40k input and returned 200) and nevermore's 0.3 rerank threshold (0 flips in 2,000
|
||||
scores; low sensitivity, since few scores sit near 0.3).
|
||||
|
||||
**Cut-over, 0820–0830 PT:**
|
||||
1. esh-ml1: vLLM pair down, TEI `tei-embed` :8001 + `tei-rerank` :8013 up (same
|
||||
ports). The old `.env` is kept as `.env.vllm-retired-2026-09-25`. The vLLM image
|
||||
(~30 GB) is kept in case the reward classifier (vLLM-only) moves here.
|
||||
2. Gateway: `qwen3-embedding` → esh-ml1 only (`hosted_vllm/`, unchanged
|
||||
address); `reranker` → **`huggingface/`** provider (hosted_vllm 422s on TEI's
|
||||
`texts` body); `order` fields removed; LiteLLM restarted (~58 s). The DB
|
||||
alias `reranker-a3-bge-v2-m3` was PATCHed to the same TEI target.
|
||||
3. Verified through the gateway against the still-running FV seats: embed
|
||||
cosine median 0.999927 / min 0.999881 (n=203); rerank top-1/top-3 29/30 (2 of
|
||||
30 lists had a duplicate-source tie; the disagreeing query wasn't logged).
|
||||
4. **No direct (non-gateway) consumers existed**: per day, fv-ml1 seat POSTs =
|
||||
gateway spend-log rows exactly (09-20→09-24). After the repoint, the only FV hits
|
||||
were my own verification calls.
|
||||
5. fv-ml1: `vllm-embed` + `vllm-rerank-a3` stopped and removed; `stacks/vllm`
|
||||
compose + `.env.example` updated (reward + coder untouched, 10–11 days up);
|
||||
GPU 1 93,086 → 86,956 MiB (~6.1 GB freed).
|
||||
|
||||
**Open:** esh-ml1 is now load-bearing with no monitoring (no Beszel, Kuma or
|
||||
Homepage). Next: "what else moves". vllm-reward (9.2 GB, vLLM-only) is the
|
||||
candidate and fits beside TEI's ~2.6 GB on 16 GB; vllm-coder (6.3 GB) is
|
||||
latency-sensitive.
|
||||
+14
-21
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-08 ~0740 PT (patch night: fv-ml1 kernel 6.12.111 + NVIDIA 580.178.04 + ZFS 2.3.9, Docker 29.8.2 on nh3-docker + esh-docker-vm; nh3-svos on SVOS a737a06 with knowledge/terminal/post office live; albok-service 0.1.13 → 0.1.14; gateway glm-5.3 + gen-frontier on 5.3; heid's runaway 76 GB sweep killed; Hermes launchers repaired.)_
|
||||
_Last updated: 2026-10-09 ~1610 PT (nh3-svos: SVOS served on the LAN too, voice live, refreshed to ac42ace then kernel 0.0.8 — thaw race and the SQLite-connection fd outage both fixed; SVOS app-traffic ingress for U13 APPROVED, build deferred; fv-ml1 off the mesh by design; albok library building + token swap; Worldtree gate 10-09 filing FAIL under watch; Homepage descriptions.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -113,27 +113,29 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-10-08 ~0740 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
|
||||
_As of 2026-10-09 ~1610 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
|
||||
|
||||
### 2026-10-08: live now (as of 2026-10-08 ~0740 PT)
|
||||
### 2026-10-09: live now (as of 2026-10-09 ~1610 PT)
|
||||
|
||||
- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated).
|
||||
- **Approved, NOT built (do not start before Prime says so after this snapshot):** SVOS app-traffic ingress for U13. Details, release (`afab123`, v0.0.9, schema 12) and gotchas are in the 2026-10-09 Recent-decisions entry; svos-dev's edit doc is `docs/operations/u13-svos-edit.md`; highseat-dev owns the schedule.
|
||||
- **Owed by me (no input needed):**
|
||||
- albok `.browse/` exposure test before ANY share: pick a surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on nh3-albok, and a fresh open through the client's cached listing must FAIL; also record the failed-rename case. Report on albok-dev's thread; the operator's user needs the restricted wings' groups on the share host. → `stacks/albok-service/README.md`
|
||||
- fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024), now ~10-10, since every seat restarted at 0034 10-08.
|
||||
- fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024) ~10-10; every seat restarted at 0034 10-08.
|
||||
- Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed.
|
||||
- sfsrv-ana `pre-pbs4` snapshot delete is unverified (infra-ops has no key there; ask infra-hermes).
|
||||
- SVOS thaw fix needs repeats (one clean drill is evidence, not proof): after the nightly 21:30 nh3-pve-2 vzdumps, grep nh3-svos `journalctl -u 'svos-*' --since 21:29` for `startup or service failed`; if one reappears, send svos-dev one line.
|
||||
- Re-read nh3-nas usage after the 10-08 GC (79% at 10-07 2350; the mirror prune ran OK).
|
||||
- **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08; `pre-pve9` snapshots are gone on pfi-pve and esh-pve.
|
||||
- Re-read nh3-nas usage after the 10-08/09 GCs (79% at 10-07 2350).
|
||||
- nh3-svos housekeeping: releases `6c6abbe` and `a737a06` are no longer referenced by the front door and can be deleted (`ac42ace` holds the same static as the current `b588510`).
|
||||
- **Watching: Worldtree memory gate.** 10-09 FAILED on filing/user (9, 11, 8 of 12; median 0.75 < 0.80), the first fail that is not #416. Bootstrap from 27 prior runs puts P(median ≤ 9) ≈ 0.06 per batch, and the misses are the habitual fixtures, so it is not called a regression yet. But per-run means drifted 10.6 → 10.0 → 9.7 → 9.3 since the 10-08 fv-ml1 patch (writer + judge = flash-next on GPU 2). If the 10-10 batch also fails filing, infra-hermes frames it as a possible regression after that patch and sends it to me; then investigate flash-next. ⚠ worldtree-dev has not read mail in 6+ days, so the reports dead-end there (Prime's call, below).
|
||||
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable.
|
||||
- **Waiting on Prime:**
|
||||
- Who sees Worldtree gate failures while worldtree-dev is not reading mail (Prime, Miranda, or keep waiting).
|
||||
- Rotate the fv-ml1 BMC password? It was printed into a local session transcript on 10-08 (never left nh3-dev); infra-ops leans no.
|
||||
- albok's 200-object load check, 2.01× against a 2× bar (albok-dev: the ruling is the operator's).
|
||||
- The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and `zpool replace` line (the 2084-media-error drive is now `/dev/nvme6`, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the `parakeet` rollback).
|
||||
- Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed).
|
||||
- ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
|
||||
- **Scheduled / unchecked:** esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran.
|
||||
- **Worldtree memory gate:** every FAIL since 09-30 (0930T0809, 1004, 1007) is ONE fixture, `u-negative-ephemeral` FILED in one run — the #416 ephemeral residual, ~3 of 42 fixture-runs since bbcc1788 (~1 batch in 5 FAILs on it), NOT a recall regression; recall (filing/user median 0.83–0.92, opener/cross_session/inline_core 100%) has held throughout. Negative is a structural class, so one run's leak fails the aggregate (`_run_structural_failed`); filing alone is judged by median. 10-07: infra-hermes told to classify FAILs by failing row (msg to its skill's daily-duty section); worldtree-dev told #416 is not closed (pull-only, unread 4 days) — reopen is theirs/Prime's.
|
||||
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%).
|
||||
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
|
||||
- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
|
||||
@@ -187,6 +189,8 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-09]` **albok `library` building (fiction, nonfiction) live for unit 4.5; Nemi + albok-dev re-minted with library grants (token grants are immutable), old tokens revoked 1555 by albok-dev.** → `stacks/albok-service/README.md`
|
||||
- `[2026-10-09]` **Homepage: The High Seat's description no longer says "SVOS" (Prime: author it yourself); SVOS's now says "Internal network + mesh".**
|
||||
- `[2026-10-09]` **APPROVED by Prime (10-09), BUILD DEFERRED until after his snapshot/clear: SVOS app-traffic ingress for U13** (apps on nh3-dev reach the nh3-svos kernel). Shape agreed with svos-dev: (1) kernel `--app-listen 127.0.0.1:8410`, which never trusts loopback (svos-dev builds it in U9a); (2) a Caddy origin `svos-kernel.phasefinal.com` on 10.100.50.43, no Basic auth, bearer passed, app paths only, allowlisted to nh3-dev; (3) kernel→bridge over plain LAN HTTP; kernel→seat host (terminal) via nh3-dev's wildcard Caddy `svos-seathost.nh3.phasefinal.com` (TLS, because the leg carries secrets). nh3-dev is NOT on the mesh; the contracts now say "fleet network". `--app-listen` is ON MAIN (`68c9933`, 10-09). It ships in kernel **0.0.9** with schema **11 → 12** (migration 0012 rebuilds dispatches + summons, so the 'schema 11' acceptance line changes and a pre-refresh backup is mandatory). At the cutover the bridge gets three owner-approved runtime grants (app-postoffice → `app:app-miranda` write, `app:app-postoffice` write, `ns:pages` write, all autonomous) and installs as `svos-postoffice[push]` with `althing-core` from `~/development/althing` in apps-env. **Release is cut: svoperatingsystem `afab123` (tag v0.0.9)**: kernel 0.0.9, svos-postoffice 0.0.2, svos-terminal 0.0.3, svos-connector 0.0.7, svos-agent 0.0.3, svos-miranda 0.0.2 (U9a + U5b). The SVOS side of the cutover is written up in `docs/operations/u13-svos-edit.md` (app listener on nh3-svos, terminal app moved to nh3-dev with `[seat_host]`, bridge on nh3-dev with the converse mapping). Gotchas: the terminal app serves WITHOUT its seat host if the socket or launch dir fails (one stderr line); `serve` refuses `--app-listen` on the primary port. Build only when the edit is scheduled (highseat-dev owns the schedule) and NOT before the snapshot; the runbook refresh is ours at deploy. Tracking: svos-dev thread `01M4GPXSBKQHCQ30FVPH4R3N7G`.
|
||||
- `[2026-10-09]` **nh3-svos → kernel 0.0.8 (`b588510`, a bounded SQLite reader pool): the 10-08 22:55 EMFILE outage is fixed at the source and the LimitNOFILE stopgap is removed.** A 4-min browser soak held `svos.db` handles flat at 9–12 (0.0.7 grew +22 in 2 min). Root cause: one cached connection per short-lived anyio worker thread. → `servers/nh3-svos/README.md`
|
||||
- `[2026-10-08]` **nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the `Type=notify` unit as a pair); thaw race FIXED**: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → `servers/nh3-svos/README.md`
|
||||
@@ -280,21 +284,13 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
- `[2026-09-26]` **GPU-LXC Temperature alerts watch the GPU, not the host CPU** (`SENSORS=-coretemp_*,acpitz`); hypervisor CPU alerts at >95 °C on nh3-pve/esh-pve. The first nh3-ml1 alert was nh3-pve's CPU during vzdump. `6d901c0`.
|
||||
- `[2026-09-26]` **Abliterated LFM2.5-VL-3B parallel seat (:8032) passed brokkr's 6-image eval and stays** (direct only; stock seat kept for SFW A/B). `9cc3824`, `944bb36`.
|
||||
- `[2026-09-25]` **AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326).** — Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale… → `persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md`
|
||||
- `[2026-09-25]` **nh3-pve AMT LIVE: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None** — (Prime). → `persistent-memory.d/2026-09-25-nh3-pve-amt-live-static-10-100-250-61-on-nh3-mgmt-udm-port.md`
|
||||
- `[2026-09-25]` **pfi-gx10 AC-restore VALIDATED by Prime's AC pull; Homepage stays on esh-docker-vm (Prime) after the mmap_lock wedge reboot.** `servers/pfi-gx10/README.md`, `servers/esh-docker-vm/README.md`.
|
||||
- `[2026-09-26]` **VibeVoice ASR → Q8_0 (Prime).** WER on the 4 bundled LibriSpeech clips 3/69 → 2/69 (only the I'm/I am artifact left), ×3 identical; RTF 0.09–0.17; +1.1 GB VRAM (nh3-ml1 ~11.3/16 GB). Q4_K file removed.
|
||||
- `[2026-09-26]` **esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only** — , for ha-dev (operator-approved, relayed). → `persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md`
|
||||
- `[2026-09-26]` **Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime).** — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → `persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md`
|
||||
- `[2026-09-26]` **Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):** — LFM2.5-VL-3B on llama.cpp `:8030` (gateway `lfm25-vl-3b`, LiteLLM restarted 36 s at 0039) and… → `persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md`
|
||||
- `[2026-09-26]` **Coder seat STAYS on fv-ml1 (Prime).** — The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5×… → `persistent-memory.d/2026-09-26-coder-seat-stays-on-fv-ml1-prime.md`
|
||||
- `[2026-09-25]` **nh3-ml1 LIVE after the NH3 visit (SB off, IGFX restored): driver + CT 109 + TEI, parity vs esh-ml1 indistinguishable (cos min 0.999993 = own floor; overlap@10 1.000 vs MRL-256 control 0.684), same speed; monitoring/DNS wired. Autonomous: lxc-pve 6.0.0-2 upgrade on nh3-pve (Docker-in-LXC fix #7006). Gateway routing + AMT follow-up are Prime's.** → `persistent-memory.d/2026-09-25-nh3-ml1-live.md`
|
||||
- `[2026-09-25]` **nh3-ml1 = CT 109 @ 10.100.50.80, second TEI embed/rerank backend on nh3-pve's RTX 2000E; GPU playbooks made host-generic — BUILD DEFERRED: blocked on nh3-pve Secure Boot until the NH3 visit; SB handling and gateway routing are Prime's calls.** Tracked: `7ddd116`, Current state. → `persistent-memory.d/2026-09-25-nh3-ml1-standup.md`
|
||||
- `[2026-09-25]` **nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333).** Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: `servers/nh3-pve/README.md` (`2b49be4`, `cdd7605`).
|
||||
- `[2026-09-25]` **MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** — (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). → `persistent-memory.d/2026-09-25-ms-01-foot-gun-a-gpu-in-the-pcie-slot-renames-every-nic.md`
|
||||
- `[2026-09-25]` **Direct ESH→esh-ml1 consumer path — PARKED** (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`.
|
||||
- `[2026-09-25]` Created empty private repo `corviduo/norn` (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the `norn-dev` handle is the operator's to declare.
|
||||
- `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md`
|
||||
- `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md`
|
||||
- `[2026-09-24]` **Worldtree `admin.memory.forget` stays OFF on demo/personal** until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.
|
||||
|
||||
|
||||
@@ -309,7 +305,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
|
||||
|
||||
_228 older entries archived to archival-memory.md._
|
||||
_236 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
@@ -340,8 +336,5 @@ _228 older entries archived to archival-memory.md._
|
||||
- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md`
|
||||
- `[2026-09-26]` **Greedy exact-match as parity for a small generative seat (coder)** — same-host repeats matched only 30–38% (run B hit the prefix cache, so a different numeric path; near-tie… → `persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md`
|
||||
- `[2026-09-26]` **zsh traps in ad-hoc test loops:** `set -- $a` does NOT word-split, so alert POSTs went out with empty fields; and `local path=…` inside a function CLOBBERS `$PATH` (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and `${=var}`.
|
||||
- `[2026-09-25]` **Concluding "not on the UDM" from a port table read 15 s after link-up** — UniFi polls (~60 s); AMT was on port 6 as Prime said. Wait a poll, and prefer a which-port-lists-the-MAC check. auto-memory `feedback_polled_stats_lag_the_event`.
|
||||
- `[2026-09-25]` **The esh-pve NVIDIA DKMS recipe on nh3-pve** — it built, then the kernel refused to load it: nh3-pve has SECURE BOOT ON (esh-pve, the same MS-01, has it off). The installer rolled back. The playbook now pre-flights `mokutil --sb-state` (`7ddd116`).
|
||||
- `[2026-09-25]` **elway against an unpinned host-key name** — every `when:` hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in `when:` should fail.
|
||||
|
||||
_123 older entries archived to archival-memory.md._
|
||||
_126 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user