memory: snapshot — 2026-10-09 SVOS LAN/voice/ac42ace/kernel 0.0.8 (fd outage fixed), U13 ingress approved+deferred, albok library, gate watch; 11 entries archived
This commit is contained in:
@@ -15007,3 +15007,288 @@ reported: ana-docker `/root/.nvm` (lkraven), irv-ml1 `uv`/`uvx`, fv-ml1
|
||||
|
||||
- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage.
|
||||
_Archived 2026-10-08._
|
||||
|
||||
## Recent decisions (archived 2026-10-09 batch)
|
||||
|
||||
# `[2026-09-25]` nh3-pve AMT LIVE: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None
|
||||
|
||||
**nh3-pve AMT LIVE: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None** (Prime). Root cause of "no IP": Linux held the I226-LM down, so igc powered off the PHY; the fix is `auto enp88s0` + IPv6 off + `arp_ignore=8`. Password vaulted `nh3-pve/amt-admin`; tool `scripts/amt-wsman.py`. `0f032fa`…`50f113c`.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
- `[2026-09-25]` **pfi-gx10 AC-restore VALIDATED by Prime's AC pull; Homepage stays on esh-docker-vm (Prime) after the mmap_lock wedge reboot.** `servers/pfi-gx10/README.md`, `servers/esh-docker-vm/README.md`.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
# `[2026-09-25]` nh3-ml1 LIVE after the NH3 site visit
|
||||
|
||||
**Prime, ~1524 PT:** "secure boot disabled, amt enabled, card plugged." This is the
|
||||
resume of `2026-09-25-nh3-ml1-standup.md`. nh3-pve came up at 1523.
|
||||
|
||||
## Post-boot checklist: all green
|
||||
|
||||
- Uplink is `enp2s0f1np1`. Guests 100, 101, 102, 105 and CTs 103, 106, 107 are up;
|
||||
104 and 108 are off as expected.
|
||||
- NFS automounts are good on nh3-docker, nh3-dev and pbs-nh3. The post office is
|
||||
push/reachable. The 14 queued althing messages were all Beszel/Kuma "UP"
|
||||
recoveries, so nothing needed acting on.
|
||||
- Secure Boot is disabled. `00:02.0` is back with `boot_vga=1` and i915 loaded.
|
||||
|
||||
## Build (elway, both playbooks idempotent)
|
||||
|
||||
1. `pve-nvidia-host.yaml`: the driver installed (DKMS, 36 s), 6/6 verify.
|
||||
2. `gpu-lxc.yaml`: CT 109 built, then **verify 3 FAILED**. `docker run` died with
|
||||
`open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission
|
||||
denied`.
|
||||
- The cause is the lxc-pve version, not runc. Both CTs run runc 1.5.1. nh3-pve
|
||||
(PVE 8.4.1) had lxc-pve **6.0.0-1**, and esh-pve (8.4.20) has **6.0.0-2**,
|
||||
whose changelog is Proxmox "fix #7006: do not restrict /proc or /sys if
|
||||
nesting is enabled".
|
||||
- **Decided autonomously:** upgrade that one package (a dry-run showed
|
||||
1 upgraded, 0 new, 0 removed) and `pct reboot 109`. GPU-in-docker then passed.
|
||||
ops-log recorded.
|
||||
- The playbook gained a first step that upgrades lxc-pve when it is below
|
||||
6.0.0-2, with a quoted name, because an unquoted `#7006` truncated it as a YAML
|
||||
comment. The re-run was 5/5 verify OK. The guard skips on esh-pve (tested).
|
||||
3. `embed-rerank` deploy. `deploy-stack.sh --yes` was **denied by the auto-mode
|
||||
classifier as a blind apply**, so the pattern now is: preview with
|
||||
`echo n | deploy-stack.sh …` (it shows the diff and aborts), then `echo y |`.
|
||||
`.env` = example + `HOST_NAME=nh3-ml1`, `HOST_IP=10.100.50.80`. Healthy in about
|
||||
120 s including the image pull; 2.7 GB VRAM.
|
||||
|
||||
## Parity and speed (vs esh-ml1)
|
||||
|
||||
Script: session scratchpad, not committed. Corpus: 1,120 repo-doc paragraphs
|
||||
plus 6 fixed texts, 2 runs per host, 50 queries.
|
||||
- Embedding cosine esh↔nh3: median 0.999998, min 0.999993. The floors are esh
|
||||
0.999995 and nh3 0.999993.
|
||||
- Negative control (different texts): median 0.50.
|
||||
- overlap@10 = 1.000 both ways. Positive control, MRL-256: 0.684.
|
||||
- Rerank, 100×20: top-1 1.00, max score diff 0.0014 (the esh-esh floor is
|
||||
0.0020). Positive control, a 4-word query: top-5 order 0.09, max diff 0.97.
|
||||
- Speed on-box, 3 interleaved reps: short embed p50 6.6–6.7 vs 7.0 ms; bulk
|
||||
104–105 vs 106–108 passages/s; rerank-20 p50 156–159 vs 152–161 ms. The same.
|
||||
|
||||
## Monitoring and wiring
|
||||
|
||||
- Beszel: agent (`hosts/nh3-ml1.yaml`, same hub key as esh-ml1), system
|
||||
`1feeeq61g4mkqre`, 5 alerts, GPU samples present.
|
||||
- ⚠ zsh gotcha: `for a in "X 1 2"; do set -- $a` does NOT word-split in zsh, so
|
||||
the first batch of alert POSTs failed silently (empty fields). Use a function
|
||||
with explicit args.
|
||||
- Kuma #29/#30 (the seed also refreshed the esh-ml1 descriptions). Homepage
|
||||
`nh3-ml1-docker` shows 2 cards with no restart. Dozzle agent added, and the hub
|
||||
`.env` backed up as `.env.bak-20260925-nh3ml1` before the edit.
|
||||
- DNS `nh3-ml1.nh3.internal` on all 3 resolvers. `~/.ssh/config` `Host nh3-ml1`;
|
||||
`servers/nh3-ml1/` has ssh-target, system-details and README.
|
||||
|
||||
## Found along the way
|
||||
|
||||
- **Kernel 6.8.12-43 came from our own playbook.** The headers step's
|
||||
`proxmox-headers-6.8` install upgraded the `proxmox-kernel-6.8` meta
|
||||
(apt history 14:19:23). The visit reboot booted it. Documented in
|
||||
`pve-nvidia-host.yaml`.
|
||||
- **btmtk Bluetooth oops on every -4x boot** (2/2 lines on each of 3 -43 boots,
|
||||
0 on the -11 boots; esh-pve -42 is the same). It taints `D`. Only the BT worker
|
||||
dies. Proposed fix: blacklist btusb on both PVE hosts. Not done.
|
||||
- **AMT is not reachable.**
|
||||
- The LM port has a 1 Gb carrier (brief unbridged `ip link up` with IPv6
|
||||
disabled, then restored; ops-log recorded).
|
||||
- No UDM client or lease for `58:47:ca:76:96:0e`.
|
||||
- A /dev/tcp sweep of 16992/16993 over 10.100.{0,10,50,250}.0/24 found nothing.
|
||||
Its positive controls, :8390 and :8006, did answer.
|
||||
- The USW Pro 24's up ports (19, 22, 23, 26) are all other devices. So the cable
|
||||
is probably on nh3-sw1 (`10.100.250.2`), where SSH times out and no creds are
|
||||
held.
|
||||
- Asked Prime.
|
||||
|
||||
## Open (Prime's)
|
||||
|
||||
- Gateway routing for `qwen3-embedding` / `reranker`. Recommendation: load-share
|
||||
(identical cards, and a second site).
|
||||
- AMT network settings and switch port. The NanoKVM stays on nh3-pve until AMT KVM
|
||||
shows the console.
|
||||
- btusb blacklist on nh3-pve and esh-pve. Low priority.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
# `[2026-09-25]` nh3-ml1 — second embed/rerank GPU LXC on nh3-pve (DEFERRED: Secure Boot)
|
||||
|
||||
**Ask (Prime, ~1410 PT):** "stand up an nh3-ml1 that takes advantage of that GPU"
|
||||
(the RTX 2000E Ada installed in nh3-pve that morning, `01:00.0`).
|
||||
|
||||
## Decisions taken on Prime's behalf (he said "clear to proceed" at ~1418)
|
||||
|
||||
- **Purpose: the second TEI embed/rerank backend.** esh-ml1's README had been
|
||||
waiting on "the second RTX 2000" for failover. No reward-seat copy is planned,
|
||||
because that seat has no working consumer.
|
||||
- **Placement: CT 109, `10.100.50.80/24`, VLAN 50, gw `.1`, rootfs on `local-zfs`,
|
||||
6 cores / 16 GB / 80 GB, startup order 30.** The address came from the UDM
|
||||
(`nh3-servers` DHCP pool `.150–.249`; in use `.40 .42 .45 .46 .50 .60 .71 .90`).
|
||||
`.80` mirrors esh-ml1. nh3-pve had 40 of 62 GB RAM free.
|
||||
- **Not backed up**, like esh-ml1. nh3-pve's vzdump job is `all 1` (exclude 107),
|
||||
so `gpu-lxc.yaml` adds the CT to that job's `exclude` list. The guard and the
|
||||
step were tested against sample configs: one positive case, plus negatives for
|
||||
already excluded, a vmid-list job, a non-vzdump `all 1` stanza and a missing
|
||||
file.
|
||||
- **Playbooks made host-generic (`bc278d4`):** `esh-pve-nvidia-host` →
|
||||
`pve-nvidia-host`, which gained headers/dkms/build-essential, a nouveau
|
||||
blacklist and an unload that refuses if nouveau has bound a device.
|
||||
`esh-ml1-lxc` → `gpu-lxc`, whose host vars have **no defaults**; elway aborts on
|
||||
an undefined `{{ var }}` (tested). Per-host `--var` lines are in its header.
|
||||
- **`stacks/embed-rerank` Homepage labels** take `HOST_NAME` / `HOST_IP`. The
|
||||
defaults render byte-identical to esh-ml1's live labels, so nothing is
|
||||
recreated there.
|
||||
|
||||
## What happened
|
||||
|
||||
1. The first `pve-nvidia-host` run died on host-key verification for
|
||||
`nh3-pve.nh3.internal`. Every `when:` got ssh rc 255 and SKIPPED, and verify
|
||||
failed. I pinned the name after confirming its ed25519 key equals the known
|
||||
IP entry.
|
||||
2. The second run installed headers 6.8.12-11 + `proxmox-headers-6.8` + dkms +
|
||||
build-essential, blacklisted and unloaded nouveau (it had failed its probe
|
||||
with `gsp ctor failed: -2` and was bound to nothing), then **the DKMS module
|
||||
was refused at load: Secure Boot is ON** (`mokutil --sb-state` enabled;
|
||||
lockdown `integrity`; the only MOK is the Proxmox Secure Boot CA). esh-pve,
|
||||
the same MS-01 and BIOS, has Secure Boot OFF. The installer rolled back: no
|
||||
dkms entry, no `nvidia-smi`. All guests were unaffected.
|
||||
3. `7ddd116` added a pre-flight that refuses when Secure Boot is on without an
|
||||
enrolled `/var/lib/dkms/mok.pub`. Tested: it refuses on nh3-pve and passes on
|
||||
esh-pve.
|
||||
|
||||
## Open calls (Prime's)
|
||||
|
||||
- **Secure Boot:** (1) turn it off in the BIOS on the NH3 visit (recommended; one
|
||||
more toggle in the menu he is already in for IGFX, and it matches esh-pve);
|
||||
(2) keep it and enroll a DKMS MOK (prep remotely, then answer MokManager at
|
||||
boot; DKMS 3.x then signs rebuilds itself); (3) a VFIO VM (works without a
|
||||
visit, but reverses his 09-24 "LXC + host driver, not VFIO" rule, and the RTX is
|
||||
currently the boot VGA). Options 1 and 2 need the console at boot, which is
|
||||
blind until IGFX is set, so the visit carries both.
|
||||
- **Gateway routing** once parity passes: recommendation load-share. The two
|
||||
cards are identical, and the old "don't load-share" rule came from pairing
|
||||
esh-ml1 with the much faster fv-ml1.
|
||||
|
||||
## Resume
|
||||
|
||||
The Current-state checklist, step 9. Nothing host-side exists yet for CT 109.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
# `[2026-09-25]` MS-01 foot-gun: a GPU in the PCIe slot renames every NIC
|
||||
|
||||
**MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). On nh3-pve that means booting with no uplink. Pinned by MAC with `.link` files before the install, and the AMT port was pulled from vmbr0. → `servers/nh3-pve/README.md`, `playbooks/nh3-pve-pin-nic-names.yaml` (`2118449`)
|
||||
_Archived 2026-10-09._
|
||||
|
||||
- `[2026-09-25]` Created empty private repo `corviduo/norn` (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the `norn-dev` handle is the operator's to declare.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
# `[2026-09-25]` Reward seat audit + move to esh-ml1; esh-ml1 monitoring wired
|
||||
|
||||
**Prime:** "Audit the reward seat to see if something has superseded what we
|
||||
deployed — then move it to esh-ml1 as well, then wire up telemetry and monitoring."
|
||||
|
||||
**Audit (full record `stacks/reward-seat/README.md`).**
|
||||
- Not superseded. Skywork-Reward-V2-Llama-3.1-8B is #1 of 188 in AllenAI's official
|
||||
RewardBench 2 per-sample results (mean over the 5 non-Ties subsets, 84.7; the
|
||||
dataset was last updated 2025-12-11). No Skywork V3 exists. The -40M sibling is
|
||||
vendor-marked experimental. Caveat: 2026 models never submitted to RewardBench 2
|
||||
would not appear.
|
||||
- Our local AWQ W4A16: 0.847 vs published bf16 0.860 on 150 prompts (±2.9 pt SE),
|
||||
96.2% pairwise agreement, Pearson 0.991.
|
||||
- ⚠ vLLM adds a SECOND BOS to already-templated text unless
|
||||
`add_special_tokens:false` is sent: −2.7 pts. That was a harness confound in the
|
||||
first measurement (0.820), caught by checking `/tokenize`.
|
||||
- **No working consumer.** 0 requests 2026-09-13 → 09-25. Worldtree Domari
|
||||
hardcodes the dead `10.250.50.54:8003` and sends a non-vLLM schema
|
||||
(`prompt`/`responses` → `rewards`), so it always fell back to Selene/chat-judge.
|
||||
worldtree-dev told (single message; their call).
|
||||
|
||||
**Move.**
|
||||
- sha256-identical model copy.
|
||||
- vLLM v0.24.0 on esh-ml1 `:8003`, util 0.55 (KV 21,328 tokens = 1.30× of a 16k
|
||||
request).
|
||||
- Parity vs fv-ml1 on 600 RewardBench 2 texts:
|
||||
- 149/150 verdicts and 99.8% pairwise signs identical.
|
||||
- Raw |Δ| median 0.049; p99 0.22 vs the self-noise p99 of 0.16–0.20.
|
||||
- Gateway `/scalar-judge` → `10.0.50.80:8003` (verified 3/3).
|
||||
- fv-ml1 `vllm-reward` removed; `stacks/vllm` now = `vllm-coder` only.
|
||||
|
||||
**Monitoring.**
|
||||
- **Beszel:** NVIDIA agent (GPU util/VRAM/power/temp), system
|
||||
`ridkfdpwfq3f730`, 5 alerts incl. Temperature >85 °C.
|
||||
- **Kuma:** #27 embed, #28 rerank (the only seats inside Kuma's lane, because they
|
||||
are the sole backends). Reward excluded (no consumer).
|
||||
- **Homepage:** dockerd tcp/2375 on 10.0.50.80, which restarted docker (all
|
||||
services healthy within ~31 s), and `esh-ml1-docker` in docker.yaml. Three cards
|
||||
appeared without a Homepage restart.
|
||||
- **Dozzle:** agent v10.4.1 pinned to the hub's version. It needed an empty
|
||||
`traefik-net` network.
|
||||
|
||||
**Found while wiring (same "rots on re-IP" class as the a3 alias):**
|
||||
- The Dozzle hub still listed ana-ml2 `10.250.50.54` and irv-ml1's retired
|
||||
`10.100.79.3`. fv-ml1 and irv-ml1 logs had been missing since their moves. Fixed;
|
||||
the hub now reports 7 clients.
|
||||
- nh3-dev's Beszel agent had been down since the 09-24 NH3 power recovery (bind of
|
||||
the automounted `/mnt/smithy` failed at boot; docker never retries a failed
|
||||
create). Started; the Beszel "up" alert reached althing, an end-to-end proof of
|
||||
the alert route. Recurs on the next cold start until fixed.
|
||||
- The nh3-docker Dozzle agent has been stopped by hand for ~5 months. Left alone.
|
||||
|
||||
⚠ While checking the Beszel credential's shape, its password was printed into this
|
||||
session's transcript. The transcript stays on nh3-dev and it was not written to any
|
||||
memory; rotation offered to Prime.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
# `[2026-09-25]` TEI adopted as the fleet embed/rerank engine; esh-ml1 is the sole backend; fv-ml1 seats retired
|
||||
|
||||
**Prime's rulings (2026-09-25):**
|
||||
- Utility seats move off fv-ml1 to a utility host like esh-ml1, accepting slower
|
||||
service; the Blackwells are kept for higher-requirement work.
|
||||
- **parakeet STAYS on fv-ml1** (real-time chat path, every ms counts).
|
||||
- **"TEI is embed/reranker server for esh-ml1 and the FLEET in general, in
|
||||
future."**
|
||||
- fv-ml1 retires both embed + rerank once esh-ml1 is up. A **second RTX 2000 is
|
||||
coming to the fleet soon**; until then embed/rerank has one backend (accepted).
|
||||
|
||||
**Evidence chain:** speed A/B vs fv-ml1 (single queries a wash, bulk 3–10×
|
||||
slower on the Ada); whole-novel test (*The Stand*, 12,814 paragraphs) through
|
||||
the gateway ~12 min one per request on either site, because the gateway is the
|
||||
bottleneck; TEI bake-off (`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`):
|
||||
matches vLLM vectors, no re-embed, slower on bulk, much lighter. Dvalin
|
||||
(Grok research peer) reviewed: agree with caveats. Two caveats were actioned:
|
||||
fail-closed truncation (TEI's default silently embedded a 16,384-token prefix of a
|
||||
40k input and returned 200) and nevermore's 0.3 rerank threshold (0 flips in 2,000
|
||||
scores; low sensitivity, since few scores sit near 0.3).
|
||||
|
||||
**Cut-over, 0820–0830 PT:**
|
||||
1. esh-ml1: vLLM pair down, TEI `tei-embed` :8001 + `tei-rerank` :8013 up (same
|
||||
ports). The old `.env` is kept as `.env.vllm-retired-2026-09-25`. The vLLM image
|
||||
(~30 GB) is kept in case the reward classifier (vLLM-only) moves here.
|
||||
2. Gateway: `qwen3-embedding` → esh-ml1 only (`hosted_vllm/`, unchanged
|
||||
address); `reranker` → **`huggingface/`** provider (hosted_vllm 422s on TEI's
|
||||
`texts` body); `order` fields removed; LiteLLM restarted (~58 s). The DB
|
||||
alias `reranker-a3-bge-v2-m3` was PATCHed to the same TEI target.
|
||||
3. Verified through the gateway against the still-running FV seats: embed
|
||||
cosine median 0.999927 / min 0.999881 (n=203); rerank top-1/top-3 29/30 (2 of
|
||||
30 lists had a duplicate-source tie; the disagreeing query wasn't logged).
|
||||
4. **No direct (non-gateway) consumers existed**: per day, fv-ml1 seat POSTs =
|
||||
gateway spend-log rows exactly (09-20→09-24). After the repoint, the only FV hits
|
||||
were my own verification calls.
|
||||
5. fv-ml1: `vllm-embed` + `vllm-rerank-a3` stopped and removed; `stacks/vllm`
|
||||
compose + `.env.example` updated (reward + coder untouched, 10–11 days up);
|
||||
GPU 1 93,086 → 86,956 MiB (~6.1 GB freed).
|
||||
|
||||
**Open:** esh-ml1 is now load-bearing with no monitoring (no Beszel, Kuma or
|
||||
Homepage). Next: "what else moves". vllm-reward (9.2 GB, vLLM-only) is the
|
||||
candidate and fits beside TEI's ~2.6 GB on 16 GB; vllm-coder (6.3 GB) is
|
||||
latency-sensitive.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
## Tried and abandoned (archived 2026-10-09 batch)
|
||||
|
||||
- `[2026-09-25]` **Concluding "not on the UDM" from a port table read 15 s after link-up** — UniFi polls (~60 s); AMT was on port 6 as Prime said. Wait a poll, and prefer a which-port-lists-the-MAC check. auto-memory `feedback_polled_stats_lag_the_event`.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
- `[2026-09-25]` **The esh-pve NVIDIA DKMS recipe on nh3-pve** — it built, then the kernel refused to load it: nh3-pve has SECURE BOOT ON (esh-pve, the same MS-01, has it off). The installer rolled back. The playbook now pre-flights `mokutil --sb-state` (`7ddd116`).
|
||||
_Archived 2026-10-09._
|
||||
|
||||
- `[2026-09-25]` **elway against an unpinned host-key name** — every `when:` hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in `when:` should fail.
|
||||
_Archived 2026-10-09._
|
||||
|
||||
Reference in New Issue
Block a user