memory: snapshot for clear — SVOS v0.0.16+judge live, v0.0.17 pushed (deploy at Prime's window); albok u6 done; Worldtree b195; 30+3 entries archived
This commit is contained in:
@@ -15932,3 +15932,75 @@ independent check across all 8 found 0 leaking units. nh3-docker's scheduled uni
|
||||
|
||||
`[2026-09-27]` **Relying on the build cache surviving on esh-ml1** — the augaman v0.1.3 build missed its dependency layer there (fv-ml1 hit it; reason unexplained, suspect the `docker builder prune` runs) and the rootfs peaked at 90%. Check for ≥15 GB free, or remove the old image, before building there.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
## Recent decisions (archived 2026-10-18 batch)
|
||||
|
||||
- `[2026-10-04]` **albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime)** — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns `nh3-pve-2`; copy verified by digest (dir st_size differs on ext4, not data). → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-04]` **The whole fleet is on PVE 9.2.21 and PBS 4.2.7** — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-04]` **sudo session recordings: archived weekly, bulk data never kept (Prime)** — `BULK_DATA !log_output` on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → `services/sudo-io-archive/README.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-04]` **VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use)** — off, `onboot 0`, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-03]` **PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host")**: an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → `services/pve-nag-patch/README.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-03]` **nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 `nic3`, AMT on `nic0` with `auto` kept up, 1 TB wiped into `vmstore` (913 GiB), stale boot entry gone** → `persistent-memory.d/2026-10-03-nh3-pve-2-finished-prime-on-site-host-10-100-250.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-03]` **CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232)** → `persistent-memory.d/2026-10-03-cira-tunnel-watchdog-live-on-pfi-tacticalrmm-prime-go-1232.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-03]` **Worldtree memory-gate duty REFRAMED: regression check, not a gate** → `persistent-memory.d/2026-10-03-worldtree-memory-gate-duty-reframed-regression-check-not-a-gate.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-03]` **albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1** → `persistent-memory.d/2026-10-03-albok-service-0-1-2-live-on-nh3-docker-albok-dev-ask.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-02]` **esh-pve-2 (MS-03 at ESH, future `esh-dev` host) onboarded: infra-ops (Prime), 1 TB → LVM-thin `vmstore`, AMT phoning home as `esh-pve-2-amt`** → `persistent-memory.d/2026-10-02-esh-pve-2-ms-03-at-esh-future-esh-dev-host-onboarded-infra-ops.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-02]` **Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684)** → `persistent-memory.d/2026-10-02-demo-outage-13-min-rolled-back-worldtree-dev-urgent-6684.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-02]` **nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`)** → `persistent-memory.d/2026-10-02-nh3-pve-s-amt-is-in-tacticalrmm-s-meshcentral-group.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-02]` **Prime: dev-backup gets dailies + weeklies — DONE** → `persistent-memory.d/2026-10-02-prime-dev-backup-gets-dailies-weeklies-done.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-02]` **ESH `tank` (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629** → `persistent-memory.d/2026-10-02-esh-tank-esh-nas-pve-175-t-raidz2-2-was-degraded.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-02]` **albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): `http://albok.nh3.internal:8392`** → `persistent-memory.d/2026-10-02-albok-service-0-1-0-live-on-nh3-docker-albok-dev-ask.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-02]` **nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03** → `persistent-memory.d/2026-10-02-nh3-pve-2-status-at-end-of-day-amt-done-proxmox.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-02]` **Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as `nh3-pve-2` at NH3 (Prime, ~0910)** → `persistent-memory.d/2026-10-02-prime-ms-03s-get-proxmox-dummy-plugs-in-hand-one.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-02]` **nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot)** → `persistent-memory.d/2026-10-02-nh3-dev-root-grown-250-378-gb-prime-resized-scsi0.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-01]` **MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev)** → `persistent-memory.d/2026-10-01-mia-make-it-animatable-v2-auto-rigger-live-on-fv-ml1-gpu-3.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-01]` **Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed** → `persistent-memory.d/2026-10-01-prime-clean-up-the-1700-backups-dev-backup-retention-fixed.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-01]` **Prime: gen-small keeps its 8 GiB KV pin, and parakeet-nemo keeps CUDA graphs OFF** (~2–8 ms at short clips, accepted). GPU 0's ~1 GB of spare memory is enough with the seat's hard 3,840 MiB cap.
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-01]` **⚠ Gitea's `[webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8` silently REJECTS headscale mesh IPs (100.64.0.0/10)** → `persistent-memory.d/2026-10-01-gitea-s-webhook-allowed-host-list-external-10-0.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-01]` **vh/arbo push webhook repointed from the retired wg0 lifeline `10.100.79.3:9009` to irv-ml1's mesh address `100.64.0.6:9009`** → `persistent-memory.d/2026-10-01-vh-arbo-push-webhook-repointed-from-the-retired-wg0.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-01]` **Prime: delete the bench leftovers, no upstream for Scriberr, push** → `persistent-memory.d/2026-10-01-prime-delete-the-bench-leftovers-no-upstream-for-scriberr.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-01]` **irv-ml1 /storetank reclaim done:** Prime ruled through comfy-dev, which deleted 84 files of its own (260 → 299 GB free). Tiers B/C/D got no ruling (infra-hermes thread `01M3TCSYRSFNAPA9BPQTMFQ6KJ`).
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-09-27]` **Worldtree reward path: d8726a13 + `providers.yaml` `reward_models.scalar-judge` live on demo; personal waited on a newer image.** → `persistent-memory.d/2026-09-27-worldtree-reward-path-live.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-09-26]` **esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only** — , for ha-dev (operator-approved, relayed). → `persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-09-26]` **Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime).** — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → `persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md`
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-09-26]` **Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):** — LFM2.5-VL-3B on llama.cpp `:8030` (gateway `lfm25-vl-3b`, LiteLLM restarted 36 s at 0039) and… → `persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md`
|
||||
_Archived 2026-10-18._
|
||||
|
||||
## Tried and abandoned (archived 2026-10-18 batch)
|
||||
|
||||
- `[2026-10-04]` **`apt-get` run in-band through `qm guest exec`** — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a `systemd-run` unit. (auto-memory `feedback_no_inband_apt_through_guest_agent`)
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-04]` **Pulling files with `ssh host 'sudo cat …'`** — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and `log_output` re-recorded the whole file as a new session. Read without sudo (group-readable file).
|
||||
_Archived 2026-10-18._
|
||||
- `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** → `persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md`
|
||||
_Archived 2026-10-18._
|
||||
|
||||
+15
-43
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-18 (SVOS v0.0.14/v0.0.15 live; asset-engine retired; gate keeps scratch). Previously: 2026-10-14 (snapshot for clear, elway note updated: SVOS v0.0.11 live, pewpewstudio deployed, albok inbox live, decision seat on h2o; earlier: the h2o swap saga, U13.)_
|
||||
_Last updated: 2026-10-18 ~2125 (snapshot for clear: SVOS v0.0.16 + judge live, v0.0.17 pushed not deployed; albok u6 done; Worldtree b195; asset-engine retired). Previously: 2026-10-14 (snapshot for clear, elway note updated: SVOS v0.0.11 live, pewpewstudio deployed, albok inbox live, decision seat on h2o; earlier: the h2o swap saga, U13.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -116,20 +116,21 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-10-14 ~0005 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
|
||||
_As of 2026-10-18 ~2125 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
|
||||
|
||||
### 2026-10-14: live now (as of 2026-10-14 ~0005 PT)
|
||||
### 2026-10-18: live now (as of 2026-10-18 ~2125 PT)
|
||||
|
||||
- **Nothing is mid-action.** No ops-log claims held. eshpfi `main` is pushed through `d1192ca` (infra-ops has STANDING push on main since 2026-10-13, CLAUDE.md § Working rules); this snapshot's commit follows and is pushed. ⚠ The working tree and index are shared with infra-hermes, albok-dev and subagents: commit with `git commit -- <paths>` and read `git log origin/main..main` before pushing. `graphify-out/GRAPH_REPORT.md` stays modified on purpose.
|
||||
- **SVOS: v0.0.16 `b51aa5e` on nh3-svos since 2026-10-18 1634, with app-judge (drafting gen-large; six local task models; `[tasks].judge` on since 1642, gen `13241a40`). Card audio VERIFIED 1640 (3/3 accepted, 4.5–5.5 s).** ⚠ voice.toml `utterance_seconds` is 30 again (since 10-18 1646): the web client refused >30 and broke the voice panel. svos-dev's fix (client 0.1..120 + long replies spoken in pieces) ships with the U17 release (interface 0.0.21); after that deploy, 60 is optional. Open: one drafted scratch task from /tasks (Draft → launch → approve; expect events routing, admitted, routed, launching, started), which also verifies v0.0.14's gate. Before that, **v0.0.15 `57d90d6` on nh3-svos since 2026-10-17 1101** (kernel 0.0.15, the short spoken summons with no ids; voice 0.0.2; `utterance_seconds = 60`; `[notifications.matrix] request_timeout_seconds = 30`; config gen `40f469b75b5d0782`, schema 14). It sits on **v0.0.14 `3726d73` (0908–0910)**: kernel 0.0.14, migration 0014; terminal 0.0.5 with `scan_patterns`; interface 0.0.6 with briefs v2; both cards approved by Prime 0909. nh3-dev `current → 3726d73`, and only svos-terminal was restarted. **Hermes task runtime on the query-file route** (Prime 10-17; backup `instance.toml.pre-queryfile`). ⚠ **Two checks still open:** (1) one scratch task from /tasks with check `exit 0`; Prime must launch it, since every spawn is an owner card. It verifies the success gate and the query route: task.md 0400 under the tasks root, the exit report's first_line, check.exit.code 0. (2) card audio: VERIFIED 10-18 1640. Rollback material: `backup/pre-3726d73`, `backup/pre-57d90d6`, `instance.toml.pre-{3726d73,57d90d6,queryfile,matrix-timeout}`, `voice.toml.pre-57d90d6`, nh3-dev `terminal.toml.pre-3726d73`. Details: servers/nh3-svos/README.md. (Before: main `33a7077` (v0.0.13+2: kernel 0.0.13, interface 0.0.5 with briefs, postoffice 0.0.3, UI 0.0.18) since 2026-10-14 1108–1111; config_gen `33f001f4e2dbc7dc`, apps up 7.) ⚠ Lesson (paid 10-14): a manifest revision whose verbs change is HELD for the owner, and the app reads DOWN (`refused wrong-revision`) until Prime approves its card (15-min expiry). Roll such apps only while he is at the board, and the post-office bridge last (held, it carries no Miranda mail). The interface unit needs the drop-in `svos-interface.service.d/20-artifact-host.conf` (`SVOS_ARTIFACT_HOST=nh3-svos`), and the briefs mapping host must equal it. Before that, kernel 0.0.12 (`d95078c`, 0416) on top of v0.0.11 (452a592, U15 task seats slice 1). The `hermes` runtime is the only task-capable one (one-off `hermes -z`, `process = "python3"`, no `--yolo`: Prime's call); its old argv is in `instance.toml.pre-hermes-task`. Earlier: v0.0.11 is live on nh3-svos (schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7) and nh3-dev (`/opt/svos/current → releases/452a592`; svos-terminal 0.0.4 runs from it). svos-dev accepted it. The task command is ruled with svos (Prime, 2026-10-14): infra-ops edits runtimes only at svos-dev's ask. `GET /tasks` as Prime is his acceptance check. Static-only interface releases since: UI 0.0.14 `89dad7a`, 0.0.15 `7f95e54` (the web build carried into 452a592). ⚠ Process→release map: nh3-svos kernel + local apps run `33a7077`; on nh3-dev svos-postoffice runs `33a7077`, svos-terminal `452a592`; svos-miranda runs `adc862e`, svos-agent still runs `dc241f8` (check `/proc/<pid>/maps` before deleting any release dir). Rollback material: nh3-svos `/var/lib/svos/backup/pre-452a592/` (schema-12 DB + config tar), `instance.toml.pre-452a592`, `instance.toml.pre-a171`; nh3-dev `terminal.toml.pre-452a592`. U13 history and the static-release chain: → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`. Phase R (24 h soak, High Seat removal) is highseat-dev's. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510`, `afab123`, `adc862e` and nh3-dev `/opt/svos/releases/afab123` can go once no process maps them.
|
||||
- **Nothing is mid-action.** No ops-log claims held, no background watchers. eshpfi `main` is pushed (standing push since 2026-10-13); vh/worldtree-instance-configs `main` also has STANDING push since 2026-10-16 (Prime; auto-memory `feedback_git_autonomous`). On Prime's word ~2120 10-18: svoperatingsystem `main` + tag `v0.0.17` (15c2d04) and dotfiles `f909dcf` pushed. ⚠ The working tree and index are shared with infra-hermes, albok-dev and subagents: commit with `git commit -- <paths>` and read `git log origin/main..main` before pushing. `graphify-out/GRAPH_REPORT.md` stays modified on purpose.
|
||||
- **SVOS v0.0.17 (15c2d04, U17 seat activity + the voice client fix) is PUSHED but NOT DEPLOYED** — Prime's next window (svos-dev thread `01M595YP8R62Y665C38MHPH6YX`). Plan: stage kit (kernel 0.0.17, terminal 0.0.6, UI 0.0.21) on both hosts; on nh3-svos add each `[[seats.runtimes]]`' `prompt`/`working`/`window` copied VERBATIM from `docs/operations/u13-edit-payload.md` @15c2d04, check-config, kernel + UI; on nh3-dev restart svos-terminal 0.0.6 (raises ONE held-manifest card: do it only with Prime at the board); then voice.toml `utterance_seconds` back to 60 + restart svos-voice; verify GET /seats `activity`, `/events?kind=seat`, the hero "Fleet · working · N". Rollback = v0.0.16 + take the new runtime keys out (an old kernel refuses them).
|
||||
- **SVOS live: v0.0.16 `b51aa5e` on nh3-svos (since 10-18 1634)** with app-judge (drafting gen-large; six local task models; `[tasks].judge` on, gen `13241a40`); nh3-dev on `3726d73`. Card audio VERIFIED 10-18 (3/3, 4.5–5.5 s). voice.toml `utterance_seconds = 30` (the old web client refused >30; fixed in v0.0.17). **Open:** one drafted scratch task from /tasks (Draft → launch → approve; expect routing, admitted, routed, launching, started) — verifies v0.0.14's gate, the hermes query-file route and U19; Prime must launch it (every spawn is an owner card). Details + rollback material: servers/nh3-svos/README.md.
|
||||
- **Worldtree:** all three on v1.0.0b195 `a416ac01c5a2` since 1112 10-16; Matrix bridge profile-gated (`COMPOSE_PROFILES=matrix` on demo + personal box .env; pinned none, renders api only). Retired-config strips DONE (demo `2240110`, personal `4d1aae2`). ⚠ Pinned's seeded host config dir `/opt/worldtree-pinned/config` is NOT tracked in vh/worldtree-instance-configs and `deploy-wt-config` refuses pinned: capture it. The daily memory gate keeps its harness scratch since 10-18 (`--keep-data`, `~/.local/state/wt-memory-gate-batch/scratch/<batch>/`, 14-day user tmpfiles); 10-18 PASS. Keep defaults `library:` (thread `01M4WA58P2M001K1420NN72WFM`).
|
||||
- **Decision seat = h2o-lightning-4b (production, Prime 1805 10-13: fix forward, no INC fallback).** The one open margin is the judge's personal-question reading (h2o 18 vs INC 6 on fleet/memory); albok-dev re-fits it forward in small batches on the live seat.
|
||||
- **albok note intake (Prime 1530 10-13):** gate 2 DONE (wing `personal/inbox`, `/srv/nemi/inbox` on nh3-dev, nemi token `01M4VX207PJ09ZSECASCSZPCFE`). Gate 1 (ruling 6) DONE 2026-10-15 0140: Prime's notes are `/home/lkraven/kb` = gitea `vh/VuongKB` (he syncs them with git, not Syncthing), so Nemi's copy is a fresh clone at `/srv/nemi/vault`, pulled hourly at :50 by the lkraven user timer `nemi-vault-pull` (`configs/nemi-vault/`). It holds 129 md outside dot-dirs; the untracked Muninn output (`main/`, `fiction/`) is not in it. **Walked 0143–0147 10-15:** 232 objects (129 notes + 103 children), 2 judged personal, 0 adult. The nemi token was re-minted again (`01M4ZBJG3DMRF9Q9XEVZ8Q20GH`, + personal/notes) and the old one revoked (thread `01M4VWV5FEF64EP71W60N6Q1N0`). Prime says the kb needs cleaning up (its last commit is 2026-04-16).
|
||||
- **Worldtree:** demo was deploying `beba08a4` (albok 4.7 the rest; Worldtree is a pure albok-service HTTP customer) at 2120 10-13. Retired-config strip: **demo DONE 2026-10-15 2339 on Prime's go** (instance-configs `2240110`: providers `graph_store` + `library`, plus the b193-retired `memory.*` keys; the boot shows 0 retired-key warnings, down from 7). (0823 10-16: all three moved to v1.0.0b194 `daf587396f7a`; pinned via Deploy-pinned 11136, from its May 446e5807 build.) **Personal's strip DONE 2026-10-16 1036 on Prime's yes** (instance-configs `4d1aae2`, the same patch as demo; boot shows 0 retired-key warnings, down from 7). **The Matrix bridge becomes profile-gated** in worldtree-dev's next release: `COMPOSE_PROFILES=matrix` is set on the demo and personal box .env (and in env.public, `5bc796a`); pinned sets NOTHING. **All three on v1.0.0b195 (`a416ac01c5a2`) since 1112 10-16** (demo/personal via CI 11145/11146, pinned via Deploy-pinned 11147). Pinned renders api only, and its dead bridge container was removed. The image no longer carries KB_PATH. ⚠ Pinned's new compose bind-mounts `/opt/worldtree-pinned/config`: the entrypoint SEEDED it from the image's config-defaults on first start, so pinned now HAS a host config dir that vh/worldtree-instance-configs does not yet track (pinned was out of scope while its config was baked in). Capture it. Keep defaults `library:` (thread `01M4WA58P2M001K1420NN72WFM`). Daily memory gate: 10-13 re-run `20261013T160414Z` PASS; worldtree-dev owns the tagger-latch fix. muninn-gate + mimir-inbox are STOPPED. **Albok unit 6 teardown (thread `01M4WAEFZEA1YY03ZR31AEHGTS`): steps 1-4 DONE 2026-10-16 0836.** 8 tars (3 worldtree-kb volumes, mimir-inbox staging, personal `state/ingestion`, muninn-gate conf, both compose dirs) were test-restored with per-file sha256 identical, and are now on `nh3-nas:/volume1/Backup/retired/albok-u6-corviduo-dev-20261016` (0700, re-verified there; local copy `/var/tmp/albok-u6`). **Deletion (step 5) is HELD for Prime's direct go**: albok-dev relayed "land all of u6", and a relayed go does not authorize a delete. The images muninn-gate:0.0.14 + mimir-inbox:0.0.1 were already gone at 0830, cause unknown. The in-process ingestion watcher ran until the b194 deploy (last heartbeat 0821:45). worldtree-dev cleared the ingestion subtree from the b194 source (0845). I removed personal .env's no-op `COMPOSE_PROFILES=muninn` (0846). The two 07-27 `.bak-muninn` files go in step 5 with no archive (the .env one holds live secrets). personal's providers.yaml `write_agents: ["muninn"]` goes with personal's retired-config strip (on Prime's yes). **albok-service 0.1.24 (sha256:75be71c1) LIVE since 0917 on nh3-albok, with personal/sealed configured (8 wings ready).** Harness (a) PASSED 0922: LOCK 0920:08-0920:38 (423s, search `locked`), live UNLOCK 0921:53 with no restart; the synthetic object was hard-deleted. The sealed wing stays MOUNTED. **Step 5 DONE 2026-10-16 1031 on Prime's direct go:** the 3 worldtree-kb volumes, personal `state/ingestion`, the muninn/mimir dirs and staging, and the 2 .bak files are all deleted. I also pruned the build cache (345.7MB) and removed /var/tmp/albok-u6; the NAS archive is the only copy. Found during the readback: **pinned's Matrix bridge had crash-looped since the 0822 b194 deploy** (no matrix.yaml, empty MATRIX tokens: pinned never had a bridge). I stopped it at 1033. The next pinned deploy recreates it; the durable fix is worldtree-dev's compose.
|
||||
- **satellite-1:** its own key is authorized for infra-ops on esh-nas only (`playbooks/files/infra-ops-satellite-keys`). Waiting on Prime's login test from the satellite and its machine name (the key's comment came through as bare `infra-ops@`).
|
||||
- **fv-ml1: hard hang 10-11 ~0005, self-reset, no record** (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit `fv-ml1-sol-logger` (`journalctl --user -u fv-ml1-sol-logger`). ⚠ **HOLD** `zroot/ROOT/debian@pre-patch-20261008` (do not destroy) until the cause is known or a clean week passes after the last crash.
|
||||
- **albok `.browse/` share:** the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in `~vh/albok-browse-test/` on nh3-albok, waits on Prime pasting the one-liner (`scripts/oneoff/albok-browse-withdrawal/README.md`); afterwards read `results/`, report to albok-dev, remove the replica.
|
||||
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key.
|
||||
- **Waiting on Prime:** where his Obsidian vault lives (gate 1); satellite-1's login test + name; rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
|
||||
- **Waiting on Prime:** one drafted scratch task on /tasks (SVOS verify); the v0.0.17 cutover window (one terminal card); satellite-1's login test + name; rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
|
||||
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). The ana-docker gitea-runner's Homepage name changes to its runner name at its next recreate (stacks/gitea-runner/compose.yaml). nh3-dev's `/opt/svos/releases/` and the h2o eval seat on fv-ml1 GPU 3 (`stacks/h2o-eval`, down) are kept as artifacts.
|
||||
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
|
||||
- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
|
||||
@@ -169,7 +170,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
- **U11b STEP 5 DONE 2026-10-02 14:54:03Z on Prime's direct go** (in this channel, ~0750 PT). H2 re-check exit 0 on both (demo rows 4/12/5, personal 0/6/2200, agent_self 0); deleted `agents/{forseti,lofn,mimir}/memory/<agent>.chroma` + `memory/context_promotion` on demo and personal with literal paths; apis healthy, no restart; stamp sent to worldtree-dev (b193 is theirs to cut). Demo had already lost 22 context_promotion JSONL (2026-07-03) to ledger.py's 90-day `JSONL_RETENTION_DAYS` sweep on 10-01 07:10:30 — TTL, not an erasure; flagged to worldtree-dev as retirement owner.
|
||||
- **b193 is cut and pushed (c2d87263, worldtree-dev 2026-10-02); demo deploys via CI, personal on Prime's staging tag.** `scripts/wt-memory-gate-batch` now passes `--legacy-mode` only when the DEPLOYED sha's harness still has it (git grep at that sha; tested b192 → passed, b193 → omitted), so no timed edit is needed. ⚠ It will REFUSE (exit 3) while demo's deployed sha lags a tree HEAD whose pyproject/uv.lock/packages changed (b193, then the albok commit): by design. **Still mine after b193 deploys:** remove the residue `memory/context_promotion/ledger.db` a b192 restart may have re-created; then the parity config retirements (detail file). Verdict ccs come FROM `hermes-gateway` (never read); reply to `infra-hermes`.
|
||||
- **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
|
||||
- **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").
|
||||
- **TODO:** re-sweep both api logs after real traffic. (The post-b193 retired config keys are GONE: demo 10-15 `2240110`, personal 10-16 `4d1aae2`.)
|
||||
|
||||
### The decision seat `intern-decision` :8033 (h2o-lightning-4b, production since 2026-10-12 1807)
|
||||
|
||||
@@ -185,6 +186,10 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-18]` **SVOS v0.0.14 → v0.0.16 rolled on nh3-svos/nh3-dev (10-17/10-18, Prime's gos):** v0.0.14 (3726d73, success gate, cards approved 0909 10-17), the hermes task runtime on the query-file route (Prime), v0.0.15 (57d90d6, short spoken summons; card audio had failed 12/12 at the kernel's 15.0 s timeout, now 3/3 in ~5 s), v0.0.16 (b51aa5e, app-judge: gen-large drafting, six local task models, key `svos-app-judge`, Prime's 3 owner grants 1641). voice `utterance_seconds` reverted to 30 at 1646 (the web client refused >30). → servers/nh3-svos/README.md
|
||||
- `[2026-10-17]` **asset-engine's stale Homepage siblings + kept-scratch gate:** Speaches/sherpa-Parakeet ghost containers removed (images + compose kept; Parakeet rollback = compose up); the Worldtree memory gate keeps its harness scratch from 10-18 (worldtree-dev; `--keep-data` via TMPDIR, 14-day user tmpfiles). Also: LiteLLM key `sk-bootstrap` (six local models, vault `litellm/sk-bootstrap`) and private repo `vh/vh-bootstrap` for brokkr-smithy-dev (Prime).
|
||||
- `[2026-10-16]` **Albok unit 6 teardown DONE on Prime's direct go (1031):** the 3 worldtree-kb volumes, personal `state/ingestion`, muninn-gate/mimir-inbox dirs + staging + 2 `.bak` files deleted after 8 verified archives (per-file sha256 restore-identical) went to `nh3-nas:/volume1/Backup/retired/albok-u6-corviduo-dev-20261016` (the only copy now). albok-service 0.1.24 → 1.0.0a1 with `personal/sealed` (LUKS, harness passed: lock 423 / live unlock). Book drop `/srv/nemi/books` (scp only, Prime).
|
||||
- `[2026-10-16]` **Worldtree b195 everywhere; personal's retired-config strip (Prime's yes); standing push on vh/worldtree-instance-configs (Prime).** Pinned's b194 deploy had crash-looped its Matrix bridge (no config, no tokens) unseen for ~2 h: fixed by worldtree-dev's `profiles: [matrix]` + `COMPOSE_PROFILES=matrix` on demo/personal. Lesson → auto-memory `feedback_verify_every_service_a_deploy_touched`.
|
||||
- `[2026-10-17]` **asset-engine RETIRED (Prime)**, along with the stale speech entries: the Homepage Speaches (irv-ml1) and sherpa Parakeet (fv-ml1) ghosts were removed via `docker rm` of their dead containers (images and compose dirs kept); the Uptime Kuma monitor was deleted (`scripts/kuma delete`, new); asset-engine is `compose down` (data kept). Evidence: last job 08-02; all 21 catalog endpoints on the dead 10.100.79.3. FleetTools, the agent bootstrap and global CLAUDE.md (dotfiles f909dcf, unpushed) no longer list `:8200`.
|
||||
- `[2026-10-16]` **albok unit 6: the first encrypted wing `personal/sealed` provisioned on nh3-albok** (Prime's goal "land all of u6", via albok-dev): a 20 GiB LUKS2 file + ext4, key ONLY in vault `albok/wing-key-personal-sealed`, gid 1513, mountpoint `/srv/albok/sealed/personal.sealed` (immutable while unmounted), the parent bound rslave. The lock/unlock lifecycle was tested with the container running. It is not in albok.yaml until albok-dev's release. Runbook: `stacks/albok-service/README.md` § Sealed. After an nh3-albok reboot the wing is LOCKED until someone unlocks it.
|
||||
- `[2026-10-13]` **pewpewstudio LIVE on nh3-docker :8185** (Prime via pewpew-dev): api + Caddy front; the Host allowlist must sit in a `route` (foreign Host 400); DB, assets and shop config on named volumes. → `stacks/pewpewstudio/README.md`
|
||||
@@ -245,48 +250,18 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
- `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md`
|
||||
- `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md`
|
||||
- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`
|
||||
- `[2026-10-04]` **albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime)** — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns `nh3-pve-2`; copy verified by digest (dir st_size differs on ext4, not data). → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
|
||||
- `[2026-10-04]` **The whole fleet is on PVE 9.2.21 and PBS 4.2.7** — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
|
||||
- `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`
|
||||
- `[2026-10-04]` **sudo session recordings: archived weekly, bulk data never kept (Prime)** — `BULK_DATA !log_output` on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → `services/sudo-io-archive/README.md`
|
||||
- `[2026-10-04]` **VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use)** — off, `onboot 0`, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.
|
||||
- `[2026-10-04]` **PARKED (Prime): move fv-ml1's Docker data-root to tank** — needs a model-seat outage window; tracked at park `move-fv-ml1-s-docker-storage-off-the-root-ssd` (id 99). — `[2026-10-05]` DONE (entry above).
|
||||
- `[2026-10-04]` **PARKED (Prime): guest OS patch campaign** — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park `guest-os-patch-campaign-across-the-fleet-debian` (id 100). — `[2026-10-05]` DONE (entry above).
|
||||
- `[2026-10-04]` **PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway** — tracked at park `give-svos-and-miranda-a-model-path-that-doesn-t` (id 98).
|
||||
|
||||
- `[2026-10-03]` **PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host")**: an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → `services/pve-nag-patch/README.md`
|
||||
- `[2026-10-03]` **nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 `nic3`, AMT on `nic0` with `auto` kept up, 1 TB wiped into `vmstore` (913 GiB), stale boot entry gone** → `persistent-memory.d/2026-10-03-nh3-pve-2-finished-prime-on-site-host-10-100-250.md`
|
||||
- `[2026-10-03]` **CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232)** → `persistent-memory.d/2026-10-03-cira-tunnel-watchdog-live-on-pfi-tacticalrmm-prime-go-1232.md`
|
||||
- `[2026-10-03]` **Worldtree memory-gate duty REFRAMED: regression check, not a gate** → `persistent-memory.d/2026-10-03-worldtree-memory-gate-duty-reframed-regression-check-not-a-gate.md`
|
||||
- `[2026-10-03]` **albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1** → `persistent-memory.d/2026-10-03-albok-service-0-1-2-live-on-nh3-docker-albok-dev-ask.md`
|
||||
- `[2026-10-02]` **esh-pve-2 (MS-03 at ESH, future `esh-dev` host) onboarded: infra-ops (Prime), 1 TB → LVM-thin `vmstore`, AMT phoning home as `esh-pve-2-amt`** → `persistent-memory.d/2026-10-02-esh-pve-2-ms-03-at-esh-future-esh-dev-host-onboarded-infra-ops.md`
|
||||
- `[2026-10-02]` **Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684)** → `persistent-memory.d/2026-10-02-demo-outage-13-min-rolled-back-worldtree-dev-urgent-6684.md`
|
||||
- `[2026-10-02]` **nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`)** → `persistent-memory.d/2026-10-02-nh3-pve-s-amt-is-in-tacticalrmm-s-meshcentral-group.md`
|
||||
- `[2026-10-02]` **Prime: dev-backup gets dailies + weeklies — DONE** → `persistent-memory.d/2026-10-02-prime-dev-backup-gets-dailies-weeklies-done.md`
|
||||
- `[2026-10-02]` **ESH `tank` (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629** → `persistent-memory.d/2026-10-02-esh-tank-esh-nas-pve-175-t-raidz2-2-was-degraded.md`
|
||||
- `[2026-10-02]` **albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): `http://albok.nh3.internal:8392`** → `persistent-memory.d/2026-10-02-albok-service-0-1-0-live-on-nh3-docker-albok-dev-ask.md`
|
||||
- `[2026-10-02]` **nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03** → `persistent-memory.d/2026-10-02-nh3-pve-2-status-at-end-of-day-amt-done-proxmox.md`
|
||||
- `[2026-10-02]` **Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as `nh3-pve-2` at NH3 (Prime, ~0910)** → `persistent-memory.d/2026-10-02-prime-ms-03s-get-proxmox-dummy-plugs-in-hand-one.md`
|
||||
- `[2026-10-02]` **nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot)** → `persistent-memory.d/2026-10-02-nh3-dev-root-grown-250-378-gb-prime-resized-scsi0.md`
|
||||
- `[2026-10-01]` **MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev)** → `persistent-memory.d/2026-10-01-mia-make-it-animatable-v2-auto-rigger-live-on-fv-ml1-gpu-3.md`
|
||||
- `[2026-10-01]` **Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed** → `persistent-memory.d/2026-10-01-prime-clean-up-the-1700-backups-dev-backup-retention-fixed.md`
|
||||
- `[2026-10-01]` **Prime: gen-small keeps its 8 GiB KV pin, and parakeet-nemo keeps CUDA graphs OFF** (~2–8 ms at short clips, accepted). GPU 0's ~1 GB of spare memory is enough with the seat's hard 3,840 MiB cap.
|
||||
- `[2026-10-01]` **⚠ Gitea's `[webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8` silently REJECTS headscale mesh IPs (100.64.0.0/10)** → `persistent-memory.d/2026-10-01-gitea-s-webhook-allowed-host-list-external-10-0.md`
|
||||
- `[2026-10-01]` **vh/arbo push webhook repointed from the retired wg0 lifeline `10.100.79.3:9009` to irv-ml1's mesh address `100.64.0.6:9009`** → `persistent-memory.d/2026-10-01-vh-arbo-push-webhook-repointed-from-the-retired-wg0.md`
|
||||
- `[2026-10-01]` **Prime: delete the bench leftovers, no upstream for Scriberr, push** → `persistent-memory.d/2026-10-01-prime-delete-the-bench-leftovers-no-upstream-for-scriberr.md`
|
||||
- `[2026-10-01]` **irv-ml1 /storetank reclaim done:** Prime ruled through comfy-dev, which deleted 84 files of its own (260 → 299 GB free). Tiers B/C/D got no ruling (infra-hermes thread `01M3TCSYRSFNAPA9BPQTMFQ6KJ`).
|
||||
- `[2026-09-30]` **Parakeet speech seat → `parakeet-unified-en-0.6b` under NeMo (bf16) APPROVED by Prime, NVIDIA Open Model License accepted. DONE 2026-10-01 0126 by infra-hermes; infra-ops audit passed 0137.** → `persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md`
|
||||
- `[2026-09-30]` **Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30.** → `persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md`
|
||||
- `[2026-09-30]` **SemIf replaced by intern-decision (Intern-Decision-4B, the Jev bench pick): semif-compatible plus Jev `/v1/systemone` at 32k tokens on GPU 1, with a Triton warm-up cache volume.** → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`
|
||||
- `[2026-09-30]` **Scriberr moved to GPU 3 (on demand); our build carries the overlap slicer (0001) and the Parakeet gap retry (0002); v3 kept.** → `persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md`
|
||||
- `[2026-09-27]` **Worldtree reward path: d8726a13 + `providers.yaml` `reward_models.scalar-judge` live on demo; personal waited on a newer image.** → `persistent-memory.d/2026-09-27-worldtree-reward-path-live.md`
|
||||
- `[2026-09-26]` **augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy.** → `persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md` — `[2026-09-27]` **DONE:** deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (`d8f59a1`).
|
||||
- `[2026-09-26]` **`zellij-fleet@.service` installed, NOT enabled** (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling `@Claude` at boot is Prime's call when seat_up ships. Tracked: `0ad7799`, `services/zellij-fleet/README.md`.
|
||||
- `[2026-09-26]` **btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority).** Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: `servers/nh3-pve/README.md`.
|
||||
- `[2026-09-25]` **AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326).** — Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale… → `persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md`
|
||||
- `[2026-09-26]` **esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only** — , for ha-dev (operator-approved, relayed). → `persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md`
|
||||
- `[2026-09-26]` **Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime).** — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → `persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md`
|
||||
- `[2026-09-26]` **Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):** — LFM2.5-VL-3B on llama.cpp `:8030` (gateway `lfm25-vl-3b`, LiteLLM restarted 36 s at 0039) and… → `persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md`
|
||||
- `[2026-09-25]` **nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333).** Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: `servers/nh3-pve/README.md` (`2b49be4`, `cdd7605`).
|
||||
- `[2026-09-25]` **Direct ESH→esh-ml1 consumer path — PARKED** (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`.
|
||||
- `[2026-09-24]` **Worldtree `admin.memory.forget` stays OFF on demo/personal** until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.
|
||||
@@ -303,7 +278,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
|
||||
|
||||
_258 older entries archived to archival-memory.md._
|
||||
_288 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
@@ -316,8 +291,5 @@ _258 older entries archived to archival-memory.md._
|
||||
- `[2026-10-06]` **`deploy-stack.sh` on a stack whose host dir holds rollback copies** (albok-service `compose.yaml.pre-*`) — its rsync `--delete` removes them. Upload the single changed file with elway instead.
|
||||
- `[2026-10-05]` **Sizing a vLLM seat's KV from `--gpu-memory-utilization` on a shared card** — co-booting neighbours corrupt the profile (non-torch −0.56 GiB), so the KV is a lottery per boot. Pin `--kv-cache-memory`. auto-memory `reference_vllm_util_kv_coboot_race`.
|
||||
- `[2026-10-05]` **Reading GPU headroom just after a restart** — 10-01's "Free ~1.05 GB" was really a ~60 MiB plateau 2 days later. Size against Beszel history. auto-memory `feedback_vllm_headroom_at_plateau_not_boot`.
|
||||
- `[2026-10-04]` **`apt-get` run in-band through `qm guest exec`** — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a `systemd-run` unit. (auto-memory `feedback_no_inband_apt_through_guest_agent`)
|
||||
- `[2026-10-04]` **Pulling files with `ssh host 'sudo cat …'`** — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and `log_output` re-recorded the whole file as a new session. Read without sudo (group-readable file).
|
||||
- `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** → `persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md`
|
||||
|
||||
_141 older entries archived to archival-memory.md._
|
||||
_144 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user