Compare commits
23
Commits
ee137dbb04
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
16e0dac572 | ||
|
|
f49e3f18ed | ||
|
|
477de888ef | ||
|
|
05e370d2ae | ||
|
|
9b2716261c | ||
|
|
0b873dd8fe | ||
|
|
a583c54399 | ||
|
|
db88f76aea | ||
|
|
dd816662dc | ||
|
|
a84f6b00df | ||
|
|
8758c359bc | ||
|
|
9d599caa53 | ||
|
|
d1192ca868 | ||
|
|
5e8307ed7a | ||
|
|
60a6d6db59 | ||
|
|
f1cc7ab995 | ||
|
|
047ff26778 | ||
|
|
4eacb746fa | ||
|
|
7ca2f76d21 | ||
|
|
eda7025ed1 | ||
|
|
3e634bb3d0 | ||
|
|
30a7220b2c | ||
|
|
b699088e2b |
@@ -503,6 +503,7 @@ because SLAAC addresses rotate and a stale record is worse than none.
|
||||
|
||||
## Working rules
|
||||
|
||||
- **Standing push on `main` (Prime, 2026-10-13).** infra-ops pushes this repo's `main` to origin without asking each time, at natural boundaries (fast-forward only). Force-push, history rewrites on pushed commits and pushes to other branches still need Prime's per-op word. Commit with `git commit -- <paths>` (the index is shared), and check `git log origin/main..main` before pushing so you know whose commits ride along.
|
||||
- **Copies, not symlinks.** Files here reflect what's on the server at the time of the last sync. When you edit here, the server doesn't change until you deploy.
|
||||
- **Never commit secrets.** Use `.env.example` templates; real `.env` files (with tokens, passwords) live on the server and are gitignored if/when this becomes a git repo.
|
||||
- **Surgical edits.** When fixing one stack, don't touch unrelated ones. Follow AIPA-MCP's CLAUDE.md rules about scope discipline.
|
||||
|
||||
@@ -15314,3 +15314,621 @@ latency-sensitive.
|
||||
|
||||
- `[2026-09-26]` **zsh traps in ad-hoc test loops:** `set -- $a` does NOT word-split, so alert POSTs went out with empty fields; and `local path=…` inside a function CLOBBERS `$PATH` (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and `${=var}`.
|
||||
_Archived 2026-10-11._
|
||||
|
||||
|
||||
## Recent decisions (archived)
|
||||
|
||||
- `[2026-09-29]` **Worldtree U11a prepped, not flipped: a staged demo config plus the agreed U8 window plan (infra-hermes runs the batches).** → `persistent-memory.d/2026-09-29-worldtree-u11a-prepped.md`
|
||||
|
||||
# Worldtree U11a (legacy memory → read_only) is PREPPED, not flipped.
|
||||
|
||||
`[2026-09-29]` **Worldtree U11a (legacy memory → read_only) is PREPPED, not flipped.** A staged, inert `/opt/worldtree/config/defaults.yaml.staged-u11a-read_only` sits on corviduo-dev. Applying it needs Prime's go AND demo on 48bdf235 or later (demo ran 1b8746e6, which predates U11a). Apply = back up, copy over `defaults.yaml`, `docker restart`, then read the boot line `memory cutover: legacy plane read_only (writer on, reader on)`. **Once the window opens, infra-hermes runs one U8 batch per day and I own the wrapper and the escalations (proposed, worldtree-dev agreed):** on nh3-dev in `~/development/Worldtree`, at demo's deployed SHA, `python -m core.memory_acceptance.live --runs 3 --gate --legacy-mode read_only --label window`. The verdict is in `trace1.md`; FAIL goes to worldtree-dev the same day, never a retry; no git commit; never two at once. The wrapper is NOT built yet, deliberately: it waits for the go.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-28]` **Worldtree U10 backfill done on demo (5) and personal (797). model_roles drift needed a memory_tagger sync first; mimir had missing vectors.** → `persistent-memory.d/2026-09-28-worldtree-u10-backfill.md`
|
||||
|
||||
# Worldtree U10 legacy-memory backfill: demo COMMITTED (Prime, direct go 2350).
|
||||
|
||||
`[2026-09-28]` **Worldtree U10 legacy-memory backfill: demo COMMITTED (Prime, direct go 2350).** Dry run then commit, both as `-u worldtree`: 5 filed (lofn 4, mimir 1), 16 `unslotted_pre_364` skipped, 0 aborted. **The first commit exited 1 because demo's hand-managed `model_roles.yaml` lacked the `memory_tagger` role**, so I synced it verbatim from the image's config-defaults (backup `.bak-pre-memory-tagger`). **2026-09-29 0854 presync for personal (Prime: "complete u10"):** personal got `memory_tagger` and the U9 `admin.memory.forget` policy delta. **Demo had also been missing that U9 policy delta since 09-24**, which left readonly-admin's `admin.*` rule with no exclusion for the destructive forget. Fixed, and demo's api was restarted to load it. **Personal COMMITTED 2026-09-29 1007** (b191 e2f74706, a snapshot-copy build; Prime's standing "complete u10" via worldtree-dev's go): 797 filed (mimir 795, lofn 2), mimir `dropped:too_large` 377, 1 unroutable held. The first personal dry run (b190) had crashed on 45 mimir rows with no vector (HNSW frozen since 08-06, #417, accepted by Prime until U11). Pinned (446e5807; Prime: "leave pinned") keeps its legacy chroma in the container's WRITABLE LAYER (`/app/agents/*/memory/.chroma`, 7 agents, 196K each): a recreate erases them.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-28]` **Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired.** → `persistent-memory.d/2026-09-28-bonsai-ternary-spike.md`
|
||||
|
||||
# Bonsai ternary spike on fv-ml1 GPU 3 (Prime via brokkr): at the 275 W cap, PQ2_0 is 1.93x Q4_K_XL at N=1 but only 1.06x at N=8 (PTQ1_0 1.24x), recovering to ~1.37x at N=16.
|
||||
|
||||
`[2026-09-28]` **Bonsai ternary spike on fv-ml1 GPU 3 (Prime via brokkr): at the 275 W cap, PQ2_0 is 1.93x Q4_K_XL at N=1 but only 1.06x at N=8 (PTQ1_0 1.24x), recovering to ~1.37x at N=16.** Positive control passed on tg128 (+1.1%). nvidia-smi's sw_power_cap flag never fires on this card, so "at cap" is judged from board draw. Follow-up (brokkr): moving PQ2_0 onto MMQ from batch 6, like Q4_K (mmvq.cu, `ne11 <= 5`), lifts N=8 to 1.21x, so the dip was partly a kernel threshold. The residual gap is not power. Runs are in `fv-ml1:/tank/spikes/bonsai-2026-09-28/runs{,-mmvq5}/` and on the Booth. The build image was removed; `Dockerfile.build` recreates it. **Acquired for keeps (Prime, 1540):** GGUF PQ2_0/PTQ1_0/mmproj-Q8_0 + the fork source pin in `/tank/aimodels/llm/prism-ml_Ternary-Bonsai-2-27B-gguf/` (runtime/), the MLX 2-bit pack in `/tank/aimodels/mlx/`, 34/34 hash-verified. Weights single copy (no snapshots, /tank not in restic); the fork pin also at `/mnt/smithy/runtime-pins/prism-llama.cpp-87268f77/` (brokkr).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-28]` **blender-run gained `--cpu` (no GPU attached) and a fixed hostname `fv-ml1-blender` (draupnir).** The design stage never renders, so it stays off GPU 3.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-28]` **Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (`--extensions`).** SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → `stacks/blender/README.md` § Extensions
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** → `persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md`
|
||||
|
||||
# hermes-gateway restarted 0401 for highseat-dev
|
||||
|
||||
`[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
|
||||
|
||||
# `[2026-09-27]` SemIf live on fv-ml1 GPU 1 (Prime)
|
||||
|
||||
**semif-serve 0.1.2 is live** at `http://10.251.50.54:8032` (`semif.fv.internal`), stack `stacks/semif`,
|
||||
code and contract `services/semif-serve/` (`069725c`). It is a FastAPI wrapper around SemIf-OpenJev's
|
||||
direct and shared torch scorers (MIT, pinned `23cf1f39fc9534fe81437200959b6dfc7106e45a`) on
|
||||
Qwen3.5-4B `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a` (BF16, offline HF cache on `/tank`).
|
||||
Hosting assessment: infra-hermes, `/tmp/semif.md` (2026-09-25).
|
||||
|
||||
Prime's calls (2026-09-27): fv-ml1 GPU 1 under a hard VRAM cap (not the reserved-empty GPU 3, not a Q4
|
||||
GGUF on a 16 GB box); infra-ops builds it with a light process (short contract → TDD → heid bug hunt);
|
||||
no consumer named yet, so every score is labelled uncalibrated.
|
||||
|
||||
**Two defects that only the card showed**, each fixed with a test:
|
||||
- **0.1.1:** the OOM path raised `OutOfMemory(...) from exc` inside the `except` block. The chain kept the
|
||||
torch exception's traceback, and with it the failed call's frames and tensors, so 11.9 GiB stayed
|
||||
allocated after the 503. The fix raises after the block, with no chain, and runs `gc.collect()`
|
||||
before `empty_cache()`.
|
||||
- **0.1.2:** a large shared request left torch's cache holding 12.6 GB, which left scriberr 3.5 GB on GPU 1.
|
||||
Now, after each call, reserved memory above the post-warm-up baseline + 512 MiB is released
|
||||
(`empty_cache`). It costs ~16 ms on a big request.
|
||||
|
||||
**Acceptance** (`services/semif-serve/acceptance/result-2026-09-27-v0.1.2.json`), against SemIf's
|
||||
committed torch predictions on authored144:
|
||||
- 142/144 same top choice; both misses are exact bf16 ties in our output;
|
||||
- 144/144 identical prompt SHA-256;
|
||||
- A-vs-A gap 0.0;
|
||||
- negative control (rotated option descriptions) 14/144;
|
||||
- shared vs direct 72/72;
|
||||
- 21 binary criteria over one state in 159 ms.
|
||||
|
||||
The shared-mode capacity under 12 GiB is 52 decisions at a ~140-token prefix and 13 at ~3,900.
|
||||
|
||||
**Build:** torch 2.10.0+cu128 from the pytorch index (SemIf's own stack; sm_120 present). The
|
||||
Dockerfile installs dependencies from a manifest with the project version blanked, so a version bump
|
||||
reuses the ~4 GB torch layer (41 s rebuild, layer CACHED). See [[2026-09-27-semif-order-averaging]].
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port.** → `persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md` (canonical: `stacks/blender/README.md`)
|
||||
|
||||
# `[2026-09-27]` Blender 5.2 on fv-ml1 GPU 3, agent-driven (Prime)
|
||||
|
||||
(Moved out of the in-flight section at the 2026-10-06 snapshot. Canonical: `stacks/blender/README.md`.)
|
||||
|
||||
- Prime: "go ahead with gpu 3, both", and he does not use Blender, so **agents drive it through MCP**.
|
||||
Blender 5.2.2 LTS (linuxserver/selkies image, digest-pinned) with a web desktop at
|
||||
`https://10.251.50.54:3001` (vault `fv-ml1/blender-web-password`). **On demand only:**
|
||||
`scripts/blender-mcp up|down|status`. It was left DOWN (GPU 3 back to 2 MiB).
|
||||
- MCP: `mcp-for-blender` 2.1.1 (Dvalin's research pick). The add-on is vendored at `41a18432`. The
|
||||
server runs INSIDE the container, and agents reach it as ssh+docker-exec stdio via
|
||||
`scripts/blender-mcp`. No port is published, and screenshots work because server and Blender
|
||||
share a filesystem. Telemetry is off and safe mode is on. Verified end to end (render, screenshot,
|
||||
safe-mode refusal). **Batch path: `scripts/blender-run`** (one-shot `docker run --rm`, `--job` staging; first user is draupnir). **Access: the shared fleet `infra-ops` login, with no render-only key (Prime, 2026-09-28).** **Registration + up/down: PER WORKING SESSION (Prime, 2026-09-28, relayed by draupnir; superseded per-task of 09-27)**: `blender-mcp up` + `claude mcp add` at session start, `claude mcp remove` + `down` at its end. Never always-on, never user- or project-wide config.
|
||||
→ `stacks/blender/README.md`
|
||||
- **Extensions (2026-09-28, draupnir; Prime ruled Blender a MANDATORY pipeline stage):** 8 pinned
|
||||
add-ons (`stacks/blender/extensions.lock`) built by `scripts/blender-extensions sync` into
|
||||
`fv-ml1:/tank/blender-extensions/5.2/system` (LIVE), mounted read-only as the System repo;
|
||||
`fleet_extensions.py` enables them (GUI startup timer; `blender-run --extensions`). Headless
|
||||
acceptance 8/9 (CAD Sketcher sketching is GUI-only); **MCP 9/9 after Prime ran the deploy himself
|
||||
(1505; the classifier had refused mine)**. SurfacePsycho's eval() is patched to literal_eval
|
||||
(held under MCP). Open: agent-drawn CAD Sketcher geometry (its stateful ops want point picks),
|
||||
and MeasureIt overlays not seen in MCP screenshots. GUI left DOWN.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August).** → `servers/esh-docker-vm/README.md`
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only.** Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → `persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md` (canonical: `stacks/zigbee2mqtt/README.md`)
|
||||
|
||||
# `[2026-09-27]` Zigbee2MQTT on esh-docker-vm (Prime go-ahead; ha-dev request)
|
||||
|
||||
(Moved out of the in-flight section at the 2026-10-06 snapshot. Canonical: `stacks/zigbee2mqtt/README.md`.)
|
||||
|
||||
- **LIVE since 1240:** Z2M 2.14.1 (digest-pinned) on `http://10.0.50.45:8099` (auth token), radio
|
||||
SLZB-MR1U chip 0 `tcp://10.0.90.10:6638` (ember), channel 25, **PAN 0xCFF4**. State and the network
|
||||
key are in `/opt/docker/data/zigbee2mqtt` (root 0700, restic via /opt/docker, never in git). Vault:
|
||||
`esh-docker-vm/zigbee2mqtt-{network-key,frontend-token,mqtt-password}`. Broker user `zigbee2mqtt`
|
||||
added to mosquitto (passwd backup `.bak-20260927-z2m`).
|
||||
- **ha-dev deleted the ZHA entry**; pairing the Aqara T1 is theirs. ⚠ The "bridge in HA" acceptance first
|
||||
FAILED because HA had had no MQTT since 2026-09-25 06:13. Cause: the esh-docker-vm macvlan shim had no
|
||||
host route to HA (two equal /24s, ens18 wins). Fixed at 1249 with a /32 via the shim (if-up.d hook,
|
||||
`playbooks/esh-docker-vm-macvlan-shim-route.yaml`), and HA reconnected. My first report called HA
|
||||
"connected" from a client count that included my own probe; `$SYS` counts are not identities.
|
||||
- ⚠ The first credential-mint attempt was blocked by the auto-mode classifier on a peer-relayed
|
||||
approval. It went ahead only on Prime's own go-ahead in this session. Treat that as correct.
|
||||
→ `stacks/zigbee2mqtt/README.md`
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **semif-serve 0.1.4: object states ending in `)`, `;` or `}` no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart.** Prime ruled; heid bug hunt folded. → `stacks/semif/README.md`
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win.** Build nothing (Prime). Henge 88 carries it. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
|
||||
|
||||
# SemIf consumer-fit spikes: Wyrd scene change, Cicada affect gate (2026-09-27, Prime)
|
||||
|
||||
Prime, 0904: "Two semif spikes -- Wyrd has a couple flags per turn on whether the scene should
|
||||
change, check the codebase, see if semif is a clean fit and then spike a couple of sample
|
||||
scenarios. likewise, cicada has a speech module and some reactions-- spike whether or not she
|
||||
should show emotion." No service change. Code and raw output: `services/semif-serve/spike/`
|
||||
(`consumer_fit.py` harness; `wyrd_cases.py`, `cicada_cases.py` build the scenario files;
|
||||
`consumer-fit-2026-09-27/` holds scenarios, `.result.json` and `.log` per run).
|
||||
|
||||
## Method (applies to both)
|
||||
|
||||
- One `/decide/shared` per turn, every decision with `"orderings": "rotations"`. Target:
|
||||
semif-serve 0.1.3 on fv-ml1 GPU 1, client on nh3-dev.
|
||||
- **Noise floor:** the service is deterministic (acceptance: 144 rows twice, gap 0.0), and the
|
||||
harness asserts the same top on all 3 runs of every case. Repeats measure latency only.
|
||||
- **Null control:** every case is also asked over a content-free state. It always picks the
|
||||
"nothing happened" option, so its accuracy is that label's base rate (Cicada 16/31, Wyrd
|
||||
15/21). Evidence conditions are read against that, not against 0.
|
||||
- **Positive controls:** cases tagged `control`, unambiguous by construction.
|
||||
- **Limits, stated:** one labeller (me), authored inputs, 31 Cicada and 21 Wyrd cases per
|
||||
decision. Wilson 95% intervals are wide (30/31 → 0.84–0.99; 21/21 → 0.85–1.0), so nothing
|
||||
here resolves differences under ~10–15 points. The winning wordings were chosen on this same
|
||||
set, so their scores are optimistic until checked on held-out real play.
|
||||
|
||||
## Wyrd: scene change (fit: yes for the CHOICE, no for the WRITING)
|
||||
|
||||
How it works today (from the code): no dedicated flag. The GM's `gen` structured-output calls
|
||||
carry it two ways. **R8** (reflect): `plot_direction.target_node`, accepted only if it is an
|
||||
advanceable exit; `plot_direction.intrude` is "advisory" and never read anywhere. **Path B**:
|
||||
raw `AddStoryNode` + `AdvanceToNode` events, gated only by prompt prose ("Add + advance to a NEW
|
||||
node ONLY when the scene moves to a genuinely different place — never every turn"). Wyrd's
|
||||
archival memory already says "if it churns, tighten the prompt (or gate node-adds)".
|
||||
**Three of the real campaigns in `.wyrd-data/canon` have ZERO story edges, so R8 can never fire
|
||||
there and every scene change went through ungated path B.**
|
||||
|
||||
Fit: both are typed choices over one state, exactly semif's shape. Semif's exit options can only
|
||||
be "stay" plus the real advanceable exits, so it cannot name an illegal target. It cannot write the
|
||||
new node (title, summary, mood, image prompt) or the steering directive; the GM keeps those. So
|
||||
it would be a gate/selector in front of the GM, not a replacement.
|
||||
|
||||
Spike: 3 real seed graphs (The Fletcher, The Toll Road, The Last Vigil + one added edge), 21
|
||||
authored turns in the campaigns' register (the transcripts are gone from conv-api:
|
||||
`session_not_found`). State = current scene + player action + narration.
|
||||
|
||||
| decision | wording | correct | real moves found | controls |
|
||||
|---|---|---|---|---|
|
||||
| `place`: left this place? | r1 "has the scene moved … or does it still hold?" | 16/21 (null 15/21) | **1/6** | 4/7 |
|
||||
| `place2` | r2 location-anchored: "began at the {loc}; by the end, where are they?" | **21/21**, all unanimous, p ≥ 0.91 | 6/6 | 7/7 |
|
||||
| `exit`: which exit? | r1 "which one does this turn move the story into?" | **18/21** | 6/6 | 6/7 |
|
||||
| `exit2` | r2 "only once the narration actually gets there" | 16/21 | 1/6 | 4/7 |
|
||||
|
||||
- Round 1 `place` failed its positive controls (no better than the blind null), so the
|
||||
instrument, not the model, was broken. `place2` was written after seeing that.
|
||||
- `exit` r1 misses: two false advances on Toll-Road conversation turns (p 0.59 split, p 0.67
|
||||
unanimous), and `v-to-pub`, where my label said stay and semif chose "Outside the Sanctum". On
|
||||
re-reading, semif's pick is defensible (they do run outside first). All its correct advances
|
||||
had p ≥ 0.87 except one at 0.67. A threshold would be tuned on the test set, so none is proposed.
|
||||
- Latency: 4 decisions per turn, 133–179 ms end to end (101–141 ms server). The first request
|
||||
of two files took ~700–800 ms (n=2, cause not isolated). Either way it is in dead time and is
|
||||
far below the GM's reflect.
|
||||
- Side findings from the code read, not fixed (they are Wyrd's): `story_node.created_turn` is
|
||||
never written, so "scenes so far (in order)" is alphabetical and includes unvisited seeds;
|
||||
and a late "ready" for an abandoned node's backdrop can still replace the one on screen.
|
||||
|
||||
## Cicada: should she show emotion (a gate, not a mood ring)
|
||||
|
||||
Constraint found first: Cicada persistent-memory 2026-09-20, operator: **"Affect is emitted once
|
||||
and fanned out … nothing derives affect a second time … A classifier inferring emotion from
|
||||
output text is the mood-ring failure."** So semif does not pick her pose and never reads her
|
||||
reply. What is open: talk's `/face` loop measured the model gesturing on 8–11 of 18 mundane
|
||||
turns (target under 1 in 3), and its README names "the page dropping gestures" as the next lever,
|
||||
"a design call, not a tuning pass". The spike measures that lever: a gate on what the PERSON
|
||||
said, run beside the LLM.
|
||||
|
||||
| wording | correct | mundane → "ordinary" | earned → "react" | controls |
|
||||
|---|---|---|---|---|
|
||||
| w1: descriptive options (news, joke, surprise, distress, a turn / command, plain question, small talk) | **30/31** | 16/16 | 14/15 | 5/5 |
|
||||
| w2: "Should the assistant's face visibly react to this?" Yes/No | 19/31 | 16/16 | **3/15** | 4/5 |
|
||||
|
||||
- w2 said "ordinary" to "My dog died this morning" at p 0.93. **The wording carries the whole
|
||||
definition.** Terse yes/no options fail; descriptive ones work.
|
||||
- Adding the earlier lines of the conversation changed no scored answer.
|
||||
- The one w1 miss: "Order four hundred pounds of cheese" → ordinary. Open (unscored) cases:
|
||||
"Goodnight" → react (p 0.58), "Thanks, you're the best" → react (0.89), "STOP THE TIMER" →
|
||||
ordinary.
|
||||
- Latency ~105 ms end to end (76 ms server), under talk's ~0.4–0.6 s pose decode, so it adds
|
||||
nothing to the turn if it starts at end of utterance.
|
||||
|
||||
## Recommendations (surfaced to Prime; nothing built)
|
||||
|
||||
- **Cicada:** the cleanest fit of the two. Gate **gestures only**: on "ordinary", drop the model's
|
||||
gesture and leave its pose alone. A miss then costs one missing gesture; gating poses would
|
||||
leave her neutral on bad news. It reads input, never her output, and never picks WHICH feeling,
|
||||
but it does override the model's affect. That is Prime's ruling to make, and the code would
|
||||
live in tts-stack talk `/face`, not in cicada (affect is not on cicada's v1 list).
|
||||
- **Wyrd:** `place2` is the node-add gate the archival note asked for. R8 exit choice works but
|
||||
its wording is fragile. Wyrd's v1 is met and no churn has been observed (2–6 nodes per real
|
||||
campaign), so by the roadmap gate this is a parking-lot item with a trigger: build it if live
|
||||
play shows churn, after labelling ~50 real turns as a held-out set.
|
||||
|
||||
## Follow-up: SemIf as Cicada's mood source — FASTER? WORKS? (Prime, 0937: "build nothing … test cicada on latency grounds with clear emotional context")
|
||||
|
||||
Parked design, henge id 88: SemIf replaces the chat model's guided pose. Each call sees the earlier
|
||||
turns, each stamped with the pose SemIf chose, and the stamp drives the carry-over. Harness:
|
||||
`services/semif-serve/spike/cicada_mood_latency.py` + `cicada_mood_analyze.py`, raw data in
|
||||
`cicada-mood-2026-09-27/`. 24 emotionally clear single lines and three 5-turn arcs, 3 runs,
|
||||
interleaved. The prompt and schema are imported from tts-stack talk `app.py`, so A matches /face
|
||||
byte for byte. The chat model is char-rp-fast = G4-MeroMero-26B-A4B NVFP4A16 on fv-ml1:8021,
|
||||
**on GPU 1, the same GPU as semif**.
|
||||
|
||||
**Answer: no on both counts.**
|
||||
|
||||
| (singles, n=72 each) | first paragraph ready, direction known | vs today |
|
||||
|---|---|---|
|
||||
| A: today, guided `{pose, gesture, text}` | **246 ms** (IQR 230–263); the pose closes at 86 ms | — |
|
||||
| BA: SemIf and a text-only LLM call at once | 285 ms | **+32 ms** (95% CI +27..+41), faster in 10/72 |
|
||||
| BP: SemIf first, mood in the prompt | 352 ms | **+94 ms** (CI +85..+117), faster in 0/72 |
|
||||
|
||||
Noise floor A vs A: median |diff| 16.5 ms. The arcs agree: BP is +98 ms.
|
||||
- **Why it can't win on speed.** Dropping the pose+gesture header saves only ~31 ms (the LLM
|
||||
alone, text-only, reaches first paragraph in 215 ms vs 246 ms). The prompt is prefix-cached, so
|
||||
its 1,885-vs-909 tokens barely matter. SemIf's fast path costs 136 ms on its own. Run
|
||||
concurrently, it slows the LLM's decode on the shared GPU: SemIf goes to 196 ms and the LLM's
|
||||
first paragraph to 285 ms, and SemIf was the bottleneck in only 3/72 cases. DERIVED, not
|
||||
measured: with SemIf on a different GPU, the async version would be about max(215, 136) =
|
||||
~31 ms faster than today. That is the ceiling.
|
||||
- **Rotations are out.** 16 options with rotations is 550–1,200 ms, so every live arm used one
|
||||
ordering (pose + gesture ~110–136 ms).
|
||||
|
||||
| works (clear cases) | today (A) | SemIf (live config) |
|
||||
|---|---|---|
|
||||
| acceptable pose, singles | **66/72** (92%) | 48/72 (67%), 16/24 per case, deterministic |
|
||||
| controls | 12/12 | 12/12 |
|
||||
| egregious | 0 | 0 |
|
||||
| arcs acceptable | **40/45** | 25/45 |
|
||||
| mood carries through a mundane follow-up | **14/15** | **7/15** |
|
||||
| distinct poses used | 10 | 9 (no collapse to neutral) |
|
||||
| gesture rate, all / calm commands | 42/72 / 2/12 | **9/72** / 0/12 |
|
||||
|
||||
- SemIf's misses are systematic: hostility → annoyed, "someone's in the hallway" → curious,
|
||||
"someone's trying the back door" → suspicious, jokes → curious or neutral.
|
||||
- **The stamped previous mood did NOT carry.** In the vet arc, SemIf went concerned → neutral →
|
||||
neutral in all 3 runs ("I don't want to talk about it", "play something quiet") while the model
|
||||
held sad, sad, sad. The previous pose was in the state and it lost to the neutral-sounding line.
|
||||
- SemIf's pick agrees with the model 43% of the time, against the model's own run-to-run
|
||||
agreement of 83%, so this is a different judgment and not noise.
|
||||
- Wording grid (deterministic, singles): authored 16/24, short 14/24, plain **19/24** but it
|
||||
called "turn off the lights" sleepy. Rotations add about one case at 5–8× the cost. Null
|
||||
control (content-free state) → "content" 24/24, i.e. 5/24 by base rate, so SemIf is reading
|
||||
the evidence.
|
||||
- No vocal-sound vs pose clashes in any arm (0/117, 0/72, 0/117; crude detector).
|
||||
- The one bright spot matches the gate spike: **SemIf is far more sparing with gestures (13% vs
|
||||
58%)**, and the model is well over its own "fewer than 1 in 3" target on emotionally loaded
|
||||
lines.
|
||||
- Limits: one labeller, 24 + 15 cases, one model size (4B vs 26B-A4B), wordings not tuned.
|
||||
Tuning could lift accuracy some (plain 19/24), but it cannot create latency headroom that
|
||||
isn't there.
|
||||
- Found on the way (service, not fixed): `/decide/shared` 422s an object state whose last value
|
||||
ends in `)`, `;` or `}`. Also, rotations over long option lists cost options², and hit one cold
|
||||
503. Both are now in `stacks/semif/README.md`.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21.** Recommendations await Prime. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
|
||||
|
||||
# SemIf consumer-fit spikes: Wyrd scene change, Cicada affect gate (2026-09-27, Prime)
|
||||
|
||||
Prime, 0904: "Two semif spikes -- Wyrd has a couple flags per turn on whether the scene should
|
||||
change, check the codebase, see if semif is a clean fit and then spike a couple of sample
|
||||
scenarios. likewise, cicada has a speech module and some reactions-- spike whether or not she
|
||||
should show emotion." No service change. Code and raw output: `services/semif-serve/spike/`
|
||||
(`consumer_fit.py` harness; `wyrd_cases.py`, `cicada_cases.py` build the scenario files;
|
||||
`consumer-fit-2026-09-27/` holds scenarios, `.result.json` and `.log` per run).
|
||||
|
||||
## Method (applies to both)
|
||||
|
||||
- One `/decide/shared` per turn, every decision with `"orderings": "rotations"`. Target:
|
||||
semif-serve 0.1.3 on fv-ml1 GPU 1, client on nh3-dev.
|
||||
- **Noise floor:** the service is deterministic (acceptance: 144 rows twice, gap 0.0), and the
|
||||
harness asserts the same top on all 3 runs of every case. Repeats measure latency only.
|
||||
- **Null control:** every case is also asked over a content-free state. It always picks the
|
||||
"nothing happened" option, so its accuracy is that label's base rate (Cicada 16/31, Wyrd
|
||||
15/21). Evidence conditions are read against that, not against 0.
|
||||
- **Positive controls:** cases tagged `control`, unambiguous by construction.
|
||||
- **Limits, stated:** one labeller (me), authored inputs, 31 Cicada and 21 Wyrd cases per
|
||||
decision. Wilson 95% intervals are wide (30/31 → 0.84–0.99; 21/21 → 0.85–1.0), so nothing
|
||||
here resolves differences under ~10–15 points. The winning wordings were chosen on this same
|
||||
set, so their scores are optimistic until checked on held-out real play.
|
||||
|
||||
## Wyrd: scene change (fit: yes for the CHOICE, no for the WRITING)
|
||||
|
||||
How it works today (from the code): no dedicated flag. The GM's `gen` structured-output calls
|
||||
carry it two ways. **R8** (reflect): `plot_direction.target_node`, accepted only if it is an
|
||||
advanceable exit; `plot_direction.intrude` is "advisory" and never read anywhere. **Path B**:
|
||||
raw `AddStoryNode` + `AdvanceToNode` events, gated only by prompt prose ("Add + advance to a NEW
|
||||
node ONLY when the scene moves to a genuinely different place — never every turn"). Wyrd's
|
||||
archival memory already says "if it churns, tighten the prompt (or gate node-adds)".
|
||||
**Three of the real campaigns in `.wyrd-data/canon` have ZERO story edges, so R8 can never fire
|
||||
there and every scene change went through ungated path B.**
|
||||
|
||||
Fit: both are typed choices over one state, exactly semif's shape. Semif's exit options can only
|
||||
be "stay" plus the real advanceable exits, so it cannot name an illegal target. It cannot write the
|
||||
new node (title, summary, mood, image prompt) or the steering directive; the GM keeps those. So
|
||||
it would be a gate/selector in front of the GM, not a replacement.
|
||||
|
||||
Spike: 3 real seed graphs (The Fletcher, The Toll Road, The Last Vigil + one added edge), 21
|
||||
authored turns in the campaigns' register (the transcripts are gone from conv-api:
|
||||
`session_not_found`). State = current scene + player action + narration.
|
||||
|
||||
| decision | wording | correct | real moves found | controls |
|
||||
|---|---|---|---|---|
|
||||
| `place`: left this place? | r1 "has the scene moved … or does it still hold?" | 16/21 (null 15/21) | **1/6** | 4/7 |
|
||||
| `place2` | r2 location-anchored: "began at the {loc}; by the end, where are they?" | **21/21**, all unanimous, p ≥ 0.91 | 6/6 | 7/7 |
|
||||
| `exit`: which exit? | r1 "which one does this turn move the story into?" | **18/21** | 6/6 | 6/7 |
|
||||
| `exit2` | r2 "only once the narration actually gets there" | 16/21 | 1/6 | 4/7 |
|
||||
|
||||
- Round 1 `place` failed its positive controls (no better than the blind null), so the
|
||||
instrument, not the model, was broken. `place2` was written after seeing that.
|
||||
- `exit` r1 misses: two false advances on Toll-Road conversation turns (p 0.59 split, p 0.67
|
||||
unanimous), and `v-to-pub`, where my label said stay and semif chose "Outside the Sanctum". On
|
||||
re-reading, semif's pick is defensible (they do run outside first). All its correct advances
|
||||
had p ≥ 0.87 except one at 0.67. A threshold would be tuned on the test set, so none is proposed.
|
||||
- Latency: 4 decisions per turn, 133–179 ms end to end (101–141 ms server). The first request
|
||||
of two files took ~700–800 ms (n=2, cause not isolated). Either way it is in dead time and is
|
||||
far below the GM's reflect.
|
||||
- Side findings from the code read, not fixed (they are Wyrd's): `story_node.created_turn` is
|
||||
never written, so "scenes so far (in order)" is alphabetical and includes unvisited seeds;
|
||||
and a late "ready" for an abandoned node's backdrop can still replace the one on screen.
|
||||
|
||||
## Cicada: should she show emotion (a gate, not a mood ring)
|
||||
|
||||
Constraint found first: Cicada persistent-memory 2026-09-20, operator: **"Affect is emitted once
|
||||
and fanned out … nothing derives affect a second time … A classifier inferring emotion from
|
||||
output text is the mood-ring failure."** So semif does not pick her pose and never reads her
|
||||
reply. What is open: talk's `/face` loop measured the model gesturing on 8–11 of 18 mundane
|
||||
turns (target under 1 in 3), and its README names "the page dropping gestures" as the next lever,
|
||||
"a design call, not a tuning pass". The spike measures that lever: a gate on what the PERSON
|
||||
said, run beside the LLM.
|
||||
|
||||
| wording | correct | mundane → "ordinary" | earned → "react" | controls |
|
||||
|---|---|---|---|---|
|
||||
| w1: descriptive options (news, joke, surprise, distress, a turn / command, plain question, small talk) | **30/31** | 16/16 | 14/15 | 5/5 |
|
||||
| w2: "Should the assistant's face visibly react to this?" Yes/No | 19/31 | 16/16 | **3/15** | 4/5 |
|
||||
|
||||
- w2 said "ordinary" to "My dog died this morning" at p 0.93. **The wording carries the whole
|
||||
definition.** Terse yes/no options fail; descriptive ones work.
|
||||
- Adding the earlier lines of the conversation changed no scored answer.
|
||||
- The one w1 miss: "Order four hundred pounds of cheese" → ordinary. Open (unscored) cases:
|
||||
"Goodnight" → react (p 0.58), "Thanks, you're the best" → react (0.89), "STOP THE TIMER" →
|
||||
ordinary.
|
||||
- Latency ~105 ms end to end (76 ms server), under talk's ~0.4–0.6 s pose decode, so it adds
|
||||
nothing to the turn if it starts at end of utterance.
|
||||
|
||||
## Recommendations (surfaced to Prime; nothing built)
|
||||
|
||||
- **Cicada:** the cleanest fit of the two. Gate **gestures only**: on "ordinary", drop the model's
|
||||
gesture and leave its pose alone. A miss then costs one missing gesture; gating poses would
|
||||
leave her neutral on bad news. It reads input, never her output, and never picks WHICH feeling,
|
||||
but it does override the model's affect. That is Prime's ruling to make, and the code would
|
||||
live in tts-stack talk `/face`, not in cicada (affect is not on cicada's v1 list).
|
||||
- **Wyrd:** `place2` is the node-add gate the archival note asked for. R8 exit choice works but
|
||||
its wording is fragile. Wyrd's v1 is met and no churn has been observed (2–6 nodes per real
|
||||
campaign), so by the roadmap gate this is a parking-lot item with a trigger: build it if live
|
||||
play shows churn, after labelling ~50 real turns as a held-out set.
|
||||
|
||||
## Follow-up: SemIf as Cicada's mood source — FASTER? WORKS? (Prime, 0937: "build nothing … test cicada on latency grounds with clear emotional context")
|
||||
|
||||
Parked design, henge id 88: SemIf replaces the chat model's guided pose. Each call sees the earlier
|
||||
turns, each stamped with the pose SemIf chose, and the stamp drives the carry-over. Harness:
|
||||
`services/semif-serve/spike/cicada_mood_latency.py` + `cicada_mood_analyze.py`, raw data in
|
||||
`cicada-mood-2026-09-27/`. 24 emotionally clear single lines and three 5-turn arcs, 3 runs,
|
||||
interleaved. The prompt and schema are imported from tts-stack talk `app.py`, so A matches /face
|
||||
byte for byte. The chat model is char-rp-fast = G4-MeroMero-26B-A4B NVFP4A16 on fv-ml1:8021,
|
||||
**on GPU 1, the same GPU as semif**.
|
||||
|
||||
**Answer: no on both counts.**
|
||||
|
||||
| (singles, n=72 each) | first paragraph ready, direction known | vs today |
|
||||
|---|---|---|
|
||||
| A: today, guided `{pose, gesture, text}` | **246 ms** (IQR 230–263); the pose closes at 86 ms | — |
|
||||
| BA: SemIf and a text-only LLM call at once | 285 ms | **+32 ms** (95% CI +27..+41), faster in 10/72 |
|
||||
| BP: SemIf first, mood in the prompt | 352 ms | **+94 ms** (CI +85..+117), faster in 0/72 |
|
||||
|
||||
Noise floor A vs A: median |diff| 16.5 ms. The arcs agree: BP is +98 ms.
|
||||
- **Why it can't win on speed.** Dropping the pose+gesture header saves only ~31 ms (the LLM
|
||||
alone, text-only, reaches first paragraph in 215 ms vs 246 ms). The prompt is prefix-cached, so
|
||||
its 1,885-vs-909 tokens barely matter. SemIf's fast path costs 136 ms on its own. Run
|
||||
concurrently, it slows the LLM's decode on the shared GPU: SemIf goes to 196 ms and the LLM's
|
||||
first paragraph to 285 ms, and SemIf was the bottleneck in only 3/72 cases. DERIVED, not
|
||||
measured: with SemIf on a different GPU, the async version would be about max(215, 136) =
|
||||
~31 ms faster than today. That is the ceiling.
|
||||
- **Rotations are out.** 16 options with rotations is 550–1,200 ms, so every live arm used one
|
||||
ordering (pose + gesture ~110–136 ms).
|
||||
|
||||
| works (clear cases) | today (A) | SemIf (live config) |
|
||||
|---|---|---|
|
||||
| acceptable pose, singles | **66/72** (92%) | 48/72 (67%), 16/24 per case, deterministic |
|
||||
| controls | 12/12 | 12/12 |
|
||||
| egregious | 0 | 0 |
|
||||
| arcs acceptable | **40/45** | 25/45 |
|
||||
| mood carries through a mundane follow-up | **14/15** | **7/15** |
|
||||
| distinct poses used | 10 | 9 (no collapse to neutral) |
|
||||
| gesture rate, all / calm commands | 42/72 / 2/12 | **9/72** / 0/12 |
|
||||
|
||||
- SemIf's misses are systematic: hostility → annoyed, "someone's in the hallway" → curious,
|
||||
"someone's trying the back door" → suspicious, jokes → curious or neutral.
|
||||
- **The stamped previous mood did NOT carry.** In the vet arc, SemIf went concerned → neutral →
|
||||
neutral in all 3 runs ("I don't want to talk about it", "play something quiet") while the model
|
||||
held sad, sad, sad. The previous pose was in the state and it lost to the neutral-sounding line.
|
||||
- SemIf's pick agrees with the model 43% of the time, against the model's own run-to-run
|
||||
agreement of 83%, so this is a different judgment and not noise.
|
||||
- Wording grid (deterministic, singles): authored 16/24, short 14/24, plain **19/24** but it
|
||||
called "turn off the lights" sleepy. Rotations add about one case at 5–8× the cost. Null
|
||||
control (content-free state) → "content" 24/24, i.e. 5/24 by base rate, so SemIf is reading
|
||||
the evidence.
|
||||
- No vocal-sound vs pose clashes in any arm (0/117, 0/72, 0/117; crude detector).
|
||||
- The one bright spot matches the gate spike: **SemIf is far more sparing with gestures (13% vs
|
||||
58%)**, and the model is well over its own "fewer than 1 in 3" target on emotionally loaded
|
||||
lines.
|
||||
- Limits: one labeller, 24 + 15 cases, one model size (4B vs 26B-A4B), wordings not tuned.
|
||||
Tuning could lift accuracy some (plain 19/24), but it cannot create latency headroom that
|
||||
isn't there.
|
||||
- Found on the way (service, not fixed): `/decide/shared` 422s an object state whose last value
|
||||
ends in `)`, `;` or `}`. Also, rotations over long option lists cost options², and hit one cold
|
||||
503. Both are now in `stacks/semif/README.md`.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` — **DONE:** 0.1.3 live, with fast kernels adopted (`77b8cb4`).
|
||||
|
||||
# `[2026-09-27]` SemIf order-averaging: spiked, Prime ruled build + fast-kernel trial
|
||||
|
||||
**Finding (Prime's probes):** SemIf's single-ordering answer leans toward whichever option is listed
|
||||
first on ambiguous inputs. For "It's 2AM and I'm bored", the top pick flipped from casual 0.685 to booty
|
||||
call 0.760 when the order was reversed.
|
||||
|
||||
**Spike** (`739aa03`, `services/semif-serve/spike/`, no service change). SemIf authored144 +
|
||||
perturbations108, 252 rows in 72 groups, 3 options each, with all 6 orderings of every row sent in one
|
||||
shared request:
|
||||
|
||||
| method | accuracy |
|
||||
|---|---|
|
||||
| single ordering, as sent | 78.6% |
|
||||
| expected single ordering | 80.0% |
|
||||
| 3 rotations, log-mean | **87.7%** (+9.1 pts, group-bootstrap 95% CI +4.7..+13.8) |
|
||||
| all 6 orderings, log-mean | 88.1% |
|
||||
|
||||
Rows where the rotations agree unanimously: 161 rows, 94.4% accurate. Split rows: 91 rows, 75.8%.
|
||||
The service is deterministic, so the CI measures item sampling, not run noise. This is one task family
|
||||
(evidence interpretation, 3 options), not our workload.
|
||||
|
||||
**Latency** (`spike/latency-2026-09-27.txt`, from nh3-dev, network floor 31 ms):
|
||||
- `/decide` short: 71 ms end to end (38 ms server);
|
||||
- 3 rotations shared: 113 ms (79 ms);
|
||||
- 6 orderings: 137 ms (99 ms);
|
||||
- a ~2k-token state: 210 ms (169 ms).
|
||||
|
||||
**Prime ruled (2026-09-27 ~0250): build it in, and try the fast kernels too.**
|
||||
- Averaging design: opt-in `orderings: rotations|all` (all only ≤ 4 options); all orderings in one shared
|
||||
batch; per-ordering SemIf results returned unchanged; a `combined` block (log-mean, top, agreement,
|
||||
spread); orderings count toward `max_decisions`.
|
||||
- Fast kernels: `causal_conv1d` + `flash-linear-attention` are missing, so transformers runs Qwen3.5's
|
||||
reference PyTorch paths. Adopt them only if the authored144 parity re-check holds.
|
||||
|
||||
Tracked in the in-flight SemIf section. See [[2026-09-27-semif-live-on-fv-ml1-gpu1]].
|
||||
|
||||
**Outcome (0.1.3, `77b8cb4`, ~0340 PT):** built and deployed. Through the service on the same 252 rows,
|
||||
accuracy is 78.6% → 88.1% (95% CI +5.1..+14.3), and unanimous rows are 94.5% accurate. The fast kernels
|
||||
won the A/B on the empty GPU 3:
|
||||
- parity with upstream 144/144 (was 142/144);
|
||||
- a ~2k-token `/decide` 169 → 92 ms;
|
||||
- shared capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens (was 52/43/19/13).
|
||||
All of heid's bug-hunt findings were folded into the same release (thread `01M3H3F4RR7XBP90KQ3A39H4SX`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime).** Rotation stays Prime's. → `persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md`
|
||||
|
||||
# `[2026-09-27]` restic: repository URL out of the systemd units on all 8 hosts, and vaulted (Prime)
|
||||
|
||||
`resticprofile schedule` copies `env-file` values into the generated units under /etc/systemd/system
|
||||
(0644). So `RESTIC_REPOSITORY`, including the rest-server basic-auth password, was readable by any local
|
||||
user on every env-file host (found on esh-docker-vm; `systemctl cat` works without sudo). Blast radius:
|
||||
the password allows reading the encrypted blobs and appending to one repo, and nothing more (rest-server
|
||||
is `--append-only`, and the passphrase is a separate file).
|
||||
|
||||
**Fix** (`6e203dc`, `30f2c97`): `playbooks/restic-repository-file.yaml` does four things:
|
||||
- derives `/etc/restic/repository` (root 0400) from `restic.env`;
|
||||
- uploads the profile switched to `repository-file`, guarded by the live profile's pre-change sha;
|
||||
- runs `cat config` through the new profile;
|
||||
- re-schedules, then verifies the units exist and contain no `rest:http`.
|
||||
|
||||
Applied to ana-docker, fv-ml1 (`configs/restic/ana-ml2`), esh-docker-vm, esh-vm-db, irv-ml1, nh3-dev,
|
||||
nh3-docker, and vm-esh-nas once Prime had bootstrapped infra-ops there. esh-ml1 was built that way. An
|
||||
independent check across all 8 found 0 leaking units. nh3-docker's scheduled unit ran a real backup
|
||||
(`a29b889d`). Every host's URL and passphrase are vaulted as `<host>/etc/restic/{repository,password}`.
|
||||
|
||||
- `restic.env` is **kept** (root 0600): the per-host READMEs and the freshness probe source it. **A rotation
|
||||
must update the vault, `restic.env` and `repository`.**
|
||||
- The rotation itself stays Prime's (backups runbook, Known gaps). The move stops the ongoing exposure but
|
||||
does not un-expose what was readable.
|
||||
- Before the edit, esh-docker-vm's and irv-ml1's live profiles had drifted ahead of the repo, and were
|
||||
pulled in first (`c698751`).
|
||||
|
||||
## In-flight state at retirement (moved here at the 2026-10-06 snapshot)
|
||||
|
||||
- Seven hosts were moved from `env-file` to `repository-file` (`playbooks/restic-repository-file.yaml`),
|
||||
so the systemd units no longer carry the rest-server password. Secrets are vaulted as
|
||||
`<host>/etc/restic/{repository,password}`. `restic.env` is KEPT for manual snippets, so
|
||||
**a rotation must update the vault, `restic.env` and `repository`.**
|
||||
- **vm-esh-nas done too** (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). **All 8 hosts are clean
|
||||
and vaulted.**
|
||||
- The passwords were readable until today, so the rotation (Prime's) remains the real fix.
|
||||
- 2026-09-27 0859 freshness check: all restic and PBS repos ✅. Those 0100 runs PRE-date the move
|
||||
(done ~0140–0200).
|
||||
- **VERIFIED 2026-09-28 (infra-hermes, read-only):** the first nightly on `repository-file` (0100 PT)
|
||||
succeeded on all 8 hosts (unit Result=success AND a today-dated snapshot in each repo), and the
|
||||
0800 freshness check was all-green, PBS included. nh3-dev also passed a content restore (`af5580c3`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **infra-ops bootstrapped on vm-esh-nas by Prime** (the first account at the fleet-pinned uid/gid 850; `bootstrap-infra-ops-user.yaml` now pins 850 when free, `d775a01`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **augaman: second instance on fv-ml1 benched, then REMOVED (Prime).** esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (`ad484c3`, bench `docs/pfi/augaman-speed-bench/`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` Created empty private repo `corviduo/svoperatingsystem` (SVOS, the operating system) as claude-bot, at brokkr-smithy-dev's relay of Prime's ruling. `svos-dev` → the OS session; The High Seat's session is `highseat-dev` again.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
## Tried and abandoned (archived)
|
||||
|
||||
- `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix** → `persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md`
|
||||
|
||||
# Shorter Parakeet slices as Scriberr's memory fix
|
||||
|
||||
`[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix.** I predicted ~6× less memory from the attention-matrix arithmetic. Measured, 300 → 120 s only went 9,384 → 6,510 MiB: a ~5.6 GB fixed floor dominates. `expandable_segments:True` was the real lever (5,496). Measure the process peak; never extrapolate it from one tensor.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-30]` **Whole-file local-attention Parakeet in Scriberr (context 255/255).** OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-30]` **Start-time midpoint stitching of overlapped Parakeet chunks.** It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-30]` **int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime.** The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-30]` **GPU budgets computed as total − used.** nvidia-smi `Free` is ~640 MiB lower per card (driver reserve). Budget from `Free`.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel** → `persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md`
|
||||
|
||||
# The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel
|
||||
|
||||
`[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel.** Every tool worked except `get_viewport_screenshot`, which had Blender write a file for the SERVER to read ("Screenshot file was not created"). The server now runs inside the Blender container over ssh+docker-exec stdio. That also means no port is published. → `stacks/blender/README.md`
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **A TCP connect as the "is Blender ready" probe** → `persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md`
|
||||
|
||||
# A TCP connect as the "is Blender ready" probe
|
||||
|
||||
`[2026-09-27]` **A TCP connect as the "is Blender ready" probe.** docker-proxy accepts on a published port before the app behind it listens, so it said "answering" while Blender was still loading. The probe now asks the add-on to `ping`. The same trap applies to any service behind a published port.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** → `persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md`
|
||||
|
||||
# `uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project
|
||||
|
||||
`[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md`
|
||||
|
||||
# Relying on the build cache surviving on esh-ml1
|
||||
|
||||
`[2026-09-27]` **Relying on the build cache surviving on esh-ml1** — the augaman v0.1.3 build missed its dependency layer there (fv-ml1 hit it; reason unexplained, suspect the `docker builder prune` runs) and the rootfs peaked at 90%. Check for ≥15 GB free, or remove the old image, before building there.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# eshpfi configs/nemi-vault (infra-ops, 2026-10-14): keeps /srv/nemi/vault, the read-only copy of Prime's notes
|
||||
# (gitea vh/VuongKB, which he syncs with git), current for Nemi's `vault` source (albok unit 4.4 ruling 6).
|
||||
# A lkraven USER unit: the clone uses lkraven's gitea SSH key. Fast-forward only: the copy is never written to,
|
||||
# so a non-ff (a force-push upstream) fails loudly here instead of being papered over.
|
||||
[Unit]
|
||||
Description=Pull Prime's notes (vh/VuongKB) into /srv/nemi/vault for Nemi
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/git -C /srv/nemi/vault pull --ff-only --quiet
|
||||
@@ -0,0 +1,8 @@
|
||||
# Hourly at :50, ahead of albok-nemi.timer's walk at ~:01.
|
||||
[Unit]
|
||||
Description=Hourly pull of Prime's notes for Nemi
|
||||
[Timer]
|
||||
OnCalendar=*-*-* *:50:00
|
||||
Persistent=true
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,3 +0,0 @@
|
||||
# A TCP connect as the "is Blender ready" probe
|
||||
|
||||
`[2026-09-27]` **A TCP connect as the "is Blender ready" probe.** docker-proxy accepts on a published port before the app behind it listens, so it said "answering" while Blender was still loading. The probe now asks the add-on to `ping`. The same trap applies to any service behind a published port.
|
||||
@@ -1,22 +0,0 @@
|
||||
# `[2026-09-27]` Blender 5.2 on fv-ml1 GPU 3, agent-driven (Prime)
|
||||
|
||||
(Moved out of the in-flight section at the 2026-10-06 snapshot. Canonical: `stacks/blender/README.md`.)
|
||||
|
||||
- Prime: "go ahead with gpu 3, both", and he does not use Blender, so **agents drive it through MCP**.
|
||||
Blender 5.2.2 LTS (linuxserver/selkies image, digest-pinned) with a web desktop at
|
||||
`https://10.251.50.54:3001` (vault `fv-ml1/blender-web-password`). **On demand only:**
|
||||
`scripts/blender-mcp up|down|status`. It was left DOWN (GPU 3 back to 2 MiB).
|
||||
- MCP: `mcp-for-blender` 2.1.1 (Dvalin's research pick). The add-on is vendored at `41a18432`. The
|
||||
server runs INSIDE the container, and agents reach it as ssh+docker-exec stdio via
|
||||
`scripts/blender-mcp`. No port is published, and screenshots work because server and Blender
|
||||
share a filesystem. Telemetry is off and safe mode is on. Verified end to end (render, screenshot,
|
||||
safe-mode refusal). **Batch path: `scripts/blender-run`** (one-shot `docker run --rm`, `--job` staging; first user is draupnir). **Access: the shared fleet `infra-ops` login, with no render-only key (Prime, 2026-09-28).** **Registration + up/down: PER WORKING SESSION (Prime, 2026-09-28, relayed by draupnir; superseded per-task of 09-27)**: `blender-mcp up` + `claude mcp add` at session start, `claude mcp remove` + `down` at its end. Never always-on, never user- or project-wide config.
|
||||
→ `stacks/blender/README.md`
|
||||
- **Extensions (2026-09-28, draupnir; Prime ruled Blender a MANDATORY pipeline stage):** 8 pinned
|
||||
add-ons (`stacks/blender/extensions.lock`) built by `scripts/blender-extensions sync` into
|
||||
`fv-ml1:/tank/blender-extensions/5.2/system` (LIVE), mounted read-only as the System repo;
|
||||
`fleet_extensions.py` enables them (GUI startup timer; `blender-run --extensions`). Headless
|
||||
acceptance 8/9 (CAD Sketcher sketching is GUI-only); **MCP 9/9 after Prime ran the deploy himself
|
||||
(1505; the classifier had refused mine)**. SurfacePsycho's eval() is patched to literal_eval
|
||||
(held under MCP). Open: agent-drawn CAD Sketcher geometry (its stateful ops want point picks),
|
||||
and MeasureIt overlays not seen in MCP screenshots. GUI left DOWN.
|
||||
@@ -1,3 +0,0 @@
|
||||
# hermes-gateway restarted 0401 for highseat-dev
|
||||
|
||||
`[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call.
|
||||
@@ -1,3 +0,0 @@
|
||||
# Relying on the build cache surviving on esh-ml1
|
||||
|
||||
`[2026-09-27]` **Relying on the build cache surviving on esh-ml1** — the augaman v0.1.3 build missed its dependency layer there (fv-ml1 hit it; reason unexplained, suspect the `docker builder prune` runs) and the rootfs peaked at 90%. Check for ≥15 GB free, or remove the old image, before building there.
|
||||
@@ -1,40 +0,0 @@
|
||||
# `[2026-09-27]` restic: repository URL out of the systemd units on all 8 hosts, and vaulted (Prime)
|
||||
|
||||
`resticprofile schedule` copies `env-file` values into the generated units under /etc/systemd/system
|
||||
(0644). So `RESTIC_REPOSITORY`, including the rest-server basic-auth password, was readable by any local
|
||||
user on every env-file host (found on esh-docker-vm; `systemctl cat` works without sudo). Blast radius:
|
||||
the password allows reading the encrypted blobs and appending to one repo, and nothing more (rest-server
|
||||
is `--append-only`, and the passphrase is a separate file).
|
||||
|
||||
**Fix** (`6e203dc`, `30f2c97`): `playbooks/restic-repository-file.yaml` does four things:
|
||||
- derives `/etc/restic/repository` (root 0400) from `restic.env`;
|
||||
- uploads the profile switched to `repository-file`, guarded by the live profile's pre-change sha;
|
||||
- runs `cat config` through the new profile;
|
||||
- re-schedules, then verifies the units exist and contain no `rest:http`.
|
||||
|
||||
Applied to ana-docker, fv-ml1 (`configs/restic/ana-ml2`), esh-docker-vm, esh-vm-db, irv-ml1, nh3-dev,
|
||||
nh3-docker, and vm-esh-nas once Prime had bootstrapped infra-ops there. esh-ml1 was built that way. An
|
||||
independent check across all 8 found 0 leaking units. nh3-docker's scheduled unit ran a real backup
|
||||
(`a29b889d`). Every host's URL and passphrase are vaulted as `<host>/etc/restic/{repository,password}`.
|
||||
|
||||
- `restic.env` is **kept** (root 0600): the per-host READMEs and the freshness probe source it. **A rotation
|
||||
must update the vault, `restic.env` and `repository`.**
|
||||
- The rotation itself stays Prime's (backups runbook, Known gaps). The move stops the ongoing exposure but
|
||||
does not un-expose what was readable.
|
||||
- Before the edit, esh-docker-vm's and irv-ml1's live profiles had drifted ahead of the repo, and were
|
||||
pulled in first (`c698751`).
|
||||
|
||||
## In-flight state at retirement (moved here at the 2026-10-06 snapshot)
|
||||
|
||||
- Seven hosts were moved from `env-file` to `repository-file` (`playbooks/restic-repository-file.yaml`),
|
||||
so the systemd units no longer carry the rest-server password. Secrets are vaulted as
|
||||
`<host>/etc/restic/{repository,password}`. `restic.env` is KEPT for manual snippets, so
|
||||
**a rotation must update the vault, `restic.env` and `repository`.**
|
||||
- **vm-esh-nas done too** (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). **All 8 hosts are clean
|
||||
and vaulted.**
|
||||
- The passwords were readable until today, so the rotation (Prime's) remains the real fix.
|
||||
- 2026-09-27 0859 freshness check: all restic and PBS repos ✅. Those 0100 runs PRE-date the move
|
||||
(done ~0140–0200).
|
||||
- **VERIFIED 2026-09-28 (infra-hermes, read-only):** the first nightly on `repository-file` (0100 PT)
|
||||
succeeded on all 8 hosts (unit Result=success AND a today-dated snapshot in each repo), and the
|
||||
0800 freshness check was all-green, PBS included. nh3-dev also passed a content restore (`af5580c3`).
|
||||
@@ -1,160 +0,0 @@
|
||||
# SemIf consumer-fit spikes: Wyrd scene change, Cicada affect gate (2026-09-27, Prime)
|
||||
|
||||
Prime, 0904: "Two semif spikes -- Wyrd has a couple flags per turn on whether the scene should
|
||||
change, check the codebase, see if semif is a clean fit and then spike a couple of sample
|
||||
scenarios. likewise, cicada has a speech module and some reactions-- spike whether or not she
|
||||
should show emotion." No service change. Code and raw output: `services/semif-serve/spike/`
|
||||
(`consumer_fit.py` harness; `wyrd_cases.py`, `cicada_cases.py` build the scenario files;
|
||||
`consumer-fit-2026-09-27/` holds scenarios, `.result.json` and `.log` per run).
|
||||
|
||||
## Method (applies to both)
|
||||
|
||||
- One `/decide/shared` per turn, every decision with `"orderings": "rotations"`. Target:
|
||||
semif-serve 0.1.3 on fv-ml1 GPU 1, client on nh3-dev.
|
||||
- **Noise floor:** the service is deterministic (acceptance: 144 rows twice, gap 0.0), and the
|
||||
harness asserts the same top on all 3 runs of every case. Repeats measure latency only.
|
||||
- **Null control:** every case is also asked over a content-free state. It always picks the
|
||||
"nothing happened" option, so its accuracy is that label's base rate (Cicada 16/31, Wyrd
|
||||
15/21). Evidence conditions are read against that, not against 0.
|
||||
- **Positive controls:** cases tagged `control`, unambiguous by construction.
|
||||
- **Limits, stated:** one labeller (me), authored inputs, 31 Cicada and 21 Wyrd cases per
|
||||
decision. Wilson 95% intervals are wide (30/31 → 0.84–0.99; 21/21 → 0.85–1.0), so nothing
|
||||
here resolves differences under ~10–15 points. The winning wordings were chosen on this same
|
||||
set, so their scores are optimistic until checked on held-out real play.
|
||||
|
||||
## Wyrd: scene change (fit: yes for the CHOICE, no for the WRITING)
|
||||
|
||||
How it works today (from the code): no dedicated flag. The GM's `gen` structured-output calls
|
||||
carry it two ways. **R8** (reflect): `plot_direction.target_node`, accepted only if it is an
|
||||
advanceable exit; `plot_direction.intrude` is "advisory" and never read anywhere. **Path B**:
|
||||
raw `AddStoryNode` + `AdvanceToNode` events, gated only by prompt prose ("Add + advance to a NEW
|
||||
node ONLY when the scene moves to a genuinely different place — never every turn"). Wyrd's
|
||||
archival memory already says "if it churns, tighten the prompt (or gate node-adds)".
|
||||
**Three of the real campaigns in `.wyrd-data/canon` have ZERO story edges, so R8 can never fire
|
||||
there and every scene change went through ungated path B.**
|
||||
|
||||
Fit: both are typed choices over one state, exactly semif's shape. Semif's exit options can only
|
||||
be "stay" plus the real advanceable exits, so it cannot name an illegal target. It cannot write the
|
||||
new node (title, summary, mood, image prompt) or the steering directive; the GM keeps those. So
|
||||
it would be a gate/selector in front of the GM, not a replacement.
|
||||
|
||||
Spike: 3 real seed graphs (The Fletcher, The Toll Road, The Last Vigil + one added edge), 21
|
||||
authored turns in the campaigns' register (the transcripts are gone from conv-api:
|
||||
`session_not_found`). State = current scene + player action + narration.
|
||||
|
||||
| decision | wording | correct | real moves found | controls |
|
||||
|---|---|---|---|---|
|
||||
| `place`: left this place? | r1 "has the scene moved … or does it still hold?" | 16/21 (null 15/21) | **1/6** | 4/7 |
|
||||
| `place2` | r2 location-anchored: "began at the {loc}; by the end, where are they?" | **21/21**, all unanimous, p ≥ 0.91 | 6/6 | 7/7 |
|
||||
| `exit`: which exit? | r1 "which one does this turn move the story into?" | **18/21** | 6/6 | 6/7 |
|
||||
| `exit2` | r2 "only once the narration actually gets there" | 16/21 | 1/6 | 4/7 |
|
||||
|
||||
- Round 1 `place` failed its positive controls (no better than the blind null), so the
|
||||
instrument, not the model, was broken. `place2` was written after seeing that.
|
||||
- `exit` r1 misses: two false advances on Toll-Road conversation turns (p 0.59 split, p 0.67
|
||||
unanimous), and `v-to-pub`, where my label said stay and semif chose "Outside the Sanctum". On
|
||||
re-reading, semif's pick is defensible (they do run outside first). All its correct advances
|
||||
had p ≥ 0.87 except one at 0.67. A threshold would be tuned on the test set, so none is proposed.
|
||||
- Latency: 4 decisions per turn, 133–179 ms end to end (101–141 ms server). The first request
|
||||
of two files took ~700–800 ms (n=2, cause not isolated). Either way it is in dead time and is
|
||||
far below the GM's reflect.
|
||||
- Side findings from the code read, not fixed (they are Wyrd's): `story_node.created_turn` is
|
||||
never written, so "scenes so far (in order)" is alphabetical and includes unvisited seeds;
|
||||
and a late "ready" for an abandoned node's backdrop can still replace the one on screen.
|
||||
|
||||
## Cicada: should she show emotion (a gate, not a mood ring)
|
||||
|
||||
Constraint found first: Cicada persistent-memory 2026-09-20, operator: **"Affect is emitted once
|
||||
and fanned out … nothing derives affect a second time … A classifier inferring emotion from
|
||||
output text is the mood-ring failure."** So semif does not pick her pose and never reads her
|
||||
reply. What is open: talk's `/face` loop measured the model gesturing on 8–11 of 18 mundane
|
||||
turns (target under 1 in 3), and its README names "the page dropping gestures" as the next lever,
|
||||
"a design call, not a tuning pass". The spike measures that lever: a gate on what the PERSON
|
||||
said, run beside the LLM.
|
||||
|
||||
| wording | correct | mundane → "ordinary" | earned → "react" | controls |
|
||||
|---|---|---|---|---|
|
||||
| w1: descriptive options (news, joke, surprise, distress, a turn / command, plain question, small talk) | **30/31** | 16/16 | 14/15 | 5/5 |
|
||||
| w2: "Should the assistant's face visibly react to this?" Yes/No | 19/31 | 16/16 | **3/15** | 4/5 |
|
||||
|
||||
- w2 said "ordinary" to "My dog died this morning" at p 0.93. **The wording carries the whole
|
||||
definition.** Terse yes/no options fail; descriptive ones work.
|
||||
- Adding the earlier lines of the conversation changed no scored answer.
|
||||
- The one w1 miss: "Order four hundred pounds of cheese" → ordinary. Open (unscored) cases:
|
||||
"Goodnight" → react (p 0.58), "Thanks, you're the best" → react (0.89), "STOP THE TIMER" →
|
||||
ordinary.
|
||||
- Latency ~105 ms end to end (76 ms server), under talk's ~0.4–0.6 s pose decode, so it adds
|
||||
nothing to the turn if it starts at end of utterance.
|
||||
|
||||
## Recommendations (surfaced to Prime; nothing built)
|
||||
|
||||
- **Cicada:** the cleanest fit of the two. Gate **gestures only**: on "ordinary", drop the model's
|
||||
gesture and leave its pose alone. A miss then costs one missing gesture; gating poses would
|
||||
leave her neutral on bad news. It reads input, never her output, and never picks WHICH feeling,
|
||||
but it does override the model's affect. That is Prime's ruling to make, and the code would
|
||||
live in tts-stack talk `/face`, not in cicada (affect is not on cicada's v1 list).
|
||||
- **Wyrd:** `place2` is the node-add gate the archival note asked for. R8 exit choice works but
|
||||
its wording is fragile. Wyrd's v1 is met and no churn has been observed (2–6 nodes per real
|
||||
campaign), so by the roadmap gate this is a parking-lot item with a trigger: build it if live
|
||||
play shows churn, after labelling ~50 real turns as a held-out set.
|
||||
|
||||
## Follow-up: SemIf as Cicada's mood source — FASTER? WORKS? (Prime, 0937: "build nothing … test cicada on latency grounds with clear emotional context")
|
||||
|
||||
Parked design, henge id 88: SemIf replaces the chat model's guided pose. Each call sees the earlier
|
||||
turns, each stamped with the pose SemIf chose, and the stamp drives the carry-over. Harness:
|
||||
`services/semif-serve/spike/cicada_mood_latency.py` + `cicada_mood_analyze.py`, raw data in
|
||||
`cicada-mood-2026-09-27/`. 24 emotionally clear single lines and three 5-turn arcs, 3 runs,
|
||||
interleaved. The prompt and schema are imported from tts-stack talk `app.py`, so A matches /face
|
||||
byte for byte. The chat model is char-rp-fast = G4-MeroMero-26B-A4B NVFP4A16 on fv-ml1:8021,
|
||||
**on GPU 1, the same GPU as semif**.
|
||||
|
||||
**Answer: no on both counts.**
|
||||
|
||||
| (singles, n=72 each) | first paragraph ready, direction known | vs today |
|
||||
|---|---|---|
|
||||
| A: today, guided `{pose, gesture, text}` | **246 ms** (IQR 230–263); the pose closes at 86 ms | — |
|
||||
| BA: SemIf and a text-only LLM call at once | 285 ms | **+32 ms** (95% CI +27..+41), faster in 10/72 |
|
||||
| BP: SemIf first, mood in the prompt | 352 ms | **+94 ms** (CI +85..+117), faster in 0/72 |
|
||||
|
||||
Noise floor A vs A: median |diff| 16.5 ms. The arcs agree: BP is +98 ms.
|
||||
- **Why it can't win on speed.** Dropping the pose+gesture header saves only ~31 ms (the LLM
|
||||
alone, text-only, reaches first paragraph in 215 ms vs 246 ms). The prompt is prefix-cached, so
|
||||
its 1,885-vs-909 tokens barely matter. SemIf's fast path costs 136 ms on its own. Run
|
||||
concurrently, it slows the LLM's decode on the shared GPU: SemIf goes to 196 ms and the LLM's
|
||||
first paragraph to 285 ms, and SemIf was the bottleneck in only 3/72 cases. DERIVED, not
|
||||
measured: with SemIf on a different GPU, the async version would be about max(215, 136) =
|
||||
~31 ms faster than today. That is the ceiling.
|
||||
- **Rotations are out.** 16 options with rotations is 550–1,200 ms, so every live arm used one
|
||||
ordering (pose + gesture ~110–136 ms).
|
||||
|
||||
| works (clear cases) | today (A) | SemIf (live config) |
|
||||
|---|---|---|
|
||||
| acceptable pose, singles | **66/72** (92%) | 48/72 (67%), 16/24 per case, deterministic |
|
||||
| controls | 12/12 | 12/12 |
|
||||
| egregious | 0 | 0 |
|
||||
| arcs acceptable | **40/45** | 25/45 |
|
||||
| mood carries through a mundane follow-up | **14/15** | **7/15** |
|
||||
| distinct poses used | 10 | 9 (no collapse to neutral) |
|
||||
| gesture rate, all / calm commands | 42/72 / 2/12 | **9/72** / 0/12 |
|
||||
|
||||
- SemIf's misses are systematic: hostility → annoyed, "someone's in the hallway" → curious,
|
||||
"someone's trying the back door" → suspicious, jokes → curious or neutral.
|
||||
- **The stamped previous mood did NOT carry.** In the vet arc, SemIf went concerned → neutral →
|
||||
neutral in all 3 runs ("I don't want to talk about it", "play something quiet") while the model
|
||||
held sad, sad, sad. The previous pose was in the state and it lost to the neutral-sounding line.
|
||||
- SemIf's pick agrees with the model 43% of the time, against the model's own run-to-run
|
||||
agreement of 83%, so this is a different judgment and not noise.
|
||||
- Wording grid (deterministic, singles): authored 16/24, short 14/24, plain **19/24** but it
|
||||
called "turn off the lights" sleepy. Rotations add about one case at 5–8× the cost. Null
|
||||
control (content-free state) → "content" 24/24, i.e. 5/24 by base rate, so SemIf is reading
|
||||
the evidence.
|
||||
- No vocal-sound vs pose clashes in any arm (0/117, 0/72, 0/117; crude detector).
|
||||
- The one bright spot matches the gate spike: **SemIf is far more sparing with gestures (13% vs
|
||||
58%)**, and the model is well over its own "fewer than 1 in 3" target on emotionally loaded
|
||||
lines.
|
||||
- Limits: one labeller, 24 + 15 cases, one model size (4B vs 26B-A4B), wordings not tuned.
|
||||
Tuning could lift accuracy some (plain 19/24), but it cannot create latency headroom that
|
||||
isn't there.
|
||||
- Found on the way (service, not fixed): `/decide/shared` 422s an object state whose last value
|
||||
ends in `)`, `;` or `}`. Also, rotations over long option lists cost options², and hit one cold
|
||||
503. Both are now in `stacks/semif/README.md`.
|
||||
@@ -1,35 +0,0 @@
|
||||
# `[2026-09-27]` SemIf live on fv-ml1 GPU 1 (Prime)
|
||||
|
||||
**semif-serve 0.1.2 is live** at `http://10.251.50.54:8032` (`semif.fv.internal`), stack `stacks/semif`,
|
||||
code and contract `services/semif-serve/` (`069725c`). It is a FastAPI wrapper around SemIf-OpenJev's
|
||||
direct and shared torch scorers (MIT, pinned `23cf1f39fc9534fe81437200959b6dfc7106e45a`) on
|
||||
Qwen3.5-4B `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a` (BF16, offline HF cache on `/tank`).
|
||||
Hosting assessment: infra-hermes, `/tmp/semif.md` (2026-09-25).
|
||||
|
||||
Prime's calls (2026-09-27): fv-ml1 GPU 1 under a hard VRAM cap (not the reserved-empty GPU 3, not a Q4
|
||||
GGUF on a 16 GB box); infra-ops builds it with a light process (short contract → TDD → heid bug hunt);
|
||||
no consumer named yet, so every score is labelled uncalibrated.
|
||||
|
||||
**Two defects that only the card showed**, each fixed with a test:
|
||||
- **0.1.1:** the OOM path raised `OutOfMemory(...) from exc` inside the `except` block. The chain kept the
|
||||
torch exception's traceback, and with it the failed call's frames and tensors, so 11.9 GiB stayed
|
||||
allocated after the 503. The fix raises after the block, with no chain, and runs `gc.collect()`
|
||||
before `empty_cache()`.
|
||||
- **0.1.2:** a large shared request left torch's cache holding 12.6 GB, which left scriberr 3.5 GB on GPU 1.
|
||||
Now, after each call, reserved memory above the post-warm-up baseline + 512 MiB is released
|
||||
(`empty_cache`). It costs ~16 ms on a big request.
|
||||
|
||||
**Acceptance** (`services/semif-serve/acceptance/result-2026-09-27-v0.1.2.json`), against SemIf's
|
||||
committed torch predictions on authored144:
|
||||
- 142/144 same top choice; both misses are exact bf16 ties in our output;
|
||||
- 144/144 identical prompt SHA-256;
|
||||
- A-vs-A gap 0.0;
|
||||
- negative control (rotated option descriptions) 14/144;
|
||||
- shared vs direct 72/72;
|
||||
- 21 binary criteria over one state in 159 ms.
|
||||
|
||||
The shared-mode capacity under 12 GiB is 52 decisions at a ~140-token prefix and 13 at ~3,900.
|
||||
|
||||
**Build:** torch 2.10.0+cu128 from the pytorch index (SemIf's own stack; sm_120 present). The
|
||||
Dockerfile installs dependencies from a manifest with the project version blanked, so a version bump
|
||||
reuses the ~4 GB torch layer (41 s rebuild, layer CACHED). See [[2026-09-27-semif-order-averaging]].
|
||||
@@ -1,43 +0,0 @@
|
||||
# `[2026-09-27]` SemIf order-averaging: spiked, Prime ruled build + fast-kernel trial
|
||||
|
||||
**Finding (Prime's probes):** SemIf's single-ordering answer leans toward whichever option is listed
|
||||
first on ambiguous inputs. For "It's 2AM and I'm bored", the top pick flipped from casual 0.685 to booty
|
||||
call 0.760 when the order was reversed.
|
||||
|
||||
**Spike** (`739aa03`, `services/semif-serve/spike/`, no service change). SemIf authored144 +
|
||||
perturbations108, 252 rows in 72 groups, 3 options each, with all 6 orderings of every row sent in one
|
||||
shared request:
|
||||
|
||||
| method | accuracy |
|
||||
|---|---|
|
||||
| single ordering, as sent | 78.6% |
|
||||
| expected single ordering | 80.0% |
|
||||
| 3 rotations, log-mean | **87.7%** (+9.1 pts, group-bootstrap 95% CI +4.7..+13.8) |
|
||||
| all 6 orderings, log-mean | 88.1% |
|
||||
|
||||
Rows where the rotations agree unanimously: 161 rows, 94.4% accurate. Split rows: 91 rows, 75.8%.
|
||||
The service is deterministic, so the CI measures item sampling, not run noise. This is one task family
|
||||
(evidence interpretation, 3 options), not our workload.
|
||||
|
||||
**Latency** (`spike/latency-2026-09-27.txt`, from nh3-dev, network floor 31 ms):
|
||||
- `/decide` short: 71 ms end to end (38 ms server);
|
||||
- 3 rotations shared: 113 ms (79 ms);
|
||||
- 6 orderings: 137 ms (99 ms);
|
||||
- a ~2k-token state: 210 ms (169 ms).
|
||||
|
||||
**Prime ruled (2026-09-27 ~0250): build it in, and try the fast kernels too.**
|
||||
- Averaging design: opt-in `orderings: rotations|all` (all only ≤ 4 options); all orderings in one shared
|
||||
batch; per-ordering SemIf results returned unchanged; a `combined` block (log-mean, top, agreement,
|
||||
spread); orderings count toward `max_decisions`.
|
||||
- Fast kernels: `causal_conv1d` + `flash-linear-attention` are missing, so transformers runs Qwen3.5's
|
||||
reference PyTorch paths. Adopt them only if the authored144 parity re-check holds.
|
||||
|
||||
Tracked in the in-flight SemIf section. See [[2026-09-27-semif-live-on-fv-ml1-gpu1]].
|
||||
|
||||
**Outcome (0.1.3, `77b8cb4`, ~0340 PT):** built and deployed. Through the service on the same 252 rows,
|
||||
accuracy is 78.6% → 88.1% (95% CI +5.1..+14.3), and unanimous rows are 94.5% accurate. The fast kernels
|
||||
won the A/B on the empty GPU 3:
|
||||
- parity with upstream 144/144 (was 142/144);
|
||||
- a ~2k-token `/decide` 169 → 92 ms;
|
||||
- shared capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens (was 52/43/19/13).
|
||||
All of heid's bug-hunt findings were folded into the same release (thread `01M3H3F4RR7XBP90KQ3A39H4SX`).
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel
|
||||
|
||||
`[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel.** Every tool worked except `get_viewport_screenshot`, which had Blender write a file for the SERVER to read ("Screenshot file was not created"). The server now runs inside the Blender container over ssh+docker-exec stdio. That also means no port is published. → `stacks/blender/README.md`
|
||||
@@ -1,3 +0,0 @@
|
||||
# `uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project
|
||||
|
||||
`[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`).
|
||||
@@ -1,17 +0,0 @@
|
||||
# `[2026-09-27]` Zigbee2MQTT on esh-docker-vm (Prime go-ahead; ha-dev request)
|
||||
|
||||
(Moved out of the in-flight section at the 2026-10-06 snapshot. Canonical: `stacks/zigbee2mqtt/README.md`.)
|
||||
|
||||
- **LIVE since 1240:** Z2M 2.14.1 (digest-pinned) on `http://10.0.50.45:8099` (auth token), radio
|
||||
SLZB-MR1U chip 0 `tcp://10.0.90.10:6638` (ember), channel 25, **PAN 0xCFF4**. State and the network
|
||||
key are in `/opt/docker/data/zigbee2mqtt` (root 0700, restic via /opt/docker, never in git). Vault:
|
||||
`esh-docker-vm/zigbee2mqtt-{network-key,frontend-token,mqtt-password}`. Broker user `zigbee2mqtt`
|
||||
added to mosquitto (passwd backup `.bak-20260927-z2m`).
|
||||
- **ha-dev deleted the ZHA entry**; pairing the Aqara T1 is theirs. ⚠ The "bridge in HA" acceptance first
|
||||
FAILED because HA had had no MQTT since 2026-09-25 06:13. Cause: the esh-docker-vm macvlan shim had no
|
||||
host route to HA (two equal /24s, ens18 wins). Fixed at 1249 with a /32 via the shim (if-up.d hook,
|
||||
`playbooks/esh-docker-vm-macvlan-shim-route.yaml`), and HA reconnected. My first report called HA
|
||||
"connected" from a client count that included my own probe; `$SYS` counts are not identities.
|
||||
- ⚠ The first credential-mint attempt was blocked by the auto-mode classifier on a peer-relayed
|
||||
approval. It went ahead only on Prime's own go-ahead in this session. Treat that as correct.
|
||||
→ `stacks/zigbee2mqtt/README.md`
|
||||
@@ -1,3 +0,0 @@
|
||||
# Bonsai ternary spike on fv-ml1 GPU 3 (Prime via brokkr): at the 275 W cap, PQ2_0 is 1.93x Q4_K_XL at N=1 but only 1.06x at N=8 (PTQ1_0 1.24x), recovering to ~1.37x at N=16.
|
||||
|
||||
`[2026-09-28]` **Bonsai ternary spike on fv-ml1 GPU 3 (Prime via brokkr): at the 275 W cap, PQ2_0 is 1.93x Q4_K_XL at N=1 but only 1.06x at N=8 (PTQ1_0 1.24x), recovering to ~1.37x at N=16.** Positive control passed on tg128 (+1.1%). nvidia-smi's sw_power_cap flag never fires on this card, so "at cap" is judged from board draw. Follow-up (brokkr): moving PQ2_0 onto MMQ from batch 6, like Q4_K (mmvq.cu, `ne11 <= 5`), lifts N=8 to 1.21x, so the dip was partly a kernel threshold. The residual gap is not power. Runs are in `fv-ml1:/tank/spikes/bonsai-2026-09-28/runs{,-mmvq5}/` and on the Booth. The build image was removed; `Dockerfile.build` recreates it. **Acquired for keeps (Prime, 1540):** GGUF PQ2_0/PTQ1_0/mmproj-Q8_0 + the fork source pin in `/tank/aimodels/llm/prism-ml_Ternary-Bonsai-2-27B-gguf/` (runtime/), the MLX 2-bit pack in `/tank/aimodels/mlx/`, 34/34 hash-verified. Weights single copy (no snapshots, /tank not in restic); the fork pin also at `/mnt/smithy/runtime-pins/prism-llama.cpp-87268f77/` (brokkr).
|
||||
@@ -1,3 +0,0 @@
|
||||
# Worldtree U10 legacy-memory backfill: demo COMMITTED (Prime, direct go 2350).
|
||||
|
||||
`[2026-09-28]` **Worldtree U10 legacy-memory backfill: demo COMMITTED (Prime, direct go 2350).** Dry run then commit, both as `-u worldtree`: 5 filed (lofn 4, mimir 1), 16 `unslotted_pre_364` skipped, 0 aborted. **The first commit exited 1 because demo's hand-managed `model_roles.yaml` lacked the `memory_tagger` role**, so I synced it verbatim from the image's config-defaults (backup `.bak-pre-memory-tagger`). **2026-09-29 0854 presync for personal (Prime: "complete u10"):** personal got `memory_tagger` and the U9 `admin.memory.forget` policy delta. **Demo had also been missing that U9 policy delta since 09-24**, which left readonly-admin's `admin.*` rule with no exclusion for the destructive forget. Fixed, and demo's api was restarted to load it. **Personal COMMITTED 2026-09-29 1007** (b191 e2f74706, a snapshot-copy build; Prime's standing "complete u10" via worldtree-dev's go): 797 filed (mimir 795, lofn 2), mimir `dropped:too_large` 377, 1 unroutable held. The first personal dry run (b190) had crashed on 45 mimir rows with no vector (HNSW frozen since 08-06, #417, accepted by Prime until U11). Pinned (446e5807; Prime: "leave pinned") keeps its legacy chroma in the container's WRITABLE LAYER (`/app/agents/*/memory/.chroma`, 7 agents, 196K each): a recreate erases them.
|
||||
@@ -1,3 +0,0 @@
|
||||
# Worldtree U11a (legacy memory → read_only) is PREPPED, not flipped.
|
||||
|
||||
`[2026-09-29]` **Worldtree U11a (legacy memory → read_only) is PREPPED, not flipped.** A staged, inert `/opt/worldtree/config/defaults.yaml.staged-u11a-read_only` sits on corviduo-dev. Applying it needs Prime's go AND demo on 48bdf235 or later (demo ran 1b8746e6, which predates U11a). Apply = back up, copy over `defaults.yaml`, `docker restart`, then read the boot line `memory cutover: legacy plane read_only (writer on, reader on)`. **Once the window opens, infra-hermes runs one U8 batch per day and I own the wrapper and the escalations (proposed, worldtree-dev agreed):** on nh3-dev in `~/development/Worldtree`, at demo's deployed SHA, `python -m core.memory_acceptance.live --runs 3 --gate --legacy-mode read_only --label window`. The verdict is in `trace1.md`; FAIL goes to worldtree-dev the same day, never a retry; no git commit; never two at once. The wrapper is NOT built yet, deliberately: it waits for the go.
|
||||
@@ -1,3 +0,0 @@
|
||||
# Shorter Parakeet slices as Scriberr's memory fix
|
||||
|
||||
`[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix.** I predicted ~6× less memory from the attention-matrix arithmetic. Measured, 300 → 120 s only went 9,384 → 6,510 MiB: a ~5.6 GB fixed floor dominates. `expandable_segments:True` was the real lever (5,496). Measure the process peak; never extrapolate it from one tensor.
|
||||
@@ -21,3 +21,8 @@
|
||||
- A `caddy validate` of a file under a 0700 staging dir fails "permission denied" as user caddy; a `| tail` hid it once. Validate from `/etc/caddy/Caddyfile.check` and check the exit code (positive-controlled with a bogus directive → rc 1).
|
||||
- `secret` (Vaultwarden via `bw`) calls are serialized and can take minutes each when another session is reading the vault; run them in the background.
|
||||
- Printing `~/.hermes/config.yaml` sections exposed the High Seat's Bifrost dispatch key in the local transcript; Prime ruled no rotation (10-11) since the key leaves use with U13.
|
||||
|
||||
## Moved from the in-flight section on 2026-10-14 (verbatim)
|
||||
|
||||
- **Acceptance closed 1354:** Miranda's reply arrived on the thread through the bridge's own send. ⚠ RE-CORRECTED 2026-10-12: card `card:05cf119e604649838e2a4c642c161e26` (Miranda's carded `send`) WAS approved by `prime` at 2026-10-11 20:54:25Z (1354:25 PT), per the kernel DB (`/var/lib/svos/state/svos.db`, read-only). The 1412 "NOT approved" correction was itself wrong, because it came from inference, not a read. Nothing to deny; 0 cards pending. **Answered by svos-dev 1400:** the thread text was Miranda answering bus mail with her own `send` (carded) instead of letting the bridge carry her turn's reply; the fix (a bus-tag clause in `svos_miranda.turns`) SHIPPED in v0.0.10 (adc862e), deployed 1410–1411 (nh3-svos kernel 0.0.10 + UI 0.0.8; nh3-dev miranda 0.0.4, `/opt/svos/current → releases/adc862e`); bus check passed 1412 with no card. Its manifest still says 0.0.3 (hardcoded), so it did not re-register. Then a static-only release `34476a7` (interface UI 0.0.9, tree strands follow speech) at 1712: `/opt/svos/current → releases/34476a7` on nh3-svos, no restart, previous handle = adc862e; then `ad255a2` (UI 0.0.10, stronger strands, Prime found 0.0.9 too subtle) at 1719, previous = 34476a7; then `627a9fb` (UI 0.0.11: the top bar shows `v0.0.<n> · <short commit>`, from `web/build-id.txt` via git-archive export-subst, so ALWAYS build from a clean `git archive`; a plain copy reads `dev`) at 1722, previous = ad255a2; `61459d2` (UI 0.0.12: voice chorus/flanger toggle, lighter header) at 1739, previous = 627a9fb; `8fa229d` (UI 0.0.13: strands span the tree, deeper voice character) at 1800, previous = 61459d2. `89dad7a` (UI 0.0.14: the pane lease race, a frame loading before the attach reply now gets its lease) on 10-13 at 0857, previous = 8fa229d. `7f95e54` (UI 0.0.15: the terminal leaves the app list's Open; a bare embedded terminal can only ever wait for a lease, so it is attached per seat) at 0906:37, previous = 89dad7a. Static-only builds: `scratchpad svos-static-release.sh <prev> <new> <ui>` pattern (rebuild it from the 61459d2 kit if the scratchpad is gone). Static-only releases: build the full release dir (carried artifacts + new `static/`), switch `current` without restarting, regenerate the inventory. ⚠ The kernel + apps still RUN from `releases/adc862e` (verified via `/proc/<pid>/maps`): do not delete it before they restart. Same on nh3-dev: svos-terminal, svos-postoffice and svos-agent still run from `releases/dc241f8` (svos-miranda from `adc862e`). The non-autonomous `miranda → app:app-postoffice write` grant is intended (her own sends); making it autonomous is Prime's call, surfaced by svos-dev. Speech leg unconfirmed (svos-dev asked Prime; n=1 vs the earlier 15 s `synthesis_failed`). Phase R (24 h soak, doctrine + broadcast texts, High Seat removal) is highseat-dev's (svos `docs/u13-cutover-runbook.md`).
|
||||
- **Rollback material, keep until the soak ends:** nh3-svos `releases/b588510`, `instance.toml.pre-u13-{A,B}`, `/etc/svos/kernel.env.pre-u13` (absent: there was none); nh3-dev `~/.hermes/config.yaml.pre-u13` and `~/.hermes/plugins/svos_miranda`. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510` and nh3-dev `/opt/svos/releases/afab123` (unreferenced) can go.
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
# [2026-10-11] worldtree-dev answering again; #416 residual reopened
|
||||
|
||||
`[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# [2026-10-12] Decision seat: h2o tried, rolled back twice, then production on Prime's fix-forward
|
||||
|
||||
`[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). **Then:** the round 2 eval seat on GPU 3 (1428–1619, Prime 1420) found the cause. The judge's v1 note question asked two things at once: INC answered it inconsistently, h2o literally. The shim floor is ruled out (identical verdicts on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge v2) re-fitted the question. Both seats agree under v2 (work notes INC 0/22, h2o 1/22 at 0.52). **Prime ruled the cutover at 1640 ("do it", via albok-dev): h2o LIVE again since 1704** (`switch-to-h2o.sh` 170223, healthy 170454, probes clean). **The v2 walk on h2o (1705–1745) moved two margins past albok-dev's named line**: nonfiction adult 0 → 79 (the Schell chapter at 0.531 on the book profile) and memory personal 6 → 18. Notes held at 0 adult. **Rolled back to INC at 1748 on albok-dev's word.** The 1749–1751 re-stamp walk restored the counts (memory 0 adult / 6 personal, feedback 0 / 2, nonfiction 0 / 0), and the timer was back at 1751. **Then Prime, 1805 (via albok-dev): "go to h2o, and fix forward. do not fall back to inc."** albok-dev declared the two published nonfiction titles `visibility: public` (the chapter margin cannot recur). **h2o has been production since 1807** (`switch-to-h2o.sh` 180513, healthy 180740, all checks clean). The cutover walk (1809–1813, 0 degraded) left memory 0 adult / 18 personal, feedback 2 / 0, nonfiction 0 / 0 and fiction 0 / 0, and the timer is back. **The one open margin is the personal reading (18 vs INC's 6).** albok-dev is re-fitting it forward on the live seat in small batches. Threads `01M4S026GWS354WZGXZ5E34F31`, `01M4S91WPPPN0B17JBNX98K5CY`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# [2026-10-12] Shared CI runner slow = ospool fsync; fix A (nh3-docker fast runner) done
|
||||
|
||||
`[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime ruled A (2026-10-12 ~1715): an opt-in fast runner on nh3-docker.** **Done 1725** (`nh3-docker-runner`, label `nh3-docker`, no docker socket in jobs, a 4 GiB cap). vastblue's pytest went 14m17s → 4m45s; albok is green (run 11051). ⚠ albok's failures were the slow fsync, never a hang: my 10-12 "hangs" diagnosis was wrong. The ana-docker vCPU-type question (x86-64-v2-AES, no AVX2) is DROPPED: the measured bottleneck was disk fsync plus a hanging network test, not CPU, so nothing rides on it.
|
||||
@@ -0,0 +1,3 @@
|
||||
# [2026-10-12] U13 regression fixed (app panes 403)
|
||||
|
||||
`[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing.
|
||||
@@ -0,0 +1,3 @@
|
||||
# [2026-10-13] A171: Miranda's bus sends no longer card
|
||||
|
||||
`[2026-10-13]` **A171: Miranda's bus sends no longer card (Prime 1400, "give miranda more autonomy", via svos-dev).** On nh3-svos the `miranda → app:app-postoffice write` grant got `autonomous = true` (`playbooks/nh3-svos-a171-miranda-postoffice-autonomous.yaml`; backup `instance.toml.pre-a171`), and config_gen went `acae45206bbc9fcb → 9139783f413b45fa`. Revert = restore the backup and `systemctl reload svos-kernel`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# [2026-10-13] Albok note intake: gate 2 done (personal/inbox, /srv/nemi/inbox), gate 1 waits on Prime
|
||||
|
||||
`[2026-10-13]` **Albok note intake (Prime 1530: albok subsumes library/note intake; Muninn's watcher, muninn-gate and mimir-inbox retired). Gate 2 DONE 1735:** wing `personal/inbox` on nh3-albok, Nemi token re-minted `01M4VX207PJ09ZSECASCSZPCFE` (vaulted `albok/token-nemi`; the old `01M4HAP9…` revoked 1742 after albok-dev's swap; the capture source walks), and `/srv/nemi/inbox` on nh3-dev (`playbooks/nh3-dev-nemi-inbox.yaml`). **Gate 1 (ruling 6, Prime's Obsidian vault synced read-only onto nh3-dev) WAITS ON PRIME:** the vault's location, plus approval for Syncthing receive-only (recommended). Thread `01M4VWV5FEF64EP71W60N6Q1N0`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# [2026-10-13] app-terminal degraded since U13: its unit's PATH lacked ~/.cargo/bin
|
||||
|
||||
`[2026-10-13]` **app-terminal was degraded from the U13 cutover until 0856 on 10-13: its user unit's PATH lacked `~/.cargo/bin`, so every `zellij` call failed to launch** (rc 127). That broke /health AND every dump/inject. Fixed with `Environment=PATH=` in `configs/svos-apps/nh3-dev/svos-terminal.service` (3c92a12). The kernel logged app-terminal up at 15:56:40Z, the manifest was unchanged, and no card was raised; kernel health reads up 7 / degraded 0. The restart moved its process from release dc241f8 to adc862e (same svos_terminal 0.0.3). ⚠ Any new user unit that shells out to a cargo/uv/bun tool needs PATH set explicitly.
|
||||
@@ -0,0 +1,3 @@
|
||||
# [2026-10-13] muninn-gate + mimir-inbox STOPPED (retired)
|
||||
|
||||
`[2026-10-13]` **muninn-gate (corviduo-dev:8090) and mimir-inbox (:8091) STOPPED 2123** (Prime retired them at 1530; worldtree-dev: unit 6 needs the data at rest only, and up was worse because they kept accepting into an undrained queue). `compose down` only: images, compose dirs, config and `/mnt/muninn-staging/mimir-inbox` stay for albok's unit 6 (albok-dev owns the backup and teardown). Worldtree is a pure albok-service HTTP customer since beba08a4. Strip the retired config blocks (providers `library:`, `graph_store:`, defaults `kb_audit:`; NOT defaults `library:`) per instance only after it runs beba08a4+, demo first. Thread `01M4WA58P2M001K1420NN72WFM` (worldtree-dev).
|
||||
@@ -0,0 +1,3 @@
|
||||
# [2026-10-13] Prime ~2120: notes copy approved; fleet infra-ops key vaulted
|
||||
|
||||
`[2026-10-13]` **Prime ~2120: "you may copy the notes, put the key in the vault".** (1) The fleet infra-ops SSH private key is vaulted as `nh3-dev/.ssh/infra-ops_ed25519`; the restore was tested (same public key derived, login to esh-nas). (2) Gate 1 is approved for the copy of Prime's Obsidian notes onto nh3-dev, BUT the vault's location is unknown: no `.obsidian` dir on nh3-dev (bar oldAIPA's old KB) or on esh-nas's shares (depth 5). Asked Prime where. Plan: Syncthing receive-only onto nh3-dev, the root path to albok-dev.
|
||||
@@ -0,0 +1,3 @@
|
||||
# [2026-10-13] SVOS v0.0.11 (452a592, U15 task seats slice 1) deployed
|
||||
|
||||
`[2026-10-13]` **SVOS v0.0.11 (452a592, U15 task seats slice 1) deployed 2131–2133** after the 21:30 vzdump: schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7; nh3-dev current → 452a592, terminal 0.0.4 pinned (no card). The tasks artifact mapping uses host `nh3-svos` (= the terminal's artifact_host), not svos-dev's `nh3-dev`. Rollback and details: `servers/nh3-svos/README.md`. Open: Prime's `GET /tasks` check, and a `task_argv` for the claude runtime when he names it.
|
||||
+30
-48
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-12 ~1810 PT (decision seat: h2o production since 1807 on Prime's fix-forward ruling, after two rollbacks; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_
|
||||
_Last updated: 2026-10-14 (snapshot for clear, elway note updated: SVOS v0.0.11 live, pewpewstudio deployed, albok inbox live, decision seat on h2o; earlier: the h2o swap saga, U13.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -99,7 +99,10 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
- **`scripts/elway` sudo handling** — elway prompts for the sudo password
|
||||
ONCE via `getpass` before the first `sudo: true` step → can't run
|
||||
unattended from a non-TTY tool if any step needs sudo. Sudo-free
|
||||
playbooks run fully non-interactive over key SSH.
|
||||
playbooks run fully non-interactive over key SSH. — `[2026-10-13]` superseded for the
|
||||
infra-ops identity: run as `infra-ops@<ip>` (NOPASSWD sudo), elway's sudo steps run
|
||||
unattended with no prompt (every playbook run on 10-12/10-13). The getpass prompt only
|
||||
fires for an account that needs a sudo password, e.g. bootstrapping a host as lkraven.
|
||||
|
||||
- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
|
||||
on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
|
||||
@@ -113,19 +116,21 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
|
||||
_As of 2026-10-14 ~0005 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
|
||||
|
||||
### 2026-10-11: live now (as of 2026-10-11 ~1400 PT)
|
||||
### 2026-10-14: live now (as of 2026-10-14 ~0005 PT)
|
||||
|
||||
- **Nothing is mid-action.** No claims held. eshpfi is pushed through `df7161b`; this snapshot's commit follows. ⚠ The working tree and index are shared with infra-hermes and subagents: commit with `git commit -- <paths>`. `graphify-out/GRAPH_REPORT.md` stays modified on purpose.
|
||||
- **U13 cutover: executed 10-11 0721–0816, acceptance 1350.** Miranda now runs in SVOS: app-miranda + svos-agent on nh3-dev, the Hermes gateway on plugin `svos`, and the bridge app-postoffice on nh3-dev owning the `miranda` push route. The URGENT bus test (thread `01M4PBNX5PP0PGHDTY5BG7HG95`) ingested, paged per the connector receipt, and a reply came back on the thread from `svos`. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`
|
||||
- **Acceptance closed 1354:** Miranda's reply arrived on the thread through the bridge's own send. ⚠ RE-CORRECTED 2026-10-12: card `card:05cf119e604649838e2a4c642c161e26` (Miranda's carded `send`) WAS approved by `prime` at 2026-10-11 20:54:25Z (1354:25 PT), per the kernel DB (`/var/lib/svos/state/svos.db`, read-only). The 1412 "NOT approved" correction was itself wrong, because it came from inference, not a read. Nothing to deny; 0 cards pending. **Answered by svos-dev 1400:** the thread text was Miranda answering bus mail with her own `send` (carded) instead of letting the bridge carry her turn's reply; the fix (a bus-tag clause in `svos_miranda.turns`) SHIPPED in v0.0.10 (adc862e), deployed 1410–1411 (nh3-svos kernel 0.0.10 + UI 0.0.8; nh3-dev miranda 0.0.4, `/opt/svos/current → releases/adc862e`); bus check passed 1412 with no card. Its manifest still says 0.0.3 (hardcoded), so it did not re-register. Then a static-only release `34476a7` (interface UI 0.0.9, tree strands follow speech) at 1712: `/opt/svos/current → releases/34476a7` on nh3-svos, no restart, previous handle = adc862e; then `ad255a2` (UI 0.0.10, stronger strands, Prime found 0.0.9 too subtle) at 1719, previous = 34476a7; then `627a9fb` (UI 0.0.11: the top bar shows `v0.0.<n> · <short commit>`, from `web/build-id.txt` via git-archive export-subst, so ALWAYS build from a clean `git archive`; a plain copy reads `dev`) at 1722, previous = ad255a2; `61459d2` (UI 0.0.12: voice chorus/flanger toggle, lighter header) at 1739, previous = 627a9fb; `8fa229d` (UI 0.0.13: strands span the tree, deeper voice character) at 1800, previous = 61459d2. `89dad7a` (UI 0.0.14: the pane lease race, a frame loading before the attach reply now gets its lease) on 10-13 at 0857, previous = 8fa229d. `7f95e54` (UI 0.0.15: the terminal leaves the app list's Open; a bare embedded terminal can only ever wait for a lease, so it is attached per seat) at 0906:37, previous = 89dad7a. Static-only builds: `scratchpad svos-static-release.sh <prev> <new> <ui>` pattern (rebuild it from the 61459d2 kit if the scratchpad is gone). Static-only releases: build the full release dir (carried artifacts + new `static/`), switch `current` without restarting, regenerate the inventory. ⚠ The kernel + apps still RUN from `releases/adc862e` (verified via `/proc/<pid>/maps`): do not delete it before they restart. Same on nh3-dev: svos-terminal, svos-postoffice and svos-agent still run from `releases/dc241f8` (svos-miranda from `adc862e`). The non-autonomous `miranda → app:app-postoffice write` grant is intended (her own sends); making it autonomous is Prime's call, surfaced by svos-dev. Speech leg unconfirmed (svos-dev asked Prime; n=1 vs the earlier 15 s `synthesis_failed`). Phase R (24 h soak, doctrine + broadcast texts, High Seat removal) is highseat-dev's (svos `docs/u13-cutover-runbook.md`).
|
||||
- **Rollback material, keep until the soak ends:** nh3-svos `releases/b588510`, `instance.toml.pre-u13-{A,B}`, `/etc/svos/kernel.env.pre-u13` (absent: there was none); nh3-dev `~/.hermes/config.yaml.pre-u13` and `~/.hermes/plugins/svos_miranda`. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510` and nh3-dev `/opt/svos/releases/afab123` (unreferenced) can go.
|
||||
- **Nothing is mid-action.** No ops-log claims held. eshpfi `main` is pushed through `d1192ca` (infra-ops has STANDING push on main since 2026-10-13, CLAUDE.md § Working rules); this snapshot's commit follows and is pushed. ⚠ The working tree and index are shared with infra-hermes, albok-dev and subagents: commit with `git commit -- <paths>` and read `git log origin/main..main` before pushing. `graphify-out/GRAPH_REPORT.md` stays modified on purpose.
|
||||
- **SVOS: main `33a7077` (v0.0.13+2: kernel 0.0.13, interface 0.0.5 with briefs, postoffice 0.0.3, UI 0.0.18) since 2026-10-14 1108–1111; config_gen `33f001f4e2dbc7dc`, apps up 7.** ⚠ Lesson (paid 10-14): a manifest revision whose verbs change is HELD for the owner, and the app reads DOWN (`refused wrong-revision`) until Prime approves its card (15-min expiry). Roll such apps only while he is at the board, and the post-office bridge last (held, it carries no Miranda mail). The interface unit needs the drop-in `svos-interface.service.d/20-artifact-host.conf` (`SVOS_ARTIFACT_HOST=nh3-svos`), and the briefs mapping host must equal it. Before that, kernel 0.0.12 (`d95078c`, 0416) on top of v0.0.11 (452a592, U15 task seats slice 1). The `hermes` runtime is the only task-capable one (one-off `hermes -z`, `process = "python3"`, no `--yolo`: Prime's call); its old argv is in `instance.toml.pre-hermes-task`. Earlier: v0.0.11 is live on nh3-svos (schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7) and nh3-dev (`/opt/svos/current → releases/452a592`; svos-terminal 0.0.4 runs from it). svos-dev accepted it. The task command is ruled with svos (Prime, 2026-10-14): infra-ops edits runtimes only at svos-dev's ask. `GET /tasks` as Prime is his acceptance check. Static-only interface releases since: UI 0.0.14 `89dad7a`, 0.0.15 `7f95e54` (the web build carried into 452a592). ⚠ Process→release map: nh3-svos kernel + local apps run `33a7077`; on nh3-dev svos-postoffice runs `33a7077`, svos-terminal `452a592`; svos-miranda runs `adc862e`, svos-agent still runs `dc241f8` (check `/proc/<pid>/maps` before deleting any release dir). Rollback material: nh3-svos `/var/lib/svos/backup/pre-452a592/` (schema-12 DB + config tar), `instance.toml.pre-452a592`, `instance.toml.pre-a171`; nh3-dev `terminal.toml.pre-452a592`. U13 history and the static-release chain: → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`. Phase R (24 h soak, High Seat removal) is highseat-dev's. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510`, `afab123`, `adc862e` and nh3-dev `/opt/svos/releases/afab123` can go once no process maps them.
|
||||
- **Decision seat = h2o-lightning-4b (production, Prime 1805 10-13: fix forward, no INC fallback).** The one open margin is the judge's personal-question reading (h2o 18 vs INC 6 on fleet/memory); albok-dev re-fits it forward in small batches on the live seat.
|
||||
- **albok note intake (Prime 1530 10-13):** gate 2 DONE (wing `personal/inbox`, `/srv/nemi/inbox` on nh3-dev, nemi token `01M4VX207PJ09ZSECASCSZPCFE`). Gate 1 (ruling 6) DONE 2026-10-15 0140: Prime's notes are `/home/lkraven/kb` = gitea `vh/VuongKB` (he syncs them with git, not Syncthing), so Nemi's copy is a fresh clone at `/srv/nemi/vault`, pulled hourly at :50 by the lkraven user timer `nemi-vault-pull` (`configs/nemi-vault/`). It holds 129 md outside dot-dirs; the untracked Muninn output (`main/`, `fiction/`) is not in it. **Walked 0143–0147 10-15:** 232 objects (129 notes + 103 children), 2 judged personal, 0 adult. The nemi token was re-minted again (`01M4ZBJG3DMRF9Q9XEVZ8Q20GH`, + personal/notes) and the old one revoked (thread `01M4VWV5FEF64EP71W60N6Q1N0`). Prime says the kb needs cleaning up (its last commit is 2026-04-16).
|
||||
- **Worldtree:** demo was deploying `beba08a4` (albok 4.7 the rest; Worldtree is a pure albok-service HTTP customer) at 2120 10-13. Retired-config strip: **demo DONE 2026-10-15 2339 on Prime's go** (instance-configs `2240110`: providers `graph_store` + `library`, plus the b193-retired `memory.*` keys; the boot shows 0 retired-key warnings, down from 7). Personal and pinned follow ONLY after each runs beba08a4+ (pinned has no bind-mounted config). Keep defaults `library:` (thread `01M4WA58P2M001K1420NN72WFM`). Daily memory gate: 10-13 re-run `20261013T160414Z` PASS; worldtree-dev owns the tagger-latch fix. muninn-gate + mimir-inbox are STOPPED (their data at rest kept for albok unit 6).
|
||||
- **satellite-1:** its own key is authorized for infra-ops on esh-nas only (`playbooks/files/infra-ops-satellite-keys`). Waiting on Prime's login test from the satellite and its machine name (the key's comment came through as bare `infra-ops@`).
|
||||
- **fv-ml1: hard hang 10-11 ~0005, self-reset, no record** (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit `fv-ml1-sol-logger` (`journalctl --user -u fv-ml1-sol-logger`). ⚠ **HOLD** `zroot/ROOT/debian@pre-patch-20261008` (do not destroy) until the cause is known or a clean week passes after the last crash.
|
||||
- **albok `.browse/` share:** the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in `~vh/albok-browse-test/` on nh3-albok, waits on Prime pasting the one-liner (`scripts/oneoff/albok-browse-withdrawal/README.md`); afterwards read `results/`, report to albok-dev, remove the replica.
|
||||
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key.
|
||||
- **Waiting on Prime:** rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
|
||||
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%).
|
||||
- **Waiting on Prime:** where his Obsidian vault lives (gate 1); satellite-1's login test + name; rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
|
||||
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). The ana-docker gitea-runner's Homepage name changes to its runner name at its next recreate (stacks/gitea-runner/compose.yaml). nh3-dev's `/opt/svos/releases/` and the h2o eval seat on fv-ml1 GPU 3 (`stacks/h2o-eval`, down) are kept as artifacts.
|
||||
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
|
||||
- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
|
||||
|
||||
@@ -180,12 +185,20 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-13]` **app-terminal was degraded from the U13 cutover until 0856 on 10-13: its user unit's PATH lacked `~/.cargo/bin`, so every `zellij` call failed to launch** (rc 127). That broke /health AND every dump/inject. Fixed with `Environment=PATH=` in `configs/svos-apps/nh3-dev/svos-terminal.service` (3c92a12). The kernel logged app-terminal up at 15:56:40Z, the manifest was unchanged, and no card was raised; kernel health reads up 7 / degraded 0. The restart moved its process from release dc241f8 to adc862e (same svos_terminal 0.0.3). ⚠ Any new user unit that shells out to a cargo/uv/bun tool needs PATH set explicitly.
|
||||
- `[2026-10-13]` **pewpewstudio LIVE on nh3-docker :8185** (Prime via pewpew-dev): api + Caddy front; the Host allowlist must sit in a `route` (foreign Host 400); DB, assets and shop config on named volumes. → `stacks/pewpewstudio/README.md`
|
||||
- `[2026-10-13]` **infra-ops on esh-nas (CT 103) + per-host satellite keys (Prime)** — bootstrapped via pct from esh-pve-nas; satellite-1 authorized on esh-nas only (`playbooks/files/infra-ops-satellite-keys`); its login test + name pending.
|
||||
- `[2026-10-13]` **Worldtree memory gate: 0730 harness error (the tagger's MalformedClassification latched the writer), re-run `20261013T160414Z` PASS (median 0.83)** — the latch fix is worldtree-dev's (thread `01M4V035SF73XKVG3NRVX37XQ6`).
|
||||
- [2026-10-13] **SVOS v0.0.11 (452a592, U15 task seats slice 1) deployed** — schema 13, terminal 0.0.4 pinned without a card, tasks mapping host nh3-svos. → `persistent-memory.d/2026-10-13-svos-v0-0-11-deployed.md`
|
||||
- [2026-10-13] **muninn-gate + mimir-inbox STOPPED (retired)** — data at rest kept for albok unit 6; Worldtree retired-config strip order. → `persistent-memory.d/2026-10-13-muninn-gate-mimir-inbox-stopped.md`
|
||||
- [2026-10-13] **Prime ~2120: notes copy approved; fleet infra-ops key vaulted** — restore tested; vault location still unknown. → `persistent-memory.d/2026-10-13-notes-copy-approved-fleet-key-vaulted.md`
|
||||
- [2026-10-13] **Albok note intake: gate 2 done (personal/inbox, /srv/nemi/inbox), gate 1 waits on Prime** — wing, token, ACL trap. → `persistent-memory.d/2026-10-13-albok-note-intake-gate-2.md`
|
||||
- [2026-10-13] **A171: Miranda's bus sends no longer card** — one grant made autonomous; config_gen and revert. → `persistent-memory.d/2026-10-13-a171-miranda-postoffice-autonomous.md`
|
||||
- [2026-10-13] **app-terminal degraded since U13: its unit's PATH lacked ~/.cargo/bin** — zellij unreachable; dump/inject broken too. → `persistent-memory.d/2026-10-13-app-terminal-path-fix.md`
|
||||
- `[2026-10-12]` **Prime's rulings ~1715:** CI fix A (fast runner on nh3-docker); push eshpfi main (done, `3aa715c..6a01da7`); deny old card `05cf119e` (moot: the kernel DB shows Prime approved it 10-11 1354:25, so there is nothing to deny); **no rotation of `albok/judge-token`** (it was printed only into a local transcript). The ana-docker vCPU-type change is dropped.
|
||||
- `[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). **Then:** the round 2 eval seat on GPU 3 (1428–1619, Prime 1420) found the cause. The judge's v1 note question asked two things at once: INC answered it inconsistently, h2o literally. The shim floor is ruled out (identical verdicts on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge v2) re-fitted the question. Both seats agree under v2 (work notes INC 0/22, h2o 1/22 at 0.52). **Prime ruled the cutover at 1640 ("do it", via albok-dev): h2o LIVE again since 1704** (`switch-to-h2o.sh` 170223, healthy 170454, probes clean). **The v2 walk on h2o (1705–1745) moved two margins past albok-dev's named line**: nonfiction adult 0 → 79 (the Schell chapter at 0.531 on the book profile) and memory personal 6 → 18. Notes held at 0 adult. **Rolled back to INC at 1748 on albok-dev's word.** The 1749–1751 re-stamp walk restored the counts (memory 0 adult / 6 personal, feedback 0 / 2, nonfiction 0 / 0), and the timer was back at 1751. **Then Prime, 1805 (via albok-dev): "go to h2o, and fix forward. do not fall back to inc."** albok-dev declared the two published nonfiction titles `visibility: public` (the chapter margin cannot recur). **h2o has been production since 1807** (`switch-to-h2o.sh` 180513, healthy 180740, all checks clean). The cutover walk (1809–1813, 0 degraded) left memory 0 adult / 18 personal, feedback 2 / 0, nonfiction 0 / 0 and fiction 0 / 0, and the timer is back. **The one open margin is the personal reading (18 vs INC's 6).** albok-dev is re-fitting it forward on the live seat in small batches. Threads `01M4S026GWS354WZGXZ5E34F31`, `01M4S91WPPPN0B17JBNX98K5CY`.
|
||||
- `[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing.
|
||||
- `[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime ruled A (2026-10-12 ~1715): an opt-in fast runner on nh3-docker.** **Done 1725** (`nh3-docker-runner`, label `nh3-docker`, no docker socket in jobs, a 4 GiB cap). vastblue's pytest went 14m17s → 4m45s; albok is green (run 11051). ⚠ albok's failures were the slow fsync, never a hang: my 10-12 "hangs" diagnosis was wrong. The ana-docker vCPU-type question (x86-64-v2-AES, no AVX2) is DROPPED: the measured bottleneck was disk fsync plus a hanging network test, not CPU, so nothing rides on it.
|
||||
- `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`.
|
||||
- [2026-10-12] **Decision seat: h2o tried, rolled back twice, then production on Prime's fix-forward** — the judge's v1 question, the margins, the lessons. → `persistent-memory.d/2026-10-12-decision-seat-h2o-rollbacks-and-cutover.md`
|
||||
- [2026-10-12] **U13 regression fixed (app panes 403)** — base_url back to the terminal origin, seat routes allowed there. → `persistent-memory.d/2026-10-12-u13-regression-fixed.md`
|
||||
- [2026-10-12] **Shared CI runner slow = ospool fsync; fix A (nh3-docker fast runner) done** — measurements, the corrected albok diagnosis. → `persistent-memory.d/2026-10-12-shared-ci-runner-slow-fsync.md`
|
||||
- [2026-10-11] **worldtree-dev answering again; #416 residual reopened** — four threads answered at once. → `persistent-memory.d/2026-10-11-worldtree-dev-reading-mail.md`
|
||||
- `[2026-10-11]` **U13 cutover executed on Prime's go (0721) + his four rulings** — E1 0733, E2 steps 1–6 0734–0816, URGENT acceptance test 1350; fv-ml1 evidence capture, stale snapshots, Mac keys, no key rotation. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`
|
||||
- `[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset, no panic record; pre-patch snapshot HELD** — and infra-ops' soft-off that landed mid-boot. → `persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md`
|
||||
- `[2026-10-10]` **Prime's U13 rulings: roles dev+ops, artifact label KEPT nh3-svos (whole app dirs copied at E2), relay `matrix-relay`.** → `persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md`
|
||||
@@ -264,25 +277,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
- `[2026-09-30]` **Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30.** → `persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md`
|
||||
- `[2026-09-30]` **SemIf replaced by intern-decision (Intern-Decision-4B, the Jev bench pick): semif-compatible plus Jev `/v1/systemone` at 32k tokens on GPU 1, with a Triton warm-up cache volume.** → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`
|
||||
- `[2026-09-30]` **Scriberr moved to GPU 3 (on demand); our build carries the overlap slicer (0001) and the Parakeet gap retry (0002); v3 kept.** → `persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md`
|
||||
- `[2026-09-29]` **Worldtree U11a prepped, not flipped: a staged demo config plus the agreed U8 window plan (infra-hermes runs the batches).** → `persistent-memory.d/2026-09-29-worldtree-u11a-prepped.md`
|
||||
- `[2026-09-28]` **Worldtree U10 backfill done on demo (5) and personal (797). model_roles drift needed a memory_tagger sync first; mimir had missing vectors.** → `persistent-memory.d/2026-09-28-worldtree-u10-backfill.md`
|
||||
- `[2026-09-28]` **Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired.** → `persistent-memory.d/2026-09-28-bonsai-ternary-spike.md`
|
||||
- `[2026-09-28]` **blender-run gained `--cpu` (no GPU attached) and a fixed hostname `fv-ml1-blender` (draupnir).** The design stage never renders, so it stays off GPU 3.
|
||||
- `[2026-09-28]` **Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (`--extensions`).** SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → `stacks/blender/README.md` § Extensions
|
||||
- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** → `persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md`
|
||||
- `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
|
||||
- `[2026-09-27]` **Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port.** → `persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md` (canonical: `stacks/blender/README.md`)
|
||||
- `[2026-09-27]` **Worldtree reward path: d8726a13 + `providers.yaml` `reward_models.scalar-judge` live on demo; personal waited on a newer image.** → `persistent-memory.d/2026-09-27-worldtree-reward-path-live.md`
|
||||
- `[2026-09-27]` **esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August).** → `servers/esh-docker-vm/README.md`
|
||||
- `[2026-09-27]` **Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only.** Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → `persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md` (canonical: `stacks/zigbee2mqtt/README.md`)
|
||||
- `[2026-09-27]` **semif-serve 0.1.4: object states ending in `)`, `;` or `}` no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart.** Prime ruled; heid bug hunt folded. → `stacks/semif/README.md`
|
||||
- `[2026-09-27]` **SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win.** Build nothing (Prime). Henge 88 carries it. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
|
||||
- `[2026-09-27]` **SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21.** Recommendations await Prime. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
|
||||
- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` — **DONE:** 0.1.3 live, with fast kernels adopted (`77b8cb4`).
|
||||
- `[2026-09-27]` **restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime).** Rotation stays Prime's. → `persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md`
|
||||
- `[2026-09-27]` **infra-ops bootstrapped on vm-esh-nas by Prime** (the first account at the fleet-pinned uid/gid 850; `bootstrap-infra-ops-user.yaml` now pins 850 when free, `d775a01`).
|
||||
- `[2026-09-27]` **augaman: second instance on fv-ml1 benched, then REMOVED (Prime).** esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (`ad484c3`, bench `docs/pfi/augaman-speed-bench/`).
|
||||
- `[2026-09-27]` Created empty private repo `corviduo/svoperatingsystem` (SVOS, the operating system) as claude-bot, at brokkr-smithy-dev's relay of Prime's ruling. `svos-dev` → the OS session; The High Seat's session is `highseat-dev` again.
|
||||
- `[2026-09-26]` **augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy.** → `persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md` — `[2026-09-27]` **DONE:** deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (`d8f59a1`).
|
||||
- `[2026-09-26]` **`zellij-fleet@.service` installed, NOT enabled** (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling `@Claude` at boot is Prime's call when seat_up ships. Tracked: `0ad7799`, `services/zellij-fleet/README.md`.
|
||||
- `[2026-09-26]` **btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority).** Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: `servers/nh3-pve/README.md`.
|
||||
@@ -306,7 +301,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
|
||||
|
||||
_240 older entries archived to archival-memory.md._
|
||||
_258 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
@@ -322,18 +317,5 @@ _240 older entries archived to archival-memory.md._
|
||||
- `[2026-10-04]` **`apt-get` run in-band through `qm guest exec`** — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a `systemd-run` unit. (auto-memory `feedback_no_inband_apt_through_guest_agent`)
|
||||
- `[2026-10-04]` **Pulling files with `ssh host 'sudo cat …'`** — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and `log_output` re-recorded the whole file as a new session. Read without sudo (group-readable file).
|
||||
- `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** → `persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md`
|
||||
- `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix** → `persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md`
|
||||
- `[2026-09-30]` **Whole-file local-attention Parakeet in Scriberr (context 255/255).** OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
|
||||
- `[2026-09-30]` **Start-time midpoint stitching of overlapped Parakeet chunks.** It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
|
||||
- `[2026-09-30]` **int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime.** The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
|
||||
- `[2026-09-30]` **GPU budgets computed as total − used.** nvidia-smi `Free` is ~640 MiB lower per card (driver reserve). Budget from `Free`.
|
||||
- `[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel** → `persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md`
|
||||
- `[2026-09-27]` **A TCP connect as the "is Blender ready" probe** → `persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md`
|
||||
- `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`).
|
||||
- `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
|
||||
- `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** → `persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md`
|
||||
- `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`.
|
||||
- `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.
|
||||
- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md`
|
||||
|
||||
_128 older entries archived to archival-memory.md._
|
||||
_141 older entries archived to archival-memory.md._
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
# nh3-albok: the personal/inbox wing (unit 4.4 slice C's capture target; albok-dev 2026-10-13, after Prime's 1530 ruling
|
||||
# that albok subsumes library/note intake). Like personal/notes: building personal, default_group albok-personal (1511),
|
||||
# not service-restricted. Run between two Nemi walks (albok-nemi.timer on nh3-dev fires hourly at ~:01).
|
||||
# scripts/elway infra-ops@10.100.50.44 --playbook playbooks/albok-personal-inbox.yaml
|
||||
steps:
|
||||
- name: Back up the live config
|
||||
sudo: true
|
||||
shell: cp -p /srv/albok/etc/albok.yaml /srv/albok/etc/albok.yaml.pre-inbox
|
||||
creates: /srv/albok/etc/albok.yaml.pre-inbox
|
||||
- name: Add wing personal/inbox after personal/notes
|
||||
sudo: true
|
||||
shell: |
|
||||
python3 - <<'PY'
|
||||
import os, grp
|
||||
p = "/srv/albok/etc/albok.yaml"; s = open(p).read()
|
||||
if " - id: inbox" in s:
|
||||
print("already present"); raise SystemExit(0)
|
||||
old = " wings:\n - id: notes\n"
|
||||
assert s.count(old) == 1, s.count(old)
|
||||
s = s.replace(old, old + " - id: inbox # unit 4.4 capture wing (Nemi's drop-box /srv/nemi/inbox on nh3-dev); like notes (2026-10-13)\n")
|
||||
t = p + ".tmp"
|
||||
fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o640)
|
||||
with os.fdopen(fd, "w") as f: f.write(s)
|
||||
os.chown(t, 0, grp.getgrnam("albok").gr_gid); os.replace(t, p); print("edited")
|
||||
PY
|
||||
- name: Restart the service (config is read at start) and wait for health
|
||||
sudo: true
|
||||
shell: |
|
||||
docker restart albok-service >/dev/null
|
||||
for i in $(seq 1 60); do
|
||||
h=$(curl -s -m 3 http://127.0.0.1:8392/health) && echo "$h" | grep -q '"status":"ok"' && break; sleep 2
|
||||
done
|
||||
echo "$h" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('status'), d.get('wings'), d.get('leaks'))"
|
||||
verify:
|
||||
- name: Seven wings ready, none degraded
|
||||
shell: curl -s -m 5 http://127.0.0.1:8392/health | python3 -c "import json,sys; w=json.load(sys.stdin)['wings']; print(w); assert w['ready']==7 and w['degraded']==0 and w['locked']==0"
|
||||
changed_when: "false"
|
||||
- name: The wing's store dir carries the personal group
|
||||
sudo: true
|
||||
shell: stat -c '%A %U:%G %n' /srv/albok/store/personal/inbox* | head -3
|
||||
changed_when: "false"
|
||||
@@ -7,7 +7,8 @@
|
||||
# WHY a dedicated identity (not the agent borrowing lkraven/vh sudo):
|
||||
# - Clean audit trail — every agent action attributable to `infra-ops`,
|
||||
# separate from human accounts, with sudo I/O logging on.
|
||||
# - Revocation = pull one authorized_keys line + one sudoers file.
|
||||
# - Revocation = pull one authorized_keys line + one sudoers file. Satellite agents carry their own keys,
|
||||
# listed per host in playbooks/files/infra-ops-satellite-keys; revoke one by deleting its line and re-running.
|
||||
# - No human password ever in the agent's reach (key-gated NOPASSWD).
|
||||
#
|
||||
# RUN (operator, with YOUR sudo — infra-ops doesn't exist yet, so you bootstrap
|
||||
@@ -25,7 +26,8 @@
|
||||
# esh-pve-nas) DO get infra-ops — done that day.
|
||||
# PVE ships without sudo: `apt-get install sudo` first.
|
||||
# Also granted by operator request: esh-docker-vm,
|
||||
# esh-vm-db (2026-08-24) and vm-esh-nas (2026-09-27).
|
||||
# esh-vm-db (2026-08-24), vm-esh-nas (2026-09-27) and esh-nas
|
||||
# (CT 103, 2026-10-13, via pct from its host: playbooks/esh-nas-infra-ops.yaml).
|
||||
# - nh3-nas (Synology DSM) (no standard useradd / sudoers.d)
|
||||
#
|
||||
# Idempotent: re-running reconciles the key + sudoers without error.
|
||||
@@ -63,12 +65,23 @@ steps:
|
||||
sudo: true
|
||||
shell: install -d -m 700 -o {{ ops_user }} -g {{ ops_user }} /home/{{ ops_user }}/.ssh
|
||||
|
||||
- name: Install infra-ops authorized_keys (exact line, mode 600)
|
||||
# Satellite agents get their OWN keys, per host, from playbooks/files/infra-ops-satellite-keys (Prime 2026-10-13).
|
||||
- name: Stage the satellite key list
|
||||
upload:
|
||||
src: playbooks/files/infra-ops-satellite-keys
|
||||
dest: /tmp/infra-ops-satellite-keys
|
||||
mode: "0644"
|
||||
|
||||
- name: Install infra-ops authorized_keys (the fleet key + this host's satellite keys, exactly; mode 600)
|
||||
sudo: true
|
||||
shell: |
|
||||
line='{{ ops_pubkey }}'
|
||||
[ -n '{{ ssh_from }}' ] && line='from="{{ ssh_from }}" {{ ops_pubkey }}'
|
||||
printf '%s\n' "$line" > /home/{{ ops_user }}/.ssh/authorized_keys
|
||||
{ printf '%s\n' "$line"
|
||||
awk -v h="$(hostname -s)" '!/^#/ && $1==h { $1=""; sub(/^ /,""); print }' /tmp/infra-ops-satellite-keys
|
||||
} > /home/{{ ops_user }}/.ssh/authorized_keys
|
||||
rm -f /tmp/infra-ops-satellite-keys
|
||||
echo "satellite keys for $(hostname -s): $(($(wc -l < /home/{{ ops_user }}/.ssh/authorized_keys) - 1))"
|
||||
chown {{ ops_user }}:{{ ops_user }} /home/{{ ops_user }}/.ssh/authorized_keys
|
||||
chmod 600 /home/{{ ops_user }}/.ssh/authorized_keys
|
||||
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# Bootstrap the infra-ops identity INSIDE esh-nas (CT 103 on esh-pve-nas, 10.0.50.50), Prime 2026-10-13 ("set it up").
|
||||
# infra-ops could not log into the container (no user), so this runs on the HOST and reaches in with pct.
|
||||
# scripts/elway infra-ops@10.0.50.55 --playbook playbooks/esh-nas-infra-ops.yaml
|
||||
# The in-container steps are scripts/oneoff/esh-nas-infra-ops/bootstrap-in-ct.sh (= playbooks/bootstrap-infra-ops-user.yaml).
|
||||
vars:
|
||||
ctid: "103"
|
||||
steps:
|
||||
- name: Stage the bootstrap script on the host
|
||||
upload:
|
||||
src: scripts/oneoff/esh-nas-infra-ops/bootstrap-in-ct.sh
|
||||
dest: /tmp/esh-nas-bootstrap-infra-ops.sh
|
||||
mode: "0700"
|
||||
- name: Push it into CT {{ ctid }} and run it as root there
|
||||
sudo: true
|
||||
shell: pct push {{ ctid }} /tmp/esh-nas-bootstrap-infra-ops.sh /root/bootstrap-infra-ops.sh --perms 0700 && pct exec {{ ctid }} -- bash /root/bootstrap-infra-ops.sh
|
||||
- name: Remove the staged copies
|
||||
sudo: true
|
||||
shell: pct exec {{ ctid }} -- rm -f /root/bootstrap-infra-ops.sh && rm -f /tmp/esh-nas-bootstrap-infra-ops.sh
|
||||
verify:
|
||||
- name: The user and its sudoers file exist in CT {{ ctid }}
|
||||
sudo: true
|
||||
shell: pct exec {{ ctid }} -- sh -c 'id infra-ops && visudo -cf /etc/sudoers.d/infra-ops && grep -q NOPASSWD:ALL /etc/sudoers.d/infra-ops'
|
||||
changed_when: "false"
|
||||
@@ -0,0 +1,9 @@
|
||||
# Satellite agents' own keys for the infra-ops identity, scoped PER HOST (Prime 2026-10-13).
|
||||
# Format: <target's `hostname -s`> <public key line>. playbooks/bootstrap-infra-ops-user.yaml writes the fleet key
|
||||
# plus every line here whose first field matches the target's short hostname. Nothing else survives, so a key
|
||||
# that is not listed here is removed on the next run. Revoke a satellite by deleting its line(s) and re-running
|
||||
# the playbook on those hosts. Public keys only: never put a private key in this repo.
|
||||
#
|
||||
# satellite-1: generated by Prime on his satellite machine 2026-10-13, fingerprint
|
||||
# SHA256:KEFUjceFjuIZDuVbEq3FWYHm0rzdqx9hC+wFrA5h3xc (its own comment was "infra-ops@", with no hostname).
|
||||
esh-nas ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL0XLLZVTJUTgrWbMAx5exF3UDqPdi/30l/1lRo6d4bt infra-ops@satellite-1
|
||||
@@ -0,0 +1,38 @@
|
||||
# nh3-dev: /srv/nemi/inbox, the capture root Nemi walks into albok's personal/inbox wing (unit 4.4 slice C; albok-dev
|
||||
# 2026-10-13 after Prime's 1530 intake ruling). People and agents drop markdown here; Nemi ingests each file once
|
||||
# (capture mode never tombstones on absence, so a walked file may be moved away).
|
||||
# scripts/elway infra-ops@10.100.10.50 --playbook playbooks/nh3-dev-nemi-inbox.yaml
|
||||
# Writers: group nemi-drop (lkraven, infra-ops). Nemi runs as lkraven from the user manager, whose groups were fixed
|
||||
# at its start, so lkraven's access does not depend on the new group: the dir is lkraven-owned and a DEFAULT ACL gives
|
||||
# lkraven and nemi-drop rw on everything created inside, whoever drops it and whatever their umask.
|
||||
steps:
|
||||
- name: Group nemi-drop with lkraven and infra-ops
|
||||
sudo: true
|
||||
shell: |
|
||||
getent group nemi-drop >/dev/null || groupadd nemi-drop
|
||||
usermod -aG nemi-drop lkraven; usermod -aG nemi-drop infra-ops
|
||||
getent group nemi-drop
|
||||
- name: /srv/nemi (root:nemi-drop 0750) and /srv/nemi/inbox (lkraven:nemi-drop 2770, default ACL)
|
||||
sudo: true
|
||||
shell: |
|
||||
install -d -o root -g nemi-drop -m 0750 /srv/nemi
|
||||
setfacl -m u:lkraven:rx /srv/nemi # traversal for lkraven's user manager, whose groups predate nemi-drop
|
||||
install -d -o lkraven -g nemi-drop -m 2770 /srv/nemi/inbox
|
||||
setfacl -m u:lkraven:rwx,g:nemi-drop:rwx /srv/nemi/inbox
|
||||
setfacl -d -m u::rwX,u:lkraven:rwX,g::rwX,g:nemi-drop:rwX,o::--- /srv/nemi/inbox
|
||||
getfacl -p /srv/nemi/inbox
|
||||
verify:
|
||||
- name: Nemi's own context (lkraven's systemd user manager, groups fixed at boot) can list the inbox
|
||||
shell: runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemd-run --user --wait --pipe -q ls -ld /srv/nemi/inbox
|
||||
sudo: true
|
||||
changed_when: "false"
|
||||
- name: A file dropped by infra-ops is readable and removable by lkraven
|
||||
sudo: true
|
||||
shell: |
|
||||
set -e
|
||||
f=/srv/nemi/inbox/.acl-probe-$$
|
||||
runuser -u infra-ops -g nemi-drop -- sh -c "umask 077; echo probe > $f"
|
||||
runuser -u lkraven -- cat "$f" >/dev/null
|
||||
runuser -u lkraven -- rm -f "$f"
|
||||
test ! -e "$f" && echo "infra-ops drop -> lkraven read+remove: ok"
|
||||
changed_when: "false"
|
||||
@@ -0,0 +1,36 @@
|
||||
# nh3-svos: miranda -> app:app-postoffice write becomes autonomous (A171; Prime 2026-10-13 1400 via svos-dev).
|
||||
steps:
|
||||
- name: back up instance.toml
|
||||
sudo: true
|
||||
shell: runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-a171
|
||||
creates: /var/lib/svos/config/instance.toml.pre-a171
|
||||
- name: autonomous = true on the miranda -> app:app-postoffice write grant
|
||||
sudo: true
|
||||
shell: |
|
||||
runuser -u svc-svos -- python3 - <<'PY'
|
||||
import os
|
||||
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
|
||||
old = '[[grants]]\nprincipal = "miranda"\nobject = "app:app-postoffice"\nmode = "write"\n[[grants]]'
|
||||
new = '[[grants]]\nprincipal = "miranda"\nobject = "app:app-postoffice"\nmode = "write"\nautonomous = true # A171, Prime 2026-10-13 1400: her own bus sends no longer card\n[[grants]]'
|
||||
assert s.count(old) == 1, s.count(old)
|
||||
s = s.replace(old, new); t = p + ".tmp"
|
||||
fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w") as f: f.write(s)
|
||||
os.replace(t, p); print("edited")
|
||||
PY
|
||||
- name: the kernel accepts it (else restore)
|
||||
sudo: true
|
||||
shell: |
|
||||
if runuser -u svc-svos -- /opt/svos/current/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
|
||||
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-a171 /var/lib/svos/config/instance.toml; echo RESTORED; exit 1
|
||||
- name: reload
|
||||
sudo: true
|
||||
shell: |
|
||||
systemctl reload svos-kernel; sleep 6
|
||||
curl -s http://127.0.0.1:8400/health; echo
|
||||
journalctl -u svos-kernel --since "-20s" --no-pager -o cat | grep -i -E "refus|invalid|error|config" | tail -5 | cut -c1-200
|
||||
verify:
|
||||
- name: the grant reads autonomous
|
||||
sudo: true
|
||||
shell: runuser -u svc-svos -- python3 -c "import tomllib; d=tomllib.load(open('/var/lib/svos/config/instance.toml','rb')); g=[x for x in d['grants'] if x.get('principal')=='miranda' and x.get('object')=='app:app-postoffice' and x.get('mode')=='write']; print(g); assert len(g)==1 and g[0].get('autonomous') is True"
|
||||
changed_when: "false"
|
||||
@@ -0,0 +1,88 @@
|
||||
# nh3-svos -> SVOS main 33a7077 (v0.0.13 + 2): kernel 0.0.13 (no migration), connector base 0.0.9, app-interface 0.0.5
|
||||
# (needs SVOS_ARTIFACT_HOST), app-postoffice 0.0.3 wheel (runs on nh3-dev), UI 0.0.18. One config row: the interface briefs
|
||||
# artifact mapping (host = the interface's SVOS_ARTIFACT_HOST). svos-dev's ask on Prime's 0955 go (thread 01M4XSGTE30G33X6SR76B03JFJ).
|
||||
steps:
|
||||
- name: Private staging dir
|
||||
shell: install -d -m 0700 /tmp/svos-stage-33a7077
|
||||
- {name: stage release, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-33a7077/svos-release-33a7077.tar.gz, dest: /tmp/svos-stage-33a7077/release.tar.gz, mode: "0644"}}
|
||||
- {name: stage uv, upload: {src: /home/lkraven/bin/uv, dest: /tmp/svos-stage-33a7077/uv, mode: "0755"}}
|
||||
- {name: stage installer, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-33a7077/install-release.py, dest: /tmp/svos-stage-33a7077/install-release.py, mode: "0644"}}
|
||||
- name: Verify staged hashes
|
||||
shell: cd /tmp/svos-stage-33a7077 && printf '%s %s\n' abd2af171f3408431ab33ea097244692c846995c98308df1487413013fc5612c release.tar.gz 83aa128da93e168283ed32cf75cd819fe17b135b3fc1f7ccb3c912eec198d9fb uv | sha256sum -c -
|
||||
changed_when: "false"
|
||||
- name: Install release 33a7077 beside the live one (current NOT switched)
|
||||
sudo: true
|
||||
shell: python3 /tmp/svos-stage-33a7077/install-release.py
|
||||
- name: New kernel accepts the live configuration as it is
|
||||
sudo: true
|
||||
shell: runuser -u svc-svos -- /opt/svos/releases/33a7077/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos
|
||||
- name: The interface unit gets SVOS_ARTIFACT_HOST (0.0.5 fails closed without it); takes effect at the restart below
|
||||
sudo: true
|
||||
shell: |
|
||||
install -d -m 0755 /etc/systemd/system/svos-interface.service.d
|
||||
printf '%s\n' '# U16 (2026-10-14): names every task brief reference; must equal the instance.toml mapping host byte for byte' '[Service]' 'Environment=SVOS_ARTIFACT_HOST=nh3-svos' > /etc/systemd/system/svos-interface.service.d/20-artifact-host.conf
|
||||
systemctl daemon-reload; systemctl show svos-interface -p Environment | tr ' ' '\n' | grep ARTIFACT
|
||||
- name: Stop the local apps then the kernel (terminal, postoffice, miranda, agent run on nh3-dev since U13)
|
||||
sudo: true
|
||||
shell: systemctl stop svos-voice svos-knowledge svos-interface svos-kernel; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
|
||||
- name: Offline backup to /var/lib/svos/backup/pre-33a7077 (schema 13 database + config)
|
||||
sudo: true
|
||||
shell: |
|
||||
set -e
|
||||
d=/var/lib/svos/backup/pre-33a7077
|
||||
runuser -u svc-svos -- install -d -m 0700 $d
|
||||
runuser -u svc-svos -- python3 -c "
|
||||
import sqlite3
|
||||
src = sqlite3.connect('/var/lib/svos/state/svos.db'); dst = sqlite3.connect('/var/lib/svos/backup/pre-33a7077/svos.db')
|
||||
src.backup(dst); dst.close(); src.close()
|
||||
c = sqlite3.connect('file:/var/lib/svos/backup/pre-33a7077/svos.db?mode=ro', uri=True)
|
||||
print('integrity', c.execute('pragma integrity_check').fetchone()[0], 'user_version', c.execute('pragma user_version').fetchone()[0])"
|
||||
runuser -u svc-svos -- tar -C /var/lib/svos -cf $d/config-keys-apps.tar config state/keys state/apps state/config-generations apps
|
||||
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-33a7077
|
||||
ls -la $d | tail -3
|
||||
- name: Config for U16 (the interface briefs mapping), then the NEW kernel must accept it (else restore + old kernel back)
|
||||
sudo: true
|
||||
shell: |
|
||||
runuser -u svc-svos -- python3 - <<'PY'
|
||||
import os
|
||||
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
|
||||
anchor = '[[notifications.artifacts]]\nhost = "nh3-svos"\nroot = "/var/lib/svos/apps/terminal/tasks"\napp = "app-terminal"\nverb = "artifact"\n'
|
||||
assert s.count(anchor) == 1, s.count(anchor)
|
||||
assert "interface/briefs" not in s
|
||||
add = ('\n# U16 (main 33a7077, 2026-10-14): app-interface task briefs. host = the interface unit\'s SVOS_ARTIFACT_HOST (drop-in\n'
|
||||
'# svos-interface.service.d/20-artifact-host.conf), byte for byte; root = its SVOS_DATA_ROOT + /state/apps/interface/briefs.\n'
|
||||
'[[notifications.artifacts]]\nhost = "nh3-svos"\nroot = "/var/lib/svos/state/apps/interface/briefs"\napp = "app-interface"\nverb = "artifact"\n')
|
||||
s = s.replace(anchor, anchor + add)
|
||||
t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w") as f: f.write(s)
|
||||
os.replace(t, p); print("edited")
|
||||
PY
|
||||
if runuser -u svc-svos -- /opt/svos/releases/33a7077/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
|
||||
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-33a7077 /var/lib/svos/config/instance.toml
|
||||
systemctl start svos-kernel.service svos-interface svos-knowledge svos-voice; echo "RESTORED config, old kernel restarted"; exit 1
|
||||
- name: Switch current to 33a7077
|
||||
sudo: true
|
||||
shell: ln -sfn /opt/svos/releases/33a7077 /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current
|
||||
- name: Start the kernel (READY-gated; no migration) then the local apps
|
||||
sudo: true
|
||||
shell: |
|
||||
t0=$(date +%s.%N); systemctl start svos-kernel.service; rc=$?; t1=$(date +%s.%N)
|
||||
echo "kernel start rc=$rc after $(python3 -c "print(round($t1-$t0,2))") s"
|
||||
systemctl start svos-interface svos-knowledge svos-voice
|
||||
for i in $(seq 1 60); do h=$(curl -s http://127.0.0.1:8400/health); case "$h" in *'"down":0'*) case "$h" in *'"unknown":0'*) break;; esac;; esac; sleep 1; done
|
||||
echo "$h"; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
|
||||
journalctl -u svos-kernel --since "-90s" --no-pager -o cat | grep -i -E "migrat|0013|schema|error|refus" | tail -6 | cut -c1-200
|
||||
- name: Regenerate the front door inventory (d95078c becomes the previous release)
|
||||
sudo: true
|
||||
shell: /usr/local/sbin/svos-front-inventory 2>&1 | grep -E 'GENERATED|updated|root \*|path /assets' | head -6
|
||||
verify:
|
||||
- name: Inventory current
|
||||
sudo: true
|
||||
shell: /usr/local/sbin/svos-front-inventory --check
|
||||
changed_when: "false"
|
||||
- name: Health is schema 13, status ok, nothing down
|
||||
shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"schema_version":13' && echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0'
|
||||
changed_when: "false"
|
||||
- name: App listener still refuses an uncredentialed kernel-key (401 from the kernel)
|
||||
shell: test "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8410/auth/kernel-key)" = 401
|
||||
changed_when: "false"
|
||||
@@ -0,0 +1,85 @@
|
||||
# nh3-svos -> SVOS v0.0.11 (452a592, U15 task seats slice 1) at svos-dev's ask (Prime's 2010 go relayed). Migration 0013
|
||||
# (schema 13) and two config additions ([tasks], the tasks artifact mapping). Run AFTER tonight's 21:30 vzdump of VM 111.
|
||||
steps:
|
||||
- name: Private staging dir
|
||||
shell: install -d -m 0700 /tmp/svos-stage-452a592
|
||||
- {name: stage release, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-452a592/svos-release-452a592.tar.gz, dest: /tmp/svos-stage-452a592/release.tar.gz, mode: "0644"}}
|
||||
- {name: stage uv, upload: {src: /home/lkraven/bin/uv, dest: /tmp/svos-stage-452a592/uv, mode: "0755"}}
|
||||
- {name: stage installer, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-452a592/install-release.py, dest: /tmp/svos-stage-452a592/install-release.py, mode: "0644"}}
|
||||
- name: Verify staged hashes
|
||||
shell: cd /tmp/svos-stage-452a592 && printf '%s %s\n' b38a2a09cdaa7475705108bdebdf96c9d434d71be142b89b4fb1dc15a2d3af1e release.tar.gz 83aa128da93e168283ed32cf75cd819fe17b135b3fc1f7ccb3c912eec198d9fb uv | sha256sum -c -
|
||||
changed_when: "false"
|
||||
- name: Install release 452a592 beside the live one (current NOT switched)
|
||||
sudo: true
|
||||
shell: python3 /tmp/svos-stage-452a592/install-release.py
|
||||
- name: New kernel accepts the live configuration as it is
|
||||
sudo: true
|
||||
shell: runuser -u svc-svos -- /opt/svos/releases/452a592/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos
|
||||
- name: Stop the local apps then the kernel (terminal, postoffice, miranda, agent run on nh3-dev since U13)
|
||||
sudo: true
|
||||
shell: systemctl stop svos-voice svos-knowledge svos-interface svos-kernel; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
|
||||
- name: Offline backup to /var/lib/svos/backup/pre-452a592 (schema 12 database + config)
|
||||
sudo: true
|
||||
shell: |
|
||||
set -e
|
||||
d=/var/lib/svos/backup/pre-452a592
|
||||
runuser -u svc-svos -- install -d -m 0700 $d
|
||||
runuser -u svc-svos -- python3 -c "
|
||||
import sqlite3
|
||||
src = sqlite3.connect('/var/lib/svos/state/svos.db'); dst = sqlite3.connect('/var/lib/svos/backup/pre-452a592/svos.db')
|
||||
src.backup(dst); dst.close(); src.close()
|
||||
c = sqlite3.connect('file:/var/lib/svos/backup/pre-452a592/svos.db?mode=ro', uri=True)
|
||||
print('integrity', c.execute('pragma integrity_check').fetchone()[0], 'user_version', c.execute('pragma user_version').fetchone()[0])"
|
||||
runuser -u svc-svos -- tar -C /var/lib/svos -cf $d/config-keys-apps.tar config state/keys state/apps state/config-generations apps
|
||||
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-452a592
|
||||
ls -la $d | tail -3
|
||||
- name: Config for U15 ([tasks] + the tasks artifact mapping), then the NEW kernel must accept it (else restore + old kernel back)
|
||||
sudo: true
|
||||
shell: |
|
||||
runuser -u svc-svos -- python3 - <<'PY'
|
||||
import os
|
||||
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
|
||||
anchor = '[[notifications.artifacts]]\nhost = "nh3-svos"\nroot = "/var/lib/svos/apps/postoffice/mail"\napp = "app-postoffice"\nverb = "artifact"\n'
|
||||
assert s.count(anchor) == 1, s.count(anchor)
|
||||
assert "[tasks]" not in s
|
||||
add = ('\n# U15 (v0.0.11, 2026-10-13): app-terminal task results. host = the terminal\'s artifact_host ("nh3-svos", kept\n'
|
||||
'# through U13 although the files live on nh3-dev, like the injects mapping above); root = its [seat_host].tasks_root.\n'
|
||||
'[[notifications.artifacts]]\nhost = "nh3-svos"\nroot = "/var/lib/svos/apps/terminal/tasks"\napp = "app-terminal"\nverb = "artifact"\n')
|
||||
s = s.replace(anchor, anchor + add)
|
||||
s = s.rstrip("\n") + ('\n\n# Task seats (U15 slice 1, v0.0.11, 2026-10-13): the seat host\'s tasks root on nh3-dev (= terminal.toml\n'
|
||||
'# [seat_host].tasks_root); every ceiling at its default. No runtime declares task_argv yet, so a task-spawn is\n'
|
||||
'# refused routing_unavailable: runtime_not_task_capable until Prime names the command.\n'
|
||||
'[tasks]\nscratch_root = "/var/lib/svos/apps/terminal/tasks"\n')
|
||||
t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w") as f: f.write(s)
|
||||
os.replace(t, p); print("edited")
|
||||
PY
|
||||
if runuser -u svc-svos -- /opt/svos/releases/452a592/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
|
||||
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-452a592 /var/lib/svos/config/instance.toml
|
||||
systemctl start svos-kernel.service svos-interface svos-knowledge svos-voice; echo "RESTORED config, old kernel restarted"; exit 1
|
||||
- name: Switch current to 452a592
|
||||
sudo: true
|
||||
shell: ln -sfn /opt/svos/releases/452a592 /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current
|
||||
- name: Start the kernel (READY-gated; migration 0013 runs) then the local apps
|
||||
sudo: true
|
||||
shell: |
|
||||
t0=$(date +%s.%N); systemctl start svos-kernel.service; rc=$?; t1=$(date +%s.%N)
|
||||
echo "kernel start rc=$rc after $(python3 -c "print(round($t1-$t0,2))") s"
|
||||
systemctl start svos-interface svos-knowledge svos-voice
|
||||
for i in $(seq 1 60); do h=$(curl -s http://127.0.0.1:8400/health); case "$h" in *'"down":0'*) case "$h" in *'"unknown":0'*) break;; esac;; esac; sleep 1; done
|
||||
echo "$h"; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
|
||||
journalctl -u svos-kernel --since "-90s" --no-pager -o cat | grep -i -E "migrat|0013|schema|error|refus" | tail -6 | cut -c1-200
|
||||
- name: Regenerate the front door inventory (7f95e54 becomes the previous release)
|
||||
sudo: true
|
||||
shell: /usr/local/sbin/svos-front-inventory 2>&1 | grep -E 'GENERATED|updated|root \*|path /assets' | head -6
|
||||
verify:
|
||||
- name: Inventory current
|
||||
sudo: true
|
||||
shell: /usr/local/sbin/svos-front-inventory --check
|
||||
changed_when: "false"
|
||||
- name: Health is schema 13, status ok, nothing down
|
||||
shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"schema_version":13' && echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0'
|
||||
changed_when: "false"
|
||||
- name: App listener still refuses an uncredentialed kernel-key (401 from the kernel)
|
||||
shell: test "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8410/auth/kernel-key)" = 401
|
||||
changed_when: "false"
|
||||
@@ -0,0 +1,61 @@
|
||||
# nh3-svos -> SVOS v0.0.12 (d95078c): the kernel wheel alone (runtime `process`, placeholders inside task_argv args).
|
||||
# No migration (schema stays 13), no config change in this playbook; the hermes runtime edit follows on 0.0.12.
|
||||
steps:
|
||||
- name: Private staging dir
|
||||
shell: install -d -m 0700 /tmp/svos-stage-d95078c
|
||||
- {name: stage release, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-d95078c/svos-release-d95078c.tar.gz, dest: /tmp/svos-stage-d95078c/release.tar.gz, mode: "0644"}}
|
||||
- {name: stage uv, upload: {src: /home/lkraven/bin/uv, dest: /tmp/svos-stage-d95078c/uv, mode: "0755"}}
|
||||
- {name: stage installer, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-d95078c/install-release.py, dest: /tmp/svos-stage-d95078c/install-release.py, mode: "0644"}}
|
||||
- name: Verify staged hashes
|
||||
shell: cd /tmp/svos-stage-d95078c && printf '%s %s\n' c1be1519edf4fc91b4a87c384a645bd28b655ff64ae0abbd353a76cc9a50f195 release.tar.gz 83aa128da93e168283ed32cf75cd819fe17b135b3fc1f7ccb3c912eec198d9fb uv | sha256sum -c -
|
||||
changed_when: "false"
|
||||
- name: Install release d95078c beside the live one (current NOT switched)
|
||||
sudo: true
|
||||
shell: python3 /tmp/svos-stage-d95078c/install-release.py
|
||||
- name: New kernel accepts the live configuration as it is
|
||||
sudo: true
|
||||
shell: runuser -u svc-svos -- /opt/svos/releases/d95078c/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos
|
||||
- name: Stop the local apps then the kernel (terminal, postoffice, miranda, agent run on nh3-dev since U13)
|
||||
sudo: true
|
||||
shell: systemctl stop svos-voice svos-knowledge svos-interface svos-kernel; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
|
||||
- name: Offline backup to /var/lib/svos/backup/pre-d95078c (schema 13 database + config)
|
||||
sudo: true
|
||||
shell: |
|
||||
set -e
|
||||
d=/var/lib/svos/backup/pre-d95078c
|
||||
runuser -u svc-svos -- install -d -m 0700 $d
|
||||
runuser -u svc-svos -- python3 -c "
|
||||
import sqlite3
|
||||
src = sqlite3.connect('/var/lib/svos/state/svos.db'); dst = sqlite3.connect('/var/lib/svos/backup/pre-d95078c/svos.db')
|
||||
src.backup(dst); dst.close(); src.close()
|
||||
c = sqlite3.connect('file:/var/lib/svos/backup/pre-d95078c/svos.db?mode=ro', uri=True)
|
||||
print('integrity', c.execute('pragma integrity_check').fetchone()[0], 'user_version', c.execute('pragma user_version').fetchone()[0])"
|
||||
runuser -u svc-svos -- tar -C /var/lib/svos -cf $d/config-keys-apps.tar config state/keys state/apps state/config-generations apps
|
||||
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-d95078c
|
||||
ls -la $d | tail -3
|
||||
- name: Switch current to d95078c
|
||||
sudo: true
|
||||
shell: ln -sfn /opt/svos/releases/d95078c /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current
|
||||
- name: Start the kernel (READY-gated; no migration) then the local apps
|
||||
sudo: true
|
||||
shell: |
|
||||
t0=$(date +%s.%N); systemctl start svos-kernel.service; rc=$?; t1=$(date +%s.%N)
|
||||
echo "kernel start rc=$rc after $(python3 -c "print(round($t1-$t0,2))") s"
|
||||
systemctl start svos-interface svos-knowledge svos-voice
|
||||
for i in $(seq 1 60); do h=$(curl -s http://127.0.0.1:8400/health); case "$h" in *'"down":0'*) case "$h" in *'"unknown":0'*) break;; esac;; esac; sleep 1; done
|
||||
echo "$h"; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
|
||||
journalctl -u svos-kernel --since "-90s" --no-pager -o cat | grep -i -E "migrat|0013|schema|error|refus" | tail -6 | cut -c1-200
|
||||
- name: Regenerate the front door inventory (452a592 becomes the previous release)
|
||||
sudo: true
|
||||
shell: /usr/local/sbin/svos-front-inventory 2>&1 | grep -E 'GENERATED|updated|root \*|path /assets' | head -6
|
||||
verify:
|
||||
- name: Inventory current
|
||||
sudo: true
|
||||
shell: /usr/local/sbin/svos-front-inventory --check
|
||||
changed_when: "false"
|
||||
- name: Health is schema 13, status ok, nothing down
|
||||
shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"schema_version":13' && echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0'
|
||||
changed_when: "false"
|
||||
- name: App listener still refuses an uncredentialed kernel-key (401 from the kernel)
|
||||
shell: test "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8410/auth/kernel-key)" = 401
|
||||
changed_when: "false"
|
||||
@@ -0,0 +1,48 @@
|
||||
# nh3-svos: the hermes seat runtime becomes task-capable (Prime 2026-10-14 via svos-dev, thread 01M4X1ZE8RDF63NARVT4CFJBT2).
|
||||
# Merged into the EXISTING [[seats.runtimes]] name="hermes" entry: svos-dev's argv/process/task_argv replace its old argv
|
||||
# (venv python + hermes-agent/hermes). Needs kernel 0.0.12 (0.0.11's check-config refuses `process`). No --yolo (Prime's call).
|
||||
steps:
|
||||
- name: back up instance.toml
|
||||
sudo: true
|
||||
shell: runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-hermes-task
|
||||
creates: /var/lib/svos/config/instance.toml.pre-hermes-task
|
||||
- name: merge the task runtime keys into the hermes entry
|
||||
sudo: true
|
||||
shell: |
|
||||
runuser -u svc-svos -- python3 - <<'PY'
|
||||
import os
|
||||
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
|
||||
old = ('[[seats.runtimes]]\nname = "hermes"\nprogram = "hermes"\n'
|
||||
'argv = ["/home/lkraven/.hermes/hermes-agent/venv/bin/python", "/home/lkraven/.hermes/hermes-agent/hermes"]\n'
|
||||
'credential_env = "SVOS_CREDENTIAL"\n')
|
||||
assert s.count(old) == 1, s.count(old)
|
||||
new = ('[[seats.runtimes]] # U15 task runtime (Prime 2026-10-14 via svos-dev): hermes one-off, no --yolo\n'
|
||||
'name = "hermes"\n'
|
||||
'argv = ["/home/lkraven/.local/bin/hermes", "chat"]\n'
|
||||
'program = "hermes"\n'
|
||||
'process = "python3" # the launcher execs python3; the live process never shows hermes\n'
|
||||
'credential_env = "SVOS_CREDENTIAL"\n'
|
||||
'task_argv = ["/home/lkraven/.local/bin/hermes", "-z", "Your brief is the file {brief}. Do what it asks. Write every deliverable into {result}. Print a one-line summary first, then the details.", "-m", "{model}", "--accept-hooks"]\n')
|
||||
s = s.replace(old, new)
|
||||
s = s.replace("# [seat_host].tasks_root); every ceiling at its default. No runtime declares task_argv yet, so a task-spawn is\n# refused routing_unavailable: runtime_not_task_capable until Prime names the command.\n",
|
||||
"# [seat_host].tasks_root); every ceiling at its default. Since 2026-10-14 the hermes runtime declares task_argv\n# (the only task-capable runtime).\n")
|
||||
t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w") as f: f.write(s)
|
||||
os.replace(t, p); print("edited")
|
||||
PY
|
||||
- name: the kernel accepts it (else restore)
|
||||
sudo: true
|
||||
shell: |
|
||||
if runuser -u svc-svos -- /opt/svos/current/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
|
||||
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-hermes-task /var/lib/svos/config/instance.toml; echo RESTORED; exit 1
|
||||
- name: reload
|
||||
sudo: true
|
||||
shell: |
|
||||
systemctl reload svos-kernel; sleep 6
|
||||
curl -s http://127.0.0.1:8400/health; echo
|
||||
journalctl -u svos-kernel --since "-20s" --no-pager -o cat | grep -i -E "refus|invalid|error|config" | tail -5 | cut -c1-200
|
||||
verify:
|
||||
- name: the hermes runtime reads task-capable
|
||||
sudo: true
|
||||
shell: runuser -u svc-svos -- python3 -c "import tomllib; d=tomllib.load(open('/var/lib/svos/config/instance.toml','rb')); h=[r for r in d['seats']['runtimes'] if r['name']=='hermes']; assert len(h)==1 and h[0]['process']=='python3' and h[0]['task_argv'][1]=='-z'; print(h[0]['argv'], h[0]['process'], len(h[0]['task_argv']))"
|
||||
changed_when: "false"
|
||||
@@ -0,0 +1,27 @@
|
||||
#!/bin/bash
|
||||
# Run INSIDE esh-nas (CT 103 on esh-pve-nas) as root, via `pct exec 103`. The same steps as
|
||||
# playbooks/bootstrap-infra-ops-user.yaml, for a container infra-ops cannot log into yet (Prime 2026-10-13: "set it up").
|
||||
set -euo pipefail
|
||||
U=infra-ops; ID=850
|
||||
KEY='ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+1HBwfXrkfTYWdcnWCjLJ6VLAGC87gxH5h5vKaaA3c infra-ops@pfi-fleet'
|
||||
if ! id "$U" >/dev/null 2>&1; then
|
||||
if ! getent passwd "$ID" >/dev/null && ! getent group "$ID" >/dev/null; then
|
||||
groupadd -g "$ID" "$U"; useradd -m -s /bin/bash -u "$ID" -g "$ID" "$U"
|
||||
else
|
||||
echo "uid/gid $ID taken; letting the OS allocate"; useradd -m -s /bin/bash "$U"
|
||||
fi
|
||||
echo "created $U"
|
||||
fi
|
||||
chmod 0700 /home/$U
|
||||
getent group docker >/dev/null && usermod -aG docker "$U" || echo "no docker group, skipped"
|
||||
install -d -m 700 -o "$U" -g "$U" /home/$U/.ssh
|
||||
printf '%s\n' "$KEY" > /home/$U/.ssh/authorized_keys
|
||||
chown "$U:$U" /home/$U/.ssh/authorized_keys; chmod 600 /home/$U/.ssh/authorized_keys
|
||||
f=/etc/sudoers.d/$U
|
||||
printf 'Defaults:%s log_output\n%s ALL=(ALL) NOPASSWD:ALL\n' "$U" "$U" > "$f.tmp"
|
||||
printf '%s\n' '# Bulk data movers: keep the audit line, do not record the bytes (eshpfi playbooks/sudoers-infra-ops-bulk-no-log-output.yaml)' \
|
||||
'Cmnd_Alias BULK_DATA = /usr/bin/rsync, /usr/bin/tar, /usr/bin/cat, /usr/bin/dd, /usr/bin/zstd, /usr/bin/zcat, /usr/bin/gzip, /usr/bin/pv, /usr/bin/docker save *, /usr/bin/docker export *, /usr/bin/docker cp *' \
|
||||
'Defaults!BULK_DATA !log_output' >> "$f.tmp"
|
||||
chmod 440 "$f.tmp"
|
||||
if visudo -cf "$f.tmp"; then mv "$f.tmp" "$f"; else rm -f "$f.tmp"; echo "VISUDO VALIDATION FAILED"; exit 1; fi
|
||||
id "$U"; stat -c '%a %U %n' /home/$U /home/$U/.ssh /home/$U/.ssh/authorized_keys "$f"
|
||||
@@ -152,6 +152,10 @@ set +e
|
||||
# WORLDTREE_ROOT from its own location, so point that back at the tree under test.
|
||||
# shellcheck disable=SC1091
|
||||
source "$WT_TREE/env.sh"
|
||||
# Since 2026-10-16 every agent shell on nh3-dev exports ALBOK_SERVICE/ALBOK_TOKEN (~/.config/secrets/env.sh, the
|
||||
# fleet-agents read token). The harness has always run WITHOUT albok; unset both so the daily numbers stay comparable.
|
||||
# Remove this only on worldtree-dev's word, and then say so in the run notes.
|
||||
unset ALBOK_SERVICE ALBOK_TOKEN
|
||||
export WORLDTREE_ROOT="$run_tree"
|
||||
# The editable install maps packages/ to the main tree; put the tree under
|
||||
# test first (identical content is guaranteed by step 2, this keeps it explicit).
|
||||
|
||||
@@ -60,7 +60,7 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
|
||||
rebooted this node without accounting for it. Before any planned reboot of either
|
||||
node, either accept the read-only window on the survivor or set
|
||||
`pvecm expected 1` on it for the duration. Discovered 2026-08-18.
|
||||
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.**
|
||||
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.** Since 2026-10-13 it has the `infra-ops` identity (uid 850, NOPASSWD sudo, Prime's ruling): `ssh infra-ops@10.0.50.50` works directly. Before that, the only way in was `pct exec 103` from this host.
|
||||
Never reboot this host casually — quiesce the clients first. Measured from the
|
||||
server on 2026-08-18 (`ss -tn '( sport = :2049 )'` inside CT 103), there are
|
||||
**five** clients, not the two long documented here:
|
||||
|
||||
@@ -229,3 +229,24 @@ Available to **every user/project** on this box — no per-home `playwright inst
|
||||
/ `heid run` pollers) into Claude Code session orchestrators (heid commit
|
||||
`12aa5a9`); the `~/development/volva` dir + venvs are gone. `volva.service` had
|
||||
been crash-looping `203/EXEC`. Cleanup done by infra-ops at heid's request.
|
||||
|
||||
## `/srv/nemi/inbox`: albok's drop-box (2026-10-13)
|
||||
|
||||
Nemi (lkraven's `albok-nemi.timer`) ingests markdown dropped here into albok's `personal/inbox` wing, once per file;
|
||||
capture mode never tombstones, so a walked file may be moved away. Writers are group `nemi-drop` (lkraven,
|
||||
infra-ops). The dir is `lkraven:nemi-drop 2770` with a default ACL (u:lkraven, g:nemi-drop rwX), and `/srv/nemi`
|
||||
carries `u:lkraven:rx`.
|
||||
⚠ lkraven's systemd user manager started before `nemi-drop` existed, so Nemi gets in through those ACL entries
|
||||
for lkraven, NOT through the group. Do not "simplify" them away until lkraven's user manager has restarted.
|
||||
`playbooks/nh3-dev-nemi-inbox.yaml` verifies both paths.
|
||||
|
||||
## `/srv/nemi/vault`: Prime's notes for Nemi (2026-10-15)
|
||||
|
||||
A fresh clone of gitea `vh/VuongKB` (Prime's notes, which he syncs with git; his working copy is `/home/lkraven/kb`, never
|
||||
touched by this). Nemi's `vault` source walks it into albok's `personal/notes` (unit 4.4 ruling 6). It is read-only by
|
||||
convention: nothing writes into it but the pull. The lkraven user timer `nemi-vault-pull` (`configs/nemi-vault/`)
|
||||
runs `git pull --ff-only` hourly at :50, ahead of Nemi's walk at ~:01. A force-push upstream fails the pull loudly
|
||||
(`systemctl --user status nemi-vault-pull`) and the copy stays as it was. Re-clone it on purpose if that happens.
|
||||
Because it is a CLONE, it holds only tracked files. The untracked `main/` (Muninn job dirs) and `fiction/` (Muninn
|
||||
fiction concepts, 2,389 files) in Prime's working copy are deliberately not in it.
|
||||
|
||||
|
||||
@@ -23,6 +23,27 @@ configuration, enrollment and acceptance.
|
||||
| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
|
||||
| Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
|
||||
|
||||
## Release 452a592 / v0.0.11, U15 task seats slice 1 (2026-10-13 2131–2133; Prime's 2010 go via svos-dev, thread `01M4WAGAS2Z0P8J6GZRSCRPH2Q`)
|
||||
|
||||
- **Order:** tonight's 21:30 vzdump of VM 111 finished 21:31:30, then `playbooks/svos-releases/nh3-svos-452a592-v0.0.11.yaml`
|
||||
(the kit was built in-session: kernel 0.0.11, connector base 0.0.8 and app-terminal 0.0.4 wheels from a clean archive of
|
||||
the tag; everything else, including the web build of 7f95e54, carried byte-identical).
|
||||
- **Backup before the migration:** `/var/lib/svos/backup/pre-452a592/` (the schema 12 `svos.db` via the sqlite backup API,
|
||||
integrity ok, plus the config/keys/apps tar) and `config/instance.toml.pre-452a592`.
|
||||
- **Config:** a `[tasks]` table with `scratch_root = "/var/lib/svos/apps/terminal/tasks"` (all ceilings at their defaults),
|
||||
plus a `[[notifications.artifacts]]` for that root. Its **host is `nh3-svos`, not `nh3-dev`**: the terminal's
|
||||
`artifact_host` stayed `nh3-svos` through U13, and its ResultReader names results by that host, so the mapping must
|
||||
match it, as the injects mapping does. No runtime has `task_argv` yet, so a task-spawn is refused
|
||||
`runtime_not_task_capable` until Prime names the command.
|
||||
- **Result:** kernel 0.0.11 READY in 1.17 s, migration 0013 run, `/health` schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7.
|
||||
- **nh3-dev:** `/opt/svos/current → releases/452a592` (terminal 0.0.4, connector 0.0.8). `/var/lib/svos/apps/terminal/tasks`
|
||||
(0700) and `terminal.toml [seat_host].tasks_root` (backup `.pre-452a592`). Only svos-terminal was restarted. Its manifest
|
||||
revision 4 (0.0.4, `seat-host`) was **pinned directly, with no card**. The seats read unknown for 6 s, then live.
|
||||
svos-miranda, svos-postoffice and svos-agent still run adc862e's code until their next restart.
|
||||
- **Rollback:** stop the kernel, put `svos.db` back from `backup/pre-452a592` (schema 12), restore `instance.toml.pre-452a592`,
|
||||
`current → releases/7f95e54`, then start. On nh3-dev: `current → releases/adc862e`, restore `terminal.toml.pre-452a592`,
|
||||
then restart svos-terminal.
|
||||
|
||||
## Release afab123 / v0.0.9 + U13 phase B (2026-10-10 1646–1648, Prime-approved 1639; svos-dev thread `01M4M2Z2H3BQH375M2G30GQHDM`)
|
||||
|
||||
- **`/opt/svos/current` → `releases/afab123`: kernel 0.0.9, schema 11 → 12** (migration 0012, the page verb's tables;
|
||||
|
||||
@@ -126,6 +126,19 @@ grants plus read+write on both library wings) and albok-dev `01M4HAP9HFYZCFV76WX
|
||||
`read:library/*`), vaulted over `albok/token-nemi` / `albok/token-albok-dev`. albok-dev swaps them into nemi.yaml and
|
||||
albok-check (swapped 1459, proven by the 1503 walk) and revoked the previous nemi `01M40JWJ3NZ6VJ77KS8B06BCYZ` and
|
||||
albok-dev `01M3ZMCXE8P66MRX16MZN1D44D` at 1555 10-09. Live set: bootstrap-admin, svos-knowledge-nh3-svos, the new nemi, the new albok-dev.
|
||||
**Wing `personal/inbox` (2026-10-13, unit 4.4 slice C's capture target, albok-dev's ask after Prime's 1530 intake ruling):**
|
||||
like `personal/notes`, so `default_group: albok-personal` (1511) and not service-restricted. The hiding is the sensitivity
|
||||
classes plus Nemi's `restricted_wings`. vh reads it through 1511, and no API token besides Nemi's holds it. The live
|
||||
config backup is `albok.yaml.pre-inbox`; added by `playbooks/albok-personal-inbox.yaml`, with health reading 7 wings
|
||||
ready. Nemi's token was re-minted for it: `01M4VX207PJ09ZSECASCSZPCFE`, its previous ten grants plus read+write on
|
||||
`personal/inbox`, vaulted over `albok/token-nemi`. The previous nemi token `01M4HAP9G90J79HBRJB6VW6PQA` was
|
||||
revoked at 1742 after albok-dev's swap; the capture source walks (generation 0, empty root).
|
||||
**2026-10-15:** re-minted again for the vault source (Prime's notes, `/srv/nemi/vault` on nh3-dev → `personal/notes`):
|
||||
`01M4ZBJG3DMRF9Q9XEVZ8Q20GH` = the twelve grants above plus read+write on `personal/notes`, vaulted over `albok/token-nemi`.
|
||||
`01M4VX207PJ09ZSECASCSZPCFE` was revoked at 0148 after albok-dev's swap. The first vault walk created 232 objects (129 notes
|
||||
+ 103 children of the 7 split ones); the judge classed 2 personal, 0 adult, 0 secret. The capture ROOT is on nh3-dev: `/srv/nemi/inbox`
|
||||
(`playbooks/nh3-dev-nemi-inbox.yaml`).
|
||||
|
||||
**Nemi (albok's tagger, runs on nh3-dev):** `albok/token-nemi`, a service token minted 0255 with read+write on
|
||||
`fleet/memory`, `fleet/stash` and `personal/agent-feedback`; it sees exactly those three wings (`personal/notes` hidden).
|
||||
The previous token (`01M40979…`, without agent-feedback) is still live until albok-dev revokes it.
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
# 1512 albok-feedback (restricted wing personal/agent-feedback) is default-deny: no viewer gets it.
|
||||
services:
|
||||
albok-service:
|
||||
image: gitea.phasefinal.com/pfi/albok-service:0.1.20@sha256:f43068322ce64be2b03b12c292d4994b9a24c9eb59c28f674b1551a0930d5058
|
||||
image: gitea.phasefinal.com/pfi/albok-service:0.1.23@sha256:026de87f3ca3907e9eb0a2a2489a7507f69f9f0eca50e8e8be3fcfcb36a03de3
|
||||
container_name: albok-service
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
|
||||
@@ -49,6 +49,7 @@ buildings:
|
||||
default_group: albok-personal # host gid 1511
|
||||
wings:
|
||||
- id: notes
|
||||
- id: inbox # unit 4.4 capture wing (Nemi's drop-box /srv/nemi/inbox on nh3-dev); like notes (2026-10-13)
|
||||
- id: agent-feedback # auto-memory (Nemi's third source); restricted, default-deny (2026-10-03)
|
||||
restricted: true
|
||||
group: albok-feedback # host gid 1512
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
# mimir-inbox
|
||||
|
||||
> **RETIRED 2026-10-13** (Prime 1530: albok subsumes library/note intake). `docker compose down` on corviduo-dev at 2123;
|
||||
> the image, compose dir, config and the staging mount `/mnt/muninn-staging/mimir-inbox` are left in place for albok's unit 6
|
||||
> (backup, then teardown, albok-dev's). Bring-back = `docker compose up -d` in the compose dir, nothing else changed.
|
||||
|
||||
|
||||
UI front end for large-document / novel / book ingestion into the Muninn KB —
|
||||
server-side FastAPI + HTMX over the **muninn-gate** API (#377). The submit path
|
||||
is: **browser → mimir-inbox `/upload` (writes to staging) → path-addressed
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
# muninn-gate
|
||||
|
||||
> **RETIRED 2026-10-13** (Prime 1530: albok subsumes library/note intake). `docker compose down` on corviduo-dev at 2123;
|
||||
> the image, compose dir, config and the staging mount `/mnt/muninn-staging/mimir-inbox` are left in place for albok's unit 6
|
||||
> (backup, then teardown, albok-dev's). Bring-back = `docker compose up -d` in the compose dir, nothing else changed.
|
||||
|
||||
|
||||
WG-internal HTTP front door for the Muninn ingestion queue (Worldtree #377).
|
||||
Path-addressed submit, list/status, cancel/retry, watcher-liveness health.
|
||||
Upstream: `vh/muninn-gate` (no Dockerfile there — this stack owns containerization).
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
PEWPEW_REV=26bba3c
|
||||
PORT=8185
|
||||
# Set both Worldtree vars or neither (WORLDTREE_BASE_URL is in compose.yaml):
|
||||
WORLDTREE_API_KEY= # secret get nh3-dev/.config/worldtree/pewpewstudio-demo-key
|
||||
@@ -0,0 +1,51 @@
|
||||
# pewpewstudio
|
||||
|
||||
Laser job studio (FastAPI backend + React SPA, gitea `vh/pewpewstudio`). It composes Thunder Aurora marking jobs and
|
||||
returns LightBurn `.lbrn` files as browser downloads, and never touches hardware. Deployed 2026-10-13 on Prime's directive
|
||||
via pewpew-dev (thread `01M4WATRFQX3V7TSJ4XPGABWMN`).
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **URL** | `http://10.100.50.40:8185/` (= `http://nh3-docker.nh3.internal:8185/`). Homepage tile in *AI - Studios*. |
|
||||
| **Host** | nh3-docker: fast local disk for SQLite WAL (fsync ~2.7 ms, unlike ana-docker's HDD pool), and it reaches arbo `100.64.0.6:8201` and Worldtree DEMO `10.250.50.152:8080`. |
|
||||
| **Auth** | NONE in v1. LAN/mesh only: published on the host's LAN address, with no public ingress. |
|
||||
| **Images** | `pewpewstudio:<rev>` (api) and `pewpewstudio-web:<rev>` (Caddy + `web/dist`), built on nh3-dev and loaded here; `PEWPEW_REV` in `.env` picks the pair. **Live: `3f73448`** (since 2026-10-15 2327; was `26bba3c`, whose images stay on the host for rollback). |
|
||||
| **State** | volume `pewpewstudio_state` (`/state`: `shop.db` + `assets/`) and volume `pewpewstudio_config` (`/config`: machine profiles, lenses, denylist, sizing, fonts, which **the app edits from the browser**). Both are covered by nh3-pve's nightly 21:00 whole-VM vzdump of VM 100. |
|
||||
| **Secrets** | `.env` (0600, infra-ops): `WORLDTREE_API_KEY` = vault `nh3-dev/.config/worldtree/pewpewstudio-demo-key`. |
|
||||
|
||||
## Topology
|
||||
|
||||
`web` (Caddy, published `10.100.50.40:8185 → :80`) serves the SPA at `/`. It sends `/api/*` to `api:8000` with the
|
||||
prefix **stripped** (the backend has no `/api` routes and no CORS, so both must be one origin) and Host rewritten to
|
||||
`localhost`, which the backend's TrustedHost accepts. Because of that rewrite, **Caddy refuses any Host other than
|
||||
`10.100.50.40` / `nh3-docker.nh3.internal` with 400**, to keep the backend's DNS-rebinding defence. Add a name to
|
||||
`conf/Caddyfile` before publishing one. Verified 2026-10-13: foreign Host → 400; IP and name → 200.
|
||||
⚠ That check must stay inside `route { }`. Outside it, Caddy orders `respond` after `handle_path`, and the check
|
||||
silently never runs: a foreign Host got 200 until it was wrapped.
|
||||
⚠ The Caddy config is mounted as the DIRECTORY `/opt/docker/conf/pewpewstudio → /etc/caddy`. deploy-stack replaces the
|
||||
file, and a single-file bind mount keeps showing the old inode.
|
||||
|
||||
## Build and deploy (from nh3-dev)
|
||||
|
||||
```bash
|
||||
REV=<short sha>; C=$(mktemp -d)
|
||||
git -C ~/development/pewpewstudio archive $REV | tar -x -C $C
|
||||
cp stacks/pewpewstudio/build/Dockerfile $C/Dockerfile.eshpfi
|
||||
# hamr is a git dependency over gitea SSH (10.250.50.70:222); BuildKit forwards lkraven's gitea key:
|
||||
(cd $C && docker build -f Dockerfile.eshpfi --ssh default=$HOME/.ssh/id_rsa --target api -t pewpewstudio:$REV . \
|
||||
&& docker build -f Dockerfile.eshpfi --target web -t pewpewstudio-web:$REV .)
|
||||
docker save pewpewstudio:$REV pewpewstudio-web:$REV | ssh infra-ops@10.100.50.40 docker load
|
||||
scripts/deploy-stack.sh nh3-docker pewpewstudio # compose + conf/Caddyfile; never .env
|
||||
# on nh3-docker: set PEWPEW_REV=$REV in .env (in place, keep 0600), then
|
||||
cd /opt/docker/compose/pewpewstudio && docker compose up -d
|
||||
```
|
||||
|
||||
**First deploy only:** the config volume was seeded once from the repo's `config/` plus the gitignored
|
||||
`config/machine.local.toml` from nh3-dev's checkout (machine parameters, no secrets). Without a `machine.*.toml` (the
|
||||
registry skips `machine.example.toml`), startup fails "no machine.*.toml files". **Never reseed a populated volume**:
|
||||
it holds the shop's own edits. The database started fresh; the dev `shop.db` was smoke-test data.
|
||||
|
||||
## Checks
|
||||
|
||||
- `GET /` 200 (the SPA); `GET /api/machines` 200 (one machine, `aurora-mopa-100w-f160`); `GET /api/generate/workflows` 200.
|
||||
- The api healthcheck is `GET /machines` on loopback.
|
||||
@@ -0,0 +1,33 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
# pewpewstudio images, built on nh3-dev from a clean `git archive` of vh/pewpewstudio (see ../README.md).
|
||||
# api: the FastAPI backend (core.api), venv built with uv; hamr is a git dependency over gitea SSH (BuildKit --ssh).
|
||||
# web: Caddy serving the SPA (web/dist) and proxying /api/* to the api with the prefix stripped (Caddyfile mounted).
|
||||
|
||||
FROM python:3.12-slim-bookworm AS build
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends git openssh-client build-essential \
|
||||
&& rm -rf /var/lib/apt/lists/* && pip install --no-cache-dir uv==0.13.0
|
||||
# gitea's SSH is 10.250.50.70:222; the lockfile names ssh://git@gitea.phasefinal.com/vh/hamr.git, as nh3-dev's ssh config maps it
|
||||
RUN install -d -m 700 /root/.ssh \
|
||||
&& printf 'Host gitea.phasefinal.com\n HostName 10.250.50.70\n Port 222\n User git\n' > /root/.ssh/config \
|
||||
&& ssh-keyscan -p 222 10.250.50.70 > /root/.ssh/known_hosts 2>/dev/null
|
||||
ENV UV_LINK_MODE=copy UV_PYTHON_DOWNLOADS=never UV_PROJECT_ENVIRONMENT=/app/.venv
|
||||
WORKDIR /app
|
||||
COPY . /app
|
||||
RUN --mount=type=ssh uv sync --frozen --no-dev && uv --version > /app/.uv-version
|
||||
|
||||
FROM python:3.12-slim-bookworm AS api
|
||||
COPY --from=build /app /app
|
||||
RUN install -d -o 1000 -g 1000 /state /state/assets /config
|
||||
WORKDIR /app
|
||||
ENV PATH=/app/.venv/bin:$PATH PYTHONUNBUFFERED=1
|
||||
USER 1000:1000
|
||||
EXPOSE 8000
|
||||
CMD ["python", "-c", "import uvicorn; from core.api.app import create_app; from core.api.config import ApiConfig; uvicorn.run(create_app(ApiConfig.from_env()), host='0.0.0.0', port=8000)"]
|
||||
|
||||
FROM oven/bun:1.3 AS webbuild
|
||||
WORKDIR /web
|
||||
COPY web/ /web/
|
||||
RUN bun install --frozen-lockfile && bun run build
|
||||
|
||||
FROM caddy:2-alpine AS web
|
||||
COPY --from=webbuild /web/dist /srv
|
||||
@@ -0,0 +1,51 @@
|
||||
# pewpewstudio on nh3-docker: laser job studio (FastAPI + React SPA), composes Thunder Aurora marking jobs and exports
|
||||
# LightBurn .lbrn. Prime 2026-10-13 via pewpew-dev. NO user auth in v1, so LAN/mesh only: published on the host's LAN
|
||||
# address, never on a public ingress. Images are built on nh3-dev and loaded here (see README).
|
||||
# State: the SQLite DB + assets on the named volume `state` (local disk, never NFS). The shop's own config (machine
|
||||
# profiles, lenses, denylist, sizing) on the named volume `config`, because the app EDITS it from the browser (temp
|
||||
# file + os.replace in the directory); a volume, not /opt/docker/conf, so deploy-stack's --delete can never touch it.
|
||||
# Seeded once at first deploy (README). Both volumes ride nh3-docker's whole-VM backup.
|
||||
name: pewpewstudio
|
||||
services:
|
||||
api:
|
||||
image: pewpewstudio:${PEWPEW_REV:?set PEWPEW_REV}
|
||||
container_name: pewpewstudio-api
|
||||
restart: unless-stopped
|
||||
env_file: .env # WORLDTREE_API_KEY (secret get nh3-dev/.config/worldtree/pewpewstudio-demo-key)
|
||||
environment:
|
||||
PEWPEW_DB_PATH: /state/shop.db
|
||||
PEWPEW_ASSETS_ROOT: /state/assets
|
||||
PEWPEW_PROFILE_DIR: /config
|
||||
PEWPEW_DENYLIST: /config/materials.toml
|
||||
PEWPEW_SIZING: /config/sizing.toml
|
||||
PEWPEW_FONTS_DIR: /config/fonts
|
||||
PEWPEW_BASELINE: /config/settings_baseline.toml
|
||||
ARBO_BASE_URL: http://100.64.0.6:8201
|
||||
WORLDTREE_BASE_URL: http://10.250.50.152:8080
|
||||
volumes:
|
||||
- state:/state
|
||||
- config:/config
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/machines', timeout=5)"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
web:
|
||||
image: pewpewstudio-web:${PEWPEW_REV:?set PEWPEW_REV}
|
||||
container_name: pewpewstudio
|
||||
restart: unless-stopped
|
||||
depends_on: [api]
|
||||
ports:
|
||||
- "10.100.50.40:${PORT:-8185}:80"
|
||||
volumes:
|
||||
- /opt/docker/conf/pewpewstudio:/etc/caddy:ro # the DIRECTORY: deploy-stack replaces the file, and a single-file bind would keep the old inode
|
||||
labels:
|
||||
- homepage.group=AI - Studios
|
||||
- homepage.name=pewpewstudio
|
||||
- homepage.icon=mdi-laser-pointer
|
||||
- "homepage.description=Laser job studio: composes Thunder Aurora marking jobs, exports LightBurn .lbrn"
|
||||
- homepage.href=http://10.100.50.40:${PORT:-8185}/
|
||||
volumes:
|
||||
state:
|
||||
config:
|
||||
@@ -0,0 +1,21 @@
|
||||
# pewpewstudio's front: the SPA at /, /api/* to the backend with the prefix STRIPPED (the backend has no /api routes and
|
||||
# no CORS, so both must be one origin). The backend's TrustedHost accepts localhost, so Host is rewritten to it.
|
||||
# No request-body cap here: asset uploads are capped at 32 MB by the app itself. LAN/mesh only, no auth in v1.
|
||||
:80 {
|
||||
encode zstd gzip
|
||||
# `route` keeps this literal order; outside it Caddy sorts `respond` AFTER handle/handle_path, and the Host check
|
||||
# would never run (measured: a foreign Host got 200 before this was a route).
|
||||
route {
|
||||
# Keep the backend's TrustedHost meaning (DNS-rebinding defence): the proxy rewrites Host to localhost below,
|
||||
# so it must refuse any Host the app would refuse. Add a name here when one is published for it.
|
||||
@foreign not host 10.100.50.40 nh3-docker.nh3.internal
|
||||
respond @foreign 400
|
||||
handle_path /api/* {
|
||||
reverse_proxy api:8000 {
|
||||
header_up Host localhost
|
||||
}
|
||||
}
|
||||
root * /srv
|
||||
file_server
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user