memory: snapshot — SVOS v0.0.11, pewpewstudio, albok inbox, h2o decision seat; 31 old entries archived
This commit is contained in:
@@ -15314,3 +15314,621 @@ latency-sensitive.
|
||||
|
||||
- `[2026-09-26]` **zsh traps in ad-hoc test loops:** `set -- $a` does NOT word-split, so alert POSTs went out with empty fields; and `local path=…` inside a function CLOBBERS `$PATH` (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and `${=var}`.
|
||||
_Archived 2026-10-11._
|
||||
|
||||
|
||||
## Recent decisions (archived)
|
||||
|
||||
- `[2026-09-29]` **Worldtree U11a prepped, not flipped: a staged demo config plus the agreed U8 window plan (infra-hermes runs the batches).** → `persistent-memory.d/2026-09-29-worldtree-u11a-prepped.md`
|
||||
|
||||
# Worldtree U11a (legacy memory → read_only) is PREPPED, not flipped.
|
||||
|
||||
`[2026-09-29]` **Worldtree U11a (legacy memory → read_only) is PREPPED, not flipped.** A staged, inert `/opt/worldtree/config/defaults.yaml.staged-u11a-read_only` sits on corviduo-dev. Applying it needs Prime's go AND demo on 48bdf235 or later (demo ran 1b8746e6, which predates U11a). Apply = back up, copy over `defaults.yaml`, `docker restart`, then read the boot line `memory cutover: legacy plane read_only (writer on, reader on)`. **Once the window opens, infra-hermes runs one U8 batch per day and I own the wrapper and the escalations (proposed, worldtree-dev agreed):** on nh3-dev in `~/development/Worldtree`, at demo's deployed SHA, `python -m core.memory_acceptance.live --runs 3 --gate --legacy-mode read_only --label window`. The verdict is in `trace1.md`; FAIL goes to worldtree-dev the same day, never a retry; no git commit; never two at once. The wrapper is NOT built yet, deliberately: it waits for the go.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-28]` **Worldtree U10 backfill done on demo (5) and personal (797). model_roles drift needed a memory_tagger sync first; mimir had missing vectors.** → `persistent-memory.d/2026-09-28-worldtree-u10-backfill.md`
|
||||
|
||||
# Worldtree U10 legacy-memory backfill: demo COMMITTED (Prime, direct go 2350).
|
||||
|
||||
`[2026-09-28]` **Worldtree U10 legacy-memory backfill: demo COMMITTED (Prime, direct go 2350).** Dry run then commit, both as `-u worldtree`: 5 filed (lofn 4, mimir 1), 16 `unslotted_pre_364` skipped, 0 aborted. **The first commit exited 1 because demo's hand-managed `model_roles.yaml` lacked the `memory_tagger` role**, so I synced it verbatim from the image's config-defaults (backup `.bak-pre-memory-tagger`). **2026-09-29 0854 presync for personal (Prime: "complete u10"):** personal got `memory_tagger` and the U9 `admin.memory.forget` policy delta. **Demo had also been missing that U9 policy delta since 09-24**, which left readonly-admin's `admin.*` rule with no exclusion for the destructive forget. Fixed, and demo's api was restarted to load it. **Personal COMMITTED 2026-09-29 1007** (b191 e2f74706, a snapshot-copy build; Prime's standing "complete u10" via worldtree-dev's go): 797 filed (mimir 795, lofn 2), mimir `dropped:too_large` 377, 1 unroutable held. The first personal dry run (b190) had crashed on 45 mimir rows with no vector (HNSW frozen since 08-06, #417, accepted by Prime until U11). Pinned (446e5807; Prime: "leave pinned") keeps its legacy chroma in the container's WRITABLE LAYER (`/app/agents/*/memory/.chroma`, 7 agents, 196K each): a recreate erases them.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-28]` **Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired.** → `persistent-memory.d/2026-09-28-bonsai-ternary-spike.md`
|
||||
|
||||
# Bonsai ternary spike on fv-ml1 GPU 3 (Prime via brokkr): at the 275 W cap, PQ2_0 is 1.93x Q4_K_XL at N=1 but only 1.06x at N=8 (PTQ1_0 1.24x), recovering to ~1.37x at N=16.
|
||||
|
||||
`[2026-09-28]` **Bonsai ternary spike on fv-ml1 GPU 3 (Prime via brokkr): at the 275 W cap, PQ2_0 is 1.93x Q4_K_XL at N=1 but only 1.06x at N=8 (PTQ1_0 1.24x), recovering to ~1.37x at N=16.** Positive control passed on tg128 (+1.1%). nvidia-smi's sw_power_cap flag never fires on this card, so "at cap" is judged from board draw. Follow-up (brokkr): moving PQ2_0 onto MMQ from batch 6, like Q4_K (mmvq.cu, `ne11 <= 5`), lifts N=8 to 1.21x, so the dip was partly a kernel threshold. The residual gap is not power. Runs are in `fv-ml1:/tank/spikes/bonsai-2026-09-28/runs{,-mmvq5}/` and on the Booth. The build image was removed; `Dockerfile.build` recreates it. **Acquired for keeps (Prime, 1540):** GGUF PQ2_0/PTQ1_0/mmproj-Q8_0 + the fork source pin in `/tank/aimodels/llm/prism-ml_Ternary-Bonsai-2-27B-gguf/` (runtime/), the MLX 2-bit pack in `/tank/aimodels/mlx/`, 34/34 hash-verified. Weights single copy (no snapshots, /tank not in restic); the fork pin also at `/mnt/smithy/runtime-pins/prism-llama.cpp-87268f77/` (brokkr).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-28]` **blender-run gained `--cpu` (no GPU attached) and a fixed hostname `fv-ml1-blender` (draupnir).** The design stage never renders, so it stays off GPU 3.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-28]` **Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (`--extensions`).** SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → `stacks/blender/README.md` § Extensions
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** → `persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md`
|
||||
|
||||
# hermes-gateway restarted 0401 for highseat-dev
|
||||
|
||||
`[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
|
||||
|
||||
# `[2026-09-27]` SemIf live on fv-ml1 GPU 1 (Prime)
|
||||
|
||||
**semif-serve 0.1.2 is live** at `http://10.251.50.54:8032` (`semif.fv.internal`), stack `stacks/semif`,
|
||||
code and contract `services/semif-serve/` (`069725c`). It is a FastAPI wrapper around SemIf-OpenJev's
|
||||
direct and shared torch scorers (MIT, pinned `23cf1f39fc9534fe81437200959b6dfc7106e45a`) on
|
||||
Qwen3.5-4B `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a` (BF16, offline HF cache on `/tank`).
|
||||
Hosting assessment: infra-hermes, `/tmp/semif.md` (2026-09-25).
|
||||
|
||||
Prime's calls (2026-09-27): fv-ml1 GPU 1 under a hard VRAM cap (not the reserved-empty GPU 3, not a Q4
|
||||
GGUF on a 16 GB box); infra-ops builds it with a light process (short contract → TDD → heid bug hunt);
|
||||
no consumer named yet, so every score is labelled uncalibrated.
|
||||
|
||||
**Two defects that only the card showed**, each fixed with a test:
|
||||
- **0.1.1:** the OOM path raised `OutOfMemory(...) from exc` inside the `except` block. The chain kept the
|
||||
torch exception's traceback, and with it the failed call's frames and tensors, so 11.9 GiB stayed
|
||||
allocated after the 503. The fix raises after the block, with no chain, and runs `gc.collect()`
|
||||
before `empty_cache()`.
|
||||
- **0.1.2:** a large shared request left torch's cache holding 12.6 GB, which left scriberr 3.5 GB on GPU 1.
|
||||
Now, after each call, reserved memory above the post-warm-up baseline + 512 MiB is released
|
||||
(`empty_cache`). It costs ~16 ms on a big request.
|
||||
|
||||
**Acceptance** (`services/semif-serve/acceptance/result-2026-09-27-v0.1.2.json`), against SemIf's
|
||||
committed torch predictions on authored144:
|
||||
- 142/144 same top choice; both misses are exact bf16 ties in our output;
|
||||
- 144/144 identical prompt SHA-256;
|
||||
- A-vs-A gap 0.0;
|
||||
- negative control (rotated option descriptions) 14/144;
|
||||
- shared vs direct 72/72;
|
||||
- 21 binary criteria over one state in 159 ms.
|
||||
|
||||
The shared-mode capacity under 12 GiB is 52 decisions at a ~140-token prefix and 13 at ~3,900.
|
||||
|
||||
**Build:** torch 2.10.0+cu128 from the pytorch index (SemIf's own stack; sm_120 present). The
|
||||
Dockerfile installs dependencies from a manifest with the project version blanked, so a version bump
|
||||
reuses the ~4 GB torch layer (41 s rebuild, layer CACHED). See [[2026-09-27-semif-order-averaging]].
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port.** → `persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md` (canonical: `stacks/blender/README.md`)
|
||||
|
||||
# `[2026-09-27]` Blender 5.2 on fv-ml1 GPU 3, agent-driven (Prime)
|
||||
|
||||
(Moved out of the in-flight section at the 2026-10-06 snapshot. Canonical: `stacks/blender/README.md`.)
|
||||
|
||||
- Prime: "go ahead with gpu 3, both", and he does not use Blender, so **agents drive it through MCP**.
|
||||
Blender 5.2.2 LTS (linuxserver/selkies image, digest-pinned) with a web desktop at
|
||||
`https://10.251.50.54:3001` (vault `fv-ml1/blender-web-password`). **On demand only:**
|
||||
`scripts/blender-mcp up|down|status`. It was left DOWN (GPU 3 back to 2 MiB).
|
||||
- MCP: `mcp-for-blender` 2.1.1 (Dvalin's research pick). The add-on is vendored at `41a18432`. The
|
||||
server runs INSIDE the container, and agents reach it as ssh+docker-exec stdio via
|
||||
`scripts/blender-mcp`. No port is published, and screenshots work because server and Blender
|
||||
share a filesystem. Telemetry is off and safe mode is on. Verified end to end (render, screenshot,
|
||||
safe-mode refusal). **Batch path: `scripts/blender-run`** (one-shot `docker run --rm`, `--job` staging; first user is draupnir). **Access: the shared fleet `infra-ops` login, with no render-only key (Prime, 2026-09-28).** **Registration + up/down: PER WORKING SESSION (Prime, 2026-09-28, relayed by draupnir; superseded per-task of 09-27)**: `blender-mcp up` + `claude mcp add` at session start, `claude mcp remove` + `down` at its end. Never always-on, never user- or project-wide config.
|
||||
→ `stacks/blender/README.md`
|
||||
- **Extensions (2026-09-28, draupnir; Prime ruled Blender a MANDATORY pipeline stage):** 8 pinned
|
||||
add-ons (`stacks/blender/extensions.lock`) built by `scripts/blender-extensions sync` into
|
||||
`fv-ml1:/tank/blender-extensions/5.2/system` (LIVE), mounted read-only as the System repo;
|
||||
`fleet_extensions.py` enables them (GUI startup timer; `blender-run --extensions`). Headless
|
||||
acceptance 8/9 (CAD Sketcher sketching is GUI-only); **MCP 9/9 after Prime ran the deploy himself
|
||||
(1505; the classifier had refused mine)**. SurfacePsycho's eval() is patched to literal_eval
|
||||
(held under MCP). Open: agent-drawn CAD Sketcher geometry (its stateful ops want point picks),
|
||||
and MeasureIt overlays not seen in MCP screenshots. GUI left DOWN.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August).** → `servers/esh-docker-vm/README.md`
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only.** Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → `persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md` (canonical: `stacks/zigbee2mqtt/README.md`)
|
||||
|
||||
# `[2026-09-27]` Zigbee2MQTT on esh-docker-vm (Prime go-ahead; ha-dev request)
|
||||
|
||||
(Moved out of the in-flight section at the 2026-10-06 snapshot. Canonical: `stacks/zigbee2mqtt/README.md`.)
|
||||
|
||||
- **LIVE since 1240:** Z2M 2.14.1 (digest-pinned) on `http://10.0.50.45:8099` (auth token), radio
|
||||
SLZB-MR1U chip 0 `tcp://10.0.90.10:6638` (ember), channel 25, **PAN 0xCFF4**. State and the network
|
||||
key are in `/opt/docker/data/zigbee2mqtt` (root 0700, restic via /opt/docker, never in git). Vault:
|
||||
`esh-docker-vm/zigbee2mqtt-{network-key,frontend-token,mqtt-password}`. Broker user `zigbee2mqtt`
|
||||
added to mosquitto (passwd backup `.bak-20260927-z2m`).
|
||||
- **ha-dev deleted the ZHA entry**; pairing the Aqara T1 is theirs. ⚠ The "bridge in HA" acceptance first
|
||||
FAILED because HA had had no MQTT since 2026-09-25 06:13. Cause: the esh-docker-vm macvlan shim had no
|
||||
host route to HA (two equal /24s, ens18 wins). Fixed at 1249 with a /32 via the shim (if-up.d hook,
|
||||
`playbooks/esh-docker-vm-macvlan-shim-route.yaml`), and HA reconnected. My first report called HA
|
||||
"connected" from a client count that included my own probe; `$SYS` counts are not identities.
|
||||
- ⚠ The first credential-mint attempt was blocked by the auto-mode classifier on a peer-relayed
|
||||
approval. It went ahead only on Prime's own go-ahead in this session. Treat that as correct.
|
||||
→ `stacks/zigbee2mqtt/README.md`
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **semif-serve 0.1.4: object states ending in `)`, `;` or `}` no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart.** Prime ruled; heid bug hunt folded. → `stacks/semif/README.md`
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win.** Build nothing (Prime). Henge 88 carries it. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
|
||||
|
||||
# SemIf consumer-fit spikes: Wyrd scene change, Cicada affect gate (2026-09-27, Prime)
|
||||
|
||||
Prime, 0904: "Two semif spikes -- Wyrd has a couple flags per turn on whether the scene should
|
||||
change, check the codebase, see if semif is a clean fit and then spike a couple of sample
|
||||
scenarios. likewise, cicada has a speech module and some reactions-- spike whether or not she
|
||||
should show emotion." No service change. Code and raw output: `services/semif-serve/spike/`
|
||||
(`consumer_fit.py` harness; `wyrd_cases.py`, `cicada_cases.py` build the scenario files;
|
||||
`consumer-fit-2026-09-27/` holds scenarios, `.result.json` and `.log` per run).
|
||||
|
||||
## Method (applies to both)
|
||||
|
||||
- One `/decide/shared` per turn, every decision with `"orderings": "rotations"`. Target:
|
||||
semif-serve 0.1.3 on fv-ml1 GPU 1, client on nh3-dev.
|
||||
- **Noise floor:** the service is deterministic (acceptance: 144 rows twice, gap 0.0), and the
|
||||
harness asserts the same top on all 3 runs of every case. Repeats measure latency only.
|
||||
- **Null control:** every case is also asked over a content-free state. It always picks the
|
||||
"nothing happened" option, so its accuracy is that label's base rate (Cicada 16/31, Wyrd
|
||||
15/21). Evidence conditions are read against that, not against 0.
|
||||
- **Positive controls:** cases tagged `control`, unambiguous by construction.
|
||||
- **Limits, stated:** one labeller (me), authored inputs, 31 Cicada and 21 Wyrd cases per
|
||||
decision. Wilson 95% intervals are wide (30/31 → 0.84–0.99; 21/21 → 0.85–1.0), so nothing
|
||||
here resolves differences under ~10–15 points. The winning wordings were chosen on this same
|
||||
set, so their scores are optimistic until checked on held-out real play.
|
||||
|
||||
## Wyrd: scene change (fit: yes for the CHOICE, no for the WRITING)
|
||||
|
||||
How it works today (from the code): no dedicated flag. The GM's `gen` structured-output calls
|
||||
carry it two ways. **R8** (reflect): `plot_direction.target_node`, accepted only if it is an
|
||||
advanceable exit; `plot_direction.intrude` is "advisory" and never read anywhere. **Path B**:
|
||||
raw `AddStoryNode` + `AdvanceToNode` events, gated only by prompt prose ("Add + advance to a NEW
|
||||
node ONLY when the scene moves to a genuinely different place — never every turn"). Wyrd's
|
||||
archival memory already says "if it churns, tighten the prompt (or gate node-adds)".
|
||||
**Three of the real campaigns in `.wyrd-data/canon` have ZERO story edges, so R8 can never fire
|
||||
there and every scene change went through ungated path B.**
|
||||
|
||||
Fit: both are typed choices over one state, exactly semif's shape. Semif's exit options can only
|
||||
be "stay" plus the real advanceable exits, so it cannot name an illegal target. It cannot write the
|
||||
new node (title, summary, mood, image prompt) or the steering directive; the GM keeps those. So
|
||||
it would be a gate/selector in front of the GM, not a replacement.
|
||||
|
||||
Spike: 3 real seed graphs (The Fletcher, The Toll Road, The Last Vigil + one added edge), 21
|
||||
authored turns in the campaigns' register (the transcripts are gone from conv-api:
|
||||
`session_not_found`). State = current scene + player action + narration.
|
||||
|
||||
| decision | wording | correct | real moves found | controls |
|
||||
|---|---|---|---|---|
|
||||
| `place`: left this place? | r1 "has the scene moved … or does it still hold?" | 16/21 (null 15/21) | **1/6** | 4/7 |
|
||||
| `place2` | r2 location-anchored: "began at the {loc}; by the end, where are they?" | **21/21**, all unanimous, p ≥ 0.91 | 6/6 | 7/7 |
|
||||
| `exit`: which exit? | r1 "which one does this turn move the story into?" | **18/21** | 6/6 | 6/7 |
|
||||
| `exit2` | r2 "only once the narration actually gets there" | 16/21 | 1/6 | 4/7 |
|
||||
|
||||
- Round 1 `place` failed its positive controls (no better than the blind null), so the
|
||||
instrument, not the model, was broken. `place2` was written after seeing that.
|
||||
- `exit` r1 misses: two false advances on Toll-Road conversation turns (p 0.59 split, p 0.67
|
||||
unanimous), and `v-to-pub`, where my label said stay and semif chose "Outside the Sanctum". On
|
||||
re-reading, semif's pick is defensible (they do run outside first). All its correct advances
|
||||
had p ≥ 0.87 except one at 0.67. A threshold would be tuned on the test set, so none is proposed.
|
||||
- Latency: 4 decisions per turn, 133–179 ms end to end (101–141 ms server). The first request
|
||||
of two files took ~700–800 ms (n=2, cause not isolated). Either way it is in dead time and is
|
||||
far below the GM's reflect.
|
||||
- Side findings from the code read, not fixed (they are Wyrd's): `story_node.created_turn` is
|
||||
never written, so "scenes so far (in order)" is alphabetical and includes unvisited seeds;
|
||||
and a late "ready" for an abandoned node's backdrop can still replace the one on screen.
|
||||
|
||||
## Cicada: should she show emotion (a gate, not a mood ring)
|
||||
|
||||
Constraint found first: Cicada persistent-memory 2026-09-20, operator: **"Affect is emitted once
|
||||
and fanned out … nothing derives affect a second time … A classifier inferring emotion from
|
||||
output text is the mood-ring failure."** So semif does not pick her pose and never reads her
|
||||
reply. What is open: talk's `/face` loop measured the model gesturing on 8–11 of 18 mundane
|
||||
turns (target under 1 in 3), and its README names "the page dropping gestures" as the next lever,
|
||||
"a design call, not a tuning pass". The spike measures that lever: a gate on what the PERSON
|
||||
said, run beside the LLM.
|
||||
|
||||
| wording | correct | mundane → "ordinary" | earned → "react" | controls |
|
||||
|---|---|---|---|---|
|
||||
| w1: descriptive options (news, joke, surprise, distress, a turn / command, plain question, small talk) | **30/31** | 16/16 | 14/15 | 5/5 |
|
||||
| w2: "Should the assistant's face visibly react to this?" Yes/No | 19/31 | 16/16 | **3/15** | 4/5 |
|
||||
|
||||
- w2 said "ordinary" to "My dog died this morning" at p 0.93. **The wording carries the whole
|
||||
definition.** Terse yes/no options fail; descriptive ones work.
|
||||
- Adding the earlier lines of the conversation changed no scored answer.
|
||||
- The one w1 miss: "Order four hundred pounds of cheese" → ordinary. Open (unscored) cases:
|
||||
"Goodnight" → react (p 0.58), "Thanks, you're the best" → react (0.89), "STOP THE TIMER" →
|
||||
ordinary.
|
||||
- Latency ~105 ms end to end (76 ms server), under talk's ~0.4–0.6 s pose decode, so it adds
|
||||
nothing to the turn if it starts at end of utterance.
|
||||
|
||||
## Recommendations (surfaced to Prime; nothing built)
|
||||
|
||||
- **Cicada:** the cleanest fit of the two. Gate **gestures only**: on "ordinary", drop the model's
|
||||
gesture and leave its pose alone. A miss then costs one missing gesture; gating poses would
|
||||
leave her neutral on bad news. It reads input, never her output, and never picks WHICH feeling,
|
||||
but it does override the model's affect. That is Prime's ruling to make, and the code would
|
||||
live in tts-stack talk `/face`, not in cicada (affect is not on cicada's v1 list).
|
||||
- **Wyrd:** `place2` is the node-add gate the archival note asked for. R8 exit choice works but
|
||||
its wording is fragile. Wyrd's v1 is met and no churn has been observed (2–6 nodes per real
|
||||
campaign), so by the roadmap gate this is a parking-lot item with a trigger: build it if live
|
||||
play shows churn, after labelling ~50 real turns as a held-out set.
|
||||
|
||||
## Follow-up: SemIf as Cicada's mood source — FASTER? WORKS? (Prime, 0937: "build nothing … test cicada on latency grounds with clear emotional context")
|
||||
|
||||
Parked design, henge id 88: SemIf replaces the chat model's guided pose. Each call sees the earlier
|
||||
turns, each stamped with the pose SemIf chose, and the stamp drives the carry-over. Harness:
|
||||
`services/semif-serve/spike/cicada_mood_latency.py` + `cicada_mood_analyze.py`, raw data in
|
||||
`cicada-mood-2026-09-27/`. 24 emotionally clear single lines and three 5-turn arcs, 3 runs,
|
||||
interleaved. The prompt and schema are imported from tts-stack talk `app.py`, so A matches /face
|
||||
byte for byte. The chat model is char-rp-fast = G4-MeroMero-26B-A4B NVFP4A16 on fv-ml1:8021,
|
||||
**on GPU 1, the same GPU as semif**.
|
||||
|
||||
**Answer: no on both counts.**
|
||||
|
||||
| (singles, n=72 each) | first paragraph ready, direction known | vs today |
|
||||
|---|---|---|
|
||||
| A: today, guided `{pose, gesture, text}` | **246 ms** (IQR 230–263); the pose closes at 86 ms | — |
|
||||
| BA: SemIf and a text-only LLM call at once | 285 ms | **+32 ms** (95% CI +27..+41), faster in 10/72 |
|
||||
| BP: SemIf first, mood in the prompt | 352 ms | **+94 ms** (CI +85..+117), faster in 0/72 |
|
||||
|
||||
Noise floor A vs A: median |diff| 16.5 ms. The arcs agree: BP is +98 ms.
|
||||
- **Why it can't win on speed.** Dropping the pose+gesture header saves only ~31 ms (the LLM
|
||||
alone, text-only, reaches first paragraph in 215 ms vs 246 ms). The prompt is prefix-cached, so
|
||||
its 1,885-vs-909 tokens barely matter. SemIf's fast path costs 136 ms on its own. Run
|
||||
concurrently, it slows the LLM's decode on the shared GPU: SemIf goes to 196 ms and the LLM's
|
||||
first paragraph to 285 ms, and SemIf was the bottleneck in only 3/72 cases. DERIVED, not
|
||||
measured: with SemIf on a different GPU, the async version would be about max(215, 136) =
|
||||
~31 ms faster than today. That is the ceiling.
|
||||
- **Rotations are out.** 16 options with rotations is 550–1,200 ms, so every live arm used one
|
||||
ordering (pose + gesture ~110–136 ms).
|
||||
|
||||
| works (clear cases) | today (A) | SemIf (live config) |
|
||||
|---|---|---|
|
||||
| acceptable pose, singles | **66/72** (92%) | 48/72 (67%), 16/24 per case, deterministic |
|
||||
| controls | 12/12 | 12/12 |
|
||||
| egregious | 0 | 0 |
|
||||
| arcs acceptable | **40/45** | 25/45 |
|
||||
| mood carries through a mundane follow-up | **14/15** | **7/15** |
|
||||
| distinct poses used | 10 | 9 (no collapse to neutral) |
|
||||
| gesture rate, all / calm commands | 42/72 / 2/12 | **9/72** / 0/12 |
|
||||
|
||||
- SemIf's misses are systematic: hostility → annoyed, "someone's in the hallway" → curious,
|
||||
"someone's trying the back door" → suspicious, jokes → curious or neutral.
|
||||
- **The stamped previous mood did NOT carry.** In the vet arc, SemIf went concerned → neutral →
|
||||
neutral in all 3 runs ("I don't want to talk about it", "play something quiet") while the model
|
||||
held sad, sad, sad. The previous pose was in the state and it lost to the neutral-sounding line.
|
||||
- SemIf's pick agrees with the model 43% of the time, against the model's own run-to-run
|
||||
agreement of 83%, so this is a different judgment and not noise.
|
||||
- Wording grid (deterministic, singles): authored 16/24, short 14/24, plain **19/24** but it
|
||||
called "turn off the lights" sleepy. Rotations add about one case at 5–8× the cost. Null
|
||||
control (content-free state) → "content" 24/24, i.e. 5/24 by base rate, so SemIf is reading
|
||||
the evidence.
|
||||
- No vocal-sound vs pose clashes in any arm (0/117, 0/72, 0/117; crude detector).
|
||||
- The one bright spot matches the gate spike: **SemIf is far more sparing with gestures (13% vs
|
||||
58%)**, and the model is well over its own "fewer than 1 in 3" target on emotionally loaded
|
||||
lines.
|
||||
- Limits: one labeller, 24 + 15 cases, one model size (4B vs 26B-A4B), wordings not tuned.
|
||||
Tuning could lift accuracy some (plain 19/24), but it cannot create latency headroom that
|
||||
isn't there.
|
||||
- Found on the way (service, not fixed): `/decide/shared` 422s an object state whose last value
|
||||
ends in `)`, `;` or `}`. Also, rotations over long option lists cost options², and hit one cold
|
||||
503. Both are now in `stacks/semif/README.md`.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21.** Recommendations await Prime. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
|
||||
|
||||
# SemIf consumer-fit spikes: Wyrd scene change, Cicada affect gate (2026-09-27, Prime)
|
||||
|
||||
Prime, 0904: "Two semif spikes -- Wyrd has a couple flags per turn on whether the scene should
|
||||
change, check the codebase, see if semif is a clean fit and then spike a couple of sample
|
||||
scenarios. likewise, cicada has a speech module and some reactions-- spike whether or not she
|
||||
should show emotion." No service change. Code and raw output: `services/semif-serve/spike/`
|
||||
(`consumer_fit.py` harness; `wyrd_cases.py`, `cicada_cases.py` build the scenario files;
|
||||
`consumer-fit-2026-09-27/` holds scenarios, `.result.json` and `.log` per run).
|
||||
|
||||
## Method (applies to both)
|
||||
|
||||
- One `/decide/shared` per turn, every decision with `"orderings": "rotations"`. Target:
|
||||
semif-serve 0.1.3 on fv-ml1 GPU 1, client on nh3-dev.
|
||||
- **Noise floor:** the service is deterministic (acceptance: 144 rows twice, gap 0.0), and the
|
||||
harness asserts the same top on all 3 runs of every case. Repeats measure latency only.
|
||||
- **Null control:** every case is also asked over a content-free state. It always picks the
|
||||
"nothing happened" option, so its accuracy is that label's base rate (Cicada 16/31, Wyrd
|
||||
15/21). Evidence conditions are read against that, not against 0.
|
||||
- **Positive controls:** cases tagged `control`, unambiguous by construction.
|
||||
- **Limits, stated:** one labeller (me), authored inputs, 31 Cicada and 21 Wyrd cases per
|
||||
decision. Wilson 95% intervals are wide (30/31 → 0.84–0.99; 21/21 → 0.85–1.0), so nothing
|
||||
here resolves differences under ~10–15 points. The winning wordings were chosen on this same
|
||||
set, so their scores are optimistic until checked on held-out real play.
|
||||
|
||||
## Wyrd: scene change (fit: yes for the CHOICE, no for the WRITING)
|
||||
|
||||
How it works today (from the code): no dedicated flag. The GM's `gen` structured-output calls
|
||||
carry it two ways. **R8** (reflect): `plot_direction.target_node`, accepted only if it is an
|
||||
advanceable exit; `plot_direction.intrude` is "advisory" and never read anywhere. **Path B**:
|
||||
raw `AddStoryNode` + `AdvanceToNode` events, gated only by prompt prose ("Add + advance to a NEW
|
||||
node ONLY when the scene moves to a genuinely different place — never every turn"). Wyrd's
|
||||
archival memory already says "if it churns, tighten the prompt (or gate node-adds)".
|
||||
**Three of the real campaigns in `.wyrd-data/canon` have ZERO story edges, so R8 can never fire
|
||||
there and every scene change went through ungated path B.**
|
||||
|
||||
Fit: both are typed choices over one state, exactly semif's shape. Semif's exit options can only
|
||||
be "stay" plus the real advanceable exits, so it cannot name an illegal target. It cannot write the
|
||||
new node (title, summary, mood, image prompt) or the steering directive; the GM keeps those. So
|
||||
it would be a gate/selector in front of the GM, not a replacement.
|
||||
|
||||
Spike: 3 real seed graphs (The Fletcher, The Toll Road, The Last Vigil + one added edge), 21
|
||||
authored turns in the campaigns' register (the transcripts are gone from conv-api:
|
||||
`session_not_found`). State = current scene + player action + narration.
|
||||
|
||||
| decision | wording | correct | real moves found | controls |
|
||||
|---|---|---|---|---|
|
||||
| `place`: left this place? | r1 "has the scene moved … or does it still hold?" | 16/21 (null 15/21) | **1/6** | 4/7 |
|
||||
| `place2` | r2 location-anchored: "began at the {loc}; by the end, where are they?" | **21/21**, all unanimous, p ≥ 0.91 | 6/6 | 7/7 |
|
||||
| `exit`: which exit? | r1 "which one does this turn move the story into?" | **18/21** | 6/6 | 6/7 |
|
||||
| `exit2` | r2 "only once the narration actually gets there" | 16/21 | 1/6 | 4/7 |
|
||||
|
||||
- Round 1 `place` failed its positive controls (no better than the blind null), so the
|
||||
instrument, not the model, was broken. `place2` was written after seeing that.
|
||||
- `exit` r1 misses: two false advances on Toll-Road conversation turns (p 0.59 split, p 0.67
|
||||
unanimous), and `v-to-pub`, where my label said stay and semif chose "Outside the Sanctum". On
|
||||
re-reading, semif's pick is defensible (they do run outside first). All its correct advances
|
||||
had p ≥ 0.87 except one at 0.67. A threshold would be tuned on the test set, so none is proposed.
|
||||
- Latency: 4 decisions per turn, 133–179 ms end to end (101–141 ms server). The first request
|
||||
of two files took ~700–800 ms (n=2, cause not isolated). Either way it is in dead time and is
|
||||
far below the GM's reflect.
|
||||
- Side findings from the code read, not fixed (they are Wyrd's): `story_node.created_turn` is
|
||||
never written, so "scenes so far (in order)" is alphabetical and includes unvisited seeds;
|
||||
and a late "ready" for an abandoned node's backdrop can still replace the one on screen.
|
||||
|
||||
## Cicada: should she show emotion (a gate, not a mood ring)
|
||||
|
||||
Constraint found first: Cicada persistent-memory 2026-09-20, operator: **"Affect is emitted once
|
||||
and fanned out … nothing derives affect a second time … A classifier inferring emotion from
|
||||
output text is the mood-ring failure."** So semif does not pick her pose and never reads her
|
||||
reply. What is open: talk's `/face` loop measured the model gesturing on 8–11 of 18 mundane
|
||||
turns (target under 1 in 3), and its README names "the page dropping gestures" as the next lever,
|
||||
"a design call, not a tuning pass". The spike measures that lever: a gate on what the PERSON
|
||||
said, run beside the LLM.
|
||||
|
||||
| wording | correct | mundane → "ordinary" | earned → "react" | controls |
|
||||
|---|---|---|---|---|
|
||||
| w1: descriptive options (news, joke, surprise, distress, a turn / command, plain question, small talk) | **30/31** | 16/16 | 14/15 | 5/5 |
|
||||
| w2: "Should the assistant's face visibly react to this?" Yes/No | 19/31 | 16/16 | **3/15** | 4/5 |
|
||||
|
||||
- w2 said "ordinary" to "My dog died this morning" at p 0.93. **The wording carries the whole
|
||||
definition.** Terse yes/no options fail; descriptive ones work.
|
||||
- Adding the earlier lines of the conversation changed no scored answer.
|
||||
- The one w1 miss: "Order four hundred pounds of cheese" → ordinary. Open (unscored) cases:
|
||||
"Goodnight" → react (p 0.58), "Thanks, you're the best" → react (0.89), "STOP THE TIMER" →
|
||||
ordinary.
|
||||
- Latency ~105 ms end to end (76 ms server), under talk's ~0.4–0.6 s pose decode, so it adds
|
||||
nothing to the turn if it starts at end of utterance.
|
||||
|
||||
## Recommendations (surfaced to Prime; nothing built)
|
||||
|
||||
- **Cicada:** the cleanest fit of the two. Gate **gestures only**: on "ordinary", drop the model's
|
||||
gesture and leave its pose alone. A miss then costs one missing gesture; gating poses would
|
||||
leave her neutral on bad news. It reads input, never her output, and never picks WHICH feeling,
|
||||
but it does override the model's affect. That is Prime's ruling to make, and the code would
|
||||
live in tts-stack talk `/face`, not in cicada (affect is not on cicada's v1 list).
|
||||
- **Wyrd:** `place2` is the node-add gate the archival note asked for. R8 exit choice works but
|
||||
its wording is fragile. Wyrd's v1 is met and no churn has been observed (2–6 nodes per real
|
||||
campaign), so by the roadmap gate this is a parking-lot item with a trigger: build it if live
|
||||
play shows churn, after labelling ~50 real turns as a held-out set.
|
||||
|
||||
## Follow-up: SemIf as Cicada's mood source — FASTER? WORKS? (Prime, 0937: "build nothing … test cicada on latency grounds with clear emotional context")
|
||||
|
||||
Parked design, henge id 88: SemIf replaces the chat model's guided pose. Each call sees the earlier
|
||||
turns, each stamped with the pose SemIf chose, and the stamp drives the carry-over. Harness:
|
||||
`services/semif-serve/spike/cicada_mood_latency.py` + `cicada_mood_analyze.py`, raw data in
|
||||
`cicada-mood-2026-09-27/`. 24 emotionally clear single lines and three 5-turn arcs, 3 runs,
|
||||
interleaved. The prompt and schema are imported from tts-stack talk `app.py`, so A matches /face
|
||||
byte for byte. The chat model is char-rp-fast = G4-MeroMero-26B-A4B NVFP4A16 on fv-ml1:8021,
|
||||
**on GPU 1, the same GPU as semif**.
|
||||
|
||||
**Answer: no on both counts.**
|
||||
|
||||
| (singles, n=72 each) | first paragraph ready, direction known | vs today |
|
||||
|---|---|---|
|
||||
| A: today, guided `{pose, gesture, text}` | **246 ms** (IQR 230–263); the pose closes at 86 ms | — |
|
||||
| BA: SemIf and a text-only LLM call at once | 285 ms | **+32 ms** (95% CI +27..+41), faster in 10/72 |
|
||||
| BP: SemIf first, mood in the prompt | 352 ms | **+94 ms** (CI +85..+117), faster in 0/72 |
|
||||
|
||||
Noise floor A vs A: median |diff| 16.5 ms. The arcs agree: BP is +98 ms.
|
||||
- **Why it can't win on speed.** Dropping the pose+gesture header saves only ~31 ms (the LLM
|
||||
alone, text-only, reaches first paragraph in 215 ms vs 246 ms). The prompt is prefix-cached, so
|
||||
its 1,885-vs-909 tokens barely matter. SemIf's fast path costs 136 ms on its own. Run
|
||||
concurrently, it slows the LLM's decode on the shared GPU: SemIf goes to 196 ms and the LLM's
|
||||
first paragraph to 285 ms, and SemIf was the bottleneck in only 3/72 cases. DERIVED, not
|
||||
measured: with SemIf on a different GPU, the async version would be about max(215, 136) =
|
||||
~31 ms faster than today. That is the ceiling.
|
||||
- **Rotations are out.** 16 options with rotations is 550–1,200 ms, so every live arm used one
|
||||
ordering (pose + gesture ~110–136 ms).
|
||||
|
||||
| works (clear cases) | today (A) | SemIf (live config) |
|
||||
|---|---|---|
|
||||
| acceptable pose, singles | **66/72** (92%) | 48/72 (67%), 16/24 per case, deterministic |
|
||||
| controls | 12/12 | 12/12 |
|
||||
| egregious | 0 | 0 |
|
||||
| arcs acceptable | **40/45** | 25/45 |
|
||||
| mood carries through a mundane follow-up | **14/15** | **7/15** |
|
||||
| distinct poses used | 10 | 9 (no collapse to neutral) |
|
||||
| gesture rate, all / calm commands | 42/72 / 2/12 | **9/72** / 0/12 |
|
||||
|
||||
- SemIf's misses are systematic: hostility → annoyed, "someone's in the hallway" → curious,
|
||||
"someone's trying the back door" → suspicious, jokes → curious or neutral.
|
||||
- **The stamped previous mood did NOT carry.** In the vet arc, SemIf went concerned → neutral →
|
||||
neutral in all 3 runs ("I don't want to talk about it", "play something quiet") while the model
|
||||
held sad, sad, sad. The previous pose was in the state and it lost to the neutral-sounding line.
|
||||
- SemIf's pick agrees with the model 43% of the time, against the model's own run-to-run
|
||||
agreement of 83%, so this is a different judgment and not noise.
|
||||
- Wording grid (deterministic, singles): authored 16/24, short 14/24, plain **19/24** but it
|
||||
called "turn off the lights" sleepy. Rotations add about one case at 5–8× the cost. Null
|
||||
control (content-free state) → "content" 24/24, i.e. 5/24 by base rate, so SemIf is reading
|
||||
the evidence.
|
||||
- No vocal-sound vs pose clashes in any arm (0/117, 0/72, 0/117; crude detector).
|
||||
- The one bright spot matches the gate spike: **SemIf is far more sparing with gestures (13% vs
|
||||
58%)**, and the model is well over its own "fewer than 1 in 3" target on emotionally loaded
|
||||
lines.
|
||||
- Limits: one labeller, 24 + 15 cases, one model size (4B vs 26B-A4B), wordings not tuned.
|
||||
Tuning could lift accuracy some (plain 19/24), but it cannot create latency headroom that
|
||||
isn't there.
|
||||
- Found on the way (service, not fixed): `/decide/shared` 422s an object state whose last value
|
||||
ends in `)`, `;` or `}`. Also, rotations over long option lists cost options², and hit one cold
|
||||
503. Both are now in `stacks/semif/README.md`.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` — **DONE:** 0.1.3 live, with fast kernels adopted (`77b8cb4`).
|
||||
|
||||
# `[2026-09-27]` SemIf order-averaging: spiked, Prime ruled build + fast-kernel trial
|
||||
|
||||
**Finding (Prime's probes):** SemIf's single-ordering answer leans toward whichever option is listed
|
||||
first on ambiguous inputs. For "It's 2AM and I'm bored", the top pick flipped from casual 0.685 to booty
|
||||
call 0.760 when the order was reversed.
|
||||
|
||||
**Spike** (`739aa03`, `services/semif-serve/spike/`, no service change). SemIf authored144 +
|
||||
perturbations108, 252 rows in 72 groups, 3 options each, with all 6 orderings of every row sent in one
|
||||
shared request:
|
||||
|
||||
| method | accuracy |
|
||||
|---|---|
|
||||
| single ordering, as sent | 78.6% |
|
||||
| expected single ordering | 80.0% |
|
||||
| 3 rotations, log-mean | **87.7%** (+9.1 pts, group-bootstrap 95% CI +4.7..+13.8) |
|
||||
| all 6 orderings, log-mean | 88.1% |
|
||||
|
||||
Rows where the rotations agree unanimously: 161 rows, 94.4% accurate. Split rows: 91 rows, 75.8%.
|
||||
The service is deterministic, so the CI measures item sampling, not run noise. This is one task family
|
||||
(evidence interpretation, 3 options), not our workload.
|
||||
|
||||
**Latency** (`spike/latency-2026-09-27.txt`, from nh3-dev, network floor 31 ms):
|
||||
- `/decide` short: 71 ms end to end (38 ms server);
|
||||
- 3 rotations shared: 113 ms (79 ms);
|
||||
- 6 orderings: 137 ms (99 ms);
|
||||
- a ~2k-token state: 210 ms (169 ms).
|
||||
|
||||
**Prime ruled (2026-09-27 ~0250): build it in, and try the fast kernels too.**
|
||||
- Averaging design: opt-in `orderings: rotations|all` (all only ≤ 4 options); all orderings in one shared
|
||||
batch; per-ordering SemIf results returned unchanged; a `combined` block (log-mean, top, agreement,
|
||||
spread); orderings count toward `max_decisions`.
|
||||
- Fast kernels: `causal_conv1d` + `flash-linear-attention` are missing, so transformers runs Qwen3.5's
|
||||
reference PyTorch paths. Adopt them only if the authored144 parity re-check holds.
|
||||
|
||||
Tracked in the in-flight SemIf section. See [[2026-09-27-semif-live-on-fv-ml1-gpu1]].
|
||||
|
||||
**Outcome (0.1.3, `77b8cb4`, ~0340 PT):** built and deployed. Through the service on the same 252 rows,
|
||||
accuracy is 78.6% → 88.1% (95% CI +5.1..+14.3), and unanimous rows are 94.5% accurate. The fast kernels
|
||||
won the A/B on the empty GPU 3:
|
||||
- parity with upstream 144/144 (was 142/144);
|
||||
- a ~2k-token `/decide` 169 → 92 ms;
|
||||
- shared capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens (was 52/43/19/13).
|
||||
All of heid's bug-hunt findings were folded into the same release (thread `01M3H3F4RR7XBP90KQ3A39H4SX`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime).** Rotation stays Prime's. → `persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md`
|
||||
|
||||
# `[2026-09-27]` restic: repository URL out of the systemd units on all 8 hosts, and vaulted (Prime)
|
||||
|
||||
`resticprofile schedule` copies `env-file` values into the generated units under /etc/systemd/system
|
||||
(0644). So `RESTIC_REPOSITORY`, including the rest-server basic-auth password, was readable by any local
|
||||
user on every env-file host (found on esh-docker-vm; `systemctl cat` works without sudo). Blast radius:
|
||||
the password allows reading the encrypted blobs and appending to one repo, and nothing more (rest-server
|
||||
is `--append-only`, and the passphrase is a separate file).
|
||||
|
||||
**Fix** (`6e203dc`, `30f2c97`): `playbooks/restic-repository-file.yaml` does four things:
|
||||
- derives `/etc/restic/repository` (root 0400) from `restic.env`;
|
||||
- uploads the profile switched to `repository-file`, guarded by the live profile's pre-change sha;
|
||||
- runs `cat config` through the new profile;
|
||||
- re-schedules, then verifies the units exist and contain no `rest:http`.
|
||||
|
||||
Applied to ana-docker, fv-ml1 (`configs/restic/ana-ml2`), esh-docker-vm, esh-vm-db, irv-ml1, nh3-dev,
|
||||
nh3-docker, and vm-esh-nas once Prime had bootstrapped infra-ops there. esh-ml1 was built that way. An
|
||||
independent check across all 8 found 0 leaking units. nh3-docker's scheduled unit ran a real backup
|
||||
(`a29b889d`). Every host's URL and passphrase are vaulted as `<host>/etc/restic/{repository,password}`.
|
||||
|
||||
- `restic.env` is **kept** (root 0600): the per-host READMEs and the freshness probe source it. **A rotation
|
||||
must update the vault, `restic.env` and `repository`.**
|
||||
- The rotation itself stays Prime's (backups runbook, Known gaps). The move stops the ongoing exposure but
|
||||
does not un-expose what was readable.
|
||||
- Before the edit, esh-docker-vm's and irv-ml1's live profiles had drifted ahead of the repo, and were
|
||||
pulled in first (`c698751`).
|
||||
|
||||
## In-flight state at retirement (moved here at the 2026-10-06 snapshot)
|
||||
|
||||
- Seven hosts were moved from `env-file` to `repository-file` (`playbooks/restic-repository-file.yaml`),
|
||||
so the systemd units no longer carry the rest-server password. Secrets are vaulted as
|
||||
`<host>/etc/restic/{repository,password}`. `restic.env` is KEPT for manual snippets, so
|
||||
**a rotation must update the vault, `restic.env` and `repository`.**
|
||||
- **vm-esh-nas done too** (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). **All 8 hosts are clean
|
||||
and vaulted.**
|
||||
- The passwords were readable until today, so the rotation (Prime's) remains the real fix.
|
||||
- 2026-09-27 0859 freshness check: all restic and PBS repos ✅. Those 0100 runs PRE-date the move
|
||||
(done ~0140–0200).
|
||||
- **VERIFIED 2026-09-28 (infra-hermes, read-only):** the first nightly on `repository-file` (0100 PT)
|
||||
succeeded on all 8 hosts (unit Result=success AND a today-dated snapshot in each repo), and the
|
||||
0800 freshness check was all-green, PBS included. nh3-dev also passed a content restore (`af5580c3`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **infra-ops bootstrapped on vm-esh-nas by Prime** (the first account at the fleet-pinned uid/gid 850; `bootstrap-infra-ops-user.yaml` now pins 850 when free, `d775a01`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **augaman: second instance on fv-ml1 benched, then REMOVED (Prime).** esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (`ad484c3`, bench `docs/pfi/augaman-speed-bench/`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` Created empty private repo `corviduo/svoperatingsystem` (SVOS, the operating system) as claude-bot, at brokkr-smithy-dev's relay of Prime's ruling. `svos-dev` → the OS session; The High Seat's session is `highseat-dev` again.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
## Tried and abandoned (archived)
|
||||
|
||||
- `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix** → `persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md`
|
||||
|
||||
# Shorter Parakeet slices as Scriberr's memory fix
|
||||
|
||||
`[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix.** I predicted ~6× less memory from the attention-matrix arithmetic. Measured, 300 → 120 s only went 9,384 → 6,510 MiB: a ~5.6 GB fixed floor dominates. `expandable_segments:True` was the real lever (5,496). Measure the process peak; never extrapolate it from one tensor.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-30]` **Whole-file local-attention Parakeet in Scriberr (context 255/255).** OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-30]` **Start-time midpoint stitching of overlapped Parakeet chunks.** It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-30]` **int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime.** The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-30]` **GPU budgets computed as total − used.** nvidia-smi `Free` is ~640 MiB lower per card (driver reserve). Budget from `Free`.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel** → `persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md`
|
||||
|
||||
# The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel
|
||||
|
||||
`[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel.** Every tool worked except `get_viewport_screenshot`, which had Blender write a file for the SERVER to read ("Screenshot file was not created"). The server now runs inside the Blender container over ssh+docker-exec stdio. That also means no port is published. → `stacks/blender/README.md`
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **A TCP connect as the "is Blender ready" probe** → `persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md`
|
||||
|
||||
# A TCP connect as the "is Blender ready" probe
|
||||
|
||||
`[2026-09-27]` **A TCP connect as the "is Blender ready" probe.** docker-proxy accepts on a published port before the app behind it listens, so it said "answering" while Blender was still loading. The probe now asks the add-on to `ping`. The same trap applies to any service behind a published port.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** → `persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md`
|
||||
|
||||
# `uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project
|
||||
|
||||
`[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`).
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md`
|
||||
|
||||
# Relying on the build cache surviving on esh-ml1
|
||||
|
||||
`[2026-09-27]` **Relying on the build cache surviving on esh-ml1** — the augaman v0.1.3 build missed its dependency layer there (fv-ml1 hit it; reason unexplained, suspect the `docker builder prune` runs) and the rootfs peaked at 90%. Check for ≥15 GB free, or remove the old image, before building there.
|
||||
_Archived 2026-10-14._
|
||||
|
||||
Reference in New Issue
Block a user