Commit Graph
1677 Commits
Author SHA1 Message Date
vh feef8330ca ops(svos): U13 phase B — nh3-svos on v0.0.9 (schema 12), kernel app origin, seat-host and high-seat origins, bridge env on nh3-dev
Prime approved svos-dev's U13 pre-edit ask (2026-10-10 1639).
- B1: release afab123 on nh3-svos (kernel 0.0.9, migration 11->12) after a
  vzdump and an offline svos.db copy; app-terminal and app-postoffice held on
  b588510 by drop-ins until the edit.
- B2: --app-listen 127.0.0.1:8410 (drop-in) fronted by Caddy as
  svos-kernel.phasefinal.com on the LAN, allowlisted to nh3-dev; the path list
  adds /auth/kernel-key, which every connector client needs. DNS rewrite on
  the three internal AdGuards.
- B3/B4: svos-seathost (-> :8404) and svos-highseat (-> :8773) on nh3-dev's
  fleet Caddy, allowlisted to nh3-svos; /opt/svos apps env on nh3-dev with
  svos-postoffice[push] (althing-core 3.7.0 from althing 7a2a1c1) and
  svos-terminal. The fleet Caddyfile is now tracked in configs/.
2026-10-10 16:50:18 -07:00
vh 129122d9b4 memory: 10-10 gate PASS closes the filing watch; esh-nas-pve pre-pve9 snapshot destroyed 2026-10-10 07:50:50 -07:00
vh 70b7281182 ops(albok): vh joins every albok read group; macOS withdrawal-test run staged
Prime, 2026-10-09: "vh gets albok-everything". playbooks/nh3-albok-vh-viewer.yaml
joins vh to albok-read, albok-personal and albok-feedback on nh3-albok and
authorizes his four Macs' public keys for vh; verified at a fresh login that
every wing's .browse/ lists and reads.

The withdrawal harness gains a macOS driver (run-mac.sh: every mountable FUSE
backend, a read-only smoke of the real wings, results copied back), env
overrides for a non-container client, and summarize.py. Sanity-checked as vh
from the Linux container before staging.
2026-10-09 22:45:52 -07:00
vh 172d893cdf test(albok): sshfs pathname-withdrawal test for the .browse/ view passes (unit 4.3 exposure gate)
Replica of a browse view on nh3-albok, Linux sshfs 3.7.3 client in an ephemeral
container on nh3-dev, N=5 per condition. Fresh open of a never-opened child
through a cached listing: ENOENT 5/5 with sshfs defaults and 5/5 with hour-long
client caches. Null control (no rename) opens 5/5; held fd and failed-rename
positive controls read 5/5. Replica removed afterwards.

Also closes the post-clear checks in persistent-memory: fv-ml1 GPU 0/1 plateau,
nh3-nas usage, and the mislabelled sfsrv-ana pre-pbs4 item.
2026-10-09 20:23:30 -07:00
vh 9ba2ce476b memory: correct the snapshot time stamp (1955, not 1610) 2026-10-09 19:56:07 -07:00
vh d89864b5c4 memory: snapshot — 2026-10-09 SVOS LAN/voice/ac42ace/kernel 0.0.8 (fd outage fixed), U13 ingress approved+deferred, albok library, gate watch; 11 entries archived 2026-10-09 19:56:00 -07:00
vh d0358b562d feat(homepage): SVOS description reflects the LAN front door 2026-10-09 19:54:08 -07:00
vh 3bd092a1aa feat(homepage): The High Seat's description no longer says SVOS (rename-back) 2026-10-09 19:53:56 -07:00
vh 4611bb0e0a deploy(albok-service): 0.1.16 on nh3-albok (unit 4.5 slice A: the book source kind, the chapter cut, the book record)
Pinned to gitea.phasefinal.com/pfi/albok-service:0.1.16@sha256:b28559ec782f1ce93e89c5eb55c7418561667dde38ff819bb18bb53e2d1f2ddf,
deployed by albok-dev at 1943 (healthy in 20 s; compose.yaml.pre-0.1.16 beside it on the host).
2026-10-09 19:51:02 -07:00
vh 5cb610b433 docs(albok-service): old nemi/albok-dev tokens revoked after the library swap 2026-10-09 15:56:14 -07:00
vh fa934cc574 feat(albok-service): library building (fiction, nonfiction) for unit 4.5; widened Nemi and albok-dev tokens minted 2026-10-09 14:57:38 -07:00
vh 65fffe21aa memory: SVOS v0.0.9 (afab123) cut for U13; edit doc docs/operations/u13-svos-edit.md 2026-10-09 12:53:07 -07:00
vh 841cde1029 memory: --app-listen on main (68c9933); U9a release notes (kernel 0.0.9, schema 12, bridge grants, push extra) 2026-10-09 09:40:02 -07:00
vh 8a149a9719 memory: Prime approved the SVOS app-traffic ingress shape; build deferred until after snapshot 2026-10-09 09:39:09 -07:00
vh f613047ccf memory: pending Prime — SVOS app-traffic ingress shape for U13 2026-10-09 09:14:18 -07:00
vh 51da9ff5ea docs(nh3-svos): kernel 0.0.8 reader pool live; LimitNOFILE stopgap removed; soak flat 2026-10-09 05:45:14 -07:00
vh 457707c9ff memory: SVOS leak root-caused; kernel 0.0.8 (b588510) awaits Prime's push 2026-10-09 00:01:52 -07:00
vh 3591f63c54 albok-service: 0.1.15 on nh3-albok (albok 0.1.7 core patch; deployed 2026-10-08 by albok-dev under operator authorization) 2026-10-08 23:49:19 -07:00
vh 6b2de2b356 memory: SVOS kernel 0.0.7 SQLite connection leak — EMFILE outage, LimitNOFILE stopgap on nh3-svos 2026-10-08 23:25:24 -07:00
vh 00d990d485 memory: watch the nightly nh3-svos thaws for app restarts (repeats for the Type=notify fix) 2026-10-08 15:20:45 -07:00
vh d2f9f561c2 docs(nh3-svos): refreshed to SVOS ac42ace — kernel 0.0.7 + Type=notify pair, thaw race fixed (drill 2 -> 0) 2026-10-08 15:17:56 -07:00
vh b04cee5579 feat(nh3-svos): SVOS voice (app-voice) live — /voice route before Basic auth (lease bearer), unit + quiesce hook; app credentials vaulted 2026-10-08 13:55:13 -07:00
vh a3184f66b9 docs(nh3-svos): front door on mesh + LAN, internal DNS answers the LAN address; fv-ml1 off-mesh by design 2026-10-08 13:32:03 -07:00
vh 998d0dca2f feat(nh3-svos): SVOS front door also on the LAN (10.100.50.43); internal DNS points svos names at it 2026-10-08 13:31:34 -07:00
vh cab00585ca docs(nh3-svos): svos names now also rewritten on the three internal AdGuards (mesh-reachable only) 2026-10-08 13:26:53 -07:00
vh 2f587b0d92 feat(homepage): list SVOS (https://svos.phasefinal.com) under Apps, no site monitor (mesh-only, Basic auth) 2026-10-08 13:23:26 -07:00
vh 65c3c8d920 memory: SVOS kernel 0.0.7 (Type=notify) is on main — install as a pair with the next nh3-svos refresh 2026-10-08 11:22:39 -07:00
vh 10c0e342ee memory: snapshot — 2026-10-08 patch night, nh3-svos a737a06 + post office live, albok 0.1.13/0.1.14, gateway glm-5.3, heid sweep incident; 16 entries archived 2026-10-08 07:35:01 -07:00
vh 423bc946df docs(fv-ml1): patched to kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1; ZBM boot and the hand-verified driver path recorded 2026-10-08 00:42:50 -07:00
vh e741784e8d memory: 10-07 verifications closed; Prime: svos bridge handle, no z.ai vision alias 2026-10-08 00:12:03 -07:00
vh 469b3ba3cf feat(nh3-svos): SVOS a737a06 refresh — app-terminal on its own mesh origin (svos-terminal.phasefinal.com), app-knowledge, app-postoffice units; quiesce hook covers the new units 2026-10-07 23:17:18 -07:00
vh 9e972872e6 chore(albok-service): deploy 0.1.14 on nh3-albok (unit 4.3 write side + browse view, detector 2; albok-dev ask 8524) 2026-10-07 18:29:18 -07:00
vh 8a6d2f9433 feat(litellm): add glm-5.3, move gen-frontier pair to glm-5.3 (effort low/high), advertise 1M/128K on GLM 5.2/5.3 with a 128K default max_tokens 2026-10-07 17:30:39 -07:00
vh d99f3a1a8d chore(albok-service): deploy 0.1.13 on nh3-albok (unit 4.2 sensitivity gate; albok-dev ask 8413) 2026-10-07 11:38:05 -07:00
vh 7641012448 memory: Worldtree gate FAILs are the #416 ephemeral residual, not recall regressions 2026-10-07 07:49:45 -07:00
vh 3a9eca354a memory: snapshot — 2026-10-06 fv-ml1 rebalance, backup checker, Scriberr diarization, albok 0.1.12, gateway retention; in-flight compacted; 17 entries archived 2026-10-06 21:45:09 -07:00
vh 0f6c7a0912 feat(restic/irv-ml1): exclude arbo incognito uploads from the nightly backup
comfy-dev's ask for Prime's arbo 'incognito' runs: incognito uploads and chained
images are written to /worktank/comfyui/basedir/input/incognito-* and swept after
a TTL. input/ is a backup source, so one alive at 01:00 would be kept for up to
3 years. Verified on irv-ml1: resticprofile show lists the exclude, and a restic
dry-run of input/ skips an incognito-* canary while keeping a control canary.
2026-10-06 15:38:21 -07:00
vh ad939810e7 docs(albok-service): all wings migrated to generation collections; 0.1.10 rollback now needs a full reindex 2026-10-06 15:36:50 -07:00
vh 847424c3cf docs(litellm): mark store_prompts_in_spend_logs=false as a consumer guarantee (arbo incognito)
Verified 2026-10-06 for comfy-dev: the gateway keeps no request or response
content. A canary sent through 'gen' appears in none of SpendLogs' text columns,
LiteLLM's container log, any ana-docker container log, or flash-next's log
(positive controls hit on the same stores). In 24h, 0 of 281,432 spend rows
carried messages, response or proxy_server_request. Langfuse is retired, there
are no DB-stored callbacks, and LiteLLM_ErrorLogs is empty. The per-request
no-log flag skips third-party callbacks but not the proxy's own spend-log
callback (v1.97 source), so the guarantee is this setting. Comment-only change;
the live file was updated in place with no restart.
2026-10-06 15:36:09 -07:00
vh 6ec5a3060f memory: gen-large contention from jekyll; Prime moved it to sec-reasoning 2026-10-06 14:39:40 -07:00
vh efd31b4ef1 docs(scriberr): saved GPU+speaker profiles; first diarized job after the fixes 2026-10-06 13:36:31 -07:00
vh ec888c3130 fix(scriberr): model caches on /tank; HF_TOKEN is a placeholder (no real credential)
HF_HOME, TORCH_HOME and NLTK_DATA now point under /app/whisperx-env (the
/tank bind mount). Before, ~6 GB of models sat in the container layer and
every recreate threw them away. The cache was copied across (98/2/78 files,
apparent sizes identical).

HF_TOKEN is now a placeholder. Measured: with an invalid token, cached gated
pyannote models still load (huggingface_hub falls back to the cache on the 401),
and public models still download (HTTP 200). Scriberr only checks that the token
is non-empty. Re-verified on GPU 3: pyannote community-1 and WhisperX+3.1
diarization both exit 0 from the /tank cache with no downloads.
2026-10-06 13:33:17 -07:00
vh eaa32d6249 memory: mirror retention set; YaRN stays off (Prime) 2026-10-06 13:20:29 -07:00
vh 5896a7479c fix(scriberr): diarization works again: HF_TOKEN passthrough, pyannote env torchcodec pin (patch 0003); mirror retention; YaRN stays off
Scriberr: pyannote's diarization models are gated, and Scriberr had no Hugging
Face token, so every diarized job died at the diarize step (WhisperX after a full
CPU transcription). HF_TOKEN now passes through compose; the value is only in the
host .env (vault fv-ml1/hf-token). The separate pyannote env had torchcodec 0.16
(needs torch >=2.11/CUDA 13) beside torch 2.8, so pyannote 4 had no
AudioDecoder. Pinned to 0.7 by hand on the live env; patch 0003 makes it
durable at the next rebuild. WhisperX's adapter defaults to cpu/small/float32,
now documented. Both paths verified on GPU 3 with a real multi-speaker clip.

pbs-nh3: the backups-mirror prune job now keeps the primary's history
(last 3, daily 14, weekly 8, monthly 12, yearly 5). Dry-run: keep 738,
remove 3,194 across 32 groups; first run at the job's 00:00.
2026-10-06 13:20:21 -07:00
vh ef67546b9c memory: destroyed-guest backup residue pruned on pbs-ana and the pbs-nh3 mirror; mirror has no retention 2026-10-06 13:08:41 -07:00
vh e931b319a2 chore(albok-service): deploy 0.1.12 on nh3-albok (unit 4.1 bug-hunt fold; albok-dev ask 8199)
Image built from a clean archive of vh/albok 751a729 (tag albok-service/v0.1.12),
pushed as gitea.phasefinal.com/pfi/albok-service:0.1.12@sha256:a29a7e46...,
pinned by digest. Recreated between Nemi walks at 1254; healthy at 1254:48,
all four wings ready, canaries alive, write embedder agrees. Config unchanged.
Rollback: compose.yaml.pre-0.1.12 (0.1.11).
2026-10-06 12:54:55 -07:00
vh 24b7195bcd docs(backups): the PBS loop's defect triple of 2026-10-05/06 and the review rule
Three defects in one loop in ninety minutes, none visible by reading the
file, all caught by running it against known truth. The runbook's
tested-note now says what is actually tested (one positive control) and
what that implies for future edits.
2026-10-06 08:07:52 -07:00
vh e7bc70bc48 memory: backup checker fixes for destroyed guests, stdin swallow and snapshot age 2026-10-06 08:07:18 -07:00
vh d920cf3816 fix(check-backup-freshness): age PBS snapshots by their name, not the directory mtime
PBS rewrites index.json.blob inside a snapshot directory at every verify
(the next 02:30, then every 30 days per verification.cfg outdated-after 30)
and on notes/protect edits, which moves the directory mtime forward.
Measured: ana-pve/vm/101's 2026-10-04T10:00:04Z snapshot read as 2026-10-06.
A covered guest whose backups stopped would have alarmed up to a day late,
then gone quiet for ~2 days after each monthly re-verify.

The age now comes from the snapshot name (its backup start time).
Verified: the default run exits 0 with 28 fresh PBS groups and vm/101 dated
2026-10-04. Positive control BACKUP_MAX_AGE_HOURS=5 flags 10 groups by name
age, exit 1.
2026-10-06 08:07:04 -07:00
vh 0a348e548c fix(check-backup-freshness): report destroyed guests as policy residue, not STALE
A guest destroyed with 'qm destroy --purge' also drops out of the vzdump
job's exclude list, so its last snapshot silently ages into a false
🔴 once it passes the threshold (ana-pve/vm/110, destroyed 10-04, fired
this morning). Before reporting coverage, consult the owning cluster's
pvesh inventory: a guest the cluster no longer lists is reported in the
policy section as residue, never as stale. Inventory unreadable fails
loud (falls through to normal alarm).

Two traps pinned the hard way during verification:
- 'guest' is namespaced (ana-pve/vm/110); the case must match the
  segment (*/vm/*) and pre-set pveid so a foreign path shape fails the
  match, not the script, under set -u (infra-ops caught the first cut).
- ssh inside the while-read loop MUST take -n; without it ssh swallows
  the loop's remaining stdin and the entire PBS layer silently vanishes
  from the report while exit code and verdict stay green.
2026-10-06 08:04:17 -07:00