Commit Graph
183 Commits
Author SHA1 Message Date
vh d920cf3816 fix(check-backup-freshness): age PBS snapshots by their name, not the directory mtime
PBS rewrites index.json.blob inside a snapshot directory at every verify
(the next 02:30, then every 30 days per verification.cfg outdated-after 30)
and on notes/protect edits, which moves the directory mtime forward.
Measured: ana-pve/vm/101's 2026-10-04T10:00:04Z snapshot read as 2026-10-06.
A covered guest whose backups stopped would have alarmed up to a day late,
then gone quiet for ~2 days after each monthly re-verify.

The age now comes from the snapshot name (its backup start time).
Verified: the default run exits 0 with 28 fresh PBS groups and vm/101 dated
2026-10-04. Positive control BACKUP_MAX_AGE_HOURS=5 flags 10 groups by name
age, exit 1.
2026-10-06 08:07:04 -07:00
vh 0a348e548c fix(check-backup-freshness): report destroyed guests as policy residue, not STALE
A guest destroyed with 'qm destroy --purge' also drops out of the vzdump
job's exclude list, so its last snapshot silently ages into a false
🔴 once it passes the threshold (ana-pve/vm/110, destroyed 10-04, fired
this morning). Before reporting coverage, consult the owning cluster's
pvesh inventory: a guest the cluster no longer lists is reported in the
policy section as residue, never as stale. Inventory unreadable fails
loud (falls through to normal alarm).

Two traps pinned the hard way during verification:
- 'guest' is namespaced (ana-pve/vm/110); the case must match the
  segment (*/vm/*) and pre-set pveid so a foreign path shape fails the
  match, not the script, under set -u (infra-ops caught the first cut).
- ssh inside the while-read loop MUST take -n; without it ssh swallows
  the loop's remaining stdin and the entire PBS layer silently vanishes
  from the report while exit code and verdict stay green.
2026-10-06 08:04:17 -07:00
vh c35f6510bf feat(fv-ml1): rebalance GPUs 0/1 by usage; contexts to published maximums; gen-small default output cap
Layout: GPU 0 = cyberprev (KV 16 -> 15 GiB, 1.29x), erp-seat (moved from
GPU 1, KV 8.49 -> 5.5 GiB, 1.32x), voices (ctx 8,192 -> 40,960, its native
maximum, KV 6 GiB, 1.07x), coder (moved from GPU 1, ctx 16,384 -> 32,768,
the card maximum, 2.35x), parakeet-nemo. GPU 1 = gen-small (moved from
GPU 0, 5.5 GiB, 1.76x), meromero-rp (22 GiB, 1.32x), intern-decision.

Sizing rule: KV >= 1.25x max context rounded up to 0.5 GiB, checked against
7 days of LiteLLM traffic and the engines' own KV-usage logs. Each card keeps
spare memory for its bursty tenant (parakeet's cap on GPU 0, intern-decision's
32k call on GPU 1). YaRN extensions are not enabled: they do not fit, and the
Qwen cards warn static YaRN degrades short prompts.

gen-small gets --override-generation-config max_new_tokens 32768. It applies
only to requests that omit max_tokens (vLLM get_max_tokens) and answers 9
runaway ~260k-token replies in 7 days; the longest normal reply was 6,357.

Applied with playbooks/fv-ml1-rebalance-20261005.yaml (boot-check guard and
health/KV verification per seat); all nine seats healthy, gateway smoke
tests pass.
2026-10-05 22:51:17 -07:00
vh 1210b4a0d5 feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2
Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
2026-10-05 20:22:16 -07:00
vh c9ace41340 feat(check-backup-freshness): attach the documented reason to each policy exclusion
The policy section said WHICH guests are uncovered but not WHY, so every
reader had to reconstruct the ruling from memory or the bus. A missing
reason now prints as 'NO DOCUMENTED REASON', which turns an undocumented
new exclusion into a visible flag instead of a silent line.
2026-10-05 08:03:47 -07:00
vh af2f77a9c6 fix(oneoff): fv-ml1 docker move run 2 — one-filesystem rsync, skip overlay merged views, path-list verify, auto-restore before swap 2026-10-05 02:30:24 -07:00
vh bf46beaa96 docs(runbooks): fleet guest patch campaign 2026-10-05 (park 100); fv-ml1 docker-to-tank one-off 2026-10-05 01:50:55 -07:00
vh d3f59aa2c5 feat(albok): move albok-service to a dedicated VM, nh3-albok (VM 110 on nh3-pve-2)
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.

- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
  pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
  (pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
  x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
  infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
  mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
  until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
2026-10-04 23:33:24 -07:00
vh 513f6ef38f ops: phasefinal.com moved to Cloudflare Pages; pfi-ana-webhost destroyed and removed from inventory, DNS and the FortiGate 2026-10-04 17:28:29 -07:00
vh d0c1bc628a ops(pbs): pbs-nh3 + pbs-ana upgraded to PBS 4.2.7 / Debian 13; guest-agent driver script, plan and log 2026-10-04 11:44:57 -07:00
vh ad0322bf83 ops(meshcentral): diagnose and fix HW Connect stuck at Setup (stale CIRA tunnel); add relay probe 2026-10-02 22:49:27 -07:00
vh 9bcb9417fb feat(amt): amt-cira-setup.py — configure Intel AMT phone-home (CIRA) to MeshCentral over WS-Man
MeshCentral only pushes CIRA through an on-host agent, so this mirrors its
amtmanager.js sequence directly over the LAN: trust MeshCentral's root,
add the MPS (username = 16-char meshid prefix), a periodic policy,
BIOS+OS user-initiated connections and a random environment-detection
domain. Idempotent, with a read-only state report. Enumerate uses
Enumerate+Pull (AMT 16 ignores OptimizeEnumeration; caught by a
positive control that first read 0 instances of a class that has one).

Applied to nh3-pve's AMT 2026-10-02 alongside MeshCentral mpsPass and an
ana-gw VIP/policy for 4433; the AMT does not dial out yet (static IP).
2026-10-02 08:50:47 -07:00
vh cd30592e55 fix(wt-memory-gate-batch): pass --legacy-mode only when the deployed harness has it
b193 (U11b) retired the legacy plane and removed --legacy-mode from the
harness; passing it there fails the daily batch. The wrapper now greps
core/memory_acceptance at the deployed sha and omits the flag when it is
gone, so it is correct on both sides of the b192 -> b193 deploy (checked:
7a83f2f -> passes it, c2d87263 -> omits it).
2026-10-02 07:59:03 -07:00
vh 59fb6b77a8 ops(nh3-dev): seat healthz watchdog — the 5-day pull-only outage must page, not age
althing-seat-daemon took an outside SIGTERM on 2026-09-27 and exited 0, so
Restart=on-failure never revived it; hermes-gateway sat pull-only for five
days with a yellow verdict nobody was watching. Fixed the death mode with a
Restart=always drop-in (also on the jekyll twin, same latent bug) and added
this watchdog for every other death mode: 5-min timer, alert on the second
consecutive non-200 (~10 min sustained), re-alert at most hourly, recovery
mail on green, exit 2 distinguishes a broken alarm wire from a down seat
(backup-freshness precedent). Lifecycle exercised end-to-end against a dead
port before install.
2026-10-02 07:47:17 -07:00
vh 902e16630f feat(dev-backup): add daily and weekly retention (48 hourly + 30 daily + 12 weekly)
Prime's ruling 2026-10-02. retention.py picks the snapshots to delete:
the newest 48, plus the newest of each of the last 30 days and of each of
the last 12 ISO weeks, counting only days and weeks that have snapshots.
Names that are not exactly YYYY-MM-DD_HHMM are never selected, and the
NAS side refuses any path outside that pattern. The unit fails unless
the number kept equals the number expected. Live run: deleted 1, 0
errors, 48 kept as expected.
2026-10-02 07:42:45 -07:00
vh f44280e6b2 feat(mia): one-shot Make-It-Animatable v2 auto-rigger on fv-ml1 GPU 3 (scripts/mia-run)
Image local/mia:0.1.0 built from stacks/mia: MIA v2 @ bbd8b158 (MIT) with
its pinned submodules, dread-dev's proven Python lock with the torch family
swapped to cu129, and a driver adapted from dread-dev's run_mia.py that
seeds every mesh (fix_random + trimesh's module RNG) and writes
weights_effective into the npz. Weights stay in fv-ml1's shared HF cache
at pinned revisions, mounted read-only.

scripts/mia-run mirrors blender-run: --job DIR is shipped to
fv-ml1:/tank/mia/jobs, one docker run --rm rigs every mesh, out/ comes back.

Acceptance on the four Dread Naught characters: 3.9-4.7 s a mesh (median
of 3) plus 12.7 s model load, peak 3,394 MiB; seeded runs bit-identical
across rotated mesh order; GPU-vs-CPU distances the same size as sampling
noise, with an unseeded GPU run as the positive control.
2026-10-01 09:45:22 -07:00
vh de11e00dfb fix(dev-backup): chmod before prune so retention actually deletes; fail the unit on a bad prune
rsync -a copies a read-only source dir (0555) as read-only, so the hourly
prune's rm -rf could not unlink inside it. From 2026-07-18 every pruned
snapshot was left as a 22-entry husk while the run still logged OK. The
1,740 husks on nh3-nas were removed (0 errors; 48 full snapshots kept).

The prune now runs chmod -R u+w before rm -rf, logs its error count and
the number of snapshots kept, and exits 3 on a failed prune (exit 1 on a
failed rsync) so the systemd unit shows failed instead of passing.
2026-10-01 05:59:07 -07:00
vh 8c68bacf2e scriberr: carry patch 0002 (gap retry + PARAKEET_MODEL_PATH), live as dropout2
Prime: Scriberr gets the basic fix, v3 stays (no NeMo 3.0.0 surgery). 0002 moves
from proposed/ into the carried set; scriberr-rebuild now applies 0001+0002 by
default (suffix dropout2) and its memory budget becomes a 5,600 MiB regression
guard (Scriberr is on GPU 3). Live on fv-ml1 1602: scripts rewritten from the
patched embed, a 20-min file at 5,502 MiB with retried_gaps reported.
2026-09-30 16:03:07 -07:00
vh f65e27b08f feat(intern-decision): persist the triton autotune cache across recreates
The ~6.5-9 s first-call-per-bucket autotune lived in the container's writable
layer and died on every recreate. 0.1.3 creates /tmp/triton-cache in the image
owned by 10001 so the named volume intern-decision_triton-cache inherits a
writable mount point, and compose mounts it.

Bucket model PROVEN, not inferred: 2,048-token buckets, 16 up to 32,768. After
one warmed call per bucket, 12 random sizes across 8k-32k were all warm (worst
2.09 s); cold entries cost 6.5-9 s. Full cold warm-up 109 s; warm re-run 17 s.

scripts/intern-decision-warmup: one noul call per bucket, MAX_TOKENS from
/health, two-point live calibration of the tokenizer's linear token model (a
single probe overcorrects and the aim oscillates around the bucket edge),
per-bucket wall times, non-zero exit on a missed bucket. Run it after an IMAGE
CHANGE only; the volume carries ordinary recreates (measured: force-recreate,
then a warmed 32k call answered in 2.11 s).

Acceptance on 0.1.3: JevBench 202/231, hard 83/111, 0 diffs / 924; warm 32k GPU
1 peak 15,218 MiB (budget 15,220; a COLD autotune touched 15,224 once, README
caveat); /decide answers. Artifacts in the acceptance dir.
2026-09-30 16:02:16 -07:00
vh 38015a1977 docs(scriberr): Parakeet dropout investigation; proposed 0002 (gap retry + model path)
Prime's ask (via the coordinator): investigate the "Parakeet skips
stretches of speech" finding, including other Parakeet weights.
Investigation only; nothing deployed.

Against ground truth (official SCOTUS transcript, Gutenberg #38916) the
drops are real: production v3 loses 140 / 66 clean words per transcript on
the two public files and ~50 on each private one (Whisper-referenced,
Canary-confirmed; adjudicator 129/129 correct on the calibration). Cause:
the v2/v3 0.6B weights collapse deep inside long full-attention windows;
the encoder output is degraded, the audio alone transcribes fine, and
1.1B TDT/RNNT/CTC and CTC-0.6B never do it. Decoding (CUDA graphs, greedy
variants, max_symbols, beam), slice length, local attention, loudness,
resampling and a noise floor do not fix it. Controls: A-vs-A, silence
positive control (>=15 words 36/36), null control, bootstrap floor.

Proposed patch 0002 re-transcribes >=3 s stretches where the audio holds
speech but no word came out (-80 to -90 % lost words on all four
recordings, lower WER, no invented text, +10 MiB) and adds an explicit
PARAKEET_MODEL_PATH with the loaded model recorded in JSON and ModelUsed.
Reviewed at high effort, all findings fixed; built and tested as
scriberr:local-blackwell-a353078-dropout2, not deployed.

scriberr-rebuild: --patches takes DIR[:DIR...]; embeds and seam-checks
both Parakeet scripts (seam-check --standard for the short-audio one).
2026-09-30 15:48:02 -07:00
vh 92501a29c1 fix(scriberr-rebuild): memory stage works on a shared GPU 3
Scriberr moved to fv-ml1 GPU 3 on 2026-09-30, so the card is no longer idle when the rebuild runs. The memory stage now requires >= 20 GB free instead of an idle card (GPUs 0-2 still fail that) and attributes the peak only to the host PIDs of its own container, captured with docker top alongside the 0.2 s nvidia-smi samples. Verified with five processes on the card: peak 5,496 MiB, identical to the exclusive-card figure.
2026-09-30 13:30:29 -07:00
vh ee3db68db1 feat(scriberr): overlap-and-stitch Parakeet slicer patch, rebuild script, bench
Carry patches/0001 on our Scriberr build (upstream a353078): adjacent
buffered chunks overlap by 4 s inside --chunk-len and hand over at a word
both chunks transcribed alike, instead of cutting at fixed marks with no
overlap. Pause-aware cutting is included as an opt-in (--pause-search);
it measured neutral once the stitch was right. The Go<->Python CLI and
JSON seam is unchanged.

Bench (4 recordings, 118 min, 3 cut placements each, against a no-cut
whole-file reference; metrics only, private audio stays on fv-ml1):
cuts with an error within +-3 s fall from 52% (93/179) to 22% (41/184)
against a 19% background; floor +-0.08. Positive control: upstream's
cutter +0.33 over background. A-vs-A byte-identical in-process and
across CLI processes. Peak GPU memory unchanged at 5,496 MiB (n=3).
Also found: Parakeet skips runs of >=10 words mid-chunk with any
slicer, upstream's included; not addressed here.

scripts/scriberr-rebuild clones a pinned upstream sha into a new
/opt/docker/src dir, git-apply-checks the patches, builds a distinct
tag, and checks embed, unit tests, the JSON seam (scriberr-seam-check.py)
and the memory budget on idle GPU 3. Deploy stays manual. The upstream
PR is prepared under patches/upstream-pr/ and not opened.
2026-09-30 12:10:32 -07:00
vh cc211ef10c scripts: wt-h2-count.py — worldtree-dev's U11b H2 gate check (verbatim); rehearsed on copies, 0 on both 2026-09-30 09:07:50 -07:00
vh aee8de3395 wt-memory-gate-batch: busy is exit 4 (retry later), and PASS is reported too
Both verdicts now build the n for the U11b data-deletion gate (three
consecutive PASS batches at off). First real run 20260930T090608Z via the
detached-worktree path: PASS, user median 0.83.
2026-09-30 02:24:05 -07:00
vh 5186b3dc9a scripts: wt-memory-gate-batch — one Worldtree U8 gate batch against demo's deployed sha
For infra-hermes's daily batch through the U11 off window (operator ruling
2026-09-29 2340; mode revised to off 2026-09-30). Enforces worldtree-dev's
terms: flock plus a refusal while any harness batch runs, code under test =
demo's deployed sha (a detached worktree when the main tree has moved; refuses
on pyproject/uv.lock/packages drift the shared venv cannot honour), no git
commit, no retry. Exit 0 PASS / 1 FAIL / 2 harness error / 3 refused.
2026-09-30 01:26:46 -07:00
vh fd20183cbb feat(blender): extension set live in the GUI; MCP acceptance 9/9, probe made safe-mode compliant 2026-09-28 15:08:22 -07:00
vh a51188c76a feat(blender-run): --cpu runs with no GPU attached (modelling/IO jobs stay off GPU 3) 2026-09-28 14:23:12 -07:00
vh 5e6a6f3b26 feat(blender-run): fixed container hostname fv-ml1-blender, so gethostname() is stable provenance 2026-09-28 14:00:07 -07:00
vh 83dc497b40 feat(blender): pinned extension set in a read-only System repo, for the GUI and blender-run --extensions
Blender is now a mandatory stage in draupnir's pipeline (Prime, 2026-09-28), and draupnir asked
for eight add-ons from extensions.blender.org: SurfacePsycho 0.10.4, CAD Sketcher 0.32.1,
3D-Print Toolbox 1.4.1, STEP Importer 1.2.1, Bool Tool 2.1.0, LoopTools 4.7.7, MeasureIt 1.8.4,
3MF Import/Export 2.7.7.

- stacks/blender/extensions.lock pins each by version and archive sha256.
- scripts/blender-extensions sync builds fv-ml1:/tank/blender-extensions/5.2/system with Blender's
  own install-file, pre-warms and byte-compiles it, checks a read-only enable, then swaps it in.
  It refuses while the GUI or a blender-run job holds the old directory.
- conf/scripts/startup/fleet_extensions.py enables every package in the System repo: in a timer
  in the GUI (after the prefs load), and as --python ahead of the caller's args in
  blender-run --extensions (a failed enable exits 1 before the caller's script).
- It also patches SurfacePsycho's sp_overwrite_segment_selection from eval() to literal_eval():
  the eval walked past MCP safe mode (control: unpatched ran code, patched refuses).
- blender-run: --extensions (bind mounts via --mount so a missing source fails instead of being
  created); USER/LOGNAME set, which CAD Sketcher's getpass needs.
- compose.yaml mounts the repo read-only and the hook into the GUI container. NOT yet deployed.
- scripts/blender-probes/extensions_acceptance.py: one operator run per add-on, safe-mode
  compliant. Headless 8/9 online and with --network none; CAD Sketcher sketching is GUI-only.
  A Python audit hook saw no network/process events (positive control fired).
2026-09-28 12:50:57 -07:00
vh 46276611a7 fix(blender-run): unique remote job dir per local path, mirrored with --delete
draupnir review: the remote job dir was keyed on the basename alone, so two
local dirs with the same name shared one remote dir, and a rerun inherited
stale files. The dir is now <basename>-<8 hex of sha256(abs path)>, and it is
mirrored with rsync --delete, confined to that one directory. There is no rm on
a computed path. Verified: a file deleted locally is gone from the rerun's
remote dir. The /defaults stderr line is documented as harmless noise.
2026-09-28 08:43:55 -07:00
vh d0f68a3b18 feat(blender): blender-run one-shot headless wrapper + FLEETTOOLS entry
scripts/blender-run launches each call as a docker run --rm of the Blender
image on fv-ml1 GPU 3, capped at 64g / 48 CPUs. It needs no desktop and does
not affect the GUI container's lifecycle. --job DIR stages a local directory
to /tank/blender/jobs/<name>/, runs Blender with that as the cwd, and copies
results back. It always passes --python-exit-code 1, because Blender otherwise
exits 0 when a --python script raises (measured).

Tested headless: Cycles GPU and CPU, EEVEE via EGL, Workbench, an STL
round-trip, and exit codes (3, 7 and 1 pass through). There is no STEP
importer. Written for draupnir's design work, and indexed in FLEETTOOLS with a
detail file.
2026-09-28 08:39:43 -07:00
vh ac1cd29afa feat(blender): agent control via mcp-for-blender (in-container, ssh stdio)
The MCP server (mcp-for-blender 2.1.1, frozen requirements) runs inside the Blender
container. Its add-on is vendored at upstream 41a18432 (MIT) and started by a
startup hook. scripts/blender-mcp carries the stdio over ssh + docker exec, so the
add-on socket, which runs arbitrary Python with no auth, stays on the container's
localhost with no published port. It also runs there because viewport screenshots
need a filesystem shared by server and Blender. Telemetry is off and safe mode is
on. The hook also defaults Cycles to OptiX on GPU 3, because safe mode forbids
agents from touching preferences.

Verified end to end from nh3-dev: 36 tools; a GPU render of an agent-built scene;
a viewport screenshot; and safe mode refusing 'import os'. Blender left down
(on demand).
2026-09-27 14:34:07 -07:00
vh d8f59a15d9 feat(esh-ml1): restic backup of augaman's gallery; augaman v0.1.2
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only
through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the
same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned,
sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily
0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC,
so both schedules name the zone explicitly.

pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure,
including a stopped container, aborts the run. Tested with a stub docker that
exits 1: the run returned 1, and neither the snapshot count nor last-success
moved. The restore was verified at identity level against augaman-dev's
public-domain canary (snapshot fd3061a1: the restored copy's digest over
identities and samples matches the live gallery). That meets the operator gate
for real enrollments.

The repository URL is read through repository-file rather than restic.env.
resticprofile schedule copies env-file values into world-readable systemd
units, which publishes the rest-server password on the env-file hosts
(observed on esh-docker-vm). This is recorded in the backups runbook under
Known gaps, and the playbook verifies no generated unit contains the URL.

esh-ml1 is added to the freshness check's expected ana-side repos and to the
runbook tables.

augaman moves to v0.1.2 (dependency layer keyed on the lock without the
project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the
card, and the canary survived the container recreate.
2026-09-27 00:06:33 -07:00
vh 50f113cd7d feat(nh3-pve): AMT static on nh3-mgmt with KVM enabled and opt-in none; WS-Man helper
- AMT re-IP'd over WS-Man to static 10.100.250.61/24 (gw/DNS .1): AMT keeps
  its old DHCP lease across a VLAN move, so it was unreachable until changed
  (reached via a temporary on-link /32 on vmbr0 + ssh tunnel, since removed).
- KVM redirection enabled, redirection listener on, IPS_OptInService
  OptInRequired=0; all read back. Password vaulted as nh3-pve/amt-admin.
- scripts/amt-wsman.py: stdlib WS-Man get/put/invoke client (digest auth,
  AMT 16 legacy-renegotiation TLS), password from $AMT_PW.
2026-09-25 23:08:53 -07:00
vh e6da607767 chore(task-board): mothball it; superseded by the High Seat and ledger
Operator ruling 2026-09-24. On ana-docker the stack is `docker compose
down`: the container is removed and port 7878 is closed. Kept for revival:
- the data dir /opt/docker/conf/task-board/data (tasks.db, last written
  2026-09-11)
- the task-board:local image
- stacks/task-board/ and the host's compose + .env

The Uptime Kuma monitor (id 3) was deleted before the stop so it could
not page, and its row is removed from monitors.yaml. Homepage drops the
card on its own, since it reads the container's labels.

Hooks: the container log showed no hook POSTs in 30 days. The only
traffic was open browser tabs holding /events, and the plugin was already
uninstalled on nh3-dev. Removed the paragraph that told sessions to call
task_* tools (CLAUDE.md, and the fork-fleet.sh template that seeds new
repos), the vestigial TASK_BOARD_SESSION env in .claude/settings.json, and
the listings in README and FLEETTOOLS.
2026-09-24 09:22:50 -07:00
vh 466f7aa4e6 fix(elway): sudo uploads land root:root, validated and staged; add fleet ownership audit
elway's sudo upload did scp-as-user then `sudo mv`, and mv keeps the owner,
so every file it installed "as root" (systemd units, /etc configs, root-run
scripts) ended up owned by the SSH user. A sudoers drop-in installed that
way would be rejected by sudo outright.

elway:
- Sudo uploads now chown to root:root by default. Playbooks can override
  with `upload.owner:` and ad-hoc runs with `--owner`. An owner is refused
  on a non-sudo upload, and `--owner` outside an ad-hoc `--upload` is an
  error rather than silently ignored.
- Ownership and mode are applied to the STAGED file, then a single mv
  publishes it, so a failed chown can no longer leave the live path owned
  by the SSH user. chown runs before chmod so setuid bits survive. A trap
  removes the staged file on every exit path.
- A directory dest is refused before anything moves. mv would otherwise
  drop the file inside the directory under a staging name.
- `mode` was spliced unquoted into the remote root shell line. It is now
  validated as octal (^[0-7]{3,4}$) and shell-quoted, on both the sudo
  and non-sudo paths.
- `mode` and `owner` must be quoted YAML strings. YAML 1.1 turns a bare
  0644 into 420, a bare 1000:0 into 60000 and a bare yes into True, and
  str() of each is a different value that looks valid. All 85 playbooks
  already quote them.
- New preflight(): every step is resolved before any remote action. An
  undefined var or a templated owner that resolves badly is now refused
  up front, not after earlier steps have already changed the host.

scripts/fleet-ownership-audit.sh (new, read-only) finds files in root's
territory owned by a normal user. Tier A lists /etc, /usr/local, /root and
cron, judging symlinks by their target. Tier X covers files that root-run
units exec from /opt, /srv, /home or /volume1, reading drop-ins and quoted
paths. Tier B only summarises /opt app trees. A host that is unprivileged,
whose find fails, or whose run dies is reported INCOMPLETE (exit 5), never
clean. Completion is marked with a per-run nonce.

Verification: 24 unit tests in scripts/test_elway.py. Live checks on
esh-docker-vm: default lands root:root; the override lands infra-ops:adm at
4755 with setuid intact; a bad group fails with dest untouched and no
staging left; a directory dest is refused; a bad mode is refused before
any copy. Audit positive controls on nh3-dev (a drop-in-only root Exec, a
quoted path containing a space, a symlink to a user-owned target) were all
flagged; esh-docker-vm negative control was 0. Probes removed. Cross-model
bug-hunt (heid, Gróa arm + seat) findings folded.
2026-09-23 09:59:15 -07:00
vh 4979869731 feat(backups): discover restic repos instead of enumerating them
Adding nh3-dev to the host list fixed the instance. This fixes the class, on
svos-dev's framing: a hand-maintained list of things to watch, sitting beside a
NAS that already knows which repos exist, means the next repo added is
unwatched BY DEFAULT and nothing says so. The list of what to check can
silently disagree with the set of what exists -- the same shape as every other
instrument fault found this day, only slower-acting.

The check now asks each NAS. A directory is a repository when it has a
snapshots/ child, which cleanly separates real repos from container dirs
(/volume1/Backup/restic/repo/ holds ana|esh|nh3 namespaces and no snapshots of
its own -- verified rather than assumed before building discovery on the
layout).

The hand-written list survives DEMOTED to an EXPECTED set, used only to report
a repo that has VANISHED. Two facts that would otherwise both read as silence
stay distinct:
    "a repo exists that nobody watches"  -> impossible now, it is discovered
    "a repo we expected is gone"         -> EXPECTED REPO NOT FOUND

Preventive, not corrective: all 8 repos currently discovered are already in the
expected sets, so this found no live gap. It removes the possibility of the
next one.

Controls run, since a check only ever seen passing is untested: a bogus
expected repo reports EXPECTED REPO NOT FOUND and turns the verdict STALE;
unchanged expectations still report all-fresh; all 8 repos report their age.

Observed while testing, not a fault: restic/ana/esh-docker-vm is 36h old
against 12h for every other repo. Inside the 48h threshold so correctly green,
but it is a day behind the fleet and worth a look.
2026-09-22 13:36:57 -07:00
vh ba60fda16c feat(backups): assert restic CONTENT, and add the host that was never checked
Operator ruling 2026-09-22 (relayed via svos-dev): have the freshness check
assert snapshot content rather than REST-server reachability, so the green
light is a statement about DATA rather than about a daemon.

⚠ FIRST, A CORRECTION TO MY OWN REPORT. I ran `grep -ic restic` against
backup-freshness-alert.sh -- the WRAPPER -- got 0, and told the operator that
nothing on the fleet verified restic. Wrong. check-backup-freshness.sh has
always checked restic snapshot ages, for seven hosts. I grepped one file of a
two-file tool and generalised, which is the identical error a peer had just
made with dev-backup, made while correcting them.

THE REAL GAP was narrower and still real: the NH3 host list read
`irv-ml1 nh3-docker` and omitted nh3-dev -- the repo holding every Claude Code
session transcript, althing routes, hermes run history and Miranda's
conversation. /volume1/Backup/restic/nh3-dev/ has always existed and always
been written nightly; it was simply never enumerated, so a total failure of the
one repo nobody could reconstruct would have gone unreported indefinitely.

THE CONTENT ASSERTION, built on svos-dev's ladder (list proves the repo
answers; ls proves the index; check proves structure; RESTORE proves the bytes
come back):
  - CONJUNCTIVE: latest snapshot newer than MAX_AGE_H AND the probe path
    present AND a restore returns non-zero bytes. "A snapshot exists containing
    X" is satisfied by a three-month-old one; "the latest is recent" is
    satisfied by an empty one. Age alone was the old problem; content alone is
    the same problem rotated.
  - IT RESTORES. Metadata is what survives the failures worth fearing -- a
    pruned or partially-written repo can list a path whose blobs are gone.
    ~493KB from ~/.local/state/svos, sub-second. Large paths are not restored:
    a repo that returns one file will return others.
  - IDENTITY: the timer runs as lkraven, which has no NOPASSWD sudo on nh3-dev,
    so the probe hops through infra-ops@localhost. /etc/restic is root-only by
    design.

⚠ restic ls ALWAYS PRINTS A HEADER LINE, matched or not. A path absent from the
repo returns 1 line; a real one returned 6. Counting with `grep -c .` read the
header as a hit, so the ABSENT case fell through and reported "blobs gone" --
telling an operator the repository was corrupt when the truth was a mistyped
path. Now `grep -c '^/'`.

TESTED BY MAKING IT FAIL, because a check only ever seen passing is untested:
bogus probe path -> "absent from snapshot"; BACKUP_MAX_AGE_HOURS=1 -> "12h old
(>1h)"; healthy -> snapshot id, age, entries, bytes restored.
2026-09-22 13:33:58 -07:00
vh 94899d6fa3 feat(uptimekuma): normalize names off Homepage, publish the status page, restore the widget
NAMES. Homepage already answers "what is this service called", so the monitor
name is now that name verbatim -- a second naming authority is how drift starts,
and an alert reading "[Uptime Kuma] Beszel hub is DOWN" sends you hunting for a
card that does not exist. Only two rows moved (Beszel hub -> Beszel, Dozzle hub
-> Dozzle); the " hub" suffixes were mine, not the services'.

The remaining mixed case is deliberate and is now documented as such. talk, vor
and task-board are lowercase on Homepage and in their own repos; title-casing
them here would make this board disagree with both. What actually looked messy
was scripts/kuma's own ASCII-ordinal sort, which buried every lowercase name
below every capitalised one. Fixed to case-insensitive.

⚠ RENAME SAFETY, which this pass needed and did not have. The seed keys on NAME,
so editing a name would have read as a brand-new monitor: added fresh, with the
old row orphaned, still checking, still alerting, and holding all the history.
`rename_from:` names the old row for one run. Verified: both renamed monitors
kept their IDs and all 67 heartbeats.

Added with it, an orphan warning for any row on the board the spec no longer
names -- because a forgotten monitor keeps paging. Its first cut diffed against
the PRE-EDIT snapshot and so cried wolf on its own successful renames; it
re-reads the board now. A warning that fires on its own correct work is worse
than no warning.

STATUS PAGE + WIDGET. The Homepage uptimekuma widget reads a PUBLISHED status
page (/api/status-page/<slug>), not the admin API -- which is why the widget
labels were deliberately absent from the rebuild: a dashboard widget pointed at
a 404 is the suspected mechanism behind both of Homepage's unkillable D-state
wedges, so shipping one on purpose would have been daft.

The page now exists at slug `nethealth` (the pre-rebuild slug, so old references
still resolve) and is DECLARED IN monitors.yaml, applied by `kuma seed`. Same
principle as the notification channel: a from-scratch rebuild restores the page,
the channel and the monitors together, and nothing the widget depends on lives
only in Kuma's database. Verified in a browser: "13 SITES UP / 0 SITES DOWN /
100% UPTIME" on the dashboard.

⚠ saveStatusPage calls imgDataUrl.startsWith() unconditionally, so passing null
throws and leaves the page CREATED BUT EMPTY -- which reads as success from
/api/status-page (200, correct title) while the group list is silently blank.
Pass "" instead. Commented at the call site.
2026-09-22 00:55:29 -07:00
vh 6f0a9b9fae feat(alerts): generalize the althing bridge, wire Kuma to it, retire chamber
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.

Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.

Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.

⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.

Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.

parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.

KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.

⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.

End-to-end, both shapes, read back from the inbox:
  [Uptime Kuma] Homepage is DOWN          + target + board link
  [Uptime Kuma] althing (infra-ops) Testing   (no fabricated subject)
  [Beszel] BRIDGE RENAME CHECK            + hub footer, unchanged

ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
2026-09-21 23:11:54 -07:00
vh 3a85a6bce1 feat(uptimekuma): rebuild on 2.5.5 as the fleet's service layer, with a client
Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.

Measured before changing anything:
  - Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
    table is (system, name, value, min) -- there is NO url column, so it is
    structurally incapable of "this endpoint should return 200". Not a config
    gap; the data model.
  - Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
  - So the two are NOT redundant. They are disjoint, and the service layer
    between them was empty.

REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.

  - Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
    the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
    running 1.23.16. Verified by digest -- latest and 1 share one digest while
    2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
  - Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
    services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
    unkillably twice in four months. A monitor also cannot report the failure
    of the host it runs on, so it should not share a failure domain with the
    host layer.
  - Normalised restart: always -> unless-stopped, which the 2026-08-18 README
    flagged as worth doing on the next deliberate touch.
  - UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
    screen, so the stack comes up ready rather than parked on a form.

scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.

  ⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
  SEPARATE pushed monitorList event. Reading the ack yields an empty board that
  looks authoritative -- which duplicated all 13 rows on the first re-seed
  before the bug was found. The client now waits for the push, and carries a
  dedupe verb because of it.

13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.

Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.

⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.
2026-09-21 22:59:37 -07:00
Vuong Hoang 968ec71cce chore(r49): stage lv-mccarthy ckpt300 on fv-ml1, and guard the ship script against a local ~
The adapter is STAGED, NOT SHIPPED: adapter_config.json and
adapter_model.safetensors are in /tank/aimodels/voice-adapters/lv-mccarthy-4b-v1
on fv-ml1, verified byte-identical to gx10's checkpoint-300 at the source, after
the local hop and at the destination. The seat CANNOT load it -- nothing in
--lora-modules references it -- and /v1/models was checked after staging and still
serves exactly the three registered voices plus base.

The adapter carries a README recording the full gate result AND its cost, so it
cannot be read as clean by anyone who finds the directory without the gate record.
A copy is committed at scripts/mccarthy-corpus/gate-results/ADAPTER-README.md for
review. It leads with "STAGED, NOT SHIPPED" and says that if lv-mccarthy is absent
from the seat's model list, the decision has not been made.

The compose edit that would actually register it is prepared in the working tree
and deliberately LEFT UNCOMMITTED. stacks/ is canonical intent that deploy-stack.sh
reads; committing it would assert an intent that is precisely the open question.

Ship-script fix: <run-dir> must be an absolute REMOTE path. A leading ~ is expanded
by the local shell, so ~/r49-runs/... became /home/lkraven/... and gx10 answered
"Permission denied" -- a confusing error for a path problem. It now refuses a
non-absolute path outright, with the reason. Verified: the guard fires on a ~ path
and the real staging succeeded on the absolute one.
2026-09-21 17:57:12 -07:00
Vuong Hoang e8eb1594d9 docs(lv-mccarthy): five-arm ladder -- ckpt300 wins every axis, held at the gate by 0.02
1,200 generations across five arms. ckpt300 (epoch 0.652) is the best arm in the
run on every axis that resolves:

  VOICE          +0.177 at 3.2x its pairwise floor (primary), +0.128 at 2.8x with
                 every punctuation mark stripped. Best point estimate AND best
                 margin of any arm, spread 0.055/0.038 with no outlier seed.
  MEMORISATION   0.12 against real unseen McCarthy's own 0.12 -- identical -- with
                 a longest match of 10 words against the author's coincidental 12.
                 All 31 matches read: stock grammar, names are the renamed
                 inventions, nothing protectable.
  DAMAGE         ran-on +0.12. Clears the operator's ratified v2 floor of 0.200 by
                 40%. FAILS AMENDMENT 3's self-imposed 0.100 bar by 0.02.

NOT SHIPPED, and the reason is the bar rather than the adapter. AMENDMENT 3 fixed
ran-on <= 0.100 before either new arm existed, precisely so a marginal number could
not be talked into a ship, and shipping at 0.12 would make that pre-registration
theatre. But the bar's stated rationale was written against ckpt450's pass by 0.01
-- 5% of the threshold -- and ckpt300 clears by 40%. The number excludes a candidate
the reasoning does not. That is an operator call.

ckpt325/350/375 are on disk and one may sit under 0.100. They were deliberately NOT
gated: searching the checkpoint space until something clears is candidate-shopping,
the same family as threshold-shopping approached from the other side.

THREE CLAIMS FROM EARLIER THIS SESSION ARE REFUTED and are corrected in the record:

  1. "The damage is flat across epochs and only rotates direction" -- FALSE. ran-on
     is non-monotonic (0.38 -> 0.13 -> 0.20 -> 0.28 across epochs 0.49/0.65/0.98/
     1.96) with a real minimum near 0.65, and ckpt225 is 48% out-of-band against
     ckpt300's 35%.
  2. "ckpt300 runs far too short, ckpt225 will clear ran-on by being short" -- FALSE
     on both. ckpt225 runs LONG (median 127, 38% over-band) and is the worst arm in
     the run. I generalised from SIX generations of one arm, which is the exact n=1
     violation the measurement-discipline rule names, committed in the same breath
     as a note about being careful.
  3. The original "gate an earlier checkpoint, the overshoot may not have arrived
     yet" recommendation was RIGHT. Retracting it an hour later on a three-arm read
     was the error, not the recommendation.

What is true and unresolved by any checkpoint choice: 35% of ckpt300's generations
miss the 90-140 band against base's 11%, and in-band is 0.65 against 0.89. An
adapter that buys a voice and costs a third of the length compliance is a trade, not
a defect -- but it is the operator's trade to accept.

Raw artifacts for all five arms at scripts/mccarthy-corpus/gate-results/.
2026-09-21 17:53:43 -07:00
Vuong Hoang 3de4d2dfaf docs(lv-mccarthy): note the undershoot hole in AMENDMENT 3's bar without editing the bar
Six generations into ckpt300 its outputs are far too SHORT -- 6, 25, 46, 59 words
against a 90-140 ask. At epoch 0.65 the adapter may not have learned the length
target yet.

That exposes a hole in the bar I set an hour ago: ran-on is defined as words > 140,
so a chronically undershooting arm satisfies "ran-on <= 0.100" trivially while
being useless. Same class as the unsatisfiable criterion AMENDMENT 3 exists to
correct, inverted -- too easy rather than impossible.

The bar is NOT being edited again. Three rule-motions in one session is already too
many and a bar patched once per observation is not a bar. Instead the judgment is
stated in the open: a candidate that clears the ran-on term by being short has
cleared it vacuously and will not be shipped, and the full in-band / on-beat /
median / p90 table is reported for every arm so a reader can see which kind of pass
they are looking at.

Gate design for this corpus is evidently not settled. That is an operator call, and
the complete five-arm ladder is being measured so it can be made once rather than
iteratively.
2026-09-21 16:37:25 -07:00
Vuong Hoang 43a8d7409e fix(lv-mccarthy): the axis C I froze is UNSATISFIABLE, so it never adjudicated anything
Arithmetic, not a re-reading of results. Criterion 1 needs the candidate's in-band
rate to beat base's 0.89 by more than the 0.200 floor -- so it needs in_band >
1.09, and a rate cannot exceed 1. A PERFECT adapter, every one of 240 generations
inside the 90-140 band, scores +0.11 and fails by 0.09.

So "both candidates failed axis C" carries zero information about either candidate.
That verdict was determined before a single generation existed. It is my error.

The cause is specific: section 6 transcribed score_beats.py's v1 criteria, which the
operator AMENDED on 2026-09-15 -- three weeks before this corpus was built -- for
exactly this reason. Its own docstring: "NOT carried into v2: in-band (unresolvable
-- base maxes it) and on-beat."

The governing rule reverts to the operator's v2: axis C is ran-on only. That is the
rule they ratified and the rule lv-hemingway shipped under. Reverting a
transcription error to its source is not threshold-shopping; the source is older
than the data.

It does not rescue ckpt450 either. Under the corrected axis C its ran-on is +0.19
against a 0.200 floor -- a pass by 0.01, inside the noise the floor exists to
describe, and not a pass worth shipping on an author in copyright with a living
estate.

AMENDMENT 3 therefore gates two more arms on the SAME frozen fixture, seeds and
rule -- ckpt300 (epoch 0.65) and ckpt225 (epoch 0.49), both on disk because
--save-total-limit 60 kept all 56 checkpoints -- since the damage grows
monotonically with epoch and the voice may arrive before it does.

And it sets the ship bar BEFORE those arms exist: voice clearing the pairwise floor
on both reads, memorisation at or near the held-out author's 0.12 with every match
read, and ran-on delta AT MOST 0.100 -- half the measured floor, not merely inside
it, because a rate that clears a noise threshold by 5% of that threshold has not
been shown to clear it. If nothing clears all three, lv-mccarthy is parked pending
a retrain, and that is the report rather than a softer bar.
2026-09-21 16:34:01 -07:00
Vuong Hoang 4c3f3896f1 docs(lv-mccarthy): record the gate result -- voice passes, memorisation is the cleanest in the line, NOT shipped
720 generations, 3 arms x 60 held-out beats x 4 seeds, against the design frozen
in GATE-PREREG.md before any arm existed.

AXIS A VOICE -- PASS, both candidates, both reads. Span 0.661 -> 0.370 = 0.291
achievable; ckpt900 closed 59.1% (+0.172, but only 1.2x its floor), ckpt450 52.2%
(+0.152 at 2.9x). The normalised secondary read HOLDS at +0.124 / +0.114, so about
three quarters of the gain survives stripping every punctuation mark -- the voice
is not the cheap win the register made available.

AXIS B NOT COPIED -- ckpt450 is the cleanest result in the line. 0.12 hit-rate
against the author's own held-out 0.12, and its longest match (11 words) is
SHORTER than the author's coincidental longest (12). All 96 matched runs were
read: stock grammar in the commonest words, the name-shaped hits are the RENAMED
inventions, nothing protectable. The amendment is why this reads as clean -- the
defective base control would have shown 0.12 vs 0.00 as a 12x red flag.
Separately measured: the "his register makes collisions inevitable" story that
was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12). Neither transfers.

AXIS C NO DAMAGE -- FAIL, both, and it survives reading. 20% (ckpt450) / 28%
(ckpt900) of generations overshoot the 90-140 band against base's 1%; p90 171/190
words, max 297/279. The worst case is degenerate looping, not a long McCarthy
sentence. Base is GOOD on this axis here (0.89 in-band vs Hemingway's 0.05), so
the adapter measurably makes instruction-following worse.

NOT SHIPPED. Section 7 rule 3 makes axis C disqualifying outright.

Recorded honestly: my own prereg's axis C transcribed score_beats.py's v1
criteria, including "in-band up on base", which the operator RETIRED on
2026-09-15 for exactly the reason it fails here -- base maxes it, so it is
unsatisfiable on this corpus regardless of adapter quality. Under the operator's
v2 (ran-on only) ckpt450 passes by 0.01 against a 0.200 floor. That reading was
found AFTER the numbers and was NOT used; lv-bronte's floor defect is the in-line
precedent for finding one and declining to exploit it. The prereg gets fixed
prospectively for the next author, not re-read for this one.

And the finding worth more than the adapter: the two-epoch recipe is now 0 for 3,
and this time the loss curve was CONFIDENTLY wrong. On Bronte and Hemingway the
epoch-1/epoch-2 checkpoints were tied, so preferring the earlier one cost nothing.
Here the curve resolved epoch 2 as better at 4.9x the median neighbour jitter --
and epoch 2 lost every axis that resolves: 4.0x wider seed spread, 1.8x the
author's memorisation rate against 1.0x, more ran-on, worse on-beat. Its only win
is a 0.019 voice point estimate, inside the floor, and its spread is one outlier
seed -- the third occurrence of that shape in the later checkpoint after
lv-bronte's ckpt925 and lv-hemingway's ckpt1750.

Raw artifacts committed at scripts/mccarthy-corpus/gate-results/ so the claims can
be re-read without gx10.
2026-09-21 16:31:41 -07:00
Vuong Hoang 0d80e493a8 fix(r49): the quote-mark counter was counting apostrophes, and I saw it fire before I saw the bug
voice_distance.py's quote class shipped this morning as "'‘’“”«»‹›‚„` -- with the
apostrophe characters in it. On a corpus whose defining tic is dont/aint/wont and
whose possessives are everywhere, that made it an apostrophe counter wearing a
quote-mark label.

                          as implemented    TRUE quotes    all apostrophes
  held-out McCarthy ref            121.1            0.0              121.1
  base-unadapted                   224.7           19.9              204.8
  held-out Hemingway ref          1112.6          694.7              351.7

The corrected column is the one the pre-registration names: 0.0 for McCarthy,
which is exactly what build_corpus_mccarthy.py ASSERTS, and 694.7 for Hemingway,
the documented ~838 scale the 100-per-10k trigger line was anchored to. The
as-implemented column matched neither.

ORDER OF EVENTS, because it is the material fact: the base arm finished first, so
the trigger became evaluable while the adapted arms were still generating. I
evaluated it, saw it FIRE at 224.7, and only then -- reading the reference row of
my own table against a corpus I knew asserts 0.0 -- found the bug. No delta_cb,
memorisation rate or damage number had been read at any point.

Fixing a detector to measure the quantity the frozen rule names is not moving the
rule, but the fix un-fires the trigger and no reader should have to take my word
about my motives. So GATE-PREREG.md AMENDMENT 2 makes the trigger MOOT instead of
adjudicating it: the normalised secondary read is load-bearing UNCONDITIONALLY for
this gate, whichever reading you accept, both columns reported. The fix therefore
has no effect on the verdict.

There is a better reason than the bug anyway: base's true quote density is 19.9
against the reference's 0.0, so it did not fully comply. A small residual cheap win
IS available to the adapter, and the normalised read is what prices it. A threshold
is a blunt instrument for a residual that size.

Apostrophes now get their own column and are never folded into quotes again.
Default path stays byte-identical to the shipped lv-hemingway artifact.

The durable lesson is the one this line keeps relearning in new places: I controlled
strip_punct (2500 -> 0) and the byte-identity of the default path, but never asked
the quote counter for a value whose answer I already knew. The corpus asserts 0.0.
That check cost one line and was available before the gate ever launched.
2026-09-21 15:19:33 -07:00
Vuong Hoang a601267fa5 feat(r49): script the voice-adapter ship with sha verification across both hops
Shipping a gated adapter was ad-hoc ssh + rsync three times running. This makes
it one reproducible command for the rest of the line (Faulkner, Morrison,
Chandler are next).

The sha verification is the point, not decoration. "Copied the adapter" is a
claim with no honest form but a read-back, and a truncated or half-written
adapter loads without complaint and serves a subtly different voice. The digest
is taken at the source on gx10, after the local hop, and at the destination on
fv-ml1; any mismatch aborts before the seat is ever told the file exists.

It ships only adapter_config.json and adapter_model.safetensors. A checkpoint
dir also holds optimizer state and RNG, which are large, useless to the seat,
and would make the destination digest disagree with any future re-ship.

It deliberately does NOT edit the compose or restart the seat. Registering an
adapter is a reviewed edit to stacks/voices-seat/compose.yaml, because that file
is where the gate verdict and its caveats get written down for whoever reads it
next -- that is a feature of the current process, not friction to automate away.

Verified against the already-shipped lv-hemingway: the live adapter on fv-ml1 is
byte-identical to gx10's checkpoint-850 on both files, which confirms the file
list and the convention this script encodes.
2026-09-21 14:49:21 -07:00
Vuong Hoang 82a8e0c9b1 feat(r49): commit the instrument for READING memorisation matches
memorization_check.py answers HOW OFTEN an arm collides with the training text.
It does not answer WHAT is being reproduced, and those have different
consequences: a low rate hiding one 30-word run of distinctive imagery is worse
than a high rate of stock dialogue in the commonest words in English.

The lv-hemingway gate shipped at 7x the author's own collision rate precisely
because that reading was done -- every matched run was stock dialogue, max 9
words, no proper noun. But it was done BY HAND and left no instrument, so the
next gate had to repeat it by hand or skip it. This is it.

Prints every maximal matched run sorted by length, with arm/id/seed, and flags
any token capitalised mid-run as a possible proper noun. The flag deliberately
over-reports (sentence-initial I, He, What all trip it) because it is a reading
aid and over-reporting is the safe direction. --train-only matches
memorization_check.py so a collision with held-out text is not reported as
memorisation of training text. A corpus slice is matched against the corpus
every run, and the script REFUSES if that positive control fails -- a matcher
that only ever sees candidate text cannot tell "no matches" from "blind".

Validated against the lv-hemingway record's documented reading, which it
reproduces to the word: longest run 9 words, "swift tristan" flagged as the one
name-shaped hit (it is the RENAMED invented name, not Hemingway's), and the
record's quoted examples -- "came over and sat down at the table", "i don t
think so the girl said" -- both present.

Required by GATE-PREREG.md AMENDMENT 1, which makes reading the matches part of
axis B rather than a follow-up, because McCarthy is in copyright with a living
estate and a match carrying distinctive imagery or a proper noun is
disqualifying in a way a rate number alone is not.
2026-09-21 14:46:23 -07:00
Vuong Hoang 31e40d8ffa docs(lv-mccarthy): extend the runbook through D5 train and D6 gate
The runbook stopped at D4 pairs, so the two stages that actually produce the
shippable artifact were unrecorded -- the same gap that forced the 2026-09-17
D1->D3 chain to be recovered by rebuilding candidates and matching sha256.

D5 records the 1,380-step run, the loss curve read (median neighbour jitter
0.00393; ckpt900 the minimum, ckpt450 at +4.9x, adapter/ at +18.4x), the fact
that the epoch-3 collapse STEPS rather than drifts, and that adapter/ is the
epoch-3 weights so whatever ships is a checkpoint.

It also records the four provenance fields that look wrong and are not -- all
four reproduce on the yarros and hemingway runs -- and how to bind the run to
its pairs by RECORD COUNT rather than by the relative path it stored.

D6 records the gate invocation, the four deviations from the lv-hemingway gate
and what forced each, and the two instrument questions that had to be settled
before it could run: the punctuation confound on the voice axis (primary /
secondary / pre-registered trigger) and the defective base-unadapted control on
the memorisation axis.

Plus one thing that will otherwise be misread: gen_beats_chat_yarros.py prints
RAN-ON when it finds no paragraph break, a heuristic written for the Yarros
register. McCarthy's register asks for continuous scene prose, so an unbroken
block is the TARGET here and the tag fires on most generations without meaning
damage. The axis-C metric is a different definition and is unaffected.
2026-09-21 14:45:02 -07:00