Commit Graph
300 Commits
Author SHA1 Message Date
vh bc91b40ac5 chore: retire asset-engine (Prime 2026-10-17); kuma gains delete-by-name; FleetTools/bootstrap/catalog marked 2026-10-17 00:16:31 -07:00
vh 055d56fa45 docs(fleettools/speech): stale Homepage ASR entries cleared 2026-10-17 00:10:24 -07:00
vh b0235a1e46 docs(fleettools/speech): ASR table matches live state (Parakeet NeMo primary + LiteLLM aliases, VibeVoice, Scriberr on fv-ml1); retired Speaches/sherpa entries named 2026-10-17 00:07:30 -07:00
vh c560e37739 docs(fleettools): Albok page + index entry (draft by albok-dev, Prime's ask); albok MCP in claude-only; task-board marked mothballed 2026-10-16 19:54:34 -07:00
vh 14af41b945 docs(pfi): Amendment 1 addendum — h2o confirmation run passes all three clauses
Admission via thin wrapper (vendor shim unmodified; no shim config knob exists).
Gate re-run clean at every depth with usage honest; 16q32k warm 2,033 ms;
4x concurrent 16q32k peak 15,566 MiB vs 16,847 budget; boot matrix replaces the
unmeasured 280 s figure (cold median 153.6 s, warm-cache saves nothing, eager
slower). Wyrd itemised: 8 diffs, h2o net -2. h2o passes every Supersede clause
under the accepted 32,000-token refusing contract; seat swap is the operator's
call.
2026-10-12 12:13:31 -07:00
vh 4098a23dda docs(pfi): decision-sweep follow-on bench — H2O re-bench, Clef-flash, RSI-Jev vs the live seat
First bench-first run (prereg brokkr-smithy 2e0539f). H2O-lightning-4b is a
short-route candidate: public 205, battery CI lower bound +0.0023, fits the
GPU 1 budget at every shape — but its shipped shim silently trims state at
~32k and fails the long-input gate by the letter. Clef-flash lateral in both
precisions (Wyrd -0.0333 in both; bf16 does not fit at rest). RSI-Jev v6.1
loses on decision quality (public -23, Wyrd -0.0548). INC holds the seat.
Torn down; weights kept (model-runtime 116 GB).
2026-10-12 10:22:03 -07:00
vh a9bae04ba6 docs(pfi): D3 vs Intern-Decision-4B seat bench — final stashed findings (halted by operator call)
Measured: bf16 ladder across six sizes, calibrated 32k shape cells with
O(q*state) wire accounting, bs=1 memory/latency curve, Engine-vs-vendor
fidelity rows, Stage 4 control quant trio (battery monotone in weight error),
flash calibrated-NVFP4 with survive rules applied by the letter, and the
Amendment 2 GPU 1 budget (16,851 MiB) re-read against every fit clause.
Halted before the INC bench-copy arms and mini/27B/small-size NVFP4 arms;
stated plainly. Weights kept pending operator ruling (80 GB).
2026-10-12 08:08:39 -07:00
vh 24b7195bcd docs(backups): the PBS loop's defect triple of 2026-10-05/06 and the review rule
Three defects in one loop in ninety minutes, none visible by reading the
file, all caught by running it against known truth. The runbook's
tested-note now says what is actually tested (one positive control) and
what that implies for future edits.
2026-10-06 08:07:52 -07:00
vh 3429b5c55d ops(pfi-pteradactyl): docker.io/containerd upgraded (manual dockerd + wings restart); docs 2026-10-05 14:18:57 -07:00
vh 58bd802872 ops: docker engine 29.8.2 on esh-ml1, nh3-ml1, nh3-dev, corviduo-dev; docs 2026-10-05 14:15:12 -07:00
vh 5301ede9d2 ops(ana-docker): docker-ce from bookworm repo (29.8.2); live-restore stale-socket lesson; docs 2026-10-05 13:31:30 -07:00
vh 258a3eea68 docs: pfi-gx10 update recorded; held-engine list and Waiting-on-Prime refreshed 2026-10-05 11:34:00 -07:00
vh 06d7184158 ops: pfi-gx10 DGX OTA 7.6 + reboot, vm-esh-nas + ana-docker reboots (engine 29.8.0), crowdsec bouncer updateMaxFailure 3; docs 2026-10-05 11:33:43 -07:00
vh f6fe6f67f8 runbook(park 100): pfi-tacticalrmm rebooted + TRMM 1.5.2 (Prime) 2026-10-05 11:13:34 -07:00
vh 912d932917 runbook(park 100): 4 kernel reboots done (nh3-extdev, corviduo-dev, esh-vm-db, pfi-pteradactyl) 2026-10-05 11:09:17 -07:00
vh a808b349b8 runbook(park 100): pfi-tacticalrmm patched; reboot list now 8 hosts 2026-10-05 10:55:27 -07:00
vh aa86c37954 ops(pfi-pteradactyl): mask leftover apache2, patch (MariaDB 11.8), engine held; docs 2026-10-05 10:41:23 -07:00
vh 241b14bc66 runbook(park 100): results — 15 patched, parks and deferrals; memory 2026-10-05 03:55:58 -07:00
vh fde50b7ca7 runbook(park 100): hold kernel metas on NVIDIA-DKMS hosts (fv-ml1 6.12.111 DKMS failure) 2026-10-05 02:42:41 -07:00
vh bf46beaa96 docs(runbooks): fleet guest patch campaign 2026-10-05 (park 100); fv-ml1 docker-to-tank one-off 2026-10-05 01:50:55 -07:00
vh 867f33d672 docs(pfi): amend 10-04 bench doc per brokkr review
- Fit gap restated on nvidia-smi accounting (~17 MiB rest / ~130 MiB peak
  over, vs the live seat's 15,220/15,437 headroom); the 1.2 GiB figure
  conflated the torch pool fraction with nvidia-smi totals
- Footnote: baseline peak row is suite-only, not shape-matched to the
  candidate gate peaks
- namedef cell labeled as the description-reversal stand-in, not the
  loaded-name variant of arXiv 2609.26758; columns are overlaps
2026-10-04 23:12:41 -07:00
vh 523f086939 docs(pfi): Jev candidate bench 2026-10-04 — H2O-lightning-4b and Clef-flash vs Intern baseline
Vendor-path benches on fv-ml1 GPU 3 at the 14.4 GiB / 32k cap. Neither
candidate replaces Intern-Decision-4B: H2O reproduces its vendor figure
(205x3) and has the best calibration, but rests 1 GiB over the cap and
is 2.2x slower at the 21-criteria shape; Clef loses on accuracy (191-192),
hard tier (72-73/111), and ships a silent 16,384-token input slice.
Adds the namedef cell and JB_AUTH wiring to the 09-30 harness client.
2026-10-04 23:10:36 -07:00
vh 513f6ef38f ops: phasefinal.com moved to Cloudflare Pages; pfi-ana-webhost destroyed and removed from inventory, DNS and the FortiGate 2026-10-04 17:28:29 -07:00
vh 0e3ab447af ops: retire VM 101 (DC) and VM 104 (nh3-laser) from running + backups; infra-ops login on pbs-ana/pbs-nh3; nh3-pve-2 follow-ups verified done 2026-10-04 16:37:15 -07:00
vh d0c1bc628a ops(pbs): pbs-nh3 + pbs-ana upgraded to PBS 4.2.7 / Debian 13; guest-agent driver script, plan and log 2026-10-04 11:44:57 -07:00
vh 73129a0564 ops(esh-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20 (one reboot, Prime on site); ESH cluster uniform on 9 2026-10-04 07:45:05 -07:00
vh 9a6b23581e ops(esh-pve): PVE 9 prep done (8.4.21, microcode, systemd-boot out, removable shim, NIC pins, vmbr0 bridge-ports fixed) 2026-10-04 07:32:13 -07:00
vh b67ab48833 ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9 2026-10-04 07:16:21 -07:00
vh 075ba0930e ops(pfi-pve): PVE 9 prep done (8.4.21, grub-efi-amd64 on the UEFI/LVM box, microcode, NIC pins); upgrade plan + playbooks 2026-10-04 06:11:13 -07:00
vh 9ba8962885 ops(esh-nas-pve): upgraded to PVE 9.2.21; /boot-LV-full trap and fixes recorded 2026-10-03 16:06:33 -07:00
vh 473c52dc6b docs(pve9 plan): keep ESH DNS up during the NAS reboot (unmount 2 shares instead of stopping VM 100) 2026-10-03 15:50:31 -07:00
vh 48f28cfe93 ops(esh-nas-pve): PVE 8 reboot test done; two restart foot-guns recorded 2026-10-03 15:46:37 -07:00
vh 5f8a1311a5 memory: snapshot — 2026-10-03 ~1440 (nh3-pve clean on 9, RustDesk 1.1.16, esh-nas-pve PVE 9 assessed) 2026-10-03 15:23:19 -07:00
vh 5e25d4bae1 docs(pve9 plan): headers must be present before the upgrade (nh3-pve lesson) 2026-10-03 14:09:47 -07:00
vh 3a6d997c99 ops(esh-nas-pve): tank verified healthy after two scrubs; disk kept; PVE 9 blocker cleared 2026-10-03 06:28:48 -07:00
vh de143364fd docs: fleet Proxmox host inventory snapshot 2026-10-02 (all sites) 2026-10-02 19:23:58 -07:00
vh 62254023f3 docs: esh-pve-cluster PVE 8->9 upgrade plan (not executed); ESH tank DEGRADED finding 2026-10-02 19:20:28 -07:00
vh 902e16630f feat(dev-backup): add daily and weekly retention (48 hourly + 30 daily + 12 weekly)
Prime's ruling 2026-10-02. retention.py picks the snapshots to delete:
the newest 48, plus the newest of each of the last 30 days and of each of
the last 12 ISO weeks, counting only days and weeks that have snapshots.
Names that are not exactly YYYY-MM-DD_HHMM are never selected, and the
NAS side refuses any path outside that pattern. The unit fails unless
the number kept equals the number expected. Live run: deleted 1, 0
errors, 48 kept as expected.
2026-10-02 07:42:45 -07:00
vh f44280e6b2 feat(mia): one-shot Make-It-Animatable v2 auto-rigger on fv-ml1 GPU 3 (scripts/mia-run)
Image local/mia:0.1.0 built from stacks/mia: MIA v2 @ bbd8b158 (MIT) with
its pinned submodules, dread-dev's proven Python lock with the torch family
swapped to cu129, and a driver adapted from dread-dev's run_mia.py that
seeds every mesh (fix_random + trimesh's module RNG) and writes
weights_effective into the npz. Weights stay in fv-ml1's shared HF cache
at pinned revisions, mounted read-only.

scripts/mia-run mirrors blender-run: --job DIR is shipped to
fv-ml1:/tank/mia/jobs, one docker run --rm rigs every mesh, out/ comes back.

Acceptance on the four Dread Naught characters: 3.9-4.7 s a mesh (median
of 3) plus 12.7 s model load, peak 3,394 MiB; seeded runs bit-identical
across rotated mesh order; GPU-vs-CPU distances the same size as sampling
noise, with an unseeded GPU run as the positive control.
2026-10-01 09:45:22 -07:00
vh de11e00dfb fix(dev-backup): chmod before prune so retention actually deletes; fail the unit on a bad prune
rsync -a copies a read-only source dir (0555) as read-only, so the hourly
prune's rm -rf could not unlink inside it. From 2026-07-18 every pruned
snapshot was left as a 22-entry husk while the run still logged OK. The
1,740 husks on nh3-nas were removed (0 errors; 48 full snapshots kept).

The prune now runs chmod -R u+w before rm -rf, logs its error count and
the number of snapshots kept, and exits 3 on a failed prune (exit 1 on a
failed rsync) so the systemd unit shows failed instead of passing.
2026-10-01 05:59:07 -07:00
vh b38ec6dea0 docs(parakeet): seat A/B - clock times in Pacific military time 2026-09-30 18:51:59 -07:00
vh a6c1d3c454 docs(parakeet): seat A/B vs parakeet-unified-en-0.6b - latency is the int8-on-CPU runtime; unified wins WER
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against
nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image,
k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's
recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights).

- Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%).
- unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s
  (paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54).
- unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other,
  -3.2 to -4.4 pp AMI (paired CIs exclude 0).
- Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after
  a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only).
- B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0.

Raw requests, hypotheses, manifests and the full harness under
services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart
from 240 light test requests.
2026-09-30 18:51:44 -07:00
vh 38015a1977 docs(scriberr): Parakeet dropout investigation; proposed 0002 (gap retry + model path)
Prime's ask (via the coordinator): investigate the "Parakeet skips
stretches of speech" finding, including other Parakeet weights.
Investigation only; nothing deployed.

Against ground truth (official SCOTUS transcript, Gutenberg #38916) the
drops are real: production v3 loses 140 / 66 clean words per transcript on
the two public files and ~50 on each private one (Whisper-referenced,
Canary-confirmed; adjudicator 129/129 correct on the calibration). Cause:
the v2/v3 0.6B weights collapse deep inside long full-attention windows;
the encoder output is degraded, the audio alone transcribes fine, and
1.1B TDT/RNNT/CTC and CTC-0.6B never do it. Decoding (CUDA graphs, greedy
variants, max_symbols, beam), slice length, local attention, loudness,
resampling and a noise floor do not fix it. Controls: A-vs-A, silence
positive control (>=15 words 36/36), null control, bootstrap floor.

Proposed patch 0002 re-transcribes >=3 s stretches where the audio holds
speech but no word came out (-80 to -90 % lost words on all four
recordings, lower WER, no invented text, +10 MiB) and adds an explicit
PARAKEET_MODEL_PATH with the loaded model recorded in JSON and ModelUsed.
Reviewed at high effort, all findings fixed; built and tested as
scriberr:local-blackwell-a353078-dropout2, not deployed.

scriberr-rebuild: --patches takes DIR[:DIR...]; embeds and seam-checks
both Parakeet scripts (seam-check --standard for the short-audio one).
2026-09-30 15:48:02 -07:00
vh 3c5f1ea803 docs(scriberr): slicer patch live on fv-ml1 as local-blackwell-a353078-slicer1; live GPU 1 peak 5,496 MiB
Deployed 2026-09-30 1211 PT by pointing SCRIBERR_IMAGE at the patched tag (.env backed up as .env.bak-20260930-pre-slicer1; rollback is the unpatched scriberr:local-blackwell). PrepareEnvironment rewrote the env's parakeet_transcribe_buffered.py from the embed (sha256 matches the patched source). One live run on GPU 1 beside intern-decision peaked at 5,496 MiB. Memory records the open Parakeet mid-chunk dropout finding and the held upstream PR.
2026-09-30 12:14:09 -07:00
vh ee3db68db1 feat(scriberr): overlap-and-stitch Parakeet slicer patch, rebuild script, bench
Carry patches/0001 on our Scriberr build (upstream a353078): adjacent
buffered chunks overlap by 4 s inside --chunk-len and hand over at a word
both chunks transcribed alike, instead of cutting at fixed marks with no
overlap. Pause-aware cutting is included as an opt-in (--pause-search);
it measured neutral once the stitch was right. The Go<->Python CLI and
JSON seam is unchanged.

Bench (4 recordings, 118 min, 3 cut placements each, against a no-cut
whole-file reference; metrics only, private audio stays on fv-ml1):
cuts with an error within +-3 s fall from 52% (93/179) to 22% (41/184)
against a 19% background; floor +-0.08. Positive control: upstream's
cutter +0.33 over background. A-vs-A byte-identical in-process and
across CLI processes. Peak GPU memory unchanged at 5,496 MiB (n=3).
Also found: Parakeet skips runs of >=10 words mid-chunk with any
slicer, upstream's included; not addressed here.

scripts/scriberr-rebuild clones a pinned upstream sha into a new
/opt/docker/src dir, git-apply-checks the patches, builds a distinct
tag, and checks embed, unit tests, the JSON seam (scriberr-seam-check.py)
and the memory budget on idle GPU 3. Deploy stays manual. The upstream
PR is prepared under patches/upstream-pr/ and not opened.
2026-09-30 12:10:32 -07:00
vh 475d6d6bcb docs: Jev candidate bench vs SemIf (fv-ml1 GPU 3) — Intern-Decision-4B is the replacement if SemIf is displaced
Operator ask relayed by brokkr-smithy-dev. Positive control (SemIf 187/231,
hard 0.613) reproduced exactly; negative control and a 4-restart noise floor
(0 flips) measured. On our replaced-baseline sets no candidate beats SemIf-with-
rotations beyond the ~4-pt floor; Intern-Decision-4B native matches it at one
ordering, is better on Wyrd, fits 9.7/10.3 GB and is 1.5-2.3x faster. JevBench
rank does not transfer. Raw per-item data kept out of git.
2026-09-30 05:00:45 -07:00
vh ea5d8dfd4e docs(blender): desktop Blender + MCP registration per working session, not per task (Prime 2026-09-28) 2026-09-28 16:36:44 -07:00
vh a51188c76a feat(blender-run): --cpu runs with no GPU attached (modelling/IO jobs stay off GPU 3) 2026-09-28 14:23:12 -07:00
vh 83dc497b40 feat(blender): pinned extension set in a read-only System repo, for the GUI and blender-run --extensions
Blender is now a mandatory stage in draupnir's pipeline (Prime, 2026-09-28), and draupnir asked
for eight add-ons from extensions.blender.org: SurfacePsycho 0.10.4, CAD Sketcher 0.32.1,
3D-Print Toolbox 1.4.1, STEP Importer 1.2.1, Bool Tool 2.1.0, LoopTools 4.7.7, MeasureIt 1.8.4,
3MF Import/Export 2.7.7.

- stacks/blender/extensions.lock pins each by version and archive sha256.
- scripts/blender-extensions sync builds fv-ml1:/tank/blender-extensions/5.2/system with Blender's
  own install-file, pre-warms and byte-compiles it, checks a read-only enable, then swaps it in.
  It refuses while the GUI or a blender-run job holds the old directory.
- conf/scripts/startup/fleet_extensions.py enables every package in the System repo: in a timer
  in the GUI (after the prefs load), and as --python ahead of the caller's args in
  blender-run --extensions (a failed enable exits 1 before the caller's script).
- It also patches SurfacePsycho's sp_overwrite_segment_selection from eval() to literal_eval():
  the eval walked past MCP safe mode (control: unpatched ran code, patched refuses).
- blender-run: --extensions (bind mounts via --mount so a missing source fails instead of being
  created); USER/LOGNAME set, which CAD Sketcher's getpass needs.
- compose.yaml mounts the repo read-only and the hook into the GUI container. NOT yet deployed.
- scripts/blender-probes/extensions_acceptance.py: one operator run per add-on, safe-mode
  compliant. Headless 8/9 online and with --network none; CAD Sketcher sketching is GUI-only.
  A Python audit hook saw no network/process events (positive control fired).
2026-09-28 12:50:57 -07:00
vh 46276611a7 fix(blender-run): unique remote job dir per local path, mirrored with --delete
draupnir review: the remote job dir was keyed on the basename alone, so two
local dirs with the same name shared one remote dir, and a rerun inherited
stale files. The dir is now <basename>-<8 hex of sha256(abs path)>, and it is
mirrored with rsync --delete, confined to that one directory. There is no rm on
a computed path. Verified: a file deleted locally is gone from the rerun's
remote dir. The /defaults stderr line is documented as harmless noise.
2026-09-28 08:43:55 -07:00