vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • 1d435c9e33 memory: ana-ml2 -> fv-ml1 Fountain Valley relocation + leave-rack-dark decisions vh 2026-09-11 19:14:23 -07:00
  • 8400f3aef5 scripts: pre-script the ana-ml2->fv-ml1 rename sweep (scoped, dry-run default, history-safe) vh 2026-09-11 19:13:53 -07:00
  • ce04f9dbde docs: finalize fv-ml1 cutover — confirmed addresses, piggyback DNS, exact commands vh 2026-09-11 19:11:32 -07:00
  • a13ab598ae docs: fv-ml1 cutover runbook (ana-ml2 -> Fountain Valley colo) vh 2026-09-11 19:08:07 -07:00
  • 1405363649 memory: Anaheim recovery — crowdsec/traefik fail-closed 403 fix, ana-ml2 still needs power vh 2026-09-11 17:00:11 -07:00
  • 870f42960e memory: BabyYarros complete — both arms trained + evaluated, voice moved, beats 9/10 vh 2026-09-11 16:00:05 -07:00
  • 5558d9c7d3 R49/BabyYarros: voice + beat evaluation tooling and the seat-free delta_cb vh 2026-09-11 15:59:41 -07:00
  • 61ddcfd9a6 CLAUDE.md: remind this session it IS infra-ops, not a party to page vh 2026-09-11 15:25:13 -07:00
  • fa6e2c8228 memory: Sentinel-R3 pulled, MTP-grafted, quantized — a better-fit pen-test seat candidate vh 2026-09-11 10:56:25 -07:00
  • 5ccb6c3118 memory: concurrent training on pfi-gx10 is net slower than serial vh 2026-09-11 10:28:55 -07:00
  • 38eb40ace4 BabyYarros: stage the 4B-Base comparison arm, both concurrent and chained vh 2026-09-11 10:22:21 -07:00
  • 969d649a24 memory: BabyYarros unblocked — gate passes 0/325, instruct training live on gx10 vh 2026-09-11 10:09:50 -07:00
  • 7b0580dcbe BabyYarros: the leak gate passes, and it found three defects nobody was looking for vh 2026-09-11 10:06:04 -07:00
  • 16c144fcda memory: snapshot — BabyYarros blocked on the leak gate; R49 sweep complete; mog-sec settled vh 2026-09-11 09:34:55 -07:00
  • 6dba912324 BabyYarros: corpus built, gender resolution fixed, rename blocked on leak gate vh 2026-09-11 08:46:45 -07:00
  • e15c5ee5ea Instruct probe: voice and instruction-following coexist vh 2026-09-11 08:32:30 -07:00
  • d9cf486004 Settle the Skaldsong architecture question with a ten-format negative result vh 2026-09-11 06:24:51 -07:00
  • 8787daf04f Find the prompt recipe that works on a completion carrier vh 2026-09-11 06:04:41 -07:00
  • e47295d2ce Re-cut the 4B arms from checkpoint-75 and test an in-fiction instruction vh 2026-09-11 05:55:15 -07:00
  • 25eb65e23b Complete the R49 carrier sweep: 4B closes the continuity gap and overfits vh 2026-09-11 05:36:11 -07:00
  • 4cee4b179d Launch the 4B rung and record that voice transfer carries subject matter vh 2026-09-10 22:58:10 -07:00
  • 9e18fd58ec Complete R49 rung 2 and booth the three-way voice comparison vh 2026-09-10 22:38:11 -07:00
  • b8435ceb6f Set mog-sec's context to a measured ceiling so it refuses instead of dying vh 2026-09-10 16:04:16 -07:00
  • 935b35ac2e Unwrap the Brontë corpus and launch the 1.7B rung vh 2026-09-10 15:37:51 -07:00
  • 8842ffe1fe Pin mog-sec's KV in bytes at 16.48 GiB and take it to 320k context vh 2026-09-10 15:19:47 -07:00
  • 77224619ee Give GPU0 real headroom: mog-sec to util 0.50 and 384k context vh 2026-09-10 13:46:02 -07:00
  • 8ab794d89a Record the mog-sec GPU0 OOM crash and the spec-decode n=1 near-miss vh 2026-09-10 13:29:26 -07:00
  • 9a916a759f Swap the MeroMero A4B onto the erp-seat seat as char-rp-fast, retire the Pfish-6 alias vh 2026-09-10 11:34:05 -07:00
  • 1a5bc2ddf1 Land the MeroMero v2-31B NVFP4A16 quant and record the pinned-transformers trap vh 2026-09-10 10:55:53 -07:00
  • b8dbe71a1c memory: snapshot — R49 1-epoch pilot complete and awaiting adjudication; MeroMero A4B quantized, v2 blocked; althing 3.6.2 rolled vh 2026-09-10 10:25:52 -07:00
  • 44c853cd20 Repair ana-docker database backups and bound CI build caches vh 2026-09-10 09:10:01 -07:00
  • eb75713c1b Wire Beszel fleet filesystems, GPU telemetry, dashboard and alerts vh 2026-09-10 08:35:28 -07:00
  • 20bbb95113 fix(r49): loss-series collector silently dropped every eval record vh 2026-09-10 07:42:17 -07:00
  • 8fff722f2c feat(r49): incumbent arm generated against the concrete gen seat, not the stale H02 name vh 2026-09-10 07:39:38 -07:00
  • 375244ad05 feat(r49): D2/D3 complete and the H02 pilot is training on gx10 vh 2026-09-10 07:12:19 -07:00
  • ba8dac2c80 feat(r49): D1 corpus built and green — Charlotte Brontë, 680k words, 951k tokens vh 2026-09-10 06:51:32 -07:00
  • 6cbc9c7a2c fix(memory): kill the recurring "check on the run" cron and record how to find it vh 2026-09-10 06:29:27 -07:00
  • e08ac702c9 chore(althing): herald to 3.6.1 on nh3-dev only, operator-scoped 1 of 7 vh 2026-09-09 23:13:36 -07:00
  • 08c852792b chore(erp-tune): purge superseded merged models (~499 GB), keep run 6 and the v6 quant vh 2026-09-09 23:09:13 -07:00
  • 9f30c12da7 docs(r49-prep): close the "is there a newer small base?" question against the registry vh 2026-09-09 23:01:40 -07:00
  • 04950c2881 feat(training-probes): re-measure the R49 name pool under the Qwen3 tokenizer vh 2026-09-09 22:59:15 -07:00
  • 36f1b70a88 chore(erp-tune): purge intermediate checkpoints (~74 GB); R49 carrier settled on dense Qwen3 vh 2026-09-09 22:55:34 -07:00
  • 70f7c0e4a2 docs(training-playbook): §3.11 gains a fourth check — hybrid linear-attention carriers without a fused kernel vh 2026-09-09 22:44:13 -07:00
  • 7db6c44bcd feat(r49-prep): author-voice LoRA regime prep on gx10 — carriers staged, throughput measured, adapters secured vh 2026-09-09 22:41:47 -07:00
  • 92cf09e397 memory: in-flight rewritten for the close — Pfish-6 on ana-ml2 only, gx10 is a training box with no seat, run 7 purged (~139 GiB), no new run planned vh 2026-09-09 22:10:07 -07:00
  • 5ea35d86f4 memory: run 7 purged (~139 GiB), pfi-gx10 declared an experimental/training box with no serving seat, Pfish-6 on ana-ml2 is the sole standing seat vh 2026-09-09 22:09:33 -07:00
  • 625796b894 memory: snapshot — run 7 retired, Pfish-6 (run-6 NVFP4) is the standing seat on both boxes at 262k ctx; gate failure was a detector bug (adjective 'minor'), fixed cc42d76; ana-ml2 measured 4.1x faster than the GX10; three operator decisions left open vh 2026-09-09 22:04:53 -07:00
  • b8956b58fa feat(erp-seat): Pfish-6 replaces run 7 — run-6 NVFP4 quant on both boxes, max-num-seqs 8->32 vh 2026-09-09 19:16:36 -07:00
  • 202e7f60e1 memory: run-7 gate failure was a DETECTOR BUG (adjective 'minor' in a HARD rule) — operator-diagnosed, fixed cc42d76, all service restored, retention wired; the lesson is that rigor downstream of an unexamined premise is not rigor vh 2026-09-09 18:48:13 -07:00
  • a4888f07f8 memory: point at the run-7 decision brief (kept booth, 5-question ask, answers readable via booth answer) vh 2026-09-09 18:10:59 -07:00
  • 31c280c5d2 memory: run-07 incident CLOSED — alias removed and verified at routing, exposure quantified (2,793 tok into the operator's own Open WebUI session, no peer agents), alias-vs-artifact filter 4.7x, three reporting errors disclosed vh 2026-09-09 17:59:39 -07:00
  • b135adce99 docs(ops-lessons): filter on the artifact, not the name pointing at it — measured 4.7x inflation vh 2026-09-09 17:58:20 -07:00
  • 98da4ec1f1 docs(ops-lessons): §11 fourth variant — the instrument read a surface mid-transition vh 2026-09-09 17:50:53 -07:00
  • 4b4911e07e fix(litellm): remove the trial alias — run 7 failed the safety gate vh 2026-09-09 17:49:26 -07:00
  • 14db937624 run-07: FAILED the safety gate — CSAM drift detector fired on the tuned arm, both seats stopped, nothing deleted vh 2026-09-09 17:44:45 -07:00
  • 0e47d2a741 docs(ops-lessons): §11 check the writer and the reader together, with the three failure directions vh 2026-09-09 17:00:57 -07:00
  • 9b9f0625c9 fix(refresh): refuse to promote an empty capture over a good snapshot vh 2026-09-09 16:58:25 -07:00
  • 5a3db132aa docs(run-07): gate choreography as it ran, brokkr's pinned base results + floor addendum, and two probe failures (ssh pgrep self-match; pid file caught the launcher) vh 2026-09-09 16:55:15 -07:00
  • 6972e7ef7f feat(erp-seat): run 7 quantized to NVFP4A16 and serving as the trial seat on ana-ml2 vh 2026-09-09 15:30:15 -07:00
  • c335c38c19 fix(booth): a partial ask answer is recorded, not refused vh 2026-09-09 15:05:18 -07:00
  • 784c555dbd docs(booth): record WHEN inline ask placement earns its markup — the artifact-in-working-memory test, not just the placement rule (framing from tts-dev) vh 2026-09-09 14:23:19 -07:00
  • 6e0b85ba27 docs: refresh what today's work made stale — booth asks (inline placement promoted to its own section), ana-ml2 nvme7 settled by the scrub result, nh3-dev booth entry + the CLI-on-PATH fix, run-07 runbook outcome + serving state vh 2026-09-09 14:18:34 -07:00
  • 78c3a7c170 feat(booth): asks render INLINE in a verbatim report, placed by the author vh 2026-09-09 14:16:03 -07:00
  • 8961ca078b memory: ERP run 7 complete (542/542, adapter 13:23 PT) — erp-seat-base-ara serving for brokkr's floors, erp-tune-v7 merged and staged; Booth asks primitive + its two same-day defect fixes vh 2026-09-09 13:59:31 -07:00
  • a40f979b7a ops(nh3-dev): install sox + libsox-fmt-all for yt-voice-clipper-dev's denoiser bake-off (noisered) vh 2026-09-09 11:13:54 -07:00
  • a56743ade3 fix(booth): asks were invisible in a booth serving its own index.html vh 2026-09-09 10:39:18 -07:00
  • 2a186e4762 fix(booth): CLI resolves its source through the ~/.local/bin symlink (readlink -f), so ask/asks/answer/unlink work from any cwd vh 2026-09-09 09:21:34 -07:00
  • c85a700141 feat(booth): multi-question asks — a questions list renders one form with a radio group per question and lands as one answer sidecar keyed by question vh 2026-09-09 09:20:35 -07:00
  • 3fe01225a9 feat(booth): asks — a multiple-choice question a session poses in a booth, answered by the operator as a radio form + notes, written back as an answer sidecar vh 2026-09-09 07:24:16 -07:00
  • 5de5583762 ops(ana-ml2): pool-health actions landed — tank scrubbed clean + cleared, nvme-cli SMART inventory, zroot 91→73%; root cause of missed scrubs = nvme7 absent 04-23→09-05 (pool DEGRADED, ZED mail unrouted) vh 2026-09-09 02:03:11 -07:00
  • 5ad948bf31 memory: snapshot — run 7 training on gx10; run 6 TRANSFERRED after CSAM adjudication; erp-tune-v6-nvfp4a16 live as trial; ESH/YTVC/webhook repairs; ana-ml2 routes persisted; tank/zroot actions deferred to next session. Index 830→271 lines: 27 decisions + 8 abandoned archived, superseded in-flight blocks archived verbatim vh 2026-09-09 00:26:44 -07:00
  • 3e18a044bd memory: ana-ml2 tank cksum errors + no scrub since April + zroot 91%; ESH 10G topology and link health measured (fiber in, links clean) vh 2026-09-09 00:08:02 -07:00
  • f923d6a96e feat(ana-ml2): persist mesh return routes via ana-scale as an ifupdown if-up.d hook (playbooks/ana-ml2-mesh-routes.yaml, elway-applied, verified) vh 2026-09-08 23:49:38 -07:00
  • d8f625e1f2 memory: run 7 training on pfi-gx10; erp-seat on vLLM nightly, forced tool_choice limit recorded vh 2026-09-08 23:31:22 -07:00
  • a34a72fc09 feat(erp-tune): run 7 launched on pfi-gx10 (opening-split slot + mask union) — config, launcher, builder, runbook; erp-seat: pin vLLM nightly 311b3513 (forced tool_choice 1/9 -> 6/9, round trip clean), README on the gemma4-parser design limit vh 2026-09-08 23:31:09 -07:00
  • 3c54f59878 memory: GX10 released and cleared after brokkr's reference arm; run-7 recipe exists without a grant vh 2026-09-08 23:01:46 -07:00
  • 7f6be8a56a fix(erp-seat): tool_choice=none returned an empty turn — add --exclude-tools-when-tool-choice-none (vLLM kept the tools in the prompt, the model called one, parsing was off); 12-shape tool matrix green before/after vh 2026-09-08 23:01:26 -07:00
  • 32399d0da2 memory: run-6 CSAM soft trip adjudicated GO by the operator (no regeneration); base re-served on gx10 for the probe reference arm; cue-length probe result = run-7 lever vh 2026-09-08 22:45:52 -07:00
  • 78d5a8ccb0 memory: run-6 gate HALTED on a CSAM soft trip (operator adjudication pending); trial alias left on the NVFP4 build per the operator's prior ruling, rollback offered vh 2026-09-08 22:21:24 -07:00
  • 48aa2c6d35 feat(litellm): repoint trial alias run 5 (bf16, gx10) -> run 6 NVFP4A16 on ana-ml2 :8021 (operator: no gate) vh 2026-09-08 22:18:48 -07:00
  • 8512dd4d31 feat(erp-seat): erp-tune-v6-nvfp4a16 quantized (data-free W4A16, ~90 s) and serving on ana-ml2 GPU1 :8021; playbook §3.16 (data-free NVFP4A16 still bakes the tokenizer cap); memory: gate state, brokkr after-window asks, ana-ml2 non-persistent mesh routes vh 2026-09-08 22:14:09 -07:00
  • 911ff20356 feat(erp-seat): NVFP4A16 quant pipeline for the Gemma-4 26B-A4B MoE ERP tune + ana-ml2 GPU1 serve stack vh 2026-09-08 21:55:28 -07:00
  • 1f95728728 memory: run 6 COMPLETE (train_loss 3.259), base seat erp-seat-base-ara serving for floors, processor_config.json gap on the abliterated repo vh 2026-09-08 16:26:05 -07:00
  • f2b7384199 memory: YTVC v0.3.6 (CrisperWhisper) deployed + verified; gitea webhook repointed off the dead wg0 IP with HMAC secret re-applied vh 2026-09-08 15:59:26 -07:00
  • d2df1eb306 memory: YTVC 403 root cause = stale yt-dlp layer (no-cache rebuild), punkt_tab staged on data volume + NLTK_DATA; operator video verified 8 clips vh 2026-09-08 14:27:44 -07:00
  • 14dd9ab90c feat(egress): fleet SOCKS5 residential egress on nh3-scale (100.64.0.1:1080), successor to the retired nh3-dev danted; yt-voice-clipper repointed and back up vh 2026-09-08 14:10:08 -07:00
  • a19b7f62d9 memory: ESH static WAN follow-ups landed — FortiGate trusthost3 → 128.177.138.182 (verified from ESH), esh-ana IPsec rebound to wan1, UDP 41641 forward → esh-scale peers direct vh 2026-09-08 13:45:13 -07:00
  • 7d5de53565 memory: ESH WAN static 128.177.138.182/30 LIVE (CGNAT over), crowdsec esh allowlist updated, open follow-ups (FG trusthost3, esh-ana rebind, mesh direct path) vh 2026-09-08 13:29:42 -07:00
  • 304baddfc1 memory: run 6 TRAINING on pfi-gx10 (abliterated base 33c59654), trial alias dark, gate naming vh 2026-09-08 04:26:14 -07:00
  • 3fec668bf2 feat(erp-tune): run 6 on pfi-gx10 — jenerallee78 ARA-abliterated base (index 33c59654) pulled + byte-verified, run-5 recipe byte-held, launched under operator-2026-09-08-rnd-run6 vh 2026-09-08 04:24:38 -07:00
  • 55631e28bc memory: snapshot — run-5 RESCUED + seat live, base settled STOCK, cc-channel route, Miranda relay authority; session fleet-fixes + open items vh 2026-09-08 03:32:23 -07:00
  • 1282698e71 memory: R47 base provenance SETTLED from bytes — it is STOCK, not heretic vh 2026-09-08 03:22:33 -07:00
  • f8a7985f02 feat(litellm): repoint trial alias run 3c -> run 5 (erp-tune-v5) vh 2026-09-08 02:57:24 -07:00
  • 37f2caa2d3 memory: ERP run 5 COMPLETE — gate RESCUED (landmark R49.5) vh 2026-09-08 02:15:47 -07:00
  • 913d2d2a6c feat(homepage): wire ana-ml2 back into discovery via socat proxy on ana-docker vh 2026-09-07 16:34:13 -07:00
  • e0d1c44137 chore(fleet): repoint stale irv-ml1 refs (10.100.79.3 -> irv-ml1.nh3.internal) vh 2026-09-07 15:08:56 -07:00
  • 4506ef6fb5 feat(homepage,dns): add WhereTF card + DNS alias; fix homepage docker discovery vh 2026-09-07 14:53:33 -07:00
  • 957c8f177d fix(litellm): repoint TTS aliases at irv-ml1 via DNS name + extra_hosts vh 2026-09-07 14:32:18 -07:00
  • 265357efb2 ops(erp-tune): stage + launch R47 ERP-seat SFT run 5 on pfi-gx10 vh 2026-09-07 09:33:03 -07:00
  • 61001e69e5 memory: snapshot — run-5 authorized-but-stalled (brokkr), fleet-TLS pattern, /mnt/smithy remount, backup router-exclusions, irv DNS; session open follow-ups vh 2026-09-07 08:06:24 -07:00