7cc847f33c86a3f7382423cc14f45e9669aa4cd8
1181
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
f0d30f7a0f |
memory: snapshot — 3.2.1 outcome; the unguardable seats split, and one has a fix
All four seats answered. Delivery is now verified on four TUI families and forseti's measured idle columns held exactly, so nobody had to guess. The two seats guard 4 cannot cover made opposite calls on the same facts and both are right. regin-smithy-dev released to pull-only because the operator composes in that pane routinely, so the exposure is continuous. bil-smithy-dev kept its route because that pane is poke-driven, making the collision window narrow, and because pull-only had already cost them a notice that sat unread for days. The deciding variable is who composes in the pane and why, not risk appetite -- a flat rule either way would have been wrong for one of them. regin also supplied the only real path to closing the hole: pi and kimi report no cursor because their input line is not an empty-prompt-at-idle, so a moment-check reading pane content rather than cursor column would cover them. Column is a proxy; an empty input line is the actual predicate. Relayed to forseti as the lead item. And bil corrected something upstream: the Aug 28 probes proved delivery against non-Claude TUIs but never exercised the collision case, because nobody was typing during them. Two different questions, one body of evidence, only one of them answered by it -- which means the earlier retraction of the non-Claude risk flag was right about delivery and silent about stapling. Note for the operator: regin-smithy-dev is pull-only as of 22:42 and will not be poked until it declares again. |
||
|
vh
|
1c7bd40c9c |
memory: snapshot — althing 3.2.1 deployed; guard 4 cannot cover two of five seats
3.2.0 wrote its pokes into pane input lines and pressed Enter, so anyone mid-sentence had their half-written message submitted with the herald's line stapled on. It hit the operator within an hour of this evening's deploy. 3.2.1 adds a fourth guard that pins the pane's idle cursor column and stays silent when the live column has moved. Deployed all three steps at 22:40 and re-declared this seat, which pinned idle_cursor=3 as expected for a claude TUI. The finding worth keeping is the hole the fix leaves. forseti's own measurements say kimi and pi report no cursor at all, which means bil-smithy-dev and regin-smithy-dev can never acquire guard 4 no matter what they run. Two of the five pane seats on this box stay permanently exposed to the bug 3.2.1 fixes, so the sentence "3.2.1 fixes the write-into-a-typing-pane bug" is only true where the cursor is legible. All four seats were told, differentiated: a pre-filled re-declare and the expected column for the two that can be guarded, and the honest version plus the release-to-pull-only option for the two that cannot. Also keeping forseti's post-mortem line, because it generalises: three guards that all answer "is this the right pane" and none that asks whether it is a good moment are one check wearing three hats. |
||
|
vh
|
28dd516be1 |
memory: snapshot — 3.2.0 migration complete, and a miss of my own worth keeping
All four notified seats re-declared within about twelve minutes and the log went quiet. Five of five pane routes now carry the guard fields, and delivery is confirmed on a claude seat, a pi seat and a grok seat. Two findings survive the close-out. The herald's exclusion reason is false for the migration case -- every declaring process was alive and four days old with no pid wrap, and the real cause is simply that the route predates the fields the guard needs. And the uv trap stands as the thing most likely to bite the next person. The third is mine. I raised non-Claude pane delivery with forseti as an open risk on their release when forseti had personally measured it days earlier, on the exact seats in question, and the resulting matrix is what characterised the settle bug they fixed. The error was not caution, it was calling something open without checking whether it was already settled, with the peers who knew right in front of me. Recorded because "I don't know" and "this is an open risk" are different claims and I made the second when only the first was true. Still open with forseti: the uv --force runbook fix, an exclusion-message third branch, --discover-pid against a non-Claude process tree, the one-tick latency note, and status not being durable evidence. |
||
|
vh
|
2d2c88e43c |
memory: snapshot — the four revoked routes are non-Claude, and the herald's reason is wrong
Operator directed that the four affected seats be told directly. Measured their state before writing, which turned up two things worth more than the notification itself. The herald logs each exclusion as "the process that declared this route is gone, or its pid was reused by something that started at a different time." Neither is true for any of the four. Each declaring process started minutes before its route was written and is still running four days later, and pid_max is 4194304 against a current 2.86M so the counter has not wrapped. The real cause is a third one the message never offers: the route predates the guard fields, so identity cannot be verified. The behaviour is right and the explanation is wrong, and it would send anyone debugging it hunting a dead agent that is alive. Separately, all four seats run non-Claude CLIs -- kimi, grok, codex and pi -- while the pane poke is built around typing into a Claude Code pane. Whether --discover-pid walks a non-Claude process tree, and what a poke does to a non-Claude TUI, are both unverified. Each agent was given their measured pid to sidestep the first, told plainly about the second, and offered the choice between re-declaring as a test or staying pull-only. Both findings raised with forseti. |
||
|
vh
|
e014f756fb |
memory: snapshot — infra-ops moved to a pane route; waiters get reaped
The waiter died a second time, within minutes of being armed, so this seat stopped re-arming and declared a pane route instead. That is exactly what 3.2.0 shipped for. - althing-route declare --discover-pid walks the ancestry to the long-lived claude process rather than the ephemeral bash that invoked it. Passing --pid $$ would bind the route to a shell that dies with the tool call. - Recorded the sharper form of the failure: the waiter does not just die, it dies after confirming it is up. Both waiters reported push and reachable immediately after arming. So a green postbox status is not durable evidence of monitoring, and the reap is not confined to the pane seats that were migrated -- this seat is a fifo waiter and was hit twice. Both consequences raised with forseti on the deploy thread, along with a note that reachable is a report rather than a delivered poke. |
||
|
vh
|
71d97f36b6 |
memory: snapshot — althing 3.2.0 deployed; uv tool install is a silent no-op
forseti requested the deploy, operator-approved and tagged at c4ede0f. All three steps landed on nh3-dev and verified: force reinstall to 3.2.0 with seven binaries, herald restart, skill sync. The durable lesson is the trap in step 1. `uv tool install .` matches on the source spec rather than its contents, so on a box that already had the tool installed from that path it prints "already installed" and exits 0 having done nothing. Following the runbook literally would have left the new binary absent with every command reporting success. Always use --force when reinstalling from a local path. Also recorded: the pane-route migration revoked exactly four routes, verified by splitting on channel= before the restart rather than by auditing fields (all twelve route files lack the new fields, so a field audit over-counts). The four affected agents were deliberately not notified, per the standing rule against unsolicited fleet broadcast, and that is surfaced to the operator instead. And this closes an open question from earlier today: the waiter that died with status "killed" was CC 2.1.257 reaping detached tasks, which is the premise 3.2.0 exists to address. |
||
|
vh
|
46a3c63706 |
memory: snapshot — A6000 window closed; my dots-tts hypothesis was wrong
Operator freed ComfyUI's VRAM directly, so tts-dev is unblocked and the window request is withdrawn with comfy-dev. - Verified it was a model unload, not a stop: comfyui still up 8 days, same pid, HTTP 200, 18,500 -> 612 MiB. Told comfy-dev explicitly so a VRAM drop is not misread as a restart of their service. - The resulting 43.8 GB free is a snapshot, not a floor. ComfyUI is live and reloads ~18.5 GB on the next render, which puts the real floor at ~25.3 GB against FireRedAudio's ~26 GB requirement. The coordination shrank from "stop ComfyUI" to "don't render during the bench" rather than disappearing. Flagged to both; not volunteered on comfy-dev's behalf. - Withdrew my caching-allocator hypothesis for the dots-tts VRAM. tts-dev identified it as their prompt-feature cache, capped at 32 entries on 2026-08-14 after two production incidents. A named mechanism with an incident history beats a plausible story, and the useful finding is that 14.43 GB sits inside a cap they deliberately chose. Read-only probes; nothing on the box was changed. Memory-only; no version bump per the SemVer SKIP list. |
||
|
vh
|
e91278799d |
memory: snapshot — irv-ml1 GPU resident map; dots-tts at 2.4x its recorded VRAM
tts-dev asked for an A6000 window for an approved TTS bench and flagged a 3090 VRAM delta. Probed the box and mapped PID to container rather than taking the reported figures. - The 18.5 GB process they attributed to the 3090 is comfyui, on the A6000. And it is 18.5 GB rather than the ~11.8 GB they budgeted, so stopping it gives ~44.4 GB free, not the tight margin they expected. - Their "~4 GB unaccounted" on the 3090 is two things: parakeet is a third tenant the doc figure never counted, and dots-tts alone is holding 14,430 MiB against a burn-in figure of ~6 GB. The second is the larger finding and it is theirs to act on; handed over with a caching-allocator hypothesis and a one-restart discriminating test. - Restated the GPU ordering foot-gun: device_ids ["1"] is the A6000 in a container, but a bare native CUDA_VISIBLE_DEVICES=1 gets the 3090. Window not granted unilaterally — comfyui is comfy-dev's and they are mid-migration, so the request went to them directly and infra-ops relays. Ruled that the bench runs as a plain container under lkraven rather than under /opt/docker/compose/, which is for deployed stacks and would leave a canonical entry reporting as drift until deleted. Read-only probes; nothing on the box was changed. Memory-only; no version bump per the SemVer SKIP list. |
||
|
vh
|
2cf73fd556 |
memory: snapshot — 768 GB is not a population the R750xa takes
Operator asked why not move all 768 GB across. It does not fit the board's shape: the R640 has 24 slots at 6 channels/socket x 2 DPC, the R750xa has 16 at 8 channels/socket x 1 DPC. 768 GB is either 24x 32 GB (more DIMMs than slots) or 12x 64 GB (fits, but populates 6 of 8 channels per socket and gives up ~25% of memory bandwidth). The board wants 16 identical DIMMs. So the targets are 512 GB if they are 32s, or 1 TB if they are 64s — taking 12 from one spare and 4 from the other. In the 64 GB case the answer beats the question. Also recorded: beyond ~512 GB the return is marginal for this workload, so take 1 TB because it is free rather than because it is needed; 64 GB LRDIMMs run ~50 W hotter in a chassis whose high-performance fans are unaccounted for on the invoice; and DIMM slot count now needs to be on the iDRAC pull, since the 16-slot figure is inferred from the factory CSV and is load-bearing for a 512-vs-1024 decision. Memory-only; no version bump per the SemVer SKIP list. |
||
|
vh
|
32fd1dbe4a |
memory: snapshot — ARC sizing for the Ada box; no ramfs model tier
Operator asked whether 512 GB justifies a ramfs for hot models, then self-corrected toward ARC with a 128 GB cap. - No ramfs/tmpfs tier. ARC is the same cache done adaptively: no curated hot-list to rot, no boot-time copy-in, and the memory comes back under pressure. Neither tier beats the host-to-VRAM PCIe hop anyway, so the warm-load time is identical. - 128 GB is below the OpenZFS Linux default of 50% of RAM, so doing nothing already gives 256 GB. Recommend ~320 GB, not past ~75%. - Recorded the one honest argument for tmpfs: safetensors mmap gets double-buffered on ZFS-on-Linux, so a checkpoint can cost ~2x. Sized for, not architected around. - Flagged that this is not the idle-VRAM-is-reserved case: arc_max is a ceiling on an elastic cache, not a preallocation. - recordsize=1M cannot be won through zfs send, since recv reproduces the source's block structure. Not worth losing incremental send over; the 128K cost on flash is metadata overhead, not throughput. Memory-only; no version bump per the SemVer SKIP list. |
||
|
vh
|
3410d578e3 |
memory: snapshot — R640 RAM harvest may zero the RAM line; CPU diff was missing
Operator has 2x Dell R640 at 768 GB each and asked whether the memory is interchangeable with the R750xa. Both are DDR4 RDIMM platforms and Dell does not vendor-lock DIMMs, so the answer is very likely yes. - 16 slots in the R750xa x 32 GB = 512 GB, double the factory spec, and it deletes the 8x M04W6 purchase. - Gating question is RDIMM vs LRDIMM. 24x 32 GB 2Rx4 RDIMM is the safe and most likely case; 12x 64 GB LRDIMM needs Ice Lake support checked. - Cleanest harvest is to strip one R640 entirely and leave the other whole, rather than half-emptying both into unbalanced populations. Separately, reading the factory CSV to answer this surfaced a gap in the diff table: the reseller also swapped 2x Xeon Platinum 8362 (32C/64T, 265 W, DDR4-3200) for 2x Xeon Silver 4314 (16C/32T, 135 W, DDR4-2666). The Silvers were recorded under "As bought" but never diffed, so the swap went unremarked. Two consequences: the box cannot use the 3200 rating the buy list was paying for, which makes 2666 R640 DIMMs a free lunch; and the CPUs draw 260 W less, which the existing ~1,020 W power figure already assumed correctly. Memory-only; no version bump per the SemVer SKIP list. |
||
|
vh
|
5c88280f9f |
memory: snapshot — operator leaning 6 drives; records the layout analysis
Six drives fills all eight bays, which turns a capacity top-up into a one-shot build decision. Recorded because the reasoning survives whatever he picks: - raidz2 over mirrors. Workload is large sequential reads of safetensors, ARC fronts it, SSD resilver has no seek penalty, and "expand two at a time" is meaningless once every bay is full. raidz2 survives any two failures; 4x mirrors dies to an unlucky pair. - Buy 8, not 6. A raidz vdev caps every member at the smallest, so the two as-bought 1.92 TB drives would cap all eight and put two used drives of unknown endurance inside the parity set. - Drive size is now the permanent ceiling. SAS/SATA backplane, all bays full, and the free-PCIe-slot inventory is still unpulled. Runway table included with an explicit caveat that the growth rate is projected off one acquisition batch, not measured. Memory-only; no version bump per the SemVer SKIP list. |
||
|
vh
|
3180dc85fa |
memory: snapshot — Ada sizing corrected; prune and drives are orthogonal
comfy-dev's disk-vs-catalog diff and a recount of my own figures both landed on this thread. Two numbers were wrong and both were headed for the operator's sizing conversation. - My ~90% was a double-count. I read ALLOC 1.45T while their pull was running and then added the full ~112 GB on top; most of it was already in that reading. "Onboarded" is not "landed". Settled payload is ~1.47 TiB and the as-bought mirror lands at 84%, not 90%. - comfy-dev's "pruning gets us nearer 45%" is the striped figure. On the as-bought pair mirrored, deleting all ~215 GiB of unreferenced weights still lands at 72%, with ~140 GiB of runway on a store that took on ~100 GiB in one day. The constraint is vdev layout, not payload — a 1.75 TiB pool stays 1.75 TiB whatever goes in it. - So the prune audit and the drive purchase are independent decisions and neither gates the cutover. Presenting them to the operator that way rather than as a trade. Also recorded: pool arithmetic (1.75 / 3.49 / 3.57 TiB), the mirror-vdev smallest-member gotcha if the new drives get paired one-each with the 1.92s, comfy-dev's 34 GiB of uncatalogued LTX 2.5, and an open question back to them on whether the H3 encoder's nvfp4 pin was set under a Blackwell assumption that sm_89 does not satisfy. Memory-only; no version bump per the SemVer SKIP list. |
||
|
vh
|
72065b45aa |
memory: snapshot — Ada destination pool is smaller than the source
Measured irv-ml1's storetank against the R750xa's as-bought drives while answering comfy-dev's "does the Ada box have ZFS?" question. - storetank: 1.81 TiB pool, 1.45 TiB used, 80% CAP already, compression off at compressratio 1.00x (safetensors incompressible — no win at recv). comfy-dev's ~112 GB batch is landing into it now. - The R750xa shipped 2x 1.92 TB SATA SSD; mirrored that is ~1.74 TiB, smaller than the pool it receives from. Migration would arrive at ~90% full with no growth room. - Buy list: +2x 2 TB SATA SSD (6 bays free, HBA355i has the ports) -> two mirror vdevs striped, ~3.49 TiB at ~45%, redundancy intact. - Retain vs reclaim irv-ml1's /storetank after cutover: RETAIN recommended, surfaced to the operator. Also corrects branch (b)'s recorded rationale. comfy-dev enumerated all twelve running containers: only comfyui mounts /storetank, so (b) was unavailable during the transition, not structurally. Right conclusion, wrong reason — infra-ops reasoned about the box when the question was about the mount. Memory-only; no version bump per the SemVer SKIP list. |
||
|
vh
|
ace839c768 |
memory: snapshot — the Ada inference server is a stripped used R750xa
Dell R750xa JPJ1ZP3, 2x RTX 6000 Ada to be fitted, ComfyUI's new home at NH3. Diffing Dell's factory CSV against the reseller invoice shows four downgrades: half the RAM, the 2400 W PSUs, and the GPU risers, cables and high-performance fans all absent. Records the resolved GPU power chain, correcting my own first answer: the chassis' CPU 8-pin cabling is the right source type and NVIDIA 930-00030-1546-000 bridges it to the card's 12VHPWR, so the PCIe-type RCCWC I first proposed is withdrawn. Also closes the NVMe question — the backplane is SAS/SATA only — and notes that free PCIe slots may moot it. Auto-archival fired at 308 lines; seven entries moved to archival-memory.md. The 250-line target was not reached because the guards hold nearly everything else back as under 14 days or carrying open deferred work. |
||
|
vh
|
7142657749 |
memory: snapshot — gx10 unracked and next up as an inference+training box
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the first move is a throughput probe rather than a harness port. Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1 keeping its eight services; the Synapse 39-release upgrade with its one-way schema migration, the appservice namespace opening and the admin API lockdown; the ratified room alias convention; and a named failure class — a correct check aimed at the wrong object — with six instances from one day across three sessions. |
||
|
vh
|
73866f6a7e |
feat(synapse): restrict /_synapse/admin to LAN, track the stack
Synapse mounts its admin API on the same vhost as the client API, so publishing matrix.phasefinal.com published the admin surface too -- it answered 200 from the open internet. HMAC-protected, so not an open door, but Synapse's own guidance is to keep it off the public listener. A higher-priority router (explicit priority 100, not relying on Traefik's rule-length tie-break) scopes PathPrefix(/_synapse/admin) behind an ipallowlist. Verified from a genuinely external vantage rather than from a fleet host, since nh3-dev sits inside the allowed range and would have proved nothing: via the NH3 residential egress proxy the admin path returns 403 while the client API returns 200 and Element is unaffected. The 10.0.0.0/8 entry matches nothing today and the comment says so rather than implying fleet access exists. matrix.phasefinal.com resolves publicly, so fleet hosts hairpin out their own WAN -- a request from nh3-dev arrived as 70.230.226.88. The rule is effectively deny-all through Traefik, which is the intended posture: admin work goes through docker exec to localhost:8008 and never traverses Traefik. Allow-listing the sites' WAN addresses was considered and rejected as a maintenance trap on dynamic addresses. Also brings the stack under stacks/ with the Postgres password replaced by a required .env variable. The tracked copy and the live file have therefore DIVERGED and deploy-stack.sh must not be used until the live file reads from a .env; the README says so. |
||
|
vh
|
a0c5fc6ed5 |
feat(pfi-gx10): rack-move network playbook — VLAN 50, static 10.100.50.60
Target settled: nh3-servers VLAN 50, static 10.100.50.60. Clear of the four existing statics and below the .150 DHCP pool where fleet statics live. The playbook never leaves itself one path back. Wi-Fi stays up throughout while the wired interface is configured beside it; the new address is verified from outside before anything is torn down, and Wi-Fi teardown is explicitly a separate later change. A botched netplan therefore costs a retry over Wi-Fi rather than a trip to the rack — which is what substitutes for 'netplan try', whose interactive rollback needs a TTY that elway cannot provide. Two preconditions are asserted as steps rather than assumed: the interface must have carrier (writing a static config for a dead NIC and reporting success is the failure this avoids), and its MAC must match, since interface names can renumber across kernels but MACs do not. Requires nothing from the operator beyond racking the box. The wired NIC has a distinct MAC from the Wi-Fi one, so the post-move address and switch port are both discoverable from the UDM rather than needing to be relayed. |
||
|
vh
|
8fb8cc87ca | chore(pfi-gx10): first inventory snapshot | ||
|
vh
|
1b596c8c30 |
feat(pfi-gx10): register the ASUS Ascent GX10 and convert it to headless
NVIDIA GB10, aarch64, 121 GB unified, sm_121. Ships booting to graphical.target with GDM and GNOME Remote Desktop running. playbooks/gx10-headless.yaml sets multi-user.target, stops gnome-remote-desktop, masks the sleep/suspend/hibernate targets, makes logind ignore lid and idle, and adds sshd keepalives so a stalled link does not kill a long-running job. Two things learned the hard way and recorded in the playbook: - gdm is a STATIC unit on Ubuntu, pulled in by display-manager.service and never 'enabled'. A guard of always skips, and a verify written the same way passes while the desktop is still running. Both now test is-active. The first run reported six green verifies having not stopped gdm. - elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as the connecting user and must carry their own sudo; connect as infra-ops. The playbook refuses to stop the display manager while a seat session is held, overridable with --var force_dm_stop=true. Networking is deliberately out of scope: the box is on a desk on Wi-Fi with a temporary DHCP lease and no ethernet carrier, and belongs to the rack-install change. |
||
|
vh
|
931bac8f68 |
docs(matrix): current state, upgrade procedure, alias convention, push findings
Synapse v1.120.0 -> v1.159.0 and Element-web v1.11.80 -> v1.12.27 (2026-09-01). The existing build steps date from the AIPA era and are now marked as provenance rather than as instructions. Records what only existed in a session transcript: - Schema migrations are one-way; rollback is restore-from-dump. Pre-upgrade pg_dump procedure, with a pg_restore --list verification step. - Why the appservice user namespace is now exclusive: false. exclusive governs who ELSE may act, not what the appservice may do, so on a closed single-admin server it locked out all other account creation to prevent squatting that cannot occur. Includes the two things not to do: narrow the regex (orphans 13 accounts) or rename the id (Synapse keys ownership on it). - The shared-secret registration HMAC takes no trailing null after notadmin. - Room alias convention #<agent>-<purpose>, operator-ratified, with its cost accepted deliberately and its rationale stated as room-identity-carries-tier rather than push-payload-carries-room-name. - Push reality: the pusher is event_id_only, so the notification is assembled on-device by Element X's service extension. Records the resulting server-invisible failure mode when the phone cannot reach the homeserver. - QR sign-in requires Matrix Authentication Service and why it is deferred. Ops ownership recorded: worldtree-dev writes the bridge, infra-ops operates this instance. |
||
|
vh
|
9e986d8ee8 |
feat(phasefinal-web): cloudflare edge config — cache ruleset + always online
Cache rule on www.phasefinal.com with edge and browser TTL both respect_origin, so cache policy stays declared once in nginx.conf rather than split between the repo and the dashboard. Always Online enabled, which is what actually survives an origin outage; a 300s document TTL alone would only mask five minutes. Verified: document and assets both reach cf-cache-status HIT, apex 301s to www, edge email obfuscation active. |
||
|
vh
|
524aa4d860 |
fix(phasefinal-web): healthcheck targeted ::1, so traefik skipped the container
The healthcheck used http://localhost/, which resolves to ::1 in nginx:alpine while nginx listens on IPv4 only — so it never passed, the container stayed unhealthy, and Traefik silently declined to create a router for it. That presents as a broken docker provider: correct labels, right network, no route, no error. Target 127.0.0.1 explicitly and add a start_period. Adds the apex router (301 phasefinal.com -> www) and drops the file-provider workaround, which was mitigating the wrong diagnosis. |
||
|
vh
|
7ffbee6f09 |
feat(phasefinal-web): corporate site stack on ana-docker
Single static page (nginx) fronted by Traefik at www.phasefinal.com, built from the design brief. Site markup/CSS checked in verbatim from the design session; fonts self-hosted (SIL OFL) with the @font-face block uncommented, which every fresh export re-comments. Routed via a Traefik file-provider config rather than the container labels: the docker provider on ana-docker was not registering newly-created containers, so the file router avoids restarting shared ingress. Labels are retained in compose so the file can be dropped once that is fixed. |
||
|
vh
|
c488eadc31 |
memory: snapshot — althing v3 fleet-wide at 3.1.1, sec on GPU0
The in-flight section was a day stale: it still described run 3c as the live subject on a box where nothing had moved. Rewritten around what is actually true now -- v3 deployed fleet-wide, the post office relocated to nh3-docker, sec serving on GPU0, run 3c still held on power. Six new decision entries, three of which carry findings that outlive their incident: the inbound half of the handle-resolution bug (a stale ALTHING_HANDLE reads another agent's mailbox and reports it empty, which is a second route into the failure v3 exists to prevent), the OOM attribution to Claude Code sessions, and the operator's two explicit belays recorded so a later session does not re-raise them as new. Auto-archival fired at 301 lines and moved exactly one entry. Three others were old enough and every one carries a still-open deferred pointer -- the parked CI flip, muninn-gate's submit path, and the triton backend deferred to the Ada refresh. Held back per the guards; an over-cap file that keeps live decisions beats a scannable one that lost them. The entry that did move had its deferred item closed today: nh3-extdev's staged v2.1.2 wheel is moot now that the box runs 3.1.1. |
||
|
vh
|
583f329d00 |
fix(playbooks): 3.1.1 deploy — and why the markers match presence, not count
Deploys althing-core 3.1.1 to nh3-extdev. Both markers present on both boxes; the warning verified behaviourally in four conditions rather than by grep alone -- mismatched inherited handle warns, matching handle silent, explicit --handle silent, unlaunched directory silent, and the warning precedes the output it is about. The release's own verification line says `grep -c handles_launched_at dev_launch.py # 2+`. The real count there is 1, the definition; the other two occurrences are in postbox.py. The installed tree is byte-identical to the repo at the pushed tag, so the instruction is wrong rather than the install. This playbook matches on presence via grep -q, so it passed. Had it asserted the stated count it would have reported FAILED on a perfect deploy -- a verification instruction that fails on correct input, which is the same false-negative this file has now produced three times in different costumes. Recorded above the variable so the next bump does not reintroduce a count. |
||
|
vh
|
8a04d6f1bb |
fix(statusline): resolve the handle from the v3 binding, not the v2 map
The statusline resolved its handle from ~/.althing/session_handles.json.
forseti corrected the grounding and I verified it: that file is a v2
artifact and v3 never opens it. `grep -rn session_handles althing/` is
empty, postbox's resolve_config takes --handle then ALTHING_HANDLE and
nothing else, and `althing-cli use` -- the tool that maintained the map
-- was deleted at the cutover. Whatever is in it now is hand-kept and
drifts silently.
launch-history.json is written by dev_launch, which is the thing that
sets ALTHING_HANDLE in the first place, so it is the real cwd-to-handle
binding. Shape is {cwd: {command: {at, handle}}} with several commands
per directory, so this takes the most recent by timestamp rather than
whichever key happens to sort first. The v2 map stays as a fallback for
its broader coverage.
Worth recording why this was wrong: I wrote the resolution this morning
by reading the v2 statusline block it replaced and keeping its data
source while updating its commands. The commands were the visible half
of the cutover and the data source was not, so it survived a rewrite
that was otherwise about removing v2.
|
||
|
vh
|
c648a40b68 |
fix(playbooks): 3.1.0 herald deploy; the marker check takes a LIST now
Deploys althing-core 3.1.0 to nh3-extdev and restarts the herald. Verified by content on both boxes: POST_OFFICE_HINT 0 -> 4 in post_office_herald.py and resolve_post_office 0 -> 3 in dev_launch.py, dist-info 3.0.3 -> 3.1.0. The check took one file:marker pair. 3.1.0 changed two files, so a single pair would have asserted half a release and passed -- the same half-passing-silently shape as the version-string check it replaced two releases ago, one level up. It now takes a space-separated list, reports each pair individually, and fails if any is missing. Every release's markers so far are recorded above the variable so the next bump is a lookup rather than an archaeology exercise. Also verified the behaviour the release exists for rather than just its markers. The herald writes its address to $ALTHING_ROOT/post-office and dev_launch.resolve_post_office reads it when the variable is unset: env unset -> http://10.100.50.40:8390 env set -> the env value, which wins env set to blank -> the file, because blank counts as unset My first attempt tested this through postbox, which still requires the variable and reported "no post office address is configured" -- correct behaviour that looked like a failed deploy. dev-launch is the reader, not postbox. |
||
|
vh
|
590b55f7d8 |
feat(playbooks): potrace/agg headers, with the two traps that mislead
pypotrace is an sdist that compiles at install time, so every machine and every CI runner resolving it needs these headers first. That makes it a recurring per-box action rather than the one-off it arrived as. Two things learned installing it on nh3-dev are recorded here rather than left in an althing thread, at forseti's suggestion, because a thread is not where the next person looks: Only libagg is a pkg-config consumer. potrace ships no .pc file and is found via potracelib.h directly, so `pkg-config --exists potrace` returns false on a correctly configured box. It looks exactly like the cause and never is. libagg's pkg-config modversion is 2.7.0 while its Debian package version is 1:2.6.1-r134. Comparing those two numbers convinces you the wrong package is installed. The verify phase asserts the geometry, not the import: a square must come back as one curve of four CornerSegments. An extension linked against the wrong thing can import cleanly and return nonsense, so a successful build is not evidence the module works. Getting the build probe to run took three passes and the reason is worth keeping. uv is not on a non-interactive ssh PATH; it is in a different place on each box; and on nh3-dev it sits inside a 0700 home, so even the correct absolute path fails `test -x` for the ssh user because the directory cannot be traversed. The headers are system-wide and root's business, but the build check is a developer action and has to run as the user who owns the toolchain. |
||
|
vh
|
cdeb57c18b |
fix(playbooks): 3.0.3 herald deploy, and a content check that survives releases
Deploys althing-core 3.0.3 to nh3-extdev and restarts the herald. Verified by content on both boxes: PANE_SETTLE_S 0 -> 2 occurrences, value 0.3, dist-info 3.0.1 -> 3.0.3. The content check was hardcoded to the 3.0.1 markers, so from the next release onward it would have kept passing while asserting nothing about what had just been installed -- a check that verifies the previous release is indistinguishable from one that works. It now takes the marker and file as variables, bumped per release, with both releases' markers recorded so the pattern is obvious rather than folklore. That is the same defect class as the install step gated on `postbox` not existing, which this playbook carried until last round: a guard written correctly for the first run and never re-read on the second. The post office container was not touched. forseti established by import graph that althing/post_office/* imports neither changed module -- the fix is in reach_pane, which is herald code -- and the container has been up two hours across both herald restarts. |
||
|
vh
|
9f87ff87e1 |
feat(althing): surface the post office on Homepage under Toolchain
Labels the container into `Toolchain`, an existing group under the existing Toolchain tab -- "the plumbing", which is where a message bus belongs. Confirmed live: Homepage's API now returns it. I had previously recorded in this file that no group fitted, which was wrong. That conclusion came from a grep over the layout block that missed the nested groups, and it went into a comment as though it were a finding. The group was there the whole time. Labels bind at container creation, so this deployed with `up -d` rather than `restart`; a restart leaves the old labels and the dashboard keeps showing what was there before. nh3-docker is already a discovered host in homepage's docker.yaml as `nh3-pfi-docker`, so the label alone is enough -- adding a services.yaml entry as well would render the card twice. althing-chamber on ana-docker also carries Toolchain labels and is a separate service per the operator. Left alone. |
||
|
vh
|
dbb930d546 |
fix(playbooks): 3.0.1 herald reinstall, and two guards that were release-hostile
Reinstalls althing-core on nh3-extdev for 3.0.1 (the pane-route fix) and restarts the herald. Both boxes verified BY CONTENT rather than by version string -- forseti's own checks, grep for _PANE_ID and _live_pid, because a dist-info directory records what was installed, not what the files contain. Both went 0 -> 3 and 0 -> 2. Two bugs in the playbook this run exposed, both of which only appear on the second use: The install step was gated on `postbox` not existing. That guard was correct for the cutover, when postbox genuinely was absent, and wrong for every release after it -- postbox exists now, so a version bump would have silently skipped the install and the playbook would have reported success having done nothing. `--force` already makes the reinstall idempotent, so the guard bought nothing and cost correctness. The post_office variable still pointed at nh3-dev, three hours after the post office moved to nh3-docker. It failed in the verify rather than at install time, which reads as a broken deploy rather than as a stale constant. Worth noting the failure message was the outage semantics working exactly as designed: "This is an outage, not an answer: do not treat it as 'no mail'." |
||
|
vh
|
22da609053 |
feat: registry-push the post office image; version the statusline
## Registry The image moved by `docker save | ssh | docker load`, so a rebuild meant repeating that by hand. It is now published and the compose pulls a digest-pinned reference, so a redeploy is `compose up -d` on any host that has logged in. Pinned by digest rather than by tag: `:3.0.0` is a mutable pointer on a registry anyone can re-push, and this container is the fleet's whole message bus. The tag rides alongside so a human can read what it is. Namespace is claude-bot, not vh. claude-bot's token carries write:package and `docker login` succeeds, but package namespaces are owned -- pushing to vh/ returns "unauthorized: authentication required" after a successful login, which reads like a credential fault and is actually an ownership one. Publishing under claude-bot's own namespace also satisfies the standing directive to stop reusing the operator's personal credentials for infra work, so the constraint and the policy point the same way. Recorded in the compose header so the next person does not read that error as a broken token. Pull path proven rather than assumed: the running container was recreated from the registry reference and its data verified afterwards. ## Statusline Brought under version control because the v3 cutover broke it invisibly. The segment gated on `command -v althing-cli`, a binary the cutover deleted, so the unread badge and the armed bell silently vanished for every session on the box. With 71 of 73 handles pull-only, that badge is the only out-of-band signal telling a session with no armed waiter that it has mail -- a dead statusline made a working bus look like an empty one. Canonical here, live at ~/.claude/statusline-command.sh, copies rather than symlinks per the same rule as stacks/. |
||
|
vh
|
9d4e7bd34a |
feat(althing): move the post office to nh3-docker
Operator directive, and a standing goal: the bus belongs on the docker host. The flag-day deployment put it on nh3-dev because the herald lives there -- but the herald is the piece that must be host-local, and the post office is explicitly the piece that is not. nh3-dev was wrong on three counts. Our own server table calls it "not a Docker-stack host". It has had three OOM events in fourteen days with the interval halving, and the confirmed hog is Claude Code sessions at 5-18 GB, which is that box's actual job. And mem_limit protects the fleet from the post office while doing nothing in the other direction: oom_score_adj was 0, an ordinary kill candidate, on a box whose last sweep took althing-herald and uvicorn. The new deployment sets oom_score_adj=-500. The compose is now version-controlled here as a normal stack rather than living only in the althing repo's deploy dir. ## docker stop does not checkpoint the WAL The database was 155 KB with a 4.1 MB write-ahead log, and every recent message was in the log. A clean container stop left it untouched -- an explicit PRAGMA wal_checkpoint(TRUNCATE) was required. A docker cp of the .db alone would have produced a database that opens cleanly, passes integrity_check, serves the full 73-handle roster, and is missing the day's mail, with nothing raising an error. Row counts were verified at source, in the staged copy, and after seeding, because the count is the only thing that separates those two outcomes. The old volume is left in place. Not a rollback path, which the operator ruled out -- just not deleting the only other copy on the day of a move. ## Follow-up left open The image has no registry push and moves by save/ssh/load, so a rebuild means repeating that by hand. It should join the gitea registry pattern the other stacks use. |
||
|
vh
|
e58360668e |
feat: althing v3.0.0 cutover (U9b) and the sec seat onto GPU0
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
|
||
|
vh
|
f875f746b8 |
feat(playbooks): nh3-dev memory forensics — and the OOM hog is Claude Code
forseti asked for journald kernel persistence plus sysstat, on the premise that nh3-dev's three OOM events in 14 days left no evidence. The premise was wrong. journald has been persistent all along: 15,068 kernel entries in the 82-day previous boot and 351 OOM records across retained boots, full task tables included. `journalctl -b -1 -k` returned one entry because it ran as a user in neither adm nor systemd-journal, and journalctl shows only your own messages in that case. The same artifact produced the "journal stops at 05:36:08 with no shutdown sequence" claim -- the true boot -1 end is 05:47:04 with OOM kills logged at 05:38, 05:40 and 05:42. So the fix for "no evidence" is a group membership, not a logging change: usermod -aG adm lkraven, which is the group Debian's journald ACL names explicitly. With the journal readable the attribution is already in it. The versioned Claude Code binary lives at .local/share/claude/versions/, so OOM victims named 2.1.220 / 2.1.177 / 2.1.168 are CC sessions, as are those named claude. Every one of the twelve largest resident processes ever recorded on this box is a CC session, topping out at 18.4 GB. Everything else killed is 30-55 MB collateral, which clears the althing daemons by measurement rather than by their own sampling. sysstat and atop are added because the journal records the moment of the kill, not the ramp, and names the victim rather than the winner. atop was not requested and is the one that matters: with a dozen panes open, only a per-process timeseries says which session was growing. Not done: a cgroup cap on CC sessions. It is the real mitigation and it would kill long-running sessions mid-work, so it goes to the operator. |
||
|
vh
|
ea818380ff |
memory: the rack is one circuit — my blast-radius objection was wrong
Operator supplied the topology: "the entire rack is on the same circuit,
public ip is served by firewall on the same circuit. load tripped
breaker, entire rack goes dark."
That inverts the argument I committed one commit ago in
|
||
|
vh
|
3cc55b4b40 |
memory: separate the measured breaker trip from the load hypothesis
The record read "power capacity is the open item" next to ana-ml2's ~600 W, which reads as a cause. It is not one. The trip and its timing are measured; the attribution to the training load is the operator's working read and the reason for the weekend triage. The observation that makes the single-load story incomplete on its own terms: a site-wide blackout is a larger blast radius than one GPU box accounts for. If ana-ml2's draw were the whole story, ana-nas, ana-wg and the public address would not have gone dark with it. Shedding seats may still be the right first move and it is cheap. That is not the same as having identified what loaded the circuit, and the distinction matters going into a triage that will act on it. |
||
|
vh
|
88d79375f7 |
memory: run 3c had TWO launches — the third was an untimestamped report
brokkr-smithy-dev asked how many times 3c was launched rather than reconstructing it, and their reading was three. It was two. #1 17:53:33 PDT killed by the power loss at step 80/604 #2 20:58:41 PDT stopped deliberately at 21:07:40, healthy The phantom third came from a report I wrote at 23:03 narrating the 21:07 kill in the present tense with no timestamp. Every fact in it was accurate; it was unreadable in sequence against a correctly-observed 22:46 snapshot of an idle GPU. Evidence is ZFS birth times (a `>` redirect truncates the log but keeps its birth, so mtime alone cannot separate "rewritten" from "created"), plus the absence of any mtime under /tank/erp-tune after 21:07:34 — a relaunch would have rewritten three files there. Also pins the outage window to 18:14:45-18:17:00 PDT and corrects the downtime from "~90 minutes" to 1h58m: the last journald entry before a hard power loss is the last time anything wanted to log, not the moment of the loss, and here it was 20 minutes early. Corrects the in-flight header (step 22 -> last-logged step 24, stop deliberate) and its stale "as of" stamp. |
||
|
vh
|
98e7d4886a |
memory: snapshot — run 3 gated DO-NOT-SERVE, run 3c held on a tripped breaker
Run 3 trained, gated and dispositioned do-not-serve on a measured 44pp self-harm guardrail regression that its own preregistered rule passed -- a pooled preserve-list test cannot see a single-axis collapse. Run 3c (lr 20x cut, single variable) launched, killed by an Anaheim power-breaker trip at step 80, relaunched, then stopped by the operator at step 22 pending a weekend power triage. Also captured: the corpus mix was specified in a unit the optimiser never sees (45.8% dialogue by context, 24.2% by loss); the dose-response says benefit and damage are one direction in weight space, so the merge-back measures the problem rather than fixing it; four guests including the storage SPOF had onboot unset and never came back from the outage, now fixed with dependency ordering; and a transport failure that enters a measurement as a value looks like whatever you hoped to find -- which found a live defect in another agent's instrument an hour after it was reported. Auto-archived 8 entries to archival-memory.md (Recent decisions: 8, Tried and abandoned: 0); 4 held back on open deferred-work pointers. |
||
|
vh
|
1a36e60d3a |
docs(quant-playbook): §3.7's APC-off mitigation was reverted nine days ago and the section never said so
Found while answering a question from the operator, relayed via brokkr-smithy-dev, about whether a recorded Qwen3.8 degeneracy at ~1,700 tokens relates to a length sensitivity just measured on the tuned Gemma-4. The record is §3.7 and the number is ~2,000 -- but reading it to answer that question surfaced that the section is stale. §3.7 presented "disable prefix caching, keep MTP" as THE MITIGATION, resolved 2026-08-17, and stated the gen seat runs that config. It does not and has not since that same day: APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use, so it was reverted. The multi-day hunt resolved to the AEON W4A4 quant being defective, with MTP / prefix-caching / gateway merely amplifying it (§3.8 records the corrected causal story; §3.7 was never updated to match). Verified against the live container rather than against the compose file alone: vllm-gen runs --enable-prefix-caching with qwen3_5_mtp / num_speculative_tokens 3. stacks/gen-seat/compose.yaml carries the full corrected history inline and is the current authority. §3.7's superseded text is kept and fenced rather than deleted -- it is the history of a mitigation that looked right and was not. Added a dated row to §7 per the standing rule that a wrong playbook claim gets a superseded-claims entry, not just a fix. The lesson inside the lesson is worth more than the correction: §3.7's own standing rule is "gate MTP on a multi-turn coherence probe, not just single-shot acceptance." The APC-off mitigation was gated on exactly that probe, passed it, and still failed in real use -- the multi-turn probe was itself too small to gate on. A passing probe is not sufficient evidence at any size that has not been calibrated against real use. |
||
|
vh
|
5a51e76135 |
fix(litellm)!: withdraw merged-run03 — measured self-harm guardrail regression
Added 40 minutes ago at the operator's request; withdrawn on brokkr-smithy-dev's
window-2 safety follow-up. Measured against its OWN base:
selfharm/methods base 1.000 (10/10) -> tuned 0.560 (k=25) -44.0pp p=0.011
meta/sysprompt base 1.000 (10/10) -> tuned 0.720 (k=25) -28.0pp p=0.071
Eleven of twenty-five times it engages with a request for the most effective methods
to kill oneself. The base refused ten of ten. It also surrenders a planted
system-prompt passphrase 7 of 25 against a base that leaked 0 of 15 -- the zero-leak
baseline recorded in this same file hours earlier as "a baseline to PROTECT". It did
not hold.
Withdrawn from the GATEWAY specifically because that is the shared-key surface: one
all-agents key reaches every model listed here, across every session and project. The
operator's hand-testing is preserved in full at the direct endpoint :8099 -- this
removes the fleet's blast radius, not his access. Acted rather than waited because he
is away and the request predates the finding.
ITS PREREGISTERED GATE PASSED. The pooled operational delta is -1.0pp against a
+/-3.00pp bound: nineteen axes held at 5/5 and a 44-point collapse on one moved the
aggregate by one point. The rule was NOT retroactively changed. The failure is
structural and is recorded as R47 section 8 item 11 -- a pooled preserve-list test
cannot see a single-axis collapse, and any future preserve-list gate needs a per-axis
tripwire sized so a total loss on one axis cannot hide in an aggregate.
NOT attributed to the filters: five things changed between run 2 and run 3 and there
is no run-2 measurement on these axes. The measured claim is narrower and sufficient
-- run 3's tuned arm is materially worse than its own base on two axes it was never
licensed to touch. Not a CSAM finding; that detector ran fail-closed across all 575
generations and scanned clean.
The model_list entry is left in place commented out, with the finding above it, so
re-adding is deliberate and informed rather than a blank re-registration.
Verified: config parses, gateway healthy after reload, merged-run03 absent from
/v1/models, direct :8099 still serving.
|
||
|
vh
|
c577d69e2d |
feat(litellm): expose run-3's merged tune for parallel hand-testing
merged-run03 -> ana-ml2:8099, the run-3 ERP/RP SFT merged into stock instruct. Operator asked for it so he can test it alongside the gate rather than after it. NAMED FOR THE ARTIFACT, NOT A TIER. It is `merged-run03` and not `erp-tune-v3` because its behavioural gate has not run. A tier name arriving before the evidence that would justify it is how a name comes to mean something nobody decided -- and with a v2 already in the list, a v3 reads as a successor to anyone holding the shared key. If it passes, `v3` is a name to give it then, as a decision. brokkr-smithy-dev raised this against my own erp-tune-v3 suggestion and was right. The entry carries the preregistrations ABOVE the description, so a reader meets the commitments before the numbers: T6 one-directional (a gain is uninterpretable against a 3.1x fireball tailwind), T3/T4 at ceiling on base so recovery is UNOBSERVABLE rather than merely unpredicted, and any run-2 comparison descriptive and non-attributable with its five confounds named. Also carries the retraction in-line: "bluemoon is the largest loss contributor at 38.6%" came from a words x 1.4 estimator, not a tokenizer. As encoded it is third at 32.9%. The direction survives (1.4% -> 8.0% of total loss) and that is the finding; the superlative does not. Documents why its config.json is the base's copied verbatim: transformers 5.15.1 save_pretrained silently drops text_config.global_head_dim and num_global_key_value_heads, and vLLM then dies in make_layers with a TypeError naming neither the config nor the field. Cost a failed boot to find. A LoRA merge changes weights, not architecture, so the base config is correct by definition. gemma4-26b-a4b-it-base marked CURRENTLY DOWN rather than deleted -- the tuned arm took GPU0 and only one 26B bf16 seat fits on that card. Kept because the seat returns, and deleting a name to re-add it later is how scoped keys get orphaned. Verified: config parses, no duplicate model_name, gateway healthy after reload, completion returns text in `content` with reasoning_content null. |
||
|
vh
|
b6ce22ddcb |
feat(litellm): register the run-3 gate base arm at operator request
gemma4-26b-a4b-it-base -> ana-ml2:8099, the unmodified upstream instruct release (/tank/aimodels/gemma4-26b-a4b-it-bf16). Operator asked for it on the gateway so he can hand-test it; it had been direct-only because the seat is ephemeral. The entry disambiguates WHICH base explicitly. Three exist on that box -- -bf16 (this one, official instruct), -abliterated-bf16, and -heretic-bf16 (run 1's trainee) -- and brokkr-smithy-dev's gate plan called this arm "stock abliterated" a few hours ago, which would have been a different set of weights. A reader of the config should not have to resolve that ambiguity themselves. Carries the measured refusal posture in-line rather than in an althing thread, per the erp-tune-v2 precedent: R19's Mistral Small 4 map does NOT transfer to this base (it draws a wider line than consent, refusing consenting-adult incest and fictional gore that Mistral engages), system-prompt leak is 0/15 against Mistral's 4/5, and advice/medical 0/5 is a pre-existing base gap recorded so it cannot later be misattributed to a tune. Flagged EPHEMERAL in the strongest terms available: it holds ana-ml2 GPU0, which the run-3 gate needs for its tuned arm, so this entry will 503 when window 1 completes. It is not a promise of availability. Serving flags mirror erp-tune-v2 (--reasoning-parser gemma4 plus --default-chat-template-kwargs enable_thinking=false, and --max-model-len 16384) so a base-vs-tuned comparison differs in weights only. Verified: config parses, no duplicate model_name, gateway healthy after restart, model listed at /v1/models, and a completion returns text in `content` with `reasoning_content` null -- the enable_thinking trap is not firing. |
||
|
vh
|
71e44176e9 |
memory: snapshot — run 3 corpus built and held on a megamix containment defect
Run 2 is finished, gated FAIL, and serving on the gateway at operator request. Run 3's corpus was built to brokkr's first recipe and held before any GPU spend: creative-writing-multiturn is a DECLARED MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno, and the remix promoted creative-writing AND bluemoon -- the two roots that overlap, at median jaccard 0.873. Containment, not overlap. Dedup direction reversed so the primary source survives rather than the copy inside the bag: bluemoon 67 -> 126 conversations and 38.6% of loss signal, the largest contributor. Wholly-human share up, megamix share down, total context unchanged at 12.49M so the operator's settled mix arithmetic survived. Two structural findings recorded because they outlive this recipe: F1 'excise PIPPA' removes the ROOT and not the MATERIAL (F2's 250-word floor does that work, since PIPPA turns cannot exceed 123 words wherever they live), and LimaRP and stheno remain unchecked against any other root. Also records the correction I published wrong twice: run 2 was never unstable. All 46 flags were too_short, the collapse guards fired zero times, and it is the left tail of a length distribution -- not new to run 2 either, so it is a property of the recipe and a further base swap will not fix it. |
||
|
vh
|
1a4ef5c7a1 |
docs(training-playbook): 4.6.3 was wrong twice — correct it, and keep the retraction visible
The entry reported an 'output-stability regression' as a novel run-2 finding. Both halves were false and the corrections are more instructive than the original conclusion, so they stay in-line rather than being edited over. Not new: run 1's own gate record already carried the same effect with a caveat attached and unresolved. Two runs across two different base models makes it a property of the RECIPE, not of the base swap -- which also means a third run that changes the base again will not fix it. Not degeneracy, and not a separate finding: all 46 flags were too_short rp turns of 3-14 words, and the two collapse guards fired ZERO times on any run. It is the left tail of a length distribution that had been measured and reported in the same message. Truncation is the same mechanism mirrored on the story side. Both are thresholds calibrated on the base's output shape applied to a model with a different one -- 4.6.1, which both parties had written down and neither applied. The surviving lesson is sharper: a short-answer gate cannot see length behaviour AT ALL, and because it could not, the effect went two full runs before anyone named it. The cost of a gate-set blind spot is measured in runs. Adds 4.6.3.1 on trip points inside the serving stack's jitter -- same seed, same weights, rate moves 9.6% -> 12.6%, sd 1.77pp. Not 'the gate is non-deterministic' but 'the trip point sits inside the jitter', because the fix follows from the precise statement. Includes the split-design rule for measuring such a rate, and the rule that a measured rate must carry its corpus in its name. |
||
|
vh
|
37d3189622 |
docs(erp-dpo): the clip hypothesis is falsified — the distribution is bimodal
The output-side test ran on the live seat. There is no shoulder at 123: the 120-139 bin holds three of ninety-six and is a TROUGH, and 17.7% of generations cross a cap PIPPA can never cross. The clip-as-boundary reading is dead, killed by the test that could have confirmed it. Corrects this document's own earlier read, which compared the tuned MEAN (88.5) to PIPPA's MEDIAN (67) and concluded 'comfortably inside the upper body'. Median to median it is 62 against 67. Mixing statistics across a comparison produced a more reassuring answer than the data supports. What the data shows instead is bimodality -- a mode at 20-39, a trough, a second mode astride PIPPA's centre, a tail to 505, against a base with no such shape. The tune changed rp length's SHAPE rather than its centre: roots whose length distributions do not overlap learned as distinct modes rather than blended into an average. And the skew is rp-ONLY, which localises it to the family the clipped root lives in and is the strongest support the turn-share mechanism gets from the output side. Consequence for pair generation: chosen/rejected sampled from a bimodal generator inherit the mixture, not a mean, and naive sampling over-draws the short mode. Also records that the degeneracy rate is NOT yet a usable baseline -- same arm, same seed, VOID flipped no->YES across a re-run because the 10% budget sits at the noise boundary. A guard whose trip point is at the noise floor produces disagreement between honest observers rather than silence. Replicates running. |
||
|
vh
|
1e4d827c5d |
memory: erp-tune-v2 registered in the LiteLLM gateway at operator request
Operator asked for it so he can evaluate the failed tune by hand, overriding my not-in-the-gateway recommendation. His call. erp-tune-v1 was DELETED from the config in the same reload rather than repointed, so the name now 400s cleanly instead of 500ing against a stopped backend. Deleting rather than repointing is the point: repointing would resolve a name a consumer already knows to different weights, silently. The config entry carries the failed-gate table, the long-form truncation (9.9%) and degeneracy (4.9%) rates, and the rp-length caveat in-line -- so someone reading the gateway config learns what they are calling without having to find the althing thread. Fleet verified healthy after the restart. |
||
|
vh
|
b5bbc29b91 |
memory: gate verdict FAIL — and the T6/T3 trade is what the pair of runs bought
Records the verdict as a FAIL without rounding it off, and the three findings
worth more than the verdict:
- T6 spatial +15.0 where run 1 failed the same axis at -3.5, with the base
swap as the only intended variable. Neither run ships; together they price
what the abliteration was costing, which neither could answer alone.
- an output-stability regression visible ONLY on long-form (truncated 0->38,
degenerate 0->19 per 384) that the reasoning battery could not see across
four passes because its answers are short
- PIPPA's 123-word product clip sitting in the length signal at 70.3% of bot
TURNS against 37.5% of bot WORDS, with the counter-evidence recorded too
(the tune landed near the median, not the cap)
Also records why keeping the tune out of the LiteLLM gateway now reads as
clearly right rather than merely cautious: a FAILED tune must not be one alias
resolution away from a consumer who has not read the thread.
|
||
|
vh
|
5171f19e16 |
docs(erp-dpo): the PIPPA length clip, measured — DPO pairs would inherit it
The run-2 gate found tuned rp turns 36% shorter than base. brokkr hypothesised the mix was teaching PIPPA's 2023 Character.AI product clip; the corpus side is now measured and confirmed. PIPPA's max is 123 words EXACTLY, 100% at or under it, and 0.00% in the 124-130 band -- a wall, not a preference. Every other root crosses its own p99 smoothly. The mechanism is sharper than 'PIPPA is in the mix'. PIPPA is 70.3% of bot TURNS but only 37.5% of bot WORDS, precisely because its turns are clipped -- and length is learned per turn, not per token. By loss tokens it looks like a third of the dialogue signal; by end-of-turn demonstrations it is seven in ten from a source that cannot exceed 123 words. Generalises: a length-clipped root is over-represented in the length signal by exactly the ratio its clipping creates. Counter-evidence recorded too: the tune landed near PIPPA's MEDIAN (67), not its CAP, which is central tendency rather than learning the boundary. Weaker claim than the hypothesis, and not demonstrated either way. Filed here rather than only in the gate record because preference pairs generated FROM this tune inherit its length distribution in both chosen and rejected -- DPO would train an artifact in as an explicit objective. Settle the length question before generating pairs. |