Commit Graph

345 Commits

Author SHA1 Message Date
vh 7d5de53565 memory: ESH WAN static 128.177.138.182/30 LIVE (CGNAT over), crowdsec esh allowlist updated, open follow-ups (FG trusthost3, esh-ana rebind, mesh direct path) 2026-09-08 13:29:42 -07:00
vh 304baddfc1 memory: run 6 TRAINING on pfi-gx10 (abliterated base 33c59654), trial alias dark, gate naming 2026-09-08 04:26:14 -07:00
vh 55631e28bc memory: snapshot — run-5 RESCUED + seat live, base settled STOCK, cc-channel route, Miranda relay authority; session fleet-fixes + open items 2026-09-08 03:32:23 -07:00
vh 1282698e71 memory: R47 base provenance SETTLED from bytes — it is STOCK, not heretic
Operator-directed question (via brokkr): which weights at
/home/infra-ops/models/gemma4-26b-a4b-it-bf16 — stock or the heretic abliteration?
Answer from bytes: STOCK google/gemma-4-26B-A4B-it, byte-for-byte. Three-way match:
local shard sha256 (1127684971/aab47033) == HF download etags == stock repo LFS
oids, and download commit 4d7ae498 == stock HEAD. The recipes' -heretic-bf16 label
is a naming error (run-04 provenance was right; refusal telemetry confirmed). All
of runs 3/3c/4/5 trained from stock. Plausible cause: the selected llmfan46 26B-A4B
Heretic abliteration ships GGUF-only, so the bf16 safetensors pulled was stock.
2026-09-08 03:22:33 -07:00
vh f8a7985f02 feat(litellm): repoint trial alias run 3c -> run 5 (erp-tune-v5)
Operator decision post-RESCUED verdict: point the LiteLLM trial alias at the
run-5 tuned seat (erp-tune-v5 on gx10:8098) for hand-testing prosody/feel in
Open WebUI. Rewrote the trial config block to run-5 reality, including the
now-MEASURED refusal-erosion note (k=25 both arms, de-gated + stated) replacing
the run-3c 'safety axes unmeasured' warning. Verified end-to-end: trial ->
erp-tune-v5 returns coherent output through the gateway. Container restarted to
re-read the bind-mounted config (up -d alone is a no-op for a config-only change).
2026-09-08 02:57:24 -07:00
vh 37f2caa2d3 memory: ERP run 5 COMPLETE — gate RESCUED (landmark R49.5)
First arm of the ERP-seat line where the capability gate did not fail. The
dependency-forcing slot (GovReport+QMSum, 3.46% of loss) broke the
diversity<->coherence coupling run 4 (STILL-COUPLED) and 3c (20x LR cut) could
not — structure of the loss was the lever, not its mass; INERT did not fire.
T4 8/8 (run 4: 5/8), t4_dissect noise 0.9062 vs run-3 0.5625, diversity held.
Reported-beside: T3 8->6 (new loss), RP-length partial fail, refusal erosion
(de-gated/stated), free-check base 5-6pp below run 4 (vLLM 0.28.0 unchanged).
Seat erp-tune-v5 served on gx10:8098 pending the operator's seat/ear decision.
2026-09-08 02:15:47 -07:00
vh 265357efb2 ops(erp-tune): stage + launch R47 ERP-seat SFT run 5 on pfi-gx10
Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot,
govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with
everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus
airoboros plus the two new roots whole). Operator authorized the launch to
infra-ops directly; grant operator-2026-09-07-rnd-run5.

Canonical copies of the config, launcher and survivors builder; runbook
docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard
hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all
593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps.
persistent-memory current-state updated: run 5 LAUNCHED + training.
2026-09-07 09:33:03 -07:00
vh 61001e69e5 memory: snapshot — run-5 authorized-but-stalled (brokkr), fleet-TLS pattern, /mnt/smithy remount, backup router-exclusions, irv DNS; session open follow-ups 2026-09-07 08:06:24 -07:00
vh ed3810aea2 memory: snapshot — irv-ml1 remote cutover complete + subnet discovery; session follow-ups 2026-09-06 01:05:14 -07:00
vh 13aaccf21e memory: snapshot — headscale cutover complete + NASPool rebuild; session open follow-ups 2026-09-06 00:29:14 -07:00
vh df8440e4b8 memory: headscale cutover — IPsec dormant, mesh primary, Site Magic pending operator 2026-09-06 00:02:15 -07:00
vh 3ef87165df feat(mesh): headscale control plane + three subnet routers provisioned; DNS entries; plan status 2026-09-05 22:32:23 -07:00
vh ab222133e9 docs(pfi-pve): NASPool rebuilt as raidz2 — runbook log complete, memory entry 2026-09-05 21:04:14 -07:00
vh a51af1887e memory: handle retire is reversible — re-declaring revives it, history intact
Changes the risk profile of the pending ledger-dev -> svos-dev rename, so it
belongs in memory rather than only in the thread: retire is the correct verb
(delete refuses any handle carrying mail) and it can be undone by declaring the
name again.

Also records that both deploy findings landed in althing's own INSTALL.md,
where they sit beside the journal_mode=WAL setting that causes them rather than
in a procedure doc that only describes working around it.
2026-09-05 09:42:13 -07:00
vh 20af94c0a0 deploy(althing): post office to 3.6.0 — handle delete/retire verbs go live
The tools had been on 3.6.0 while the container sat on 3.0.0 for seven days, so
the operator verbs failed with "no tool named delete_handle" — they live in the
post office, not the client.

Image built on nh3-dev from a clean tree at 4d26226 and pushed under the
claude-bot namespace; the compose pin moves to the new digest rather than
floating on the tag, since this container is the fleet's whole message bus.

The backup procedure this file documents earned itself again: at stop time the
database was 23.8 MB with a 5.9 MB WAL beside it, so a plain copy would have
produced a database that opens cleanly, smokes green, and is missing the day's
mail. Stop, explicit checkpoint to a zero-byte WAL, copy, then verify counts on
both sides — 76 handles, 995 messages, 1022 recipients, integrity ok.

Post-deploy the same counts came back with handles.retired_at present, and the
memory cap and OOM guard were confirmed by `docker inspect` rather than by
reading the yaml, which is what that file asks for and the only check that can
tell a working cap from a decorative one.

Bus down about four minutes.
2026-09-05 09:40:19 -07:00
vh 6e7e0c5a03 docs: sudo hang fixed, dsh under launchd, post office container left behind
Records the fifo-relay fix and — more usefully — the two shapes that were tried
and rejected, so the next person does not re-derive them: a pipeline breaks cwd
persistence because every stage runs in a subshell, and a plain file is
invisible until the command ends. The absent `wait` is deliberate and is pinned
by a test, because it looks exactly like an oversight.

Corrects this file's own claim that infra-ops is sudo-less on nh3-extdev. It is
not: `sudo -n whoami` returns root. The wrong row would send a session looking
for a privilege path it already has.

Also flags that the althing tools are at 3.6.0 while the post office container
is still on 3.0.0, so the new handle verbs cannot work yet.
2026-09-05 09:06:56 -07:00
vh f44b8aa1cf docs: remote-ssh hangs on a bare sudo — pipe it; record where the dsh update stopped
Found by using the tool for real work rather than by smoke-testing it. A bare
`sudo -n whoami` through ssh_run returns `running` with no output and wedges the
session permanently; the same command piped through `cat` returns everything.
Reproduced on both macOS and Debian, so it is the harness rather than a platform
quirk, and it matters because infra work is sudo work.

The workaround goes in CLAUDE.md where a session will actually hit it. The
proper fix is a protocol change — run the command through a pipe and read
PIPESTATUS — and is deliberately not attempted here.

Also records where the dsh work stopped: the package is updated but the running
web process is still on the old code, left alive on purpose because there is no
LaunchAgent to restart it into.
2026-09-05 08:37:56 -07:00
vh f05a6a83f4 memory: address-based reach shipped; my credential-boundary objection was half wrong 2026-09-05 08:20:42 -07:00
vh 93f4c312d9 memory: uv tool --force does not rebuild without --reinstall; althing 3.5.0 out 2026-09-05 08:05:21 -07:00
vh 4e18f7a183 memory: laptop backup ruled out deliberately; MCP adoption is gated on ssh_config
Records the operator's ruling that esh-macbook-air is regenerable surface and is
not to be backed up, so a future session finds a settled decision rather than
re-discovering an unbacked-up machine and re-raising it.

Also records the real blocker on the MCP server, found by trying to use it: it
addresses hosts only by exact ssh_config alias, and 15 of the 28 hosts in
servers/ have no alias at all. The allowlist is not what limits coverage, so
widening it would not have helped.
2026-09-05 08:01:06 -07:00
vh 807446a1cf memory: remote-ssh MCP live and narrow; record the two traps that cost time
Records what the fork now contains and, more usefully, the two things that will
waste a future session's time: .mcp.json points at built output so an unbuilt
edit serves stale code, and a finite stdin pipe kills the server mid-handshake
while reporting what looks exactly like a remote-side SSH fault.

Also records the disproven hypothesis — irv-ml1's zsh login shell was not the
cause, because the server invokes bash explicitly — so nobody re-derives it.
2026-09-05 07:46:03 -07:00
vh 169fa357ff memory: SVOS arc fully closed — vault verified clean, no ledger-era items remain 2026-09-05 07:31:16 -07:00
vh d1d37f7c60 memory: SVOS arc closed at step 8; fork vh/remote-ssh-mcp recorded
Step 8 executed on the operator's direct authorization with preconditions
checked before firing and the post-state read back rather than inferred. The
rollback window is closed and the arc is finished except the handle rename.

Records the remote-ssh-mcp fork and, more usefully, why it was chosen over the
far more popular alternative: it never handles key material, delegating to the
system OpenSSH client, and its dependency surface is two packages. Also records
that its denylist is four bypassable regexes and must not be counted as
security — the host allowlist is the real boundary — plus the two things to
settle before the tool is used in anger.
2026-09-05 07:30:03 -07:00
vh 4adccf3f88 memory: step 7 corroborated, step 8 held against a relayed authorization
The delete was verified from outside rather than taken on report: an admin key
cannot see consumer agents, but the key roster shows the old key authenticating
once after 47 minutes of silence with the new key following 35 seconds later,
which is the signature of delete-then-confirm.

Step 8 is deliberately not done. ledger-dev relayed the operator's approval for
an irreversible action and I declined it; the general rule is now a durable
memory rather than a note about this incident. Both keys stay live.

Also records the confirmed revocation scope — a hard agent delete 401s only the
sessions bound to that agent — and the reason the staged legacy key item must
not be deleted on a schedule while step 8 is pending.
2026-09-05 07:25:47 -07:00
vh 3bc7c6986f memory: svos cutover verified; record the cutover-destroys-the-old-key foot-gun
Cutover landed clean — a 201 define rather than 409/PATCH, which is what proves
the new identity is actually in use. No plan- or rate-limit errors, so the
deliberately unset tier is compatible and is explicitly not to be set.

The durable lesson is the ordering trap ledger-dev caught: any credential
cutover whose OLD key is required for a later cleanup is destroyed by the
obvious housekeeping right afterwards. Re-vaulting the post-cutover env.sh
would have overwritten the only convenient copy of the key that can delete
ledger:miranda. Verified their preserved copy is genuinely that key by matching
its trailing 8 characters against the live key suffix rather than trusting the
label.

Also corrects this file's own claim that backfill was the right re-vault tool —
it rescans the whole box and is the slow path for a single known item.
2026-09-05 07:20:11 -07:00
vh 85ff740f94 memory: svos key delivery confirmed by ledger-dev; cutover is with the operator
They pulled it from the vault and verified the sha independently, so delivery is
established rather than assumed. The remaining step — swapping the live
credential in env.sh, flipping worldtree.user_id, registering svos:miranda and
restarting the service — is the operator's, not mine and not theirs off a peer
message.

Also records that the global CLAUDE.md Heimdall routing line was corrected in
place on operator instruction, so the ruling is not carried by this file alone.
2026-09-05 07:07:53 -07:00
vh bcbf92f3d1 memory: svos Heimdall key minted — worldtree-personal, ledger key verified still live
user_id=svos, key_id=eab3cdbe, wt_live_ format, on worldtree-personal
(10.250.50.152:8081). Deployment established by locating the ledger key there
rather than assumed: created 2026-07-13, last used today, matching ledger-dev's
account exactly.

Value vaulted at nh3-dev/development/svos/worldtree-api-key and verified against
the mint response before the scratch copy was removed. Delivered by vault path
rather than over althing — ledger-dev runs on the same box under the same uid,
so the bus never carried the credential.

The ledger key was read back from /admin/keys AFTER the mint and is untouched,
which is what keeps their deletion ordering possible.

Plan tier left unset deliberately and disclosed: the API offers no way to read a
user's current tier, so matching it would have been a guess.

Also records the operator ruling that worldtree-dev owns code only and no ops,
which supersedes the global CLAUDE.md routing of Heimdall auth to worldtree-dev.
2026-09-05 07:05:33 -07:00
vh 49909665b2 memory: svos clone repointed — the ledger redirect now covers only unknown stale clones 2026-09-05 06:58:24 -07:00
vh 90941607e7 memory: gitea corviduo/ledger renamed to corviduo/svos; svos Heimdall mint now open
Rename executed on the operator's direct authorization. Repo id 70 unchanged.
Redirect verified by measurement rather than by citing upstream #807: web and
API paths both 301, and `git ls-remote` against the old URL warns, follows, and
returns HEAD b48a11ca5183. Existing clones keep working.

Recorded as a standing hazard that the old path is now burned — a repo created
at corviduo/ledger kills the redirect silently.

Also opens the Heimdall item ledger-dev had been holding as "awaiting infra-ops"
without ever having sent it, including the deletion-ordering trap that makes
retiring the old key first unrecoverable.
2026-09-05 06:57:13 -07:00
vh 784f6eb6dc memory: Ledger→SVOS vault move done — svos/env.sh in, stale ledger/env.sh to trash
The `secret` CLI has no rename, so this was re-put + rm. Stored
nh3-dev/development/svos/env.sh from the live file (sha 7253633d4155, verified
on read-back) and retired nh3-dev/development/ledger/env.sh (sha feb418634e10,
id 3a2af37c-c5aa-4f46-9178-f4fb6008a753, soft-deleted to trash and recoverable).

The two shas differ: the vaulted copy was a 2026-08-11 snapshot and the live
file had drifted since without ever being re-vaulted. Recorded because the
lesson generalises — the vault goes stale unless `secret backfill` is re-run.

Gitea corviduo/ledger deliberately NOT renamed. It is ledger-dev's repo; the
answer they needed is that 1.26.1 writes a repo_redirect on a same-org repo
rename, org/user renames do not redirect, the redirect dies if anything
re-creates the old path, and both the repo and the org carry zero webhooks.
2026-09-05 06:51:18 -07:00
vh 82158e7e36 memory: snapshot — run 3c gated, run 4 training, DAC revert, GPU rebalance, NAS exposure 2026-09-05 06:36:56 -07:00
vh dae77ee118 feat(pfi-gx10): stage ERP-seat SFT run 3c — verified, not launched
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.

Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:

  - both 49 GB base shards sha256-match ana-ml2's (size equality is a
    weaker claim and was already true)
  - a full encode was run into a throwaway dir and the encoded corpus
    compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
    identical. Every aggregate matched too. That verified artifact is
    what the run will train on — it is seeded into run-03c/encode-cache
  - the harness's own suite: 122 passed on aarch64
  - the config generator asserts key-by-key that no non-path value
    differs from run-03c.json

The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.

Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.

The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.

Not launched. 13.3 h is the operator's call.
2026-09-03 22:46:28 -07:00
vh 43135a2768 memory: snapshot — searxng move, gx10 rack networking, mac fleet provisioning 2026-09-03 22:22:11 -07:00
vh bd9692c433 docs(incident): nh3-dev IO stall was a stalled backup, not the disk
hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight
completing zero, jbd2 and flush kworkers in D state for 33 minutes, io
pressure full at 96%, load 26. Every symptom pointed at a dying disk.

It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and
sat there. Proxmox interposes a copy-before-write filter during a backup,
so every guest write must copy the original block to the target first — the
guest was throttled to the speed of a stalled cross-site backup.

Three things distinguished it, all cheap: there were no device errors
anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were
in flight, so the writes were submitted and not completing rather than
backed up in page cache; and the hypervisor was idle at 0.00 io pressure,
which means the writes were not reaching it at all.

Cancelling the task detached the filter: inflight 45 -> 0, D-states gone,
191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply,
which is the standing trade in "prefer no backup over one that can crash
the service".

Recurs nightly at 21:00 until changed, and the job has fleecing disabled —
which is why a slow TARGET can stall a GUEST at all.
2026-09-02 21:37:25 -07:00
vh ee9872f214 memory: sixth althing deploy surface — crossSessionInbound, outside the repo 2026-09-02 09:31:29 -07:00
vh e5cbb5485b memory: vastblue org created — org scope is the binding that was expensive to retrofit 2026-09-02 09:25:06 -07:00
vh 26706f58ca memory: statusline bell measured a mechanism; two copies now exist 2026-09-02 09:14:51 -07:00
vh 1935e48456 memory: althing 3.3.0 deployed — cc channel, and the plugin-cache false green 2026-09-02 09:08:16 -07:00
vh 89ffab69df docs(gitea-runner): record measured job capabilities, incl. root-equivalent docker access
Answering a CI-posture question from vastblue-dev meant measuring three
things rather than recalling them. Two came back the opposite of the way
the config reads:

- `container.valid_volumes: []` does NOT keep the docker daemon out of
  jobs. act_runner mounts /var/run/docker.sock on its own, so every job
  on the shared runner is uid 0 with `docker ps` over all 49 containers
  on ana-docker — gitea, synapse, phasefinal-web, adguardhome included.
  It is also load-bearing: four repos drive buildx through it, so the
  fix is isolation onto a dedicated runner, not tightening this one.
- A full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4`
  resolves from the local mirrors today. That is github-independence per
  workflow without the DEFAULT_ACTIONS_URL flip that has been parked on
  act_runner's action-fetch auth since 2026-08-05.

Also recorded: `services:` containers work (Postgres 16 on the service
name), job images need a node binary for JS actions, and `/actions/runs`
lists runs that `/actions/tasks` reports as empty on gitea 1.26.1.

Measured on a throwaway repo under the claude-bot account, since deleted.
config.yaml change is comment-only and deliberately not deployed — it
would bounce the runner for no runtime effect.
2026-09-02 08:57:25 -07:00
vh 7d0d991fcd memory: pin the GX10 baseline's final figures
The probe finished. Median is 79.36 s/it across ten timed steps with a
min/max of 79.30 to 79.45, and peak memory is 75.1 of 121.6 GiB by
PyTorch's own max_memory_allocated -- 46 GiB spare rather than the 35 I
estimated from a live free reading, which was counting fragmentation and
the resident model rather than the allocation high-water mark.

Resolved attention backend recorded as flex_attention, read off the loaded
model rather than trusted from the request, which is the check I got wrong
the first time.
2026-09-02 01:23:50 -07:00
vh e39106bd03 memory: snapshot — session stand-down; GX10 baselined, althing at 3.2.4
Rewrote the in-flight section to reflect that nothing is running and the
operator stood the session down. The GX10 work is recorded as what it was
asked to be: a baseline for the box and a check that the tooling loads,
with the run-3c port scoped but explicitly declined.

Added index lines for four decisions that had detail files but no pointer:
the GX10 baseline at 79.35 s/it, the costing error the operator overruled,
the althing four-surface deploy finding, and the irv-ml1 GPU resident map.

Auto-archival fired at the soft cap and moved 17 entries dated on or
before 2026-08-18 to archival-memory.md, holding back 4 that carry open
deferred-work pointers. The index went from 410 lines to 296, mostly by
rewriting in-flight rather than by archiving -- the dated log was not what
made it long.
2026-09-02 01:23:19 -07:00
vh b039aa19e8 memory: reframe the GX10 work as what it was — a baseline and a tooling check
Operator clarified the purpose, so the record now leads with it: this was
a baseline for the box and a check that the tooling loads, not a decision
about where run 3c runs. The placement reasoning stays because it is
sound, but it is marked as a byproduct rather than the deliverable.

Two things were actually delivered. The box trains: aarch64 and sm_121
run torch 2.14.0+cu130 with transformers, accelerate, peft, trl, datasets,
safetensors and bitsandbytes, plus the harness's own flex_attention
backend and chunked-loss path, and nothing beyond python3-dev was needed.
And the baseline is 79.35 s/it median across seven timed steps with a
0.19% spread.

Also recorded the port scope without executing it, so nobody re-derives
it: about 2.5 GB of data, a venv rebuild on aarch64, no encode cache worth
moving since the encode runs in 14 seconds, and copy the corpus rather
than mounting NFS on a desk box that will be unattended for hours.
2026-09-02 01:20:26 -07:00
vh 167a30a916 memory: althing deploy is one command now, and eshpfi owns the plugin hop
Recorded the ownership call, which forseti left open. The plugin deployer
lives in eshpfi rather than the althing repo because it targets
per-machine paths, and althing's sync_skill.sh deliberately reaches into
no other tree. Putting a plugin installer upstream would break that
boundary for one consumer's convenience. Their repo stays the source;
this one does the installing.
2026-09-02 01:05:48 -07:00
vh c0e352a47b fix(elway): probe NOPASSWD with sudo -n true, never sudo -n -v
`sudo -v` refreshes the auth timestamp, and a NOPASSWD-only rule creates
no timestamp to refresh, so on sudo >= 1.9.15 `sudo -n -v` returns
non-zero while every real command runs passwordless. Measured:

  pfi-gx10    sudo 1.9.15p5   sudo -n -v rc=1   sudo -n true rc=0
  nh3-docker  sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0
  ana-docker  sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0
  irv-ml1     sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0

Only pfi-gx10 is new enough to hit it today, but every host does as it
moves past 1.9.13, and the failure mode is bad: elway prompts for a
password on a host with working NOPASSWD sudo, which in a
non-interactive run is an EOFError partway through a playbook.

The same probe in my own notes cost this session directly. gx10 looked
like a fleet exception with no NOPASSWD sudo when it had it from account
creation, and the operator was asked for a password that was never
needed. Corrected in auto-memory too.

Also lands the gx10 privileged outfit playbook, now green at 5/5:
NOPASSWD sudo, nvcc, docker group, a CUDA container seeing the GB10, and
the userspace torch stack still working afterward.
2026-09-01 23:37:42 -07:00
vh fad1db96a0 memory: snapshot — GX10 outfitted userspace; bare metal ruled; CUDA works on sm_121
Ruling on the architecture question: bare metal, not Proxmox. Proxmox VE
has no aarch64 build, and more fundamentally the GB10's GPU sits on an
on-package root complex cache-coherent with the CPU over NVLink-C2C,
sharing the same LPDDR5X. Passing it to a guest would mean partitioning
the unified memory that is the entire reason for the box. The fleet's
other GPU hosts are bare metal for the same class of reason.

Installed uv and a venv with torch 2.14.0+cu130 plus the full training
stack, and every one of transformers, accelerate, peft, trl, datasets,
safetensors, huggingface_hub and bitsandbytes imports clean on aarch64.
The per-arch unknowns warning did not materialise for any of them.

CUDA works: sm_121, 121.6 GiB addressable, about 93 TFLOP/s dense bf16
with tensor cores confirmed engaged by the bf16-to-fp32 ratio. That is
A6000-class throughput with two and a half times the memory, so capacity
rather than speed is what this box buys.

Two warnings worth keeping. sm_121 is not in torch's compiled arch list,
so everything runs by PTX JIT from sm_120: first use of every kernel pays
a compile, and any library shipping cubins without PTX will fail outright.
And an unwarmed benchmark read 27 TFLOP/s because it was timing that JIT,
which nearly became a phantom report that tensor cores were broken -- the
playbook's section 4 shape exactly, a run that completes and reports
plausible numbers and is wrong.

The privileged half is blocked: infra-ops has no NOPASSWD sudo on this
box, unlike the rest of the fleet, and no credential is vaulted. That
gates nvcc, the container toolkit and the docker group, but not the
run-3c throughput probe.
2026-09-01 23:18:14 -07:00
vh 18fde5902e memory: snapshot — GX10 liveness confirmed; racking is not a prerequisite
Probed the box read-only. Alive and idle at 11h48m uptime, 118 of 121 GB
memory free, 822 GB disk free, and completely unchanged since onboarding:
no torch, no nvcc, no uv, and infra-ops is not in the docker group.

The useful finding is a negative one. I assumed the temporary Wi-Fi would
gate getting a 49 GB base model onto the box and it does not. The link is
Wi-Fi 7 on 6 GHz at 2401.9 Mbit/s with a -48 dBm signal, and a measured
300 MB transfer ran at 67 MB/s over SSH, which puts the full base at
about twelve minutes. SSH's cipher is the limiter there, not the radio.
So the throughput probe can run from the desk today and racking is worth
doing for permanence rather than as a blocker.

Also recorded that nvidia-smi reporting FB Memory and BAR1 as N/A is
correct for GB10 rather than a driver fault, since the Grace Blackwell
superchip shares unified LPDDR5X between CPU and GPU and has no discrete
VRAM figure to report.
2026-09-01 23:11:21 -07:00
vh f099caa238 memory: snapshot — H3 encoder pin resolved; my question had the direction backwards
Checked the disk rather than waiting on comfy-dev. Both builds are there,
pulled a minute apart on Aug 23: a 26 GB int8 and a 15 GB nvfp4-awq.

I had asked whether their nvfp4 pin was set under a Blackwell assumption,
since Ada has no native nvfp4, which would make the int8 file the right
one on the new box. It cannot be. They pinned it on irv-ml1's A6000,
which is Ampere sm_86 and has neither native nvfp4 nor native fp8. Ada
sm_89 supports a strict superset, so a pin that was correct on the weaker
card cannot be invalidated by moving to the stronger one. The migration is
incapable of breaking it.

The pin is about VRAM, not architecture. Eleven gigabytes on a 48 GB card
that also holds a DiT and two VAEs decides whether a graph runs, and a
text encoder runs once per prompt rather than once per diffusion step, so
its throughput matters far less than the DiT's. That also explains why
this pin went the opposite way from their other one without either being
inconsistent. The RTX 6000 Ada is also 48 GB, so nothing relaxes.

Reclassified the 26 GB int8 from orphan to spare: with the extra drives
the destination lands near 14% full, so disk stops being the constraint
and the pin-rot argument says keep it. Question withdrawn to comfy-dev.
2026-09-01 23:05:19 -07:00
vh 12006d287a memory: snapshot — althing 3.2.2 deployed; discover-pid was a bug, not an unknown
Four of the five items I raised are closed and tagged. The one that
matters most: --discover-pid matched comm == "claude", which means kimi,
grok, codex and pi would each have walked to the multiplexer and refused.
Four of five pane seats could never have used it. It now matches the
pane's own command, which zellij already reports and guard 1 already
compares against, so discovery and the guard read one string.

Recording a second miss of my own alongside it. I handed all four seats
explicit measured pids because I suspected the ancestry walk was broken,
then reported the suspicion as an open question rather than spending the
same twenty seconds to settle it, with four live non-Claude seats in
front of me. That pairs with the earlier miss in the opposite direction:
asserting an open risk on something the author had already measured. Same
root -- having the means to settle a question and reporting it as open
instead.

Also keeping forseti's two dead ends as negative results, because they
are the obvious things to propose next and both fail: locating a TUI's
input box in a screen dump needs per-TUI parsing, and diffing two dumps
to detect typing refuses every poke forever, because status bars carry
live token counts and clocks so consecutive dumps differ on an idle pane.

The kimi/pi coverage gap is deliberately unfixed and now sits with the
operator, with a recommendation to leave it per-seat.
2026-09-01 22:52:35 -07:00
vh f0d30f7a0f memory: snapshot — 3.2.1 outcome; the unguardable seats split, and one has a fix
All four seats answered. Delivery is now verified on four TUI families and
forseti's measured idle columns held exactly, so nobody had to guess.

The two seats guard 4 cannot cover made opposite calls on the same facts
and both are right. regin-smithy-dev released to pull-only because the
operator composes in that pane routinely, so the exposure is continuous.
bil-smithy-dev kept its route because that pane is poke-driven, making the
collision window narrow, and because pull-only had already cost them a
notice that sat unread for days. The deciding variable is who composes in
the pane and why, not risk appetite -- a flat rule either way would have
been wrong for one of them.

regin also supplied the only real path to closing the hole: pi and kimi
report no cursor because their input line is not an empty-prompt-at-idle,
so a moment-check reading pane content rather than cursor column would
cover them. Column is a proxy; an empty input line is the actual
predicate. Relayed to forseti as the lead item.

And bil corrected something upstream: the Aug 28 probes proved delivery
against non-Claude TUIs but never exercised the collision case, because
nobody was typing during them. Two different questions, one body of
evidence, only one of them answered by it -- which means the earlier
retraction of the non-Claude risk flag was right about delivery and
silent about stapling.

Note for the operator: regin-smithy-dev is pull-only as of 22:42 and will
not be poked until it declares again.
2026-09-01 22:44:51 -07:00
vh 1c7bd40c9c memory: snapshot — althing 3.2.1 deployed; guard 4 cannot cover two of five seats
3.2.0 wrote its pokes into pane input lines and pressed Enter, so anyone
mid-sentence had their half-written message submitted with the herald's
line stapled on. It hit the operator within an hour of this evening's
deploy. 3.2.1 adds a fourth guard that pins the pane's idle cursor column
and stays silent when the live column has moved.

Deployed all three steps at 22:40 and re-declared this seat, which pinned
idle_cursor=3 as expected for a claude TUI.

The finding worth keeping is the hole the fix leaves. forseti's own
measurements say kimi and pi report no cursor at all, which means
bil-smithy-dev and regin-smithy-dev can never acquire guard 4 no matter
what they run. Two of the five pane seats on this box stay permanently
exposed to the bug 3.2.1 fixes, so the sentence "3.2.1 fixes the
write-into-a-typing-pane bug" is only true where the cursor is legible.
All four seats were told, differentiated: a pre-filled re-declare and the
expected column for the two that can be guarded, and the honest version
plus the release-to-pull-only option for the two that cannot.

Also keeping forseti's post-mortem line, because it generalises: three
guards that all answer "is this the right pane" and none that asks
whether it is a good moment are one check wearing three hats.
2026-09-01 22:42:54 -07:00