memory: snapshot — gx10 unracked and next up as an inference+training box
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the first move is a throughput probe rather than a harness port. Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1 keeping its eight services; the Synapse 39-release upgrade with its one-way schema migration, the appservice namespace opening and the admin API lockdown; the ratified room alias convention; and a named failure class — a correct check aimed at the wrong object — with six instances from one day across three sessions.
This commit is contained in:
@@ -0,0 +1,75 @@
|
||||
# `[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available
|
||||
|
||||
The Ada box (ComfyUI's new home, sm_89, x86-64) lands at **NH3**. irv-ml1 is in **Irvine**.
|
||||
comfy-dev asked whether `/storetank` rides along or the stack is rebuilt from source.
|
||||
|
||||
## The answer: (a) `zfs send`. Measured, not derived.
|
||||
|
||||
NH3 -> irv-ml1 11-26 ms, 0% loss
|
||||
throughput 99.0 MB/s (real 800 MB transfer over the WireGuard tunnel)
|
||||
payload 1.38 TB -> ~3.9 hours
|
||||
|
||||
`zfs send` is **incremental**: snapshot now, ship the base over ~4 hours while irv-ml1 keeps
|
||||
serving, then a small delta at cutover. Near-zero service interruption.
|
||||
|
||||
## Why (b) — physically moving the disks — was never on the table
|
||||
|
||||
`/storetank` is a two-disk **mirror** of Crucial MX500 2TB SATA SSDs, so it is genuinely
|
||||
portable hardware. That fact is true and **irrelevant**.
|
||||
|
||||
**irv-ml1 is NOT being decommissioned.** It runs arbo, comfyui, tts-gateway, dots-tts,
|
||||
voice-studio, waterland-studio, yt-voice-clipper and dockge — and **comfyui bind-mounts
|
||||
`/storetank/arbo/models`**. Pulling those disks does not inconvenience a source box that no
|
||||
longer needs them; it guts a live one.
|
||||
|
||||
⚠ **The lesson:** infra-ops asked whether the DATA could move and never asked whether the
|
||||
SOURCE still needed it. One `docker ps` away, at any point. Recommended (b) twice on the
|
||||
strength of a true-but-irrelevant fact. See [[2026-09-01-wrong-object-measurement]].
|
||||
|
||||
## (c) rebuild-from-source: rejected on reproducibility, not time
|
||||
|
||||
~6 hours of re-fetch at ~65 MB/s. The real objection is that **two of comfy-dev's pins are
|
||||
already paywalled** (Big Love went permanent-paid; Moody Krea 2 Mix's newest releases are
|
||||
gated while their pin is free). A rebuild today would **not reproduce today's stack**. A
|
||||
fallback that provably cannot restore what it exists to restore is not a fallback.
|
||||
|
||||
## ⚠ The two-boxes confusion — do not repeat it
|
||||
|
||||
There are **TWO new machines** and infra-ops collapsed them into one:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Ada box** | ComfyUI's target. sm_89, x86-64. Lands at NH3. Not yet arrived. |
|
||||
| **ASUS Ascent GX10** | Local inference + run 3c. **GB10, sm_121, aarch64.** On the operator's desk. → [[2026-09-01-pfi-gx10-onboarding]] |
|
||||
|
||||
infra-ops found no sm_89 part in the current inventory, saw the Ascent, and inferred the
|
||||
Ascent was the incoming box — then sent comfy-dev down an aarch64 re-platform investigation
|
||||
that was entirely void. **comfy-dev's original premise was correct throughout.**
|
||||
|
||||
Consequences of the retraction, all restored to their original state:
|
||||
- Their single-arch container image, 18 GB x86-64 venv and torch pin are **fine**.
|
||||
- Their nvfp4 pin is **RIGHT, not wrong** — sm_89 does not do native nvfp4 (that is
|
||||
Blackwell), so pinning away from the 7.74 GB nvfp4 build to the 12.84 GB int8 build was
|
||||
correct for the hardware they are actually getting.
|
||||
|
||||
**Not wasted:** comfy-dev's aarch64 runtime research (ecarmen16/SparkyUI — CUDA 13.0.2,
|
||||
torch 2.9.1+cu130 ARM64, SageAttention compiled with `TORCH_CUDA_ARCH_LIST="12.1"`, built on
|
||||
the box, ~10 min; verified from source by infra-ops) applies to the **GX10** if anything
|
||||
ComfyUI-shaped ever runs there.
|
||||
|
||||
## Their distinction, worth keeping
|
||||
|
||||
> **The weights port. The runtime does not.**
|
||||
|
||||
safetensors are architecture-neutral and travel anywhere. Container image, torch build, venv
|
||||
and attention kernels do not. "Migrate the model store" and "migrate ComfyUI" are different
|
||||
jobs, and the 1.38 TB transfer is the easy half.
|
||||
|
||||
## Open
|
||||
|
||||
- **Cutover window** — operator's, not yet set.
|
||||
- **comfy-dev's ~112 GB batch** — unheld by infra-ops, but they are deliberately NOT pulling
|
||||
until the operator approves putting that much onto his infrastructure. Their call, correct
|
||||
instinct. Manifest pinned and staged (`29324e9`).
|
||||
|
||||
Thread: `01M1EYBSYA4QRYK54PX0K1S8CS`.
|
||||
@@ -0,0 +1,102 @@
|
||||
# `[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed
|
||||
|
||||
## The upgrade
|
||||
|
||||
**Synapse v1.120.0 → v1.159.0** (21 months, 39 releases) and **Element-web v1.11.80 →
|
||||
v1.12.27**. Schema migrations applied cleanly through schema 94. Postgres deliberately left
|
||||
at 16 — changing two stateful things at once destroys failure attribution.
|
||||
|
||||
⚠ **Schema migrations are ONE-WAY.** v1.120 cannot start against a v1.159 database. Rollback
|
||||
is restore-from-dump, not revert-the-tag. Verified pre-upgrade dump (739 TOC entries from a
|
||||
33 MB database) plus all four config files at
|
||||
`/opt/docker/backups/synapse-preupgrade-20260901T174625Z/`.
|
||||
|
||||
Reviewed every upgrade note in the range; nothing applicable bit us (PG 11/12/13 drops — we
|
||||
are on 16; MSC3861/MAS items — no MAS; s3-storage-provider and worker media quarantine — not
|
||||
in use).
|
||||
|
||||
## The appservice namespace — `exclusive: true` → `false`
|
||||
|
||||
The `aipa-bridge` registration claimed `@[a-z][a-z0-9_-]*:matrix.phasefinal.com` **exclusively**
|
||||
— every localpart on the server. 14 of 14 accounts fell inside it; 13 were appservice-owned.
|
||||
|
||||
**`exclusive` governs who ELSE may act, not what the appservice may do.** On a homeserver
|
||||
with registration disabled, one admin and no competing actor, it bought anti-squatting
|
||||
protection against a threat that cannot occur, while locking out every other means of
|
||||
account creation — admin registration returned `M_EXCLUSIVE` with no explanation.
|
||||
|
||||
⚠ **Do NOT narrow the users regex to a prefix** — all 13 accounts fall inside it and would be
|
||||
orphaned. ⚠ **Do NOT rename the `id`** — Synapse keys account ownership on `aipa-bridge` in
|
||||
the `users` table. The FILE may be renamed; the id may not.
|
||||
|
||||
The narrow **aliases** namespace (`#aipa-debug-*`) was left exclusive — specific, costs nothing.
|
||||
|
||||
⚠ The registration is named `aipa`, but the service behind it is **`wt-matrix-bridge`**, the
|
||||
Worldtree PERSONAL instance on corviduo-dev `10.250.50.152:8010`. AIPA is a dead project name
|
||||
on a live service, and it is why infra-ops mis-routed a provisioning request to worldtree-dev.
|
||||
**Operator ruling: worldtree-dev writes the bridge code; infra-ops OPERATES this instance and
|
||||
has full authority over it.**
|
||||
|
||||
## `/_synapse/admin` closed to the internet
|
||||
|
||||
Synapse mounts its admin API on the same vhost as the client API, so publishing
|
||||
`matrix.phasefinal.com` published the admin surface — it **answered 200 from the open
|
||||
internet**. A higher-priority router (explicit priority 100) now scopes it behind an
|
||||
`ipallowlist`.
|
||||
|
||||
Verified from a **genuinely external vantage** — the NH3 residential egress proxy, because
|
||||
testing from a fleet host sits inside the allow-list and proves nothing: admin **403**,
|
||||
client API **200**, Element unaffected.
|
||||
|
||||
⚠ **The `10.0.0.0/8` entry matches NOTHING and that is expected.** The hostname resolves
|
||||
publicly, so fleet hosts hairpin out their own WAN — a request from nh3-dev measured as
|
||||
`70.230.226.88`. The rule is effectively **deny-all through Traefik**, which is intended:
|
||||
admin work goes via `docker exec synapse` against `localhost:8008` and never touches Traefik.
|
||||
Allow-listing the sites' WAN addresses was **rejected** — dynamic, and a stale entry either
|
||||
locks us out or hands admin to whoever inherits the address.
|
||||
|
||||
## Conventions ratified (operator, 2026-09-01)
|
||||
|
||||
#<agent>-<purpose>:matrix.phasefinal.com
|
||||
|
||||
Mirrors the existing `@<agent>:` user-ID convention. Proposed by ledger-dev. Rationale is
|
||||
**"so the room IDENTITY carries the tier"** — deliberately NOT "so the push payload carries
|
||||
the room name", which is true only for clients without a notification service extension.
|
||||
Pre-existing rooms are not renamed ("The High Seat", `!NiVVoMsyoHCBRPrrrn`).
|
||||
|
||||
## Push reality — measured, and it inverts the obvious reading
|
||||
|
||||
The registered pusher (`@vhoang`, Element X iOS) uses **`"format": "event_id_only"`** via
|
||||
matrix.org's sygnal. That payload carries event_id, room_id and counts — **no room name, no
|
||||
sender, no content**. It still produces a useful notification because `mutable-content: 1`
|
||||
means Element X runs a **Notification Service Extension**: iOS wakes it with the near-empty
|
||||
payload and it **fetches the event and renders the notification on the device**.
|
||||
|
||||
1. The tier-in-room-identity scheme works — but **via the client fetch**, not the payload.
|
||||
`m.room.name` must be set at creation; the ALIAS is not what reaches the phone. Synapse
|
||||
sends `ctx["name"]` (the `m.room.name` state event) and omits the key entirely if unset.
|
||||
2. **`push: include_content: false` is irrelevant for clients with an NSE.** It bites clients
|
||||
without one.
|
||||
3. ⚠ **Server-invisible failure mode:** if the phone cannot reach the homeserver at wake time
|
||||
the fetch fails and iOS shows the bare word "Notification". **Synapse records
|
||||
`last_success` and sees a delivered push.**
|
||||
|
||||
Self-hosted sygnal **considered and rejected** — sygnal is a relay to FCM/APNs, not a
|
||||
replacement, so it removes matrix.org and nothing else; and with `event_id_only` the path
|
||||
already carries nothing worth protecting.
|
||||
|
||||
## QR sign-in — requires MAS, deferred
|
||||
|
||||
MSC4108 hard-requires `matrix_authentication_service`; Synapse refuses to start otherwise.
|
||||
MSC4388 enables independently but is only the rendezvous **channel**, not a login flow.
|
||||
Deferred: MAS is a service, a database and a migration of every account off built-in auth,
|
||||
and the v1.139.0 note warns `/register` from **old appservice implementations may break under
|
||||
MAS** — precisely the bridge owning 13 of 15 accounts.
|
||||
|
||||
## Shared-secret registration gotcha
|
||||
|
||||
`HMAC-SHA1(secret, nonce \0 user \0 password \0 "notadmin")` — the null **separates**, it does
|
||||
not **terminate**. A trailing `\x00` yields `HMAC incorrect`. Run inside the container against
|
||||
`localhost:8008`; port 8008 is not published to the host.
|
||||
|
||||
Full doc: `docs/pfi/vm-102-matrix-synapse.md` (`931bac8`, `73866f6`).
|
||||
@@ -0,0 +1,79 @@
|
||||
# `[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c
|
||||
|
||||
ASUS Ascent GX10 arrived and was registered, converted to headless, and given a rack-move
|
||||
playbook. **It was NOT racked** — the operator ran out of day. It is still on his desk, on
|
||||
Wi-Fi, on a temporary DHCP lease.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| GPU | **NVIDIA GB10**, driver 580.173.02, **compute capability 12.1 (`sm_121`)** |
|
||||
| CPU | 20 cores, **aarch64** |
|
||||
| Memory | **121 GB UNIFIED** — CPU and GPU share it; not 121 GB *plus* VRAM |
|
||||
| Storage | 916 GB NVMe, 6% used |
|
||||
| OS | Ubuntu 24.04.4, kernel 6.17.0-1031-nvidia |
|
||||
| Access | `infra-ops` NOPASSWD sudo (operator-bootstrapped). `lkraven` has key auth but needs a password to escalate — **automation must connect as `infra-ops`**. |
|
||||
|
||||
## Purpose (operator, 2026-09-01)
|
||||
|
||||
Local inference experiments **and** the failed training — run 3c. That is the whole point:
|
||||
run 3c died on ana-ml2 when a **kilowatt-class** training box tripped a breaker
|
||||
(2× 300 W GPUs + dual EPYC 9254). The GX10 is a ~240 W appliance, roughly a fifth of the
|
||||
draw, on a different site's circuits.
|
||||
|
||||
**The memory arithmetic favours it strongly.** Run 3c is a LoRA (r=64, batch 2 × accum 8,
|
||||
gradient checkpointing) over `gemma4-26b-a4b-it-bf16` — **49 GB of base weights**, working
|
||||
set roughly 55–65 GB. The `gemma4-charrp` compose warns in capitals that "48.10 GiB of BF16
|
||||
weights CANNOT be served here" on ana-ml2's shared GPU0. 121 GB unified turns that
|
||||
constraint into a non-issue.
|
||||
|
||||
## What has NOT been established — do not assume any of it
|
||||
|
||||
1. **The box is bare.** No torch, no nvcc, no CUDA stack. Docker 29.2.1 and 822 GB free.
|
||||
2. **aarch64 dependency risk.** torch 2.9.1+cu130 ARM64 wheels exist; transformers/TRL/PEFT
|
||||
are pure Python. Compiled deps — flash-attn, bitsandbytes, liger, xformers — are open
|
||||
questions per-arch.
|
||||
3. **Triton has NO sm_121 support** (established independently by comfy-dev the same day).
|
||||
Anything reaching for `torch.compile` or Triton-backed kernels is closed on this silicon.
|
||||
4. **Throughput is unmeasured.** Run 3c ran 16.45 s/it on a 300 W RTX PRO 6000. GB10 will be
|
||||
slower; how much decides whether 604 steps is an overnight run or two days.
|
||||
**Measure this before porting anything** — the recommended first move is a probe: install
|
||||
ARM64 torch, load the base, run ten steps, report s/it.
|
||||
5. **Model transfer:** 49 GB from ana-ml2 over the Anaheim↔NH3 link at a **measured
|
||||
32.4 MB/s** — about 25 minutes. Note that is a third of the irv-ml1↔NH3 link's 99 MB/s.
|
||||
|
||||
## The headless conversion, and the lesson inside it
|
||||
|
||||
`playbooks/gx10-headless.yaml` (`1b596c8`): multi-user.target, gnome-remote-desktop stopped,
|
||||
sleep/suspend/hibernate **masked**, logind ignores lid and idle, sshd keepalives, hostname
|
||||
corrected `gx10-a745` → `pfi-gx10`.
|
||||
|
||||
⚠ **`gdm` is a STATIC unit on Ubuntu** — pulled in by `display-manager.service`, never
|
||||
"enabled". The first version guarded on `is-enabled | grep enabled`, which always skips, and
|
||||
**the verify tested the same wrong property and passed**. Six green verifies having not
|
||||
stopped the display manager. Both now test `is-active`. This is an instance of
|
||||
[[2026-09-01-wrong-object-measurement]].
|
||||
|
||||
The playbook refuses to stop GDM while a seat session is held (`--var force_dm_stop=true` to
|
||||
override) — automation should not yank a display out from under someone at the machine.
|
||||
|
||||
⚠ **elway's `--sudo` applies only to ad-hoc `--shell`/`--upload`.** Playbook steps run as the
|
||||
connecting user and must carry their own `sudo`.
|
||||
|
||||
## The rack move, written but not run
|
||||
|
||||
`playbooks/gx10-rack-network.yaml` (`a0c5fc6`) — target settled: **`nh3-servers` VLAN 50,
|
||||
static `10.100.50.60`**, clear of `.40`/`.42`/`.50`/`.90` and below the `.150` DHCP pool.
|
||||
|
||||
Requires nothing from the operator beyond racking it. The wired NIC has its own MAC
|
||||
(`30:c5:99:3d:a7:45`, distinct from Wi-Fi `50:bb:b5:a2:00:a8`), so the post-move address AND
|
||||
switch port are discoverable from the UDM rather than relayed.
|
||||
|
||||
**Design property worth preserving:** the playbook never leaves itself one path back. Wi-Fi
|
||||
stays up while the wired interface is configured beside it; Wi-Fi teardown is explicitly a
|
||||
separate later change. `netplan try`'s auto-rollback needs a TTY that elway cannot provide,
|
||||
so a live Wi-Fi link is what substitutes for it. Two preconditions are asserted as steps
|
||||
rather than assumed: carrier must be 1, and the MAC must match (interface names renumber
|
||||
across kernels; MACs do not).
|
||||
|
||||
⚠ **The GX10 is NOT the Ada box.** Two separate machines — see
|
||||
[[2026-09-01-ada-migration-branch-a]].
|
||||
@@ -0,0 +1,42 @@
|
||||
# `[2026-09-01]` A named failure class: a correct check aimed at the wrong object
|
||||
|
||||
Six instances surfaced across three sessions in a single day, independently, in unrelated
|
||||
domains. It has a distinguishing property that makes it worth naming separately from
|
||||
"a bad measurement":
|
||||
|
||||
> **Re-running the same check cannot catch it, because the check is correct and the object
|
||||
> is wrong.** The only move that breaks it is asking what the artifact *is* before trusting
|
||||
> any metric computed over it.
|
||||
|
||||
## The instances
|
||||
|
||||
| where | the metric | the artifact nobody opened |
|
||||
|---|---|---|
|
||||
| comfy-dev, civitai | a "~25 KB/s throttle" | a **9,685-byte login page** returned on failed auth |
|
||||
| comfy-dev, render harness | reported "completed" | a 4.7 KB **all-black PNG** |
|
||||
| comfy-dev, audio | RMS said healthy | degenerate audio where the **loudness WAS the noise** |
|
||||
| ledger-dev, capability 5 | healthy ping, healthy container, healthy delivery | a **requirement unmet** — a mailbox, not an away-channel |
|
||||
| infra-ops, gx10 headless | six green verifies | `is-enabled` on a **static unit**; gdm still running |
|
||||
| infra-ops, synapse admin | an `ipallowlist` comment promising fleet access | fleet traffic **hairpins out the WAN**; 10.0.0.0/8 matches nothing |
|
||||
|
||||
## Related lessons banked the same day
|
||||
|
||||
- **A caveat plus propagation is decoration.** infra-ops flagged a sample-size problem AND
|
||||
escalated the claim in the same message. If a number needs re-measuring before it can be
|
||||
repeated, hold the escalation until it has been. The caveat made the uncareful thing look
|
||||
examined.
|
||||
- **Verify with a negative control.** A 200 means nothing without a 401 beside it. Used on
|
||||
the pewpewstudio key mint; used on the `/_synapse/admin` lock (tested from a genuinely
|
||||
external vantage via the NH3 residential egress proxy, because testing from a fleet host
|
||||
sits inside the allow-list and proves nothing).
|
||||
- **Reasoning from what is VISIBLE to what EXISTS.** infra-ops found no `sm_89` part in the
|
||||
inventory, saw one new machine, and collapsed "the box I can see" into "the box that is
|
||||
coming". They were two different machines. Wrote "I cannot resolve it and will not guess",
|
||||
then built three messages on the guess.
|
||||
|
||||
## Disposition
|
||||
|
||||
Recommended for a row in `docs/pfi/training-throughput-playbook.md` §4 (the durable home for
|
||||
"why a run LIES about itself"), with attribution to comfy-dev and ledger-dev.
|
||||
**NOT YET WRITTEN — awaiting operator.** Tracking surface: this file, plus althing threads
|
||||
`01M1EYBSYA4QRYK54PX0K1S8CS` (comfy-dev) and `01M1EXVPZT66SSGCRZH3W624R7` (ledger-dev).
|
||||
Reference in New Issue
Block a user