Quantifying exposure to a gate-failed tune: counting by the gateway alias gave 363
rows, counting by the artifact gave 77, because the alias had carried three
different models that day. Wrong in the direction that looks careful.
From a near-miss brokkr-smithy-dev caught and declined to send: a post-fix
verification read raced a gateway restart and reported the alias still present.
A disagreement between two observers is not a finding until the boring
explanation is ruled out, and during an incident the pressure runs the other way.
Alias-present-with-backend-down is a different and worse state than
alias-removed: it re-arms silently the moment anything is served on
10.250.50.54:8021 again, and the next thing served there inherits a
fleet-reachable name without anyone deciding it should. Flagged by
brokkr-smithy-dev; same writer/reader-contract family as ops-lessons §11.
Commented rather than deleted so restoring is uncommenting, not reconstructing.
Verified: /v1/models returns 33 entries, trial absent, a caller gets a clean
400 'Invalid model name' instead of a connection error to a port that could
re-arm. Disposition of the artifact and the alias is the operator's.
- gx10:8098 erp-tune-v7 stopped 17:42; ana-ml2:8021 trial NVFP4 seat stopped 17:43
(infra-ops' call — the same adapter was on the SHARED-KEY gateway ~15:30-17:43
under the operator's pre-gate instruction; surfaced to him as reversible)
- probe NOT re-run, flagged generations NOT opened, length verdict left UNSET
- every artifact preserved; disposition is the operator's
- non-safety results recorded but explicitly NOT certified (primary flat +2,
diversity reduced, coherence 1.0 -> 0.875, control valid at 1.0)
The empty-snapshot promote and brokkr-smithy-dev's -s-on-a-touched-sentinel are
the same defect from opposite sides. Records the direction split (false
reassurance / false refusal / false alarm) because the mitigations differ and a
post-mortem that lumps them loses the actionable half.
ssh exiting 0 is not proof the capture is usable — the inspect script can emit
nothing and both refresh scripts would mv that over a good system-details.txt and
report 'ok (0 bytes)'. Every reader tests the snapshot with -s, so the writer was
producing an artifact its own readers call invalid: a guard whose test disagrees
with its writer's contract has quietly stopped guarding. Prompted by
brokkr-smithy-dev hitting the same shape from the other side (a -s test against a
sentinel written with touch, a precondition that could never pass).
- empty capture -> refused, previous snapshot kept, host counted as failed (exit 1)
- capture under 1/4 of the previous -> promoted but flagged, since a host can
legitimately shed services and the script should not guess
- header + CLAUDE.md contract lines corrected to say what is actually guaranteed
- verified red (empty inspect -> FAIL, snapshot intact, rc=1) then green (real host
-> ok 6727 bytes)
Operator: the form failed when a question was left blank. Refusing the whole
submission over one blank threw away the picks that were made, and the HTML
`required` on the radios blocked it in the browser before the server saw it.
- answered questions recorded; blank ones land in `unanswered`; `complete` says
whether the set is finished; a blank question carrying a note keeps the note
- `required` dropped from both templates so the browser cannot block a partial
- refused only when there is no pick anywhere AND no notes (a 400 — that would
flip an open ask to answered with no decision recorded); a choice outside the
option list is still an error
- new ◐ partial state with an n/N count; skipped questions render as skipped
- README + global CLAUDE.md tell reading sessions to check `complete`
- 154 tests; v0.1.15
Operator verdict on the separate /asks page: the question belongs with the
artifact it is about. A four-voice audition wants each voice's radio group
under that voice's audio, and one submit for the lot.
- booth/inline.py: data-booth-ask="stem" | "stem:key" | data-booth-ask-submit,
plus <!-- booth:ask ... --> comments; unknown stem left alone, not blanked
- _ask_inline.html: self-contained fragments (own scoped styles, no JS), per-question
groups bound to one form via the HTML5 form= attribute so a scattered
multi-question ask still POSTs once
- unplaced questions and a missing submit block are appended, so a partially
marked-up page can never produce an unsubmittable 400
- chip becomes a jump link to the first open ask; /asks page kept as a fallback
- 6 tests (one caught the partial-placement drop); v0.1.14
A custom index.html is returned verbatim, so booth.html's asks panel never
rendered there — a valid ask (emmie-anchor/anchor.ask.json) was listed by the
CLI and shown nowhere, with nothing to say so.
- panel extracted to _asks.html; new GET /b/<name>/asks standalone page
- verbatim pages get an amber '? N open asks' chip beside the back chip
- POST /answer honours back=asks so answering returns to that page
- single-question asks now keep an optional 'title' (was silently dropped)
- README + routes table; 8 regression tests; v0.1.12
- asks.py: single {prompt, options} and multi {title, questions:[{key, prompt, options, notes?}]} both normalise to questions[]; per-question notes; every question required on submit
- /answer reads choice.<key> / notes.<key> / notes for multi; single shape unchanged
- booth asks prints per-question picks; README + CLI header; install step symlinks the CLI to ~/.local/bin; v0.1.10; 135 tests
Config-file deployment (the trial model lives in config.yaml, so /model/update refuses it):
block rewritten to run-6 reality — abliterated base, NVFP4A16 artifact, stacks/erp-seat,
no gate on this artifact by operator ruling, same-site hop. Deployed to ana-docker and the
gateway restarted; trial -> erp-tune-v6-nvfp4a16 verified end to end (3/3).
- services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py: linearize_moe first (playbook §3.15),
asserts the expert Linear count, routers/vision/audio/norms/lm_head ignored, W4A16 for RP
long-session fidelity, post-steps restore processor configs + template and reset the
tokenizer truncation cap (§3.14); --dry-run proves targets before GPU time
- services/erp-seat-quant/run_quant_erp_v6.sh: detached container on GPU1 (vllm-llmcompressor)
- stacks/erp-seat: serve recipe copied from gemma4-charrp, true served name only, port 8021
- scripts/erp-tune-gx10/pull-verify-jenerallee78.sh + base-pin-jenerallee78-shards.txt:
revision-pinned root-shard pull, 32/32 sha256+size vs brokkr-smithy pins, index
set-equal to stock, STOCK tokenizer set installed over the repo's (which bakes in
a 256-token truncation); repo originals kept as *.repo
- scripts/erp-tune-gx10/run-06-gx10.json + launch-run-06.sh: run-05 config with the
base swapped, recipe-r6, survivors-r5 verbatim, stock template path
- docs/runbooks/gx10-run-06.md: pull/verify record, free-check result (encode
reproduces run 5 exactly), hf download --include gotcha, gate naming
(erp-seat-base-ara / erp-tune-v6)
Operator-directed question (via brokkr): which weights at
/home/infra-ops/models/gemma4-26b-a4b-it-bf16 — stock or the heretic abliteration?
Answer from bytes: STOCK google/gemma-4-26B-A4B-it, byte-for-byte. Three-way match:
local shard sha256 (1127684971/aab47033) == HF download etags == stock repo LFS
oids, and download commit 4d7ae498 == stock HEAD. The recipes' -heretic-bf16 label
is a naming error (run-04 provenance was right; refusal telemetry confirmed). All
of runs 3/3c/4/5 trained from stock. Plausible cause: the selected llmfan46 26B-A4B
Heretic abliteration ships GGUF-only, so the bf16 safetensors pulled was stock.
Operator decision post-RESCUED verdict: point the LiteLLM trial alias at the
run-5 tuned seat (erp-tune-v5 on gx10:8098) for hand-testing prosody/feel in
Open WebUI. Rewrote the trial config block to run-5 reality, including the
now-MEASURED refusal-erosion note (k=25 both arms, de-gated + stated) replacing
the run-3c 'safety axes unmeasured' warning. Verified end-to-end: trial ->
erp-tune-v5 returns coherent output through the gateway. Container restarted to
re-read the bind-mounted config (up -d alone is a no-op for a config-only change).
First arm of the ERP-seat line where the capability gate did not fail. The
dependency-forcing slot (GovReport+QMSum, 3.46% of loss) broke the
diversity<->coherence coupling run 4 (STILL-COUPLED) and 3c (20x LR cut) could
not — structure of the loss was the lever, not its mass; INERT did not fire.
T4 8/8 (run 4: 5/8), t4_dissect noise 0.9062 vs run-3 0.5625, diversity held.
Reported-beside: T3 8->6 (new loss), RP-length partial fail, refusal erosion
(de-gated/stated), free-check base 5-6pp below run 4 (vLLM 0.28.0 unchanged).
Seat erp-tune-v5 served on gx10:8098 pending the operator's seat/ear decision.
ana-ml2 is up but not reachable cross-site from ESH (return-route gap on ana-ml2,
no SSH access to fix it). New stacks/ana-ml2-proxy: a pinned alpine/socat relay on
ana-docker (host net) exposing 10.250.50.70:2354 -> ana-ml2 10.250.50.54:2375.
Homepage docker.yaml ana-ml2-docker re-enabled pointed at the proxy. Verified:
esh-docker-vm reaches ana-ml2's docker (29.3.1) through it; /api/services returns
in 0s with 111 services; ana-ml2's inference seats (gen, coder, embed, rerank,
reward, image-bench, dockge) render. Reversible: point back at 10.250.50.54:2375
and compose-down the proxy once ana-ml2 routes to ESH.
The 2026-09-06 headscale cutover retired irv-ml1's wg0 tunnel IP 10.100.79.3
(now 10.6.110.50). Repointed all LIVE canonical refs to the DNS NAME so the next
move can't re-break them: homepage.href/siteMonitor labels across 25 stack
composes, load-bearing env defaults (asset-engine INFERENCE_HOST, open-webui
AUDIO_TTS_OPENAI_API_BASE_URL, skaldsong SKALDSONG_TTS_BASE_URL, zonos-gateway
ZONOS_URL, dia), homepage services.yaml manual cards (Voice Design Studio,
IRV-ML1), and servers/irv-ml1/ssh-target. Updated the stale 'WG tunnel' comment
to the mesh reality.
Left as-is: README curl-examples and .env.example comments (docs), and historical
mentions in CLAUDE.md/persistent-memory. NOTE: applying the label repoints to the
RUNNING irv-ml1 containers needs a recreate per service (labels read at creation);
deployed .env values are separate from these canonical defaults.
WhereTF (operator-directed, brokkr-smithy-dev; nh3-dev:8093, user-systemd Bun
service): manual Homepage card under Apps (can't auto-discover) + DNS alias
wherethef.nh3.internal -> nh3-dev.
While adding the card, found Homepage's /api/services was hung (dashboard
degraded since the 2026-09-06 irv-ml1 move): its docker.yaml still pointed
irv-ml1 at the dead 10.100.79.3, and ana-ml2 (10.250.50.54:2375) is unreachable
from esh-docker-vm — either dead host hangs the whole aggregation on its TCP
connect timeout. Repointed irv-ml1 to 10.6.110.50 (verified reachable via mesh)
and disabled ana-ml2 (reversible, noted). /api/services now returns in 0s with
96 services; WhereTF renders.
ext-tts / tts-1 / tts-1-hd / gpt-4o-mini-tts (all openai/zonos) were pinned to
irv-ml1's pre-move address 10.100.79.3:8198, dead since the 2026-09-06 headscale
cutover to 10.6.110.50 — so ext-tts through the gateway hung. Reported by
tts-dev/svos-dev.
The 4 aliases are DB-backed (store_model_in_db), so their api_base was updated
via the admin API to the DNS name http://irv-ml1.nh3.internal:8198/v1 (not a
fresh IP — that just re-arms the trap on the next move). The container cannot
resolve *.nh3.internal via split-horizon DNS, so this adds an extra_hosts entry
mapping the name to 10.6.110.50. Verified: container resolves the name and a
live ext-tts/sindra call returns 200 + valid MP3.
vh