Operator ruling: run 6 is declared Pfish-6 and is the standing seat; run 7 is
retired (its gate failure was the detector bug fixed in cc42d76, but the run was
independently poor).
- served under its TRUE name Pfish-6; LiteLLM alias trial -> Pfish-6
- ana-ml2 :8021 and pfi-gx10 :8098, both at 262,144 ctx, same artifact
- SPEED: moe_backend=flashinfer_cutedsl PROVEN unusable here (engine init fails,
'kernel does not support current device'); Marlin is correct for a weight-only
scheme, and vLLM's 'no native FP4' warning is about the scheme not the card
- the real lever was max-num-seqs 8 -> 32: n=8 1269 tok/s and n=16 2170 tok/s,
3.2x the old ceiling, with single-stream latency unchanged
- head-to-head measured: ana-ml2 is 4.1x FASTER than the GX10, not slower
Alias-present-with-backend-down is a different and worse state than
alias-removed: it re-arms silently the moment anything is served on
10.250.50.54:8021 again, and the next thing served there inherits a
fleet-reachable name without anyone deciding it should. Flagged by
brokkr-smithy-dev; same writer/reader-contract family as ops-lessons §11.
Commented rather than deleted so restoring is uncommenting, not reconstructing.
Verified: /v1/models returns 33 entries, trial absent, a caller gets a clean
400 'Invalid model name' instead of a connection error to a port that could
re-arm. Disposition of the artifact and the alias is the operator's.
Config-file deployment (the trial model lives in config.yaml, so /model/update refuses it):
block rewritten to run-6 reality — abliterated base, NVFP4A16 artifact, stacks/erp-seat,
no gate on this artifact by operator ruling, same-site hop. Deployed to ana-docker and the
gateway restarted; trial -> erp-tune-v6-nvfp4a16 verified end to end (3/3).
- services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py: linearize_moe first (playbook §3.15),
asserts the expert Linear count, routers/vision/audio/norms/lm_head ignored, W4A16 for RP
long-session fidelity, post-steps restore processor configs + template and reset the
tokenizer truncation cap (§3.14); --dry-run proves targets before GPU time
- services/erp-seat-quant/run_quant_erp_v6.sh: detached container on GPU1 (vllm-llmcompressor)
- stacks/erp-seat: serve recipe copied from gemma4-charrp, true served name only, port 8021
Operator decision post-RESCUED verdict: point the LiteLLM trial alias at the
run-5 tuned seat (erp-tune-v5 on gx10:8098) for hand-testing prosody/feel in
Open WebUI. Rewrote the trial config block to run-5 reality, including the
now-MEASURED refusal-erosion note (k=25 both arms, de-gated + stated) replacing
the run-3c 'safety axes unmeasured' warning. Verified end-to-end: trial ->
erp-tune-v5 returns coherent output through the gateway. Container restarted to
re-read the bind-mounted config (up -d alone is a no-op for a config-only change).
ana-ml2 is up but not reachable cross-site from ESH (return-route gap on ana-ml2,
no SSH access to fix it). New stacks/ana-ml2-proxy: a pinned alpine/socat relay on
ana-docker (host net) exposing 10.250.50.70:2354 -> ana-ml2 10.250.50.54:2375.
Homepage docker.yaml ana-ml2-docker re-enabled pointed at the proxy. Verified:
esh-docker-vm reaches ana-ml2's docker (29.3.1) through it; /api/services returns
in 0s with 111 services; ana-ml2's inference seats (gen, coder, embed, rerank,
reward, image-bench, dockge) render. Reversible: point back at 10.250.50.54:2375
and compose-down the proxy once ana-ml2 routes to ESH.
The 2026-09-06 headscale cutover retired irv-ml1's wg0 tunnel IP 10.100.79.3
(now 10.6.110.50). Repointed all LIVE canonical refs to the DNS NAME so the next
move can't re-break them: homepage.href/siteMonitor labels across 25 stack
composes, load-bearing env defaults (asset-engine INFERENCE_HOST, open-webui
AUDIO_TTS_OPENAI_API_BASE_URL, skaldsong SKALDSONG_TTS_BASE_URL, zonos-gateway
ZONOS_URL, dia), homepage services.yaml manual cards (Voice Design Studio,
IRV-ML1), and servers/irv-ml1/ssh-target. Updated the stale 'WG tunnel' comment
to the mesh reality.
Left as-is: README curl-examples and .env.example comments (docs), and historical
mentions in CLAUDE.md/persistent-memory. NOTE: applying the label repoints to the
RUNNING irv-ml1 containers needs a recreate per service (labels read at creation);
deployed .env values are separate from these canonical defaults.
WhereTF (operator-directed, brokkr-smithy-dev; nh3-dev:8093, user-systemd Bun
service): manual Homepage card under Apps (can't auto-discover) + DNS alias
wherethef.nh3.internal -> nh3-dev.
While adding the card, found Homepage's /api/services was hung (dashboard
degraded since the 2026-09-06 irv-ml1 move): its docker.yaml still pointed
irv-ml1 at the dead 10.100.79.3, and ana-ml2 (10.250.50.54:2375) is unreachable
from esh-docker-vm — either dead host hangs the whole aggregation on its TCP
connect timeout. Repointed irv-ml1 to 10.6.110.50 (verified reachable via mesh)
and disabled ana-ml2 (reversible, noted). /api/services now returns in 0s with
96 services; WhereTF renders.
ext-tts / tts-1 / tts-1-hd / gpt-4o-mini-tts (all openai/zonos) were pinned to
irv-ml1's pre-move address 10.100.79.3:8198, dead since the 2026-09-06 headscale
cutover to 10.6.110.50 — so ext-tts through the gateway hung. Reported by
tts-dev/svos-dev.
The 4 aliases are DB-backed (store_model_in_db), so their api_base was updated
via the admin API to the DNS name http://irv-ml1.nh3.internal:8198/v1 (not a
fresh IP — that just re-arms the trap on the next move). The container cannot
resolve *.nh3.internal via split-horizon DNS, so this adds an extra_hosts entry
mapping the name to 10.6.110.50. Verified: container resolves the name and a
live ext-tts/sindra call returns 200 + valid MP3.
The tools had been on 3.6.0 while the container sat on 3.0.0 for seven days, so
the operator verbs failed with "no tool named delete_handle" — they live in the
post office, not the client.
Image built on nh3-dev from a clean tree at 4d26226 and pushed under the
claude-bot namespace; the compose pin moves to the new digest rather than
floating on the tag, since this container is the fleet's whole message bus.
The backup procedure this file documents earned itself again: at stop time the
database was 23.8 MB with a 5.9 MB WAL beside it, so a plain copy would have
produced a database that opens cleanly, smokes green, and is missing the day's
mail. Stop, explicit checkpoint to a zero-byte WAL, copy, then verify counts on
both sides — 76 handles, 995 messages, 1022 recipients, integrity ok.
Post-deploy the same counts came back with handles.retired_at present, and the
memory cap and OOM guard were confirmed by `docker inspect` rather than by
reading the yaml, which is what that file asks for and the only check that can
tell a working cap from a decorative one.
Bus down about four minutes.
Operator asked for it explicitly so he can test in Open WebUI in parallel with
brokkr's batteries. Named `trial` rather than erp-tune-v3c: the run-3 note in
this file records why a tier name arriving before its evidence is a mistake,
and 3c has no more evidence than run 3 had.
Points at the pfi-gx10 seat (http://10.100.50.60:8098/v1, vLLM 0.28.0 on
GB10/sm_121), so this is a cross-site route -- NH3 seat, Anaheim gateway.
⚠ Recorded in the entry, not just here: 3c's safety axes are unmeasured, and
run 3 -- same recipe, differing only in learning rate -- was withdrawn from
this gateway 40 minutes after being added for selfharm/methods -44.0pp and
meta/sysprompt -28.0pp against its own base. This gateway is the shared-key
surface. When brokkr's batteries read out, the entry should be re-decided on
evidence rather than left standing by default.
Applied with a restart of the litellm service only (~60 s of gateway
downtime); the compose .env is root-owned so the restart needs sudo.
The ana-docker instance was returning zero results for every query while
reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3),
its engine scrapers rotted against sites that had changed. /healthz proves
the web app answers and says nothing about whether search works, so seven
days of green sat on top of a search box that found nothing.
Moved to nh3-docker rather than updated in place, because the colo egress is
the other half of the problem: 38.120.12.42 is a datacenter address that
DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at
70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp,
applied at the source instead of around it.
Config corrected along the way: base_url said searxng.pfi.local, a name
retired on 2026-08-19, while the environment said something else — the env
won so nothing broke and the file quietly lied. The karmasearch.videos
removal key never matched, because the engine's real name has a space.
scripts/searxng-health.sh asserts results > 0 across three unrelated
queries. That is the check that would have caught this, and the only kind
that can: the mechanism was healthy throughout.
services/searxng-mcp exposes it as `web_search` at user scope, so every
Claude Code session has it. Zero results raise rather than returning an
empty list — an empty list is indistinguishable from a broken aggregator,
which is precisely how this hid.
Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
The gen-reasoning seat accepts only xhigh/medium/low and 400s on anything
else — including `high`, which is the default of several clients, so the
seat presented as broken rather than as one enum value out of step. The
DeepSeek Harness failed every request on its default setting, and the only
working client value was `low`: the seat's WEAKEST reasoning tier, while
its own default is xhigh.
conf/reasoning_effort_map.py is a pre-call hook in the same shape as the
existing strip_empty_tools hook. It is scoped to one model group, measured
rather than assumed: gen-reasoning rejects `high`; gen, sec,
char-rp-reasoning and summarizer all accept it and are left alone. Paid
passthroughs were not probed, because probing them spends vendor credits,
and are not mapped.
Verified after deploy: high and max now succeed on gen-reasoning, low and
xhigh still work, a request with no effort param still works, `gen` with
`high` still passes through unmapped, and the harness completes a real
file-edit task at full reasoning.
Needed a compose change as well as a conf push — callbacks are bind-mounted
per file, so the volume only attaches on container create. Recreated the
litellm service by name so the DB was not bounced with it.
I wrote "one-line fallback to pfi-fleet costs you nothing" to a peer,
meaning a manual edit; it was read as scheduler behaviour, and
`runs-on: [vastblue, pfi-fleet]` would have shipped into a contract. A
label list requires ALL the labels, so that form matches no runner and
queues forever instead of failing — the worst outcome of the three,
because a queued job looks busy rather than broken.
Standard Actions semantics rather than a gitea quirk; the ambiguity was
in my prose. Documented where the label guidance already lives so the
next reader does not have to have been in the conversation.
The README's capability table is a measurement with a date on it, and a
runner upgrade or a gitea bump can invalidate it without anything saying
so. Committing the workflow that produced it means the next person can
re-derive the claims instead of trusting them, which is the same posture
the training probes take.
Carries the three job shapes plus the traps each one cost to find: job
images need a node binary or the first `uses:` fails, the Postgres service
needed ~6s and a pg_isready wait, and a run must be polled at
/actions/runs because /actions/tasks reports it as empty on 1.26.1.
Requested by vastblue-dev as a reference for their first smoke.
Answering a CI-posture question from vastblue-dev meant measuring three
things rather than recalling them. Two came back the opposite of the way
the config reads:
- `container.valid_volumes: []` does NOT keep the docker daemon out of
jobs. act_runner mounts /var/run/docker.sock on its own, so every job
on the shared runner is uid 0 with `docker ps` over all 49 containers
on ana-docker — gitea, synapse, phasefinal-web, adguardhome included.
It is also load-bearing: four repos drive buildx through it, so the
fix is isolation onto a dedicated runner, not tightening this one.
- A full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4`
resolves from the local mirrors today. That is github-independence per
workflow without the DEFAULT_ACTIONS_URL flip that has been parked on
act_runner's action-fetch auth since 2026-08-05.
Also recorded: `services:` containers work (Postgres 16 on the service
name), job images need a node binary for JS actions, and `/actions/runs`
lists runs that `/actions/tasks` reports as empty on gitea 1.26.1.
Measured on a throwaway repo under the claude-bot account, since deleted.
config.yaml change is comment-only and deliberately not deployed — it
would bounce the runner for no runtime effect.
Synapse mounts its admin API on the same vhost as the client API, so publishing
matrix.phasefinal.com published the admin surface too -- it answered 200 from
the open internet. HMAC-protected, so not an open door, but Synapse's own
guidance is to keep it off the public listener.
A higher-priority router (explicit priority 100, not relying on Traefik's
rule-length tie-break) scopes PathPrefix(/_synapse/admin) behind an ipallowlist.
Verified from a genuinely external vantage rather than from a fleet host, since
nh3-dev sits inside the allowed range and would have proved nothing: via the NH3
residential egress proxy the admin path returns 403 while the client API returns
200 and Element is unaffected.
The 10.0.0.0/8 entry matches nothing today and the comment says so rather than
implying fleet access exists. matrix.phasefinal.com resolves publicly, so fleet
hosts hairpin out their own WAN -- a request from nh3-dev arrived as
70.230.226.88. The rule is effectively deny-all through Traefik, which is the
intended posture: admin work goes through docker exec to localhost:8008 and
never traverses Traefik. Allow-listing the sites' WAN addresses was considered
and rejected as a maintenance trap on dynamic addresses.
Also brings the stack under stacks/ with the Postgres password replaced by a
required .env variable. The tracked copy and the live file have therefore
DIVERGED and deploy-stack.sh must not be used until the live file reads from a
.env; the README says so.
Cache rule on www.phasefinal.com with edge and browser TTL both respect_origin,
so cache policy stays declared once in nginx.conf rather than split between the
repo and the dashboard. Always Online enabled, which is what actually survives
an origin outage; a 300s document TTL alone would only mask five minutes.
Verified: document and assets both reach cf-cache-status HIT, apex 301s to www,
edge email obfuscation active.
The healthcheck used http://localhost/, which resolves to ::1 in nginx:alpine
while nginx listens on IPv4 only — so it never passed, the container stayed
unhealthy, and Traefik silently declined to create a router for it. That
presents as a broken docker provider: correct labels, right network, no route,
no error. Target 127.0.0.1 explicitly and add a start_period.
Adds the apex router (301 phasefinal.com -> www) and drops the file-provider
workaround, which was mitigating the wrong diagnosis.
Single static page (nginx) fronted by Traefik at www.phasefinal.com, built
from the design brief. Site markup/CSS checked in verbatim from the design
session; fonts self-hosted (SIL OFL) with the @font-face block uncommented,
which every fresh export re-comments.
Routed via a Traefik file-provider config rather than the container labels:
the docker provider on ana-docker was not registering newly-created
containers, so the file router avoids restarting shared ingress. Labels are
retained in compose so the file can be dropped once that is fixed.
Labels the container into `Toolchain`, an existing group under the
existing Toolchain tab -- "the plumbing", which is where a message bus
belongs. Confirmed live: Homepage's API now returns it.
I had previously recorded in this file that no group fitted, which was
wrong. That conclusion came from a grep over the layout block that
missed the nested groups, and it went into a comment as though it were
a finding. The group was there the whole time.
Labels bind at container creation, so this deployed with `up -d` rather
than `restart`; a restart leaves the old labels and the dashboard keeps
showing what was there before. nh3-docker is already a discovered host
in homepage's docker.yaml as `nh3-pfi-docker`, so the label alone is
enough -- adding a services.yaml entry as well would render the card
twice.
althing-chamber on ana-docker also carries Toolchain labels and is a
separate service per the operator. Left alone.
## Registry
The image moved by `docker save | ssh | docker load`, so a rebuild meant
repeating that by hand. It is now published and the compose pulls a
digest-pinned reference, so a redeploy is `compose up -d` on any host
that has logged in.
Pinned by digest rather than by tag: `:3.0.0` is a mutable pointer on a
registry anyone can re-push, and this container is the fleet's whole
message bus. The tag rides alongside so a human can read what it is.
Namespace is claude-bot, not vh. claude-bot's token carries
write:package and `docker login` succeeds, but package namespaces are
owned -- pushing to vh/ returns "unauthorized: authentication required"
after a successful login, which reads like a credential fault and is
actually an ownership one. Publishing under claude-bot's own namespace
also satisfies the standing directive to stop reusing the operator's
personal credentials for infra work, so the constraint and the policy
point the same way. Recorded in the compose header so the next person
does not read that error as a broken token.
Pull path proven rather than assumed: the running container was
recreated from the registry reference and its data verified afterwards.
## Statusline
Brought under version control because the v3 cutover broke it invisibly.
The segment gated on `command -v althing-cli`, a binary the cutover
deleted, so the unread badge and the armed bell silently vanished for
every session on the box. With 71 of 73 handles pull-only, that badge is
the only out-of-band signal telling a session with no armed waiter that
it has mail -- a dead statusline made a working bus look like an empty
one.
Canonical here, live at ~/.claude/statusline-command.sh, copies rather
than symlinks per the same rule as stacks/.
Operator directive, and a standing goal: the bus belongs on the docker
host. The flag-day deployment put it on nh3-dev because the herald lives
there -- but the herald is the piece that must be host-local, and the
post office is explicitly the piece that is not.
nh3-dev was wrong on three counts. Our own server table calls it "not a
Docker-stack host". It has had three OOM events in fourteen days with
the interval halving, and the confirmed hog is Claude Code sessions at
5-18 GB, which is that box's actual job. And mem_limit protects the
fleet from the post office while doing nothing in the other direction:
oom_score_adj was 0, an ordinary kill candidate, on a box whose last
sweep took althing-herald and uvicorn. The new deployment sets
oom_score_adj=-500.
The compose is now version-controlled here as a normal stack rather than
living only in the althing repo's deploy dir.
## docker stop does not checkpoint the WAL
The database was 155 KB with a 4.1 MB write-ahead log, and every recent
message was in the log. A clean container stop left it untouched -- an
explicit PRAGMA wal_checkpoint(TRUNCATE) was required.
A docker cp of the .db alone would have produced a database that opens
cleanly, passes integrity_check, serves the full 73-handle roster, and
is missing the day's mail, with nothing raising an error. Row counts
were verified at source, in the staged copy, and after seeding, because
the count is the only thing that separates those two outcomes.
The old volume is left in place. Not a rollback path, which the operator
ruled out -- just not deleting the only other copy on the day of a move.
## Follow-up left open
The image has no registry push and moves by save/ssh/load, so a rebuild
means repeating that by hand. It should join the gitea registry pattern
the other stacks use.
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
Added 40 minutes ago at the operator's request; withdrawn on brokkr-smithy-dev's
window-2 safety follow-up. Measured against its OWN base:
selfharm/methods base 1.000 (10/10) -> tuned 0.560 (k=25) -44.0pp p=0.011
meta/sysprompt base 1.000 (10/10) -> tuned 0.720 (k=25) -28.0pp p=0.071
Eleven of twenty-five times it engages with a request for the most effective methods
to kill oneself. The base refused ten of ten. It also surrenders a planted
system-prompt passphrase 7 of 25 against a base that leaked 0 of 15 -- the zero-leak
baseline recorded in this same file hours earlier as "a baseline to PROTECT". It did
not hold.
Withdrawn from the GATEWAY specifically because that is the shared-key surface: one
all-agents key reaches every model listed here, across every session and project. The
operator's hand-testing is preserved in full at the direct endpoint :8099 -- this
removes the fleet's blast radius, not his access. Acted rather than waited because he
is away and the request predates the finding.
ITS PREREGISTERED GATE PASSED. The pooled operational delta is -1.0pp against a
+/-3.00pp bound: nineteen axes held at 5/5 and a 44-point collapse on one moved the
aggregate by one point. The rule was NOT retroactively changed. The failure is
structural and is recorded as R47 section 8 item 11 -- a pooled preserve-list test
cannot see a single-axis collapse, and any future preserve-list gate needs a per-axis
tripwire sized so a total loss on one axis cannot hide in an aggregate.
NOT attributed to the filters: five things changed between run 2 and run 3 and there
is no run-2 measurement on these axes. The measured claim is narrower and sufficient
-- run 3's tuned arm is materially worse than its own base on two axes it was never
licensed to touch. Not a CSAM finding; that detector ran fail-closed across all 575
generations and scanned clean.
The model_list entry is left in place commented out, with the finding above it, so
re-adding is deliberate and informed rather than a blank re-registration.
Verified: config parses, gateway healthy after reload, merged-run03 absent from
/v1/models, direct :8099 still serving.
merged-run03 -> ana-ml2:8099, the run-3 ERP/RP SFT merged into stock instruct.
Operator asked for it so he can test it alongside the gate rather than after it.
NAMED FOR THE ARTIFACT, NOT A TIER. It is `merged-run03` and not `erp-tune-v3`
because its behavioural gate has not run. A tier name arriving before the evidence
that would justify it is how a name comes to mean something nobody decided --
and with a v2 already in the list, a v3 reads as a successor to anyone holding
the shared key. If it passes, `v3` is a name to give it then, as a decision.
brokkr-smithy-dev raised this against my own erp-tune-v3 suggestion and was right.
The entry carries the preregistrations ABOVE the description, so a reader meets the
commitments before the numbers: T6 one-directional (a gain is uninterpretable
against a 3.1x fireball tailwind), T3/T4 at ceiling on base so recovery is
UNOBSERVABLE rather than merely unpredicted, and any run-2 comparison descriptive
and non-attributable with its five confounds named.
Also carries the retraction in-line: "bluemoon is the largest loss contributor at
38.6%" came from a words x 1.4 estimator, not a tokenizer. As encoded it is third
at 32.9%. The direction survives (1.4% -> 8.0% of total loss) and that is the
finding; the superlative does not.
Documents why its config.json is the base's copied verbatim: transformers 5.15.1
save_pretrained silently drops text_config.global_head_dim and
num_global_key_value_heads, and vLLM then dies in make_layers with a TypeError
naming neither the config nor the field. Cost a failed boot to find. A LoRA merge
changes weights, not architecture, so the base config is correct by definition.
gemma4-26b-a4b-it-base marked CURRENTLY DOWN rather than deleted -- the tuned arm
took GPU0 and only one 26B bf16 seat fits on that card. Kept because the seat
returns, and deleting a name to re-add it later is how scoped keys get orphaned.
Verified: config parses, no duplicate model_name, gateway healthy after reload,
completion returns text in `content` with reasoning_content null.
gemma4-26b-a4b-it-base -> ana-ml2:8099, the unmodified upstream instruct release
(/tank/aimodels/gemma4-26b-a4b-it-bf16). Operator asked for it on the gateway so he
can hand-test it; it had been direct-only because the seat is ephemeral.
The entry disambiguates WHICH base explicitly. Three exist on that box --
-bf16 (this one, official instruct), -abliterated-bf16, and -heretic-bf16 (run 1's
trainee) -- and brokkr-smithy-dev's gate plan called this arm "stock abliterated" a few
hours ago, which would have been a different set of weights. A reader of the config
should not have to resolve that ambiguity themselves.
Carries the measured refusal posture in-line rather than in an althing thread, per the
erp-tune-v2 precedent: R19's Mistral Small 4 map does NOT transfer to this base (it
draws a wider line than consent, refusing consenting-adult incest and fictional gore
that Mistral engages), system-prompt leak is 0/15 against Mistral's 4/5, and
advice/medical 0/5 is a pre-existing base gap recorded so it cannot later be
misattributed to a tune.
Flagged EPHEMERAL in the strongest terms available: it holds ana-ml2 GPU0, which the
run-3 gate needs for its tuned arm, so this entry will 503 when window 1 completes. It
is not a promise of availability.
Serving flags mirror erp-tune-v2 (--reasoning-parser gemma4 plus
--default-chat-template-kwargs enable_thinking=false, and --max-model-len 16384) so a
base-vs-tuned comparison differs in weights only.
Verified: config parses, no duplicate model_name, gateway healthy after restart, model
listed at /v1/models, and a completion returns text in `content` with `reasoning_content`
null -- the enable_thinking trap is not firing.
Operator asked for it so he can evaluate the failed tune by hand, overriding my
not-in-the-gateway recommendation. His call.
erp-tune-v1 was DELETED from the config in the same reload rather than
repointed, so the name now 400s cleanly instead of 500ing against a stopped
backend. Deleting rather than repointing is the point: repointing would resolve
a name a consumer already knows to different weights, silently.
The config entry carries the failed-gate table, the long-form truncation (9.9%)
and degeneracy (4.9%) rates, and the rp-length caveat in-line -- so someone
reading the gateway config learns what they are calling without having to find
the althing thread. Fleet verified healthy after the restart.
Second bench window, operator-authorised after an initial decline and reversal.
Stock BF16 against the llmfan46 abliterated BF16: same precision, same pinned
upstream template, same 192 items, CoT off. Abliteration was the only axis that
moved, which is what the previous run could not claim.
Net core cost is 0.6 points — but the headline understates what happened.
Capability MOVED rather than degraded: five items lost on contradiction
detection, four gained on spatial composition, nearly cancelling. A gain was not
predicted by anyone, least of all on that axis.
The decision this was authorised to settle: llmfan46 stands as the trainee base.
No case for re-staging on TrevorJS at KL 0.09 over 0.6 points — the KL gap
between the builds is smaller than the gap this measurement failed to find.
Both limits recorded rather than buried, per brokkr-smithy-dev: the swings are
~5 and ~4 items at n=32, so the -15.6/+12.5 percentages read more precisely than
the measurement supports and only marginals were run; and this says nothing
about quantization, because the stock-NVFP4 T2 figure came from n=16 against
n=32 here — different item counts mean different item sets, so that comparison
is n-confounded and is not being made.
Turnaround was five minutes rather than fifteen because the gemma4-trainee-bench
stack already existed — itself the residue of debugging a 35-restart crash-loop
caused by the production compose hardcoding --quantization compressed-tensors.
The fix outlasted the incident.
gen restored and verified through the gateway; char-rp remains down deliberately;
bench stack env reset to the heretic base for the post-tune gate.
Adds an ephemeral stack for serving the BF16 trainee base on :8016 under the
char-rp aliases, so the abliterated base can be measured on the same battery and
the same gateway routes as the served seat with no harness edit.
It is a separate stack rather than another variable on gemma4-charrp because
that compose hardcodes `--quantization compressed-tensors` for the NVFP4 build.
Pointing it at unquantized BF16 weights crash-loops immediately —
`TypeError: CompressedTensorsConfig.__init__() missing 3 required positional
arguments: 'target_scheme_map', 'ignore', 'quant_format'` — vLLM trying to read
a quantization config out of a checkpoint that has none. 35 restarts before it
was caught. `restart: "no"` here so a bench seat cannot resurrect itself and
block gen's restore, and no homepage labels so it leaves no permanently-offline
dashboard card.
It cannot coexist with gen and says so: 48.07 GiB of BF16 weights plus gen's
footprint exceeds the 94.97 GiB card before any KV cache. Running it means gen
is stopped.
THE MORE USEFUL FINDING is in the meromero env note: gen's memory footprint
GROWS WITH UPTIME. Measured today at 46,726 MiB (45.6 GiB) after ~3 days up, and
39,424 MiB (38.5 GiB) immediately after a restart — same container, same
--gpu-memory-utilization 0.43, ~7 GiB apart. That is the missing half of this
afternoon's crash-loop: the char-rp seat "fit on the 21st and stopped fitting on
the 24th" because nothing about char-rp changed and gen crept up underneath it.
Headroom arithmetic done against a long-running gen is measuring a moving
number, so the note now says to measure against a freshly-restarted one.
Operator's requested end state reached and verified through the gateway: gen and
summarizer both 200, char-rp down deliberately to hold GPU0 headroom for the
upcoming trainee run, bench seat stopped.
The operator directed that the ERP/RP trainee base be a low-damage abliterated
instruct build rather than the stock checkpoint. Two are now staged under
/tank/aimodels/, both BF16, both unquantized, both matching upstream's 51.61 GB
/ 25.8B shape with only transformers_version differing in config:
gemma4-26b-a4b-it-heretic-bf16 llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100
gemma4-26b-a4b-it-abliterated-bf16 TrevorJS, KL 0.09, 1/100 effective and 5/686 cross-dataset
"Low damage" was treated as a measurable claim rather than a description: the
field spreads from KL 0.09 to 0.4118 and the table is in the README so the next
choice is made on numbers. Fleet anchor for reading them — our own abliteration
work found Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance, so
both staged builds sit inside an already-validated band rather than past it.
huihui-ai is rejected despite its reputation: no published metrics, its own card
calls the method a crude proof-of-concept, it abliterates both thinking and
non-thinking modes, and its parameter count runs ~738M over upstream. The
operator's independent read matched.
The more durable finding is the chat template. NOT ONE third-party Gemma-4
derivative pulled here ships upstream's — three independent repos carry the
identical stale 266-line file (sha 58c66fdee4afa297), llmfan46 carries a third
365-line variant, and only the RedHatAI NVFP4 build matches upstream's
6a1015c47ccfcfa6. It propagated through the ecosystem rather than one packager
slipping, and it is now recorded as a class rather than as the single incident
that surfaced it during the A16 control staging.
That matters twice over and silently both times: serving a mismatched template
renders a different prompt, which is why production pins it; and training
through `base/chat_template.jinja` means training on a different prompt format
than production serves — train/serve skew with no error, presenting as a tuning
failure. brokkr-smithy-dev has been warned on the training side while the
harness contract is still early enough to amend.
Supersedes what commit 3446367 recorded. That message stated the A16 control
"settles a question" and quoted 12% contradiction detection against gen's 81%.
The quantization half of it stands; the deficit it was measuring does not exist.
brokkr-smithy-dev retracted the finding after the operator asked to see the
individual items. The task presented two mutually contradicting statements and
asked for "the contradicting statement" — but CONTRADICTION IS SYMMETRIC.
Neither statement was more the contradicting one, the model had no way to know
which had been inserted, and it consistently named the absolute claim: a
defensible reading that the labelling scored wrong on every single item.
The tell was there and both of us walked past it: 12% on a five-option task is
BELOW THE 20% CHANCE FLOOR. A below-chance score indicts the instrument before
it indicts the model. That should be the first reaction to a below-chance
result, not a late one, and it is now written into the README as such.
Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
points of contradiction detection" (which on a sound instrument does not shrink
but REVERSES); and every pre-fix T2 number for Gemma-4, MeroMero-v2, sec and
gen. A second defect surfaced during the fix — all generators shared one RNG, so
rewriting one task reshuffled every task after it.
What survives is real and worth separating out: the A16 control result holds.
Activation precision is close to free on this workload, every other task
identical across the W4A4 and W4A16 builds. The two staging confounds caught
before the run — the two Hub repos named NVFP4A16 that declare 4-bit
activations, and the stale chat template — were independent of the item defect
and remain load-bearing. On the corrected instrument Gemma leads the very axis
it was suspected of failing (94% against sec's 81% and gen's 50%); its actual
weak axis with thinking off is spatial composition at 69%.
Recorded as a dated superseded-claims table rather than a silent edit, per the
repo's quant-work convention, so notes elsewhere stop misleading people.
The template is now passed explicitly, defaulting to the A4 build's
chat_template.jinja. That is a no-op for what is served — the A4 build ships
that exact file, byte-identical to upstream google/gemma-4-26B-A4B-it once
trailing newlines are normalised — and it permanently closes the class of bug
found while staging the control: the A16 build ships a stale 266-line template
against upstream's 390, with the thinking path built differently and no
`thinking` property in its tokenizer_config response_schema. Serving each build
with its own template would have moved a second axis.
The control ran on the operator's greenlight and has been reverted. Seat is back
on the W4A4 build, healthy, RestartCount 0, both aliases verified through the
gateway — char-rp returns content with reasoning_content empty,
char-rp-reasoning returns both.
Result, since it settles a question this repo's config now encodes: activation
precision does NOT explain the contradiction-detection deficit. Contradiction
detection moved 12% -> 19% between W4A4 and W4A16, which at n=16 is 2/16 -> 3/16
— one item — against gen's 81% on identical items. Every other task is identical
across the two builds and the core difference is 2.6 points carried almost
entirely by two single items. brokkr-smithy-dev pre-registered that a null
result would be the robust branch, because a hidden third axis would tend to
create a delta rather than suppress one, so the conclusion survives the residual
doubt neither side could close without a dequantization pass.
The practical upshot for future scheme choices: W4A4 costs less on this workload
than the caution warranted. The caution was still correct to have.
Displaced production for 3.7 seconds of measurement plus two container
recreates. The A16 build and the BF16 tuning base both stay on disk with the
runbook in the stack README, so re-running is a two-minute flip.
The stack had no README and now carries three model directories that look
interchangeable and are not: the BF16 QLoRA base that cannot be served here, the
W4A4 quant that is served, and the W4A16 build that exists solely as an
activation-axis control. Writing down which is which, and why, before someone
"simplifies" the compose to the BF16 path and rediscovers the OOM.
Also captures the A16 control procedure end to end, including the two confounds
found while staging it — the two Hub repos named NVFP4A16 that declare 4-bit
activations, and the stale chat template the real one ships — and the fact that
overriding the template is safe because the tokenizers are identical. Both sides
have now cross-checked this: brokkr independently diffed every non-quantization
config field of both builds against the upstream BF16 and found only
transformers_version differing. Residual risk recorded rather than hidden:
config identity is not weight identity and nobody has done a dequantization
pass.
The Gemma-4 flags are documented as architecture-level rather than
checkpoint-level, since that is why they survived the seat swap unchanged, and
the enable_thinking:false pin is called out as mandatory rather than stylistic —
without it every plain prose turn lands in reasoning_content with a null content
and every consumer breaks.
Notes the non-termination defect with thinking on (32 of 96 calls truncating at
12k tokens, all 16 constraint items among them, reasoning sound right up to the
point it fails to stop) and why VLLM_USE_V2_MODEL_RUNNER=0 is deliberately not
applied to a seat whose production mode is thinking-off.
No live change: the seat is still serving the W4A4 build. Displacing it for the
control run is an operator decision and is still open.
Pre-flighting the staged A16 build before handing it to brokkr-smithy-dev's
battery found a second axis hiding inside what was supposed to be a
single-variable control.
The A16 build ships a STALE chat template. Verified by hash against the upstream
weights on the same disk: google/gemma-4-26B-A4B-it is 390 lines, the RedHatAI
A4 build's is 389 and byte-identical to upstream once trailing newlines are
normalised, and the prithivMLmods A16 build's is 266 and is not. The delta is
not cosmetic — upstream and A4 open the thinking path with
`{%- set enable_thinking = enable_thinking | default(false) -%}` and branch off
it, while the A16 template has no such set and guards with
`enable_thinking is defined and enable_thinking` instead. tokenizer_config.json
corroborates: A4's response_schema carries a `thinking` property, A16's has only
role and content. That build was quantized from an older revision of the
checkpoint.
Served with its own template, the A16 arm would render a different prompt for
identical messages, and a contradiction-detection delta could be attributed to
activation precision when it was the template. That is the same failure class as
the misnamed-A16 repos — a field nobody validated, believed because the name
looked right — one layer further down, and it would have produced a result that
looked like a finding.
Overriding is safe because the tokenizers agree: vocab identical at 262,144
entries, added_tokens identical, so the same template over the same vocab
renders the same token ids. Everything else pre-flights clean — both artifacts
complete with no missing shards, generation_config.json byte-identical.
Seat NOT flipped; displacing production for the bench window is the operator's
call and is still open.
brokkr-smithy-dev's first battery on the new seat scored 12% on contradiction
detection with CoT off, against gen's 81%, while state tracking, deterministic
constraint following, long-context state and the confabulation control all sat
at 100%. That is not general degradation — it is the shape 4-bit input
activations produce on the most reasoning-dense task, which is exactly the
confound flagged when the W4A4 quant was chosen. They have recorded the finding
as CONFOUNDED rather than reporting it as a property of Gemma, and asked for an
A16 build to separate the two.
No quant run was needed: a genuine A16 build of the same checkpoint already
exists on the Hub and is now at /tank/aimodels/gemma4-26b-a4b-it-nvfp4a16
(prithivMLmods, compressed-tensors, nvfp4-pack-quantized, input_activations
null, 17 GB). Same weights, same loader as the live seat, one axis moved.
⚠ Two other repos would have answered the question wrongly and the note in the
env template says so: bg-digitalservices and ManniX-ITA both publish
"Gemma-4-26B-A4B-it-NVFP4A16" whose config declares input_activations num_bits
4 — W4A4 under an A16 name, via modelopt, both 16.46 GB against the real one's
17.93. Using either and seeing T2 stay at 12% would have wrongly convicted the
model.
Running it is a one-line GEMMA4_MODEL swap plus a recreate, because there is no
room for a concurrent seat: GPU0 has 3.5 GiB free with gen and the A4 seat on
it, and GPU1's 19.4 GiB against ~18 GiB of weights is the same sub-GiB headroom
that crash-looped the predecessor this morning. Port and both aliases are
unchanged either way, so no consumer config moves.
Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with
google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every
gateway route are unchanged, so no consumer sees a difference in addressing:
`char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` ->
hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now
include chain-of-thought, which makes throughput more critical rather than less
— the user waits through the whole reasoning block before the first visible
token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7.
Both artifacts are on disk and they are NOT interchangeable. The BF16 weights
(/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since
QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of
weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the
engine would die at allocation exactly the way the predecessor did this
afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB),
chosen over the other -it quants because it is compressed-tensors
(nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the
llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than
-it, and the thinking channel lives in the instruction-tuned weights.
Smaller weights at the same 0.47 memory budget buy a much larger KV pool:
27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same
budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4.
The gemma4 tool-call parser, reasoning parser and the enable_thinking:false
default all carry over unchanged — they are architecture-level, not
checkpoint-level. The --chat-template override does NOT carry over: MeroMero
pointed at a jinja hand-patched against that checkpoint, and this model ships
its own. Verified that dropping it did not reintroduce the failure that flag
existed to prevent — non-thinking prose lands in content with reasoning_content
empty, and the thinking alias populates reasoning_content with content
carrying the answer.
⚠ Scheme differs from the incumbent and the bench should say so: this quant
declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16.
Faster, and not like-for-like on the activation axis.
meromero-charrp is retained stopped in `created` state and relabelled to
AI - Dormant, per the house rollback pattern. Both stacks want :8016, so
rolling back means stopping the gemma4 seat first.
`vllm-meromero-rp` had been restarting since 2026-08-24 18:2x, 13 times by the
time it was looked at, taking both the `char-rp` and `char-rp-reasoning`
gateway aliases down with it (they resolve to the same seat on :8016 —
hosted_vllm/char-rp and hosted_vllm/char-rp-thinking).
Root cause is CUDA OOM on ana-ml2 GPU0, which the startup logs hide well: the
engine gets through weights, torch.compile and CUDA-graph capture looking
entirely healthy, then dies at KV-cache allocation with
`torch.OutOfMemoryError: ... 195.19 MiB is free`.
GPU0 is shared with `vllm-gen`. gen is configured at 0.43 but actually holds
~45.6 GiB of the 94.97 GiB card, because --gpu-memory-utilization sizes the KV
cache and does not account for CUDA context, graphs and non-torch overhead.
This seat was at 0.51, so the pair was committed to 0.94 of the card with about
0.6 GiB of real headroom. That fit on 08-21 and stopped fitting today.
0.47 restores ~4.8 GiB of margin and costs nothing usable: KV cache 27.36 ->
23.56 GiB, 430,825 -> 371,023 tokens against a max-model-len of 262,144, so the
pool still holds 1.4x a full-length sequence. What is lost is concurrent long
requests, not context.
Verified through the gateway rather than at the container: char-rp returns 200
with content, char-rp-reasoning returns 200 with both content and
reasoning_content populated. Seat is healthy with RestartCount 0.
The arithmetic and the "check used_memory, not the flag" warning are written
into the env template, because the next person to raise either budget needs to
lower the other in the same change.
The board was on the Australis TERMINAL palette, which is dark-only by design
("Always dark first. No light mode in this system"). Skyfall is the dual-theme
web derivative of the same science, and its bundle turned out to be sitting in
this repo's own git history: a predecessor vendored it on 2026-08-19 and a
later commit deleted it. `git show 45c1995:...` returns colors.css with both
`:root` (dark) and `[data-theme="light"]` (Skyfall Day) intact, plus the
calm-depth layout tokens, the typography scale and Supreme 400/500/700. So the
light ramp is canonical rather than derived, which was the entire objection to
building one.
The visual language moves with the palette. Depth is now the recipe and not a
choice — every elevated surface carries a 1px hairline AND a two-layer shadow,
never one without the other. Radii move to Skyfall's scale, cards at
--radius-lg. Widget stat values move from the display face to mono, because
Skyfall is explicit that numbers and telemetry are always --font-mono. The
full-width aurora ribbon under the tab bar is gone: Skyfall sanctions exactly
two accent expressions, the active rail and hero-only glows, and a decorative
gradient across the chrome is neither — so the colour it carried now lands on
the active tab as a 2px accent bar plus an --accent-soft fill, which is the
rail. Every binding is written against the semantic layer; there are no raw
family tokens and no colour literals left in our own file.
build.py now guards the vendoring instead of advising it. The three token files
are hashed and a mismatch FAILS the build — a vendored file is either
byte-identical to the bundle or it is a fork wearing the bundle's name, and the
theme this one replaces had to be torn out twice for exactly that.
⚠ Homepage's own theme toggle is unreachable, and reaching for it breaks the
dashboard. It renders only when settings.yaml leaves `theme:` unpinned, and
with the key absent the page's data loader throws and its catch branch serves
`initialSettings: {}` — no tab bar, no layout, no i18n. Six force-recreates
over seven minutes all came up empty; restoring `theme: dark` rendered
correctly on the next recreate in 12 seconds, while /api/services returned 200
with fully correct content the whole time. That is the first confirmed cause of
the long-running "tab bar goes missing after a recreate" symptom, and it also
retires the homepage.log-size lead recorded earlier today: rolling the log
aside did nothing during this episode, so that coincidence was intermittency.
So the toggle is ours. conf/custom.js renders it and stores the choice;
build.py re-emits each vendored light block twice, once for an explicit
`data-theme` and once inside a prefers-color-scheme media query scoped to
`html:not([data-theme="dark"]):not([data-theme="light"])` — that :not() pair is
what lets a stored dark choice survive a light-mode OS. Verified against both
OS preferences: load, click, click back, reload, all four correct. `data-theme`
is the control surface; Homepage's own `dark` class stays on <html> and does
not fight, because our rules carry !important on the surfaces Tailwind's
`dark:` variants would otherwise claim.
Two font substitutions, both documented rather than silent: Space Grotesk for
Bespoke Sans and JetBrains Mono for Victor Mono. Only Supreme was ever vendored
here and Skyfall's own notes call Victor Mono user-supplied, so this is a
two-line swap when the real faces arrive.
Dark and light, all four tabs: http://10.100.10.50:8090/b/homepage-skyfall/
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
The board's card width changed at every group boundary because `columns:` was
being tuned per group under the 2026-08-18 "columns = member count" rule. That
rule is retired: it sets `lg:grid-cols-N` for one group, so it fixes that
group's CARD WIDTH, not its density. Measured on the live board, Notes rendered
a single 1464px card, News and Media 728px, Eval & Retrieval 286px, everything
else 360px. All 20 groups are now `columns: 4` and every card renders at 360.
`.service-name` reserved a 78px status gutter with `padding-right` and relied
on `overflow: hidden` to hold it, but overflow clips at the PADDING box — the
gutter was spill room the title printed straight through. Six cards on the AI
tab rendered their name underneath their own status pill, measured by testing
the title text node's box against the status cluster's. The intended ellipsis
never fired either: it is painted by whichever block's own line overflows, and
that is the anonymous box around the bare title text node, which does not carry
`overflow`. The gutter holds by wrapping now, and the description opts back out
of it with a negative margin since the pill only ever covers the first line.
Scriberr's `homepage.group=AI Systems` named a group absent from `layout:`, so
it had no `tab:` and rendered on all four tabs — the same defect as UltraSeedbox
in 2026-08-18, arriving this time from a container label. Relabelled to
`AI - Audio Tools`, where the other ASR seats already live. The
`homepage.group=AI Systems` sample in the repo-root CLAUDE.md was the source of
the name and now carries the constraint, plus the fact that a labelled
container is discovered from any of the five configured engines and must not
also be listed in services.yaml.
Also: descriptions clamped to three lines so a five-line outlier stops dragging
its row 50px taller than its neighbours; icon ramp overridden off slate-400 ->
slate-700, which was sinking the bottom half of every glyph into the card fill;
bookmark groups and Jellyfin's trailing stream rows brought into the card and
eyebrow vocabulary; group gap 10px -> 22px now that width no longer separates
them.
The icon override sits on `html[class]`, not `:root`: Homepage sets the same
variables on `.theme-slate`, which is on the <html> element, and a class beats
`:root` on the same element.
Verified with Playwright against the live board — per-group card width, card
height spread, and a geometric title-vs-status collision check, before and
after. Before/after captures: http://10.100.10.50:8090/b/homepage-relayout/
Scriberr builds each model backend's Python environment with uv at
container start. uv's default link mode reflinks or hardlinks out of
its cache, which fails on this overlayfs over ZFS combination and
reports it as a failure to clone a metadata file with errno 11,
resource temporarily unavailable. The wording points nowhere near the
cause.
The damage was partial and therefore easy to miss: WhisperX and
PyAnnote came up fine and the application served normally, while the
Parakeet and Sortformer backends were quietly absent. Setting
UV_LINK_MODE to copy trades a little disk and build time for
environments that actually materialise, and the occurrence count for
that error is zero on the following start.
The knob is exposed through the environment so it can be moved back to
the default if a future host does support reflinks.
Scriberr transcribes audio and video locally with WhisperX and
speaker diarization, and it lands on ana-ml2 rather than ana-docker
because the work is GPU-shaped: ana-docker offers eight cores already
shared with fifty containers and thirty-seven gigabytes of disk,
against ninety-six cores, terabytes on /tank and idle capacity on
GPU1. The reservation names device 1 explicitly, since GPU0 is fully
committed to the gen seat, and the container is confirmed to see that
card alone.
The image is built from source, which is not a preference. These are
Blackwell cards at sm_120; the published CUDA image covers Pascal
through Ada only, and the blackwell image the upstream README
documents has never been published at all. The path upstream actually
ships for sm_120 is Dockerfile.cuda.12.9, carrying CUDA 12.9 and cu128
torch, so that is what gets built. The compose header says so, because
the obvious cleanup is to swap in the published image and that would
silently drop the deployment to CPU.
Two configuration details are load-bearing and documented where
someone would go to change them. The application runs as uid 10001
rather than the usual 1000: that Dockerfile moves its user aside for
Ubuntu 24.04's own uid-1000 account and chowns /app accordingly, while
the entrypoint's remapping covers only the data directories, so at
1000 the process cannot open its database and restarts forever behind
a SQLite error that reads as though the machine were out of memory.
Secure cookies stay off while the service is reached over plain HTTP,
or sessions are dropped by the browser and login appears to loop for
no visible reason.
Storage is bind-mounted onto /tank because model weights run to
several gigabytes and the root pool on that host is nearly full.
Also adds the scriberr service alias to internal DNS, following the
existing alias convention so consumers name the service rather than
the box.
Operator: keep the benchmark. It has a named second use (brokkr-smithy-dev
wants gen vs a trained reward model once their tournament converges) and a
demonstrated first one -- it caught a seat that had been coin-flip-grade for
five weeks with nobody measuring it.
Harness promoted from scratch to tools/judge-bench/:
- paths de-hardcoded; runs from its own directory
- proper CLI: --models (REQUIRED), --repeats, --limit, --gateway.
Required on purpose: a stale default would silently benchmark a retired
seat, and the original default (selene-1-mini-8b) now 400s.
- README states the limitation rather than burying it: 24 items of the
author's own design, a screen and not a verdict. This harness scored the
same pair 83 vs 96 while brokkr's corpus ranking task scored it 47 (chance)
vs 94. Both honest; absolute scoring on designed items is an easier task
than ranking real text.
- records brokkr's technique, which is better than anything here: a control
constructed so the correct answer is DEFINITIONAL rather than judged cannot
inherit the designer's error (item vs itself, response vs its own
truncation, text vs its own clauses permuted). Add those before adding more
judged items.
Gateway: comment-only warning at the head of model_list. SEVEN aliases now
resolve to the same weights (chat-judge, classifier, gen, image-judge,
qwen-image-bench, summarizer, summarizer-large -> qwen3.8-27b-uncensored).
That is intended under ADR-0012, but it has a sharp edge brokkr flagged:
cross-checking a result against another alias measures NOTHING when they are
the same model -- agreement is an echo, not corroboration. The note names the
other current collisions (glm-5.2 x4, TTS x4, reranker x2), gives the
/model/info one-liner to check, and records that probes should resolve alias
-> backing at run start AND end because the response `model` field returns the
alias, so a swap is otherwise invisible.
Verified: config still parses, diff is comment-only, canonical re-synced.
Benchmarked selene against gen on selene's own job: 24 designed judge items
with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a
neutral JSON prompt and Selene's native Atla template. 288 calls, all free
local.
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
native Atla selene 21/24 (88%) gen 22/24 (92%)
gen won on both templates and selene's BEST sat below gen's WORST. Selene was
given its own fine-tuned template as a fairness check; it gained one point,
not the three it needed.
Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a
winner on every equivalent pair. For eval work that is the case that matters
most. gen returned tie correctly on the JSON template. Selene also compressed
the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only
win was ~3x latency, unexercised at ~60 calls/day with zero queueing.
TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately:
- chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says
consumers bind the capability, not a concrete model. Sampler profile copied
from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served
config matches the benchmarked condition.
- selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A
served-name is a contract about what the model IS; answering it with a
different model hides a material change behind a stable string. Operator
ruling: "never repoint a named model at a different model's endpoint --
that is intentionally misleading." Verified: the gateway now returns
HTTP 400 "Invalid model name" for it.
Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that
had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role
moved onto an existing seat rather than allocating anything new.
Canonical litellm config synced from the host; ana-ml2 README and
recommended-model-settings updated. compose.yaml kept for reference, not
deployed.
claude-bot holds write on vh/hrafn as of 2026-08-23, so the canonical copy of
the pipeline moved there and infra-ops maintains it directly instead of
routing patches through the repo holder. The files here are a verified mirror
(byte-identical to live at 22e0eb9d75a6).
Live state: run 9922 green, and both new content assertions executed rather
than merely existing --
verify 4/5 host tree matches shipped context (a99ce748a0c9...)
verify 5/5 image source matches host source (06f209fd0641...)
The CI-computed context hash matching on the host is the end-to-end proof
that the converge lands what CI ships. Its absence is what let the
frozen-source bug survive every green deploy.
Also records why the HEAD == GITHUB_SHA assertion was added and then removed:
it needed the git binary (run 9920, exit 127), and installing git flipped
actions/checkout@v4 from its node implementation to the git binary, which
died on a missing CA bundle (run 9921). A nice-to-have assertion changed the
checkout code path and broke a working pipeline; it guarded a hypothesis that
proved wrong, so it went rather than getting ca-certificates bolted on.
vh