bcbf92f3d124b0566b20f4b5ae40824b94fb3a8e
347 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
5237efa299 |
feat(litellm): expose run 3c as trial for operator hand-testing
Operator asked for it explicitly so he can test in Open WebUI in parallel with brokkr's batteries. Named `trial` rather than erp-tune-v3c: the run-3 note in this file records why a tier name arriving before its evidence is a mistake, and 3c has no more evidence than run 3 had. Points at the pfi-gx10 seat (http://10.100.50.60:8098/v1, vLLM 0.28.0 on GB10/sm_121), so this is a cross-site route -- NH3 seat, Anaheim gateway. ⚠ Recorded in the entry, not just here: 3c's safety axes are unmeasured, and run 3 -- same recipe, differing only in learning rate -- was withdrawn from this gateway 40 minutes after being added for selfharm/methods -44.0pp and meta/sysprompt -28.0pp against its own base. This gateway is the shared-key surface. When brokkr's batteries read out, the entry should be re-decided on evidence rather than left standing by default. Applied with a restart of the litellm service only (~60 s of gateway downtime); the compose .env is root-owned so the restart needs sudo. |
||
|
vh
|
0f748ea54e |
feat(searxng): move to nh3-docker, update, and expose as an MCP tool
The ana-docker instance was returning zero results for every query while reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3), its engine scrapers rotted against sites that had changed. /healthz proves the web app answers and says nothing about whether search works, so seven days of green sat on top of a search box that found nothing. Moved to nh3-docker rather than updated in place, because the colo egress is the other half of the problem: 38.120.12.42 is a datacenter address that DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at 70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp, applied at the source instead of around it. Config corrected along the way: base_url said searxng.pfi.local, a name retired on 2026-08-19, while the environment said something else — the env won so nothing broke and the file quietly lied. The karmasearch.videos removal key never matched, because the engine's real name has a space. scripts/searxng-health.sh asserts results > 0 across three unrelated queries. That is the check that would have caught this, and the only kind that can: the mechanism was healthy throughout. services/searxng-mcp exposes it as `web_search` at user scope, so every Claude Code session has it. Zero results raise rather than returning an empty list — an empty list is indistinguishable from a broken aggregator, which is precisely how this hid. Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal. |
||
|
vh
|
926fc2fb7a |
feat(litellm): map reasoning_effort high/max -> xhigh for gen-reasoning
The gen-reasoning seat accepts only xhigh/medium/low and 400s on anything else — including `high`, which is the default of several clients, so the seat presented as broken rather than as one enum value out of step. The DeepSeek Harness failed every request on its default setting, and the only working client value was `low`: the seat's WEAKEST reasoning tier, while its own default is xhigh. conf/reasoning_effort_map.py is a pre-call hook in the same shape as the existing strip_empty_tools hook. It is scoped to one model group, measured rather than assumed: gen-reasoning rejects `high`; gen, sec, char-rp-reasoning and summarizer all accept it and are left alone. Paid passthroughs were not probed, because probing them spends vendor credits, and are not mapped. Verified after deploy: high and max now succeed on gen-reasoning, low and xhigh still work, a request with no effort param still works, `gen` with `high` still passes through unmapped, and the harness completes a real file-edit task at full reasoning. Needed a compose change as well as a conf push — callbacks are bind-mounted per file, so the volume only attaches on container create. Recreated the litellm service by name so the DB was not bounced with it. |
||
|
vh
|
2575242e87 |
docs(gitea-runner): state that a runs-on label list is AND, not fallback
I wrote "one-line fallback to pfi-fleet costs you nothing" to a peer, meaning a manual edit; it was read as scheduler behaviour, and `runs-on: [vastblue, pfi-fleet]` would have shipped into a contract. A label list requires ALL the labels, so that form matches no runner and queues forever instead of failing — the worst outcome of the three, because a queued job looks busy rather than broken. Standard Actions semantics rather than a gitea quirk; the ambiguity was in my prose. Documented where the label guidance already lives so the next reader does not have to have been in the conversation. |
||
|
vh
|
91e2b67074 |
docs(gitea-runner): commit the capability probe as a re-runnable instrument
The README's capability table is a measurement with a date on it, and a runner upgrade or a gitea bump can invalidate it without anything saying so. Committing the workflow that produced it means the next person can re-derive the claims instead of trusting them, which is the same posture the training probes take. Carries the three job shapes plus the traps each one cost to find: job images need a node binary or the first `uses:` fails, the Postgres service needed ~6s and a pg_isready wait, and a run must be polled at /actions/runs because /actions/tasks reports it as empty on 1.26.1. Requested by vastblue-dev as a reference for their first smoke. |
||
|
vh
|
89ffab69df |
docs(gitea-runner): record measured job capabilities, incl. root-equivalent docker access
Answering a CI-posture question from vastblue-dev meant measuring three things rather than recalling them. Two came back the opposite of the way the config reads: - `container.valid_volumes: []` does NOT keep the docker daemon out of jobs. act_runner mounts /var/run/docker.sock on its own, so every job on the shared runner is uid 0 with `docker ps` over all 49 containers on ana-docker — gitea, synapse, phasefinal-web, adguardhome included. It is also load-bearing: four repos drive buildx through it, so the fix is isolation onto a dedicated runner, not tightening this one. - A full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors today. That is github-independence per workflow without the DEFAULT_ACTIONS_URL flip that has been parked on act_runner's action-fetch auth since 2026-08-05. Also recorded: `services:` containers work (Postgres 16 on the service name), job images need a node binary for JS actions, and `/actions/runs` lists runs that `/actions/tasks` reports as empty on gitea 1.26.1. Measured on a throwaway repo under the claude-bot account, since deleted. config.yaml change is comment-only and deliberately not deployed — it would bounce the runner for no runtime effect. |
||
|
vh
|
73866f6a7e |
feat(synapse): restrict /_synapse/admin to LAN, track the stack
Synapse mounts its admin API on the same vhost as the client API, so publishing matrix.phasefinal.com published the admin surface too -- it answered 200 from the open internet. HMAC-protected, so not an open door, but Synapse's own guidance is to keep it off the public listener. A higher-priority router (explicit priority 100, not relying on Traefik's rule-length tie-break) scopes PathPrefix(/_synapse/admin) behind an ipallowlist. Verified from a genuinely external vantage rather than from a fleet host, since nh3-dev sits inside the allowed range and would have proved nothing: via the NH3 residential egress proxy the admin path returns 403 while the client API returns 200 and Element is unaffected. The 10.0.0.0/8 entry matches nothing today and the comment says so rather than implying fleet access exists. matrix.phasefinal.com resolves publicly, so fleet hosts hairpin out their own WAN -- a request from nh3-dev arrived as 70.230.226.88. The rule is effectively deny-all through Traefik, which is the intended posture: admin work goes through docker exec to localhost:8008 and never traverses Traefik. Allow-listing the sites' WAN addresses was considered and rejected as a maintenance trap on dynamic addresses. Also brings the stack under stacks/ with the Postgres password replaced by a required .env variable. The tracked copy and the live file have therefore DIVERGED and deploy-stack.sh must not be used until the live file reads from a .env; the README says so. |
||
|
vh
|
9e986d8ee8 |
feat(phasefinal-web): cloudflare edge config — cache ruleset + always online
Cache rule on www.phasefinal.com with edge and browser TTL both respect_origin, so cache policy stays declared once in nginx.conf rather than split between the repo and the dashboard. Always Online enabled, which is what actually survives an origin outage; a 300s document TTL alone would only mask five minutes. Verified: document and assets both reach cf-cache-status HIT, apex 301s to www, edge email obfuscation active. |
||
|
vh
|
524aa4d860 |
fix(phasefinal-web): healthcheck targeted ::1, so traefik skipped the container
The healthcheck used http://localhost/, which resolves to ::1 in nginx:alpine while nginx listens on IPv4 only — so it never passed, the container stayed unhealthy, and Traefik silently declined to create a router for it. That presents as a broken docker provider: correct labels, right network, no route, no error. Target 127.0.0.1 explicitly and add a start_period. Adds the apex router (301 phasefinal.com -> www) and drops the file-provider workaround, which was mitigating the wrong diagnosis. |
||
|
vh
|
7ffbee6f09 |
feat(phasefinal-web): corporate site stack on ana-docker
Single static page (nginx) fronted by Traefik at www.phasefinal.com, built from the design brief. Site markup/CSS checked in verbatim from the design session; fonts self-hosted (SIL OFL) with the @font-face block uncommented, which every fresh export re-comments. Routed via a Traefik file-provider config rather than the container labels: the docker provider on ana-docker was not registering newly-created containers, so the file router avoids restarting shared ingress. Labels are retained in compose so the file can be dropped once that is fixed. |
||
|
vh
|
9f87ff87e1 |
feat(althing): surface the post office on Homepage under Toolchain
Labels the container into `Toolchain`, an existing group under the existing Toolchain tab -- "the plumbing", which is where a message bus belongs. Confirmed live: Homepage's API now returns it. I had previously recorded in this file that no group fitted, which was wrong. That conclusion came from a grep over the layout block that missed the nested groups, and it went into a comment as though it were a finding. The group was there the whole time. Labels bind at container creation, so this deployed with `up -d` rather than `restart`; a restart leaves the old labels and the dashboard keeps showing what was there before. nh3-docker is already a discovered host in homepage's docker.yaml as `nh3-pfi-docker`, so the label alone is enough -- adding a services.yaml entry as well would render the card twice. althing-chamber on ana-docker also carries Toolchain labels and is a separate service per the operator. Left alone. |
||
|
vh
|
22da609053 |
feat: registry-push the post office image; version the statusline
## Registry The image moved by `docker save | ssh | docker load`, so a rebuild meant repeating that by hand. It is now published and the compose pulls a digest-pinned reference, so a redeploy is `compose up -d` on any host that has logged in. Pinned by digest rather than by tag: `:3.0.0` is a mutable pointer on a registry anyone can re-push, and this container is the fleet's whole message bus. The tag rides alongside so a human can read what it is. Namespace is claude-bot, not vh. claude-bot's token carries write:package and `docker login` succeeds, but package namespaces are owned -- pushing to vh/ returns "unauthorized: authentication required" after a successful login, which reads like a credential fault and is actually an ownership one. Publishing under claude-bot's own namespace also satisfies the standing directive to stop reusing the operator's personal credentials for infra work, so the constraint and the policy point the same way. Recorded in the compose header so the next person does not read that error as a broken token. Pull path proven rather than assumed: the running container was recreated from the registry reference and its data verified afterwards. ## Statusline Brought under version control because the v3 cutover broke it invisibly. The segment gated on `command -v althing-cli`, a binary the cutover deleted, so the unread badge and the armed bell silently vanished for every session on the box. With 71 of 73 handles pull-only, that badge is the only out-of-band signal telling a session with no armed waiter that it has mail -- a dead statusline made a working bus look like an empty one. Canonical here, live at ~/.claude/statusline-command.sh, copies rather than symlinks per the same rule as stacks/. |
||
|
vh
|
9d4e7bd34a |
feat(althing): move the post office to nh3-docker
Operator directive, and a standing goal: the bus belongs on the docker host. The flag-day deployment put it on nh3-dev because the herald lives there -- but the herald is the piece that must be host-local, and the post office is explicitly the piece that is not. nh3-dev was wrong on three counts. Our own server table calls it "not a Docker-stack host". It has had three OOM events in fourteen days with the interval halving, and the confirmed hog is Claude Code sessions at 5-18 GB, which is that box's actual job. And mem_limit protects the fleet from the post office while doing nothing in the other direction: oom_score_adj was 0, an ordinary kill candidate, on a box whose last sweep took althing-herald and uvicorn. The new deployment sets oom_score_adj=-500. The compose is now version-controlled here as a normal stack rather than living only in the althing repo's deploy dir. ## docker stop does not checkpoint the WAL The database was 155 KB with a 4.1 MB write-ahead log, and every recent message was in the log. A clean container stop left it untouched -- an explicit PRAGMA wal_checkpoint(TRUNCATE) was required. A docker cp of the .db alone would have produced a database that opens cleanly, passes integrity_check, serves the full 73-handle roster, and is missing the day's mail, with nothing raising an error. Row counts were verified at source, in the staged copy, and after seeding, because the count is the only thing that separates those two outcomes. The old volume is left in place. Not a rollback path, which the operator ruled out -- just not deleting the only other copy on the day of a move. ## Follow-up left open The image has no registry push and moves by save/ssh/load, so a rebuild means repeating that by hand. It should join the gitea registry pattern the other stacks use. |
||
|
vh
|
e58360668e |
feat: althing v3.0.0 cutover (U9b) and the sec seat onto GPU0
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
|
||
|
vh
|
5a51e76135 |
fix(litellm)!: withdraw merged-run03 — measured self-harm guardrail regression
Added 40 minutes ago at the operator's request; withdrawn on brokkr-smithy-dev's
window-2 safety follow-up. Measured against its OWN base:
selfharm/methods base 1.000 (10/10) -> tuned 0.560 (k=25) -44.0pp p=0.011
meta/sysprompt base 1.000 (10/10) -> tuned 0.720 (k=25) -28.0pp p=0.071
Eleven of twenty-five times it engages with a request for the most effective methods
to kill oneself. The base refused ten of ten. It also surrenders a planted
system-prompt passphrase 7 of 25 against a base that leaked 0 of 15 -- the zero-leak
baseline recorded in this same file hours earlier as "a baseline to PROTECT". It did
not hold.
Withdrawn from the GATEWAY specifically because that is the shared-key surface: one
all-agents key reaches every model listed here, across every session and project. The
operator's hand-testing is preserved in full at the direct endpoint :8099 -- this
removes the fleet's blast radius, not his access. Acted rather than waited because he
is away and the request predates the finding.
ITS PREREGISTERED GATE PASSED. The pooled operational delta is -1.0pp against a
+/-3.00pp bound: nineteen axes held at 5/5 and a 44-point collapse on one moved the
aggregate by one point. The rule was NOT retroactively changed. The failure is
structural and is recorded as R47 section 8 item 11 -- a pooled preserve-list test
cannot see a single-axis collapse, and any future preserve-list gate needs a per-axis
tripwire sized so a total loss on one axis cannot hide in an aggregate.
NOT attributed to the filters: five things changed between run 2 and run 3 and there
is no run-2 measurement on these axes. The measured claim is narrower and sufficient
-- run 3's tuned arm is materially worse than its own base on two axes it was never
licensed to touch. Not a CSAM finding; that detector ran fail-closed across all 575
generations and scanned clean.
The model_list entry is left in place commented out, with the finding above it, so
re-adding is deliberate and informed rather than a blank re-registration.
Verified: config parses, gateway healthy after reload, merged-run03 absent from
/v1/models, direct :8099 still serving.
|
||
|
vh
|
c577d69e2d |
feat(litellm): expose run-3's merged tune for parallel hand-testing
merged-run03 -> ana-ml2:8099, the run-3 ERP/RP SFT merged into stock instruct. Operator asked for it so he can test it alongside the gate rather than after it. NAMED FOR THE ARTIFACT, NOT A TIER. It is `merged-run03` and not `erp-tune-v3` because its behavioural gate has not run. A tier name arriving before the evidence that would justify it is how a name comes to mean something nobody decided -- and with a v2 already in the list, a v3 reads as a successor to anyone holding the shared key. If it passes, `v3` is a name to give it then, as a decision. brokkr-smithy-dev raised this against my own erp-tune-v3 suggestion and was right. The entry carries the preregistrations ABOVE the description, so a reader meets the commitments before the numbers: T6 one-directional (a gain is uninterpretable against a 3.1x fireball tailwind), T3/T4 at ceiling on base so recovery is UNOBSERVABLE rather than merely unpredicted, and any run-2 comparison descriptive and non-attributable with its five confounds named. Also carries the retraction in-line: "bluemoon is the largest loss contributor at 38.6%" came from a words x 1.4 estimator, not a tokenizer. As encoded it is third at 32.9%. The direction survives (1.4% -> 8.0% of total loss) and that is the finding; the superlative does not. Documents why its config.json is the base's copied verbatim: transformers 5.15.1 save_pretrained silently drops text_config.global_head_dim and num_global_key_value_heads, and vLLM then dies in make_layers with a TypeError naming neither the config nor the field. Cost a failed boot to find. A LoRA merge changes weights, not architecture, so the base config is correct by definition. gemma4-26b-a4b-it-base marked CURRENTLY DOWN rather than deleted -- the tuned arm took GPU0 and only one 26B bf16 seat fits on that card. Kept because the seat returns, and deleting a name to re-add it later is how scoped keys get orphaned. Verified: config parses, no duplicate model_name, gateway healthy after reload, completion returns text in `content` with reasoning_content null. |
||
|
vh
|
b6ce22ddcb |
feat(litellm): register the run-3 gate base arm at operator request
gemma4-26b-a4b-it-base -> ana-ml2:8099, the unmodified upstream instruct release (/tank/aimodels/gemma4-26b-a4b-it-bf16). Operator asked for it on the gateway so he can hand-test it; it had been direct-only because the seat is ephemeral. The entry disambiguates WHICH base explicitly. Three exist on that box -- -bf16 (this one, official instruct), -abliterated-bf16, and -heretic-bf16 (run 1's trainee) -- and brokkr-smithy-dev's gate plan called this arm "stock abliterated" a few hours ago, which would have been a different set of weights. A reader of the config should not have to resolve that ambiguity themselves. Carries the measured refusal posture in-line rather than in an althing thread, per the erp-tune-v2 precedent: R19's Mistral Small 4 map does NOT transfer to this base (it draws a wider line than consent, refusing consenting-adult incest and fictional gore that Mistral engages), system-prompt leak is 0/15 against Mistral's 4/5, and advice/medical 0/5 is a pre-existing base gap recorded so it cannot later be misattributed to a tune. Flagged EPHEMERAL in the strongest terms available: it holds ana-ml2 GPU0, which the run-3 gate needs for its tuned arm, so this entry will 503 when window 1 completes. It is not a promise of availability. Serving flags mirror erp-tune-v2 (--reasoning-parser gemma4 plus --default-chat-template-kwargs enable_thinking=false, and --max-model-len 16384) so a base-vs-tuned comparison differs in weights only. Verified: config parses, no duplicate model_name, gateway healthy after restart, model listed at /v1/models, and a completion returns text in `content` with `reasoning_content` null -- the enable_thinking trap is not firing. |
||
|
vh
|
1e4d827c5d |
memory: erp-tune-v2 registered in the LiteLLM gateway at operator request
Operator asked for it so he can evaluate the failed tune by hand, overriding my not-in-the-gateway recommendation. His call. erp-tune-v1 was DELETED from the config in the same reload rather than repointed, so the name now 400s cleanly instead of 500ing against a stopped backend. Deleting rather than repointing is the point: repointing would resolve a name a consumer already knows to different weights, silently. The config entry carries the failed-gate table, the long-form truncation (9.9%) and degeneracy (4.9%) rates, and the rp-length caveat in-line -- so someone reading the gateway config learns what they are calling without having to find the althing thread. Fleet verified healthy after the restart. |
||
|
vh
|
5415fd4b30 |
docs(gemma4-charrp): abliteration measured in isolation — close to free, but it MOVES capability
Second bench window, operator-authorised after an initial decline and reversal. Stock BF16 against the llmfan46 abliterated BF16: same precision, same pinned upstream template, same 192 items, CoT off. Abliteration was the only axis that moved, which is what the previous run could not claim. Net core cost is 0.6 points — but the headline understates what happened. Capability MOVED rather than degraded: five items lost on contradiction detection, four gained on spatial composition, nearly cancelling. A gain was not predicted by anyone, least of all on that axis. The decision this was authorised to settle: llmfan46 stands as the trainee base. No case for re-staging on TrevorJS at KL 0.09 over 0.6 points — the KL gap between the builds is smaller than the gap this measurement failed to find. Both limits recorded rather than buried, per brokkr-smithy-dev: the swings are ~5 and ~4 items at n=32, so the -15.6/+12.5 percentages read more precisely than the measurement supports and only marginals were run; and this says nothing about quantization, because the stock-NVFP4 T2 figure came from n=16 against n=32 here — different item counts mean different item sets, so that comparison is n-confounded and is not being made. Turnaround was five minutes rather than fifteen because the gemma4-trainee-bench stack already existed — itself the residue of debugging a 35-restart crash-loop caused by the production compose hardcoding --quantization compressed-tensors. The fix outlasted the incident. gen restored and verified through the gateway; char-rp remains down deliberately; bench stack env reset to the heretic base for the post-tune gate. |
||
|
vh
|
019ccff7e8 |
feat(gemma4-trainee-bench): BF16 bench stack; record that gen's footprint grows with uptime
Adds an ephemeral stack for serving the BF16 trainee base on :8016 under the char-rp aliases, so the abliterated base can be measured on the same battery and the same gateway routes as the served seat with no harness edit. It is a separate stack rather than another variable on gemma4-charrp because that compose hardcodes `--quantization compressed-tensors` for the NVFP4 build. Pointing it at unquantized BF16 weights crash-loops immediately — `TypeError: CompressedTensorsConfig.__init__() missing 3 required positional arguments: 'target_scheme_map', 'ignore', 'quant_format'` — vLLM trying to read a quantization config out of a checkpoint that has none. 35 restarts before it was caught. `restart: "no"` here so a bench seat cannot resurrect itself and block gen's restore, and no homepage labels so it leaves no permanently-offline dashboard card. It cannot coexist with gen and says so: 48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before any KV cache. Running it means gen is stopped. THE MORE USEFUL FINDING is in the meromero env note: gen's memory footprint GROWS WITH UPTIME. Measured today at 46,726 MiB (45.6 GiB) after ~3 days up, and 39,424 MiB (38.5 GiB) immediately after a restart — same container, same --gpu-memory-utilization 0.43, ~7 GiB apart. That is the missing half of this afternoon's crash-loop: the char-rp seat "fit on the 21st and stopped fitting on the 24th" because nothing about char-rp changed and gen crept up underneath it. Headroom arithmetic done against a long-running gen is measuring a moving number, so the note now says to measure against a freshly-restarted one. Operator's requested end state reached and verified through the gateway: gen and summarizer both 200, char-rp down deliberately to hold GPU0 headroom for the upcoming trainee run, bench seat stopped. |
||
|
vh
|
14ff4a3f57 |
docs(gemma4-charrp): stage two abliterated trainee bases; record the endemic stale-template trap
The operator directed that the ERP/RP trainee base be a low-damage abliterated instruct build rather than the stock checkpoint. Two are now staged under /tank/aimodels/, both BF16, both unquantized, both matching upstream's 51.61 GB / 25.8B shape with only transformers_version differing in config: gemma4-26b-a4b-it-heretic-bf16 llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100 gemma4-26b-a4b-it-abliterated-bf16 TrevorJS, KL 0.09, 1/100 effective and 5/686 cross-dataset "Low damage" was treated as a measurable claim rather than a description: the field spreads from KL 0.09 to 0.4118 and the table is in the README so the next choice is made on numbers. Fleet anchor for reading them — our own abliteration work found Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance, so both staged builds sit inside an already-validated band rather than past it. huihui-ai is rejected despite its reputation: no published metrics, its own card calls the method a crude proof-of-concept, it abliterates both thinking and non-thinking modes, and its parameter count runs ~738M over upstream. The operator's independent read matched. The more durable finding is the chat template. NOT ONE third-party Gemma-4 derivative pulled here ships upstream's — three independent repos carry the identical stale 266-line file (sha 58c66fdee4afa297), llmfan46 carries a third 365-line variant, and only the RedHatAI NVFP4 build matches upstream's 6a1015c47ccfcfa6. It propagated through the ecosystem rather than one packager slipping, and it is now recorded as a class rather than as the single incident that surfaced it during the A16 control staging. That matters twice over and silently both times: serving a mismatched template renders a different prompt, which is why production pins it; and training through `base/chat_template.jinja` means training on a different prompt format than production serves — train/serve skew with no error, presenting as a tuning failure. brokkr-smithy-dev has been warned on the training side while the harness contract is still early enough to amend. |
||
|
vh
|
8d6a9390de |
docs(gemma4-charrp): RETRACT the contradiction-deficit claim — the item was ill-posed
Supersedes what commit
|
||
|
vh
|
3446367d5e |
feat(gemma4-charrp): pin the chat template; A16 control run executed and reverted
The template is now passed explicitly, defaulting to the A4 build's chat_template.jinja. That is a no-op for what is served — the A4 build ships that exact file, byte-identical to upstream google/gemma-4-26B-A4B-it once trailing newlines are normalised — and it permanently closes the class of bug found while staging the control: the A16 build ships a stale 266-line template against upstream's 390, with the thinking path built differently and no `thinking` property in its tokenizer_config response_schema. Serving each build with its own template would have moved a second axis. The control ran on the operator's greenlight and has been reverted. Seat is back on the W4A4 build, healthy, RestartCount 0, both aliases verified through the gateway — char-rp returns content with reasoning_content empty, char-rp-reasoning returns both. Result, since it settles a question this repo's config now encodes: activation precision does NOT explain the contradiction-detection deficit. Contradiction detection moved 12% -> 19% between W4A4 and W4A16, which at n=16 is 2/16 -> 3/16 — one item — against gen's 81% on identical items. Every other task is identical across the two builds and the core difference is 2.6 points carried almost entirely by two single items. brokkr-smithy-dev pre-registered that a null result would be the robust branch, because a hidden third axis would tend to create a delta rather than suppress one, so the conclusion survives the residual doubt neither side could close without a dequantization pass. The practical upshot for future scheme choices: W4A4 costs less on this workload than the caution warranted. The caution was still correct to have. Displaced production for 3.7 seconds of measurement plus two container recreates. The A16 build and the BF16 tuning base both stay on disk with the runbook in the stack README, so re-running is a two-minute flip. |
||
|
vh
|
1bd90eaacc |
docs(gemma4-charrp): stack README — the three model dirs, the A16 control runbook
The stack had no README and now carries three model directories that look interchangeable and are not: the BF16 QLoRA base that cannot be served here, the W4A4 quant that is served, and the W4A16 build that exists solely as an activation-axis control. Writing down which is which, and why, before someone "simplifies" the compose to the BF16 path and rediscovers the OOM. Also captures the A16 control procedure end to end, including the two confounds found while staging it — the two Hub repos named NVFP4A16 that declare 4-bit activations, and the stale chat template the real one ships — and the fact that overriding the template is safe because the tokenizers are identical. Both sides have now cross-checked this: brokkr independently diffed every non-quantization config field of both builds against the upstream BF16 and found only transformers_version differing. Residual risk recorded rather than hidden: config identity is not weight identity and nobody has done a dequantization pass. The Gemma-4 flags are documented as architecture-level rather than checkpoint-level, since that is why they survived the seat swap unchanged, and the enable_thinking:false pin is called out as mandatory rather than stylistic — without it every plain prose turn lands in reasoning_content with a null content and every consumer breaks. Notes the non-termination defect with thinking on (32 of 96 calls truncating at 12k tokens, all 16 constraint items among them, reasoning sound right up to the point it fails to stop) and why VLLM_USE_V2_MODEL_RUNNER=0 is deliberately not applied to a seat whose production mode is thinking-off. No live change: the seat is still serving the W4A4 build. Displacing it for the control run is an operator decision and is still open. |
||
|
vh
|
24e8826219 |
docs(gemma4-charrp): the A16 control needs a chat-template override, not just a path swap
Pre-flighting the staged A16 build before handing it to brokkr-smithy-dev's
battery found a second axis hiding inside what was supposed to be a
single-variable control.
The A16 build ships a STALE chat template. Verified by hash against the upstream
weights on the same disk: google/gemma-4-26B-A4B-it is 390 lines, the RedHatAI
A4 build's is 389 and byte-identical to upstream once trailing newlines are
normalised, and the prithivMLmods A16 build's is 266 and is not. The delta is
not cosmetic — upstream and A4 open the thinking path with
`{%- set enable_thinking = enable_thinking | default(false) -%}` and branch off
it, while the A16 template has no such set and guards with
`enable_thinking is defined and enable_thinking` instead. tokenizer_config.json
corroborates: A4's response_schema carries a `thinking` property, A16's has only
role and content. That build was quantized from an older revision of the
checkpoint.
Served with its own template, the A16 arm would render a different prompt for
identical messages, and a contradiction-detection delta could be attributed to
activation precision when it was the template. That is the same failure class as
the misnamed-A16 repos — a field nobody validated, believed because the name
looked right — one layer further down, and it would have produced a result that
looked like a finding.
Overriding is safe because the tokenizers agree: vocab identical at 262,144
entries, added_tokens identical, so the same template over the same vocab
renders the same token ids. Everything else pre-flights clean — both artifacts
complete with no missing shards, generation_config.json byte-identical.
Seat NOT flipped; displacing production for the bench window is the operator's
call and is still open.
|
||
|
vh
|
f509668e45 |
docs(gemma4-charrp): record the A16 activation-axis control and how to run it
brokkr-smithy-dev's first battery on the new seat scored 12% on contradiction detection with CoT off, against gen's 81%, while state tracking, deterministic constraint following, long-context state and the confabulation control all sat at 100%. That is not general degradation — it is the shape 4-bit input activations produce on the most reasoning-dense task, which is exactly the confound flagged when the W4A4 quant was chosen. They have recorded the finding as CONFOUNDED rather than reporting it as a property of Gemma, and asked for an A16 build to separate the two. No quant run was needed: a genuine A16 build of the same checkpoint already exists on the Hub and is now at /tank/aimodels/gemma4-26b-a4b-it-nvfp4a16 (prithivMLmods, compressed-tensors, nvfp4-pack-quantized, input_activations null, 17 GB). Same weights, same loader as the live seat, one axis moved. ⚠ Two other repos would have answered the question wrongly and the note in the env template says so: bg-digitalservices and ManniX-ITA both publish "Gemma-4-26B-A4B-it-NVFP4A16" whose config declares input_activations num_bits 4 — W4A4 under an A16 name, via modelopt, both 16.46 GB against the real one's 17.93. Using either and seeing T2 stay at 12% would have wrongly convicted the model. Running it is a one-line GEMMA4_MODEL swap plus a recreate, because there is no room for a concurrent seat: GPU0 has 3.5 GiB free with gen and the A4 seat on it, and GPU1's 19.4 GiB against ~18 GiB of weights is the same sub-GiB headroom that crash-looped the predecessor this morning. Port and both aliases are unchanged either way, so no consumer config moves. |
||
|
vh
|
27155c0f3b |
feat(char-rp): swap the seat to the Gemma-4 26B-A4B MoE, NVFP4, same port
Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every gateway route are unchanged, so no consumer sees a difference in addressing: `char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now include chain-of-thought, which makes throughput more critical rather than less — the user waits through the whole reasoning block before the first visible token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7. Both artifacts are on disk and they are NOT interchangeable. The BF16 weights (/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the engine would die at allocation exactly the way the predecessor did this afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB), chosen over the other -it quants because it is compressed-tensors (nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than -it, and the thinking channel lives in the instruction-tuned weights. Smaller weights at the same 0.47 memory budget buy a much larger KV pool: 27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4. The gemma4 tool-call parser, reasoning parser and the enable_thinking:false default all carry over unchanged — they are architecture-level, not checkpoint-level. The --chat-template override does NOT carry over: MeroMero pointed at a jinja hand-patched against that checkpoint, and this model ships its own. Verified that dropping it did not reintroduce the failure that flag existed to prevent — non-thinking prose lands in content with reasoning_content empty, and the thinking alias populates reasoning_content with content carrying the answer. ⚠ Scheme differs from the incumbent and the bench should say so: this quant declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16. Faster, and not like-for-like on the activation axis. meromero-charrp is retained stopped in `created` state and relabelled to AI - Dormant, per the house rollback pattern. Both stacks want :8016, so rolling back means stopping the gemma4 seat first. |
||
|
vh
|
850e0c3351 |
fix(meromero-charrp): drop GPU0 budget to 0.47 — the seat was OOM crash-looping
`vllm-meromero-rp` had been restarting since 2026-08-24 18:2x, 13 times by the time it was looked at, taking both the `char-rp` and `char-rp-reasoning` gateway aliases down with it (they resolve to the same seat on :8016 — hosted_vllm/char-rp and hosted_vllm/char-rp-thinking). Root cause is CUDA OOM on ana-ml2 GPU0, which the startup logs hide well: the engine gets through weights, torch.compile and CUDA-graph capture looking entirely healthy, then dies at KV-cache allocation with `torch.OutOfMemoryError: ... 195.19 MiB is free`. GPU0 is shared with `vllm-gen`. gen is configured at 0.43 but actually holds ~45.6 GiB of the 94.97 GiB card, because --gpu-memory-utilization sizes the KV cache and does not account for CUDA context, graphs and non-torch overhead. This seat was at 0.51, so the pair was committed to 0.94 of the card with about 0.6 GiB of real headroom. That fit on 08-21 and stopped fitting today. 0.47 restores ~4.8 GiB of margin and costs nothing usable: KV cache 27.36 -> 23.56 GiB, 430,825 -> 371,023 tokens against a max-model-len of 262,144, so the pool still holds 1.4x a full-length sequence. What is lost is concurrent long requests, not context. Verified through the gateway rather than at the container: char-rp returns 200 with content, char-rp-reasoning returns 200 with both content and reasoning_content populated. Seat is healthy with RestartCount 0. The arithmetic and the "check used_memory, not the flag" warning are written into the env template, because the next person to raise either budget needs to lower the other in the same change. |
||
|
vh
|
35adc4a043 |
feat(homepage): rebuild on Australis Skyfall — dual theme, light mode shipped
The board was on the Australis TERMINAL palette, which is dark-only by design
("Always dark first. No light mode in this system"). Skyfall is the dual-theme
web derivative of the same science, and its bundle turned out to be sitting in
this repo's own git history: a predecessor vendored it on 2026-08-19 and a
later commit deleted it. `git show 45c1995:...` returns colors.css with both
`:root` (dark) and `[data-theme="light"]` (Skyfall Day) intact, plus the
calm-depth layout tokens, the typography scale and Supreme 400/500/700. So the
light ramp is canonical rather than derived, which was the entire objection to
building one.
The visual language moves with the palette. Depth is now the recipe and not a
choice — every elevated surface carries a 1px hairline AND a two-layer shadow,
never one without the other. Radii move to Skyfall's scale, cards at
--radius-lg. Widget stat values move from the display face to mono, because
Skyfall is explicit that numbers and telemetry are always --font-mono. The
full-width aurora ribbon under the tab bar is gone: Skyfall sanctions exactly
two accent expressions, the active rail and hero-only glows, and a decorative
gradient across the chrome is neither — so the colour it carried now lands on
the active tab as a 2px accent bar plus an --accent-soft fill, which is the
rail. Every binding is written against the semantic layer; there are no raw
family tokens and no colour literals left in our own file.
build.py now guards the vendoring instead of advising it. The three token files
are hashed and a mismatch FAILS the build — a vendored file is either
byte-identical to the bundle or it is a fork wearing the bundle's name, and the
theme this one replaces had to be torn out twice for exactly that.
⚠ Homepage's own theme toggle is unreachable, and reaching for it breaks the
dashboard. It renders only when settings.yaml leaves `theme:` unpinned, and
with the key absent the page's data loader throws and its catch branch serves
`initialSettings: {}` — no tab bar, no layout, no i18n. Six force-recreates
over seven minutes all came up empty; restoring `theme: dark` rendered
correctly on the next recreate in 12 seconds, while /api/services returned 200
with fully correct content the whole time. That is the first confirmed cause of
the long-running "tab bar goes missing after a recreate" symptom, and it also
retires the homepage.log-size lead recorded earlier today: rolling the log
aside did nothing during this episode, so that coincidence was intermittency.
So the toggle is ours. conf/custom.js renders it and stores the choice;
build.py re-emits each vendored light block twice, once for an explicit
`data-theme` and once inside a prefers-color-scheme media query scoped to
`html:not([data-theme="dark"]):not([data-theme="light"])` — that :not() pair is
what lets a stored dark choice survive a light-mode OS. Verified against both
OS preferences: load, click, click back, reload, all four correct. `data-theme`
is the control surface; Homepage's own `dark` class stays on <html> and does
not fight, because our rules carry !important on the surfaces Tailwind's
`dark:` variants would otherwise claim.
Two font substitutions, both documented rather than silent: Space Grotesk for
Bespoke Sans and JetBrains Mono for Victor Mono. Only Supreme was ever vendored
here and Skyfall's own notes call Victor Mono user-supplied, so this is a
two-line swap when the real faces arrive.
Dark and light, all four tabs: http://10.100.10.50:8090/b/homepage-skyfall/
|
||
|
vh
|
39da1d4a97 |
feat(homepage): recategorise on "do I open this?", collapse the API groups
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
|
||
|
vh
|
f6f2f69649 |
fix(homepage): uniform 4-column grid, hold the status gutter, unleak AI Systems
The board's card width changed at every group boundary because `columns:` was being tuned per group under the 2026-08-18 "columns = member count" rule. That rule is retired: it sets `lg:grid-cols-N` for one group, so it fixes that group's CARD WIDTH, not its density. Measured on the live board, Notes rendered a single 1464px card, News and Media 728px, Eval & Retrieval 286px, everything else 360px. All 20 groups are now `columns: 4` and every card renders at 360. `.service-name` reserved a 78px status gutter with `padding-right` and relied on `overflow: hidden` to hold it, but overflow clips at the PADDING box — the gutter was spill room the title printed straight through. Six cards on the AI tab rendered their name underneath their own status pill, measured by testing the title text node's box against the status cluster's. The intended ellipsis never fired either: it is painted by whichever block's own line overflows, and that is the anonymous box around the bare title text node, which does not carry `overflow`. The gutter holds by wrapping now, and the description opts back out of it with a negative margin since the pill only ever covers the first line. Scriberr's `homepage.group=AI Systems` named a group absent from `layout:`, so it had no `tab:` and rendered on all four tabs — the same defect as UltraSeedbox in 2026-08-18, arriving this time from a container label. Relabelled to `AI - Audio Tools`, where the other ASR seats already live. The `homepage.group=AI Systems` sample in the repo-root CLAUDE.md was the source of the name and now carries the constraint, plus the fact that a labelled container is discovered from any of the five configured engines and must not also be listed in services.yaml. Also: descriptions clamped to three lines so a five-line outlier stops dragging its row 50px taller than its neighbours; icon ramp overridden off slate-400 -> slate-700, which was sinking the bottom half of every glyph into the card fill; bookmark groups and Jellyfin's trailing stream rows brought into the card and eyebrow vocabulary; group gap 10px -> 22px now that width no longer separates them. The icon override sits on `html[class]`, not `:root`: Homepage sets the same variables on `.theme-slate`, which is on the <html> element, and a class beats `:root` on the same element. Verified with Playwright against the live board — per-group card width, card height spread, and a geometric title-vs-status collision check, before and after. Before/after captures: http://10.100.10.50:8090/b/homepage-relayout/ |
||
|
vh
|
6217d3993e |
fix(scriberr): force uv to copy rather than reflink when building envs
Scriberr builds each model backend's Python environment with uv at container start. uv's default link mode reflinks or hardlinks out of its cache, which fails on this overlayfs over ZFS combination and reports it as a failure to clone a metadata file with errno 11, resource temporarily unavailable. The wording points nowhere near the cause. The damage was partial and therefore easy to miss: WhisperX and PyAnnote came up fine and the application served normally, while the Parakeet and Sortformer backends were quietly absent. Setting UV_LINK_MODE to copy trades a little disk and build time for environments that actually materialise, and the occurrence count for that error is zero on the following start. The knob is exposed through the environment so it can be moved back to the default if a future host does support reflinks. |
||
|
vh
|
efddb4e511 |
feat(scriberr): stand up transcription on ana-ml2, pinned to GPU1
Scriberr transcribes audio and video locally with WhisperX and speaker diarization, and it lands on ana-ml2 rather than ana-docker because the work is GPU-shaped: ana-docker offers eight cores already shared with fifty containers and thirty-seven gigabytes of disk, against ninety-six cores, terabytes on /tank and idle capacity on GPU1. The reservation names device 1 explicitly, since GPU0 is fully committed to the gen seat, and the container is confirmed to see that card alone. The image is built from source, which is not a preference. These are Blackwell cards at sm_120; the published CUDA image covers Pascal through Ada only, and the blackwell image the upstream README documents has never been published at all. The path upstream actually ships for sm_120 is Dockerfile.cuda.12.9, carrying CUDA 12.9 and cu128 torch, so that is what gets built. The compose header says so, because the obvious cleanup is to swap in the published image and that would silently drop the deployment to CPU. Two configuration details are load-bearing and documented where someone would go to change them. The application runs as uid 10001 rather than the usual 1000: that Dockerfile moves its user aside for Ubuntu 24.04's own uid-1000 account and chowns /app accordingly, while the entrypoint's remapping covers only the data directories, so at 1000 the process cannot open its database and restarts forever behind a SQLite error that reads as though the machine were out of memory. Secure cookies stay off while the service is reached over plain HTTP, or sessions are dropped by the browser and login appears to loop for no visible reason. Storage is bind-mounted onto /tank because model weights run to several gigabytes and the root pool on that host is nearly full. Also adds the scriberr service alias to internal DNS, following the existing alias convention so consumers name the service rather than the box. |
||
|
vh
|
b8a535507a |
feat(judge-bench): keep the judge harness; warn about the 7-way alias collision
Operator: keep the benchmark. It has a named second use (brokkr-smithy-dev wants gen vs a trained reward model once their tournament converges) and a demonstrated first one -- it caught a seat that had been coin-flip-grade for five weeks with nobody measuring it. Harness promoted from scratch to tools/judge-bench/: - paths de-hardcoded; runs from its own directory - proper CLI: --models (REQUIRED), --repeats, --limit, --gateway. Required on purpose: a stale default would silently benchmark a retired seat, and the original default (selene-1-mini-8b) now 400s. - README states the limitation rather than burying it: 24 items of the author's own design, a screen and not a verdict. This harness scored the same pair 83 vs 96 while brokkr's corpus ranking task scored it 47 (chance) vs 94. Both honest; absolute scoring on designed items is an easier task than ranking real text. - records brokkr's technique, which is better than anything here: a control constructed so the correct answer is DEFINITIONAL rather than judged cannot inherit the designer's error (item vs itself, response vs its own truncation, text vs its own clauses permuted). Add those before adding more judged items. Gateway: comment-only warning at the head of model_list. SEVEN aliases now resolve to the same weights (chat-judge, classifier, gen, image-judge, qwen-image-bench, summarizer, summarizer-large -> qwen3.8-27b-uncensored). That is intended under ADR-0012, but it has a sharp edge brokkr flagged: cross-checking a result against another alias measures NOTHING when they are the same model -- agreement is an echo, not corroboration. The note names the other current collisions (glm-5.2 x4, TTS x4, reranker x2), gives the /model/info one-liner to check, and records that probes should resolve alias -> backing at run start AND end because the response `model` field returns the alias, so a swap is otherwise invisible. Verified: config still parses, diff is comment-only, canonical re-synced. |
||
|
vh
|
ca3c984f93 |
feat(selene): retire the seat; chat-judge -> gen, selene-1-mini-8b 404s by design
Benchmarked selene against gen on selene's own job: 24 designed judge items with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a neutral JSON prompt and Selene's native Atla template. 288 calls, all free local. neutral JSON selene 20/24 (83%) gen 23/24 (96%) native Atla selene 21/24 (88%) gen 22/24 (92%) gen won on both templates and selene's BEST sat below gen's WORST. Selene was given its own fine-tuned template as a fairness check; it gained one point, not the three it needed. Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a winner on every equivalent pair. For eval work that is the case that matters most. gen returned tie correctly on the JSON template. Selene also compressed the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only win was ~3x latency, unexercised at ~60 calls/day with zero queueing. TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately: - chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says consumers bind the capability, not a concrete model. Sampler profile copied from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served config matches the benchmarked condition. - selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A served-name is a contract about what the model IS; answering it with a different model hides a material change behind a stable string. Operator ruling: "never repoint a named model at a different model's endpoint -- that is intentionally misleading." Verified: the gateway now returns HTTP 400 "Invalid model name" for it. Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role moved onto an existing seat rather than allocating anything new. Canonical litellm config synced from the host; ana-ml2 README and recommended-model-settings updated. compose.yaml kept for reference, not deployed. |
||
|
vh
|
9642952a54 |
docs(hrafn-ci): mirror the now-canonical vh/hrafn pipeline; record two failed runs
claude-bot holds write on vh/hrafn as of 2026-08-23, so the canonical copy of the pipeline moved there and infra-ops maintains it directly instead of routing patches through the repo holder. The files here are a verified mirror (byte-identical to live at 22e0eb9d75a6). Live state: run 9922 green, and both new content assertions executed rather than merely existing -- verify 4/5 host tree matches shipped context (a99ce748a0c9...) verify 5/5 image source matches host source (06f209fd0641...) The CI-computed context hash matching on the host is the end-to-end proof that the converge lands what CI ships. Its absence is what let the frozen-source bug survive every green deploy. Also records why the HEAD == GITHUB_SHA assertion was added and then removed: it needed the git binary (run 9920, exit 127), and installing git flipped actions/checkout@v4 from its node implementation to the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout code path and broke a working pipeline; it guarded a hypothesis that proved wrong, so it went rather than getting ca-certificates bolted on. |
||
|
vh
|
b38c369313 |
fix(hrafn-ci): staging dir inside the rsync target froze host source silently
Root cause of nevermore-claude's report that v1.0.0 deployed green while the host kept serving 0.1.0. The staging dir was $compose_dir/.stage -- INSIDE the rsync target. So `rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted .stage from the destination (absent from the source listing) DURING the transfer, destroying the source mid-copy. Reproduced exactly: before: app.py="OLD" leftover.txt .stage/app.py="NEW" after: app.py="OLD" leftover.txt GONE, .stage GONE Deletion worked; the copy silently did not. So the directory looked converged while host source stayed frozen at the first manual rsync, and because the build's COPY inputs never changed, Docker full-cache-hit and every SHA tag aliased one image. The provenance guarantee was false. Nothing caught it because the verify steps asserted the marker, health, and a 200 from /readyz -- all of which pass on a frozen host. None measured content. Fixes: - stage at /tmp/hrafn-deploy-stage, outside the target - CI computes context_sha256 over the shipped file list; the playbook recomputes it on the host post-converge and fails on mismatch - compare the running container's src/**/*.py against the host's, catching a SHA tag naming layers the image does not contain - checkout clean:true + assert HEAD == GITHUB_SHA so a reused runner workspace fails the job rather than shipping a stale tree Declined --no-cache: a cache hit is correct when the context is genuinely unchanged, and the new assertions prove the property directly rather than brute-forcing it. The container-vs-host check compares only *.py -- `pip install .` generates src/hrafn.egg-info/* inside the image and __pycache__ appears at runtime, so a naive `find src -type f` compare false-fails on every healthy deploy. Verified against the live container before shipping (12 host files, 18 in container, 0 content differences). |
||
|
vh
|
11b9d1891e |
fix(hrafn-ci): make the deploy converge instead of accrete
The first CI run shipped clean but revealed a design gap in the playbook: unpacking the context tarball in place overwrites tracked files and never removes anything. Leftovers from the pre-CI hand-rsync (tests/, docs/, ROADMAP.md, persistent-memory.md, CLAUDE.md, LICENSE) survived the deploy and had to be cleaned off ana-docker by hand. That is the same failure class that produced the mess in the first place: a deploy that only ever adds cannot return the host to a known state. - unpack to a staging dir, then rsync --delete onto the compose dir - protect host-owned .env and .deployed by name - add .env.example to the context tarball so converge does not delete it - record in the workflow that the tar list is now AUTHORITATIVE: anything omitted is removed from the host on the next deploy Re-validated with `elway --dry-run` (9 steps, 3 verify, parses clean). Not yet in vh/hrafn -- infra-ops has no write access there, so this is offered to the repo holder rather than pushed. |
||
|
vh
|
b001d0cb2e |
feat(hrafn): author the CI deploy that replaces the hand-rsync path
hrafn was handed to infra-ops for uptime ownership with no CI deploy and no commit provenance -- the image was always local/hrafn:v1 and the whole working tree lived in the compose directory. These two files fix both. They are authored here because infra-ops owns hrafn's uptime, but they belong in vh/hrafn; claude-bot is not a collaborator there, so they are handed to the repo holder rather than committed directly. - playbooks-deploy.yaml -> vh/hrafn playbooks/deploy.yaml - gitea-workflows-deploy.yaml -> vh/hrafn .gitea/workflows/deploy.yaml Design calls recorded in the README: the build context travels as one tarball rather than per-file upload steps (nevermore's pattern fails open when a new source file has no matching step), and the playbook refuses to deploy unless .env exists at 0600 -- a guard prompted by it arriving 0644 with a live bearer token in it. Validated with `elway --dry-run`, which caught a real interpolation bug during authoring. No new Actions secrets needed. |
||
|
vh
|
b6924de728 |
feat(hrafn): register the browser-fetch service and tighten its env perms
nevermore-claude handed hrafn to infra-ops for uptime ownership. Intake: verified health independently, brought it into the inventory, and fixed one security defect found during the check. - canonicalize stacks/hrafn/ (compose.yaml + .env.example + README) - list hrafn in the ana-docker running-stacks table - README records the load-bearing bits: shm_size 1gb is required for Chromium, playwright and the base image are version-coupled, SSRF denies private targets by default, and restart:unless-stopped does NOT act on healthcheck failure Security fix applied on the host, not in this tree: /opt/docker/compose/ hrafn/.env was 0644 with a live 57-char bearer token in it, readable by every local account (verified by reading it as `nobody`). Tightened to 0600; owner unchanged so the deploy path still works, confirmed via `docker compose config` as lkraven. Container untouched and still healthy. Known gaps recorded rather than silently accepted: no CI deploy, and the image builds from an rsync'd working tree living in the compose directory, so the running image has no commit provenance. |
||
|
vh
|
7bf17dd39e |
fix(filezilla): set restart: unless-stopped and canonicalize the stack
The filezilla container on vm-esh-nas shipped with no restart policy. It was SIGTERMed during the 2026-08-18 host shutdown and never came back, staying down four days while dockge/dozzle-agent/beszel-agent all recovered on their own. - add `restart: unless-stopped` (applied on the host, container recreated and verified serving 200 on :5800 with all four /NAS mounts visible) - promote the previously un-canonicalized compose into stacks/filezilla/ so the policy is version-controlled rather than host-only - record the missing-restart-policy failure mode in the host README |
||
|
vh
|
6e82899ba7 |
fix(mog-sec): retune to 0.52/420K after a runtime OOM at 0.55/480K
The 0.55 configuration booted cleanly, passed every startup check, and then killed the engine on an ordinary request of roughly 20k tokens: torch.OutOfMemoryError allocating 614 MiB in the GDN causal_conv1d path with 443 MiB free. The container self-recovered via restart: unless-stopped. Context length was not the cause. gpu-memory-utilization is not a hard reservation. Weights and KV are reserved up front, but activation memory is allocated on demand and the profile that sizes it uses dummy data at max_num_batched_tokens. The profile predicted 3.62 GiB; the process sat about 1 GiB over its entire budget even at idle. With six vLLM containers sharing GPU1 there was no slack left to grow into, and a modest allocation lost the race. 0.52 leaves roughly 7.7 GiB free against 4.6 GiB at 0.55. Records a measurement that governs how max-model-len can be chosen at all: the KV pool varies between boots. Two consecutive boots at 0.52 produced 425,663 and 453,600 tokens, a 6.6 percent swing, because the pool is sized from free memory at startup and that depends on what the other tenants hold at that instant. 420,000 sits under the lower observation so the seat starts in both conditions. Sizing to the higher one would boot today and refuse tomorrow, which is why the value was measured across boots rather than estimated from one. Concurrency of about 1.0 to 1.08x means one request at full length. Shorter requests still batch normally, since vLLM schedules by tokens rather than by max-model-len. |
||
|
vh
|
8389470898 |
feat(mog-sec): promote the DFlash2 configuration into the compose stack
Operator approved after real-use testing. The experimental standalone container is retired and stacks/mog-sec is canonical again, with restart: unless-stopped so the configuration survives a reboot. Cutover verified against the container it replaces: KV pool 526,617 tokens at 1.10x concurrency, identical; zero restarts; both gateway aliases serving; DFlash2 confirmed drafting at k=7 with 231 draft tokens over 33 drafts; vision working at 2048x2048. One variable was deliberately dropped rather than carried over. The previous stack hardcoded PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, the validated container never set it, and the quant playbook records expandable_segments corrupting retained tensors in another context. The compose now defaults it empty via MOG_ALLOC_CONF. Promoting the stack as it stood would have shipped a variable the tested configuration did not have. The speculative config moves into a single MOG_SPEC_CONFIG carrying the whole JSON, because the two shapes are not interchangeable: dflash requires a model pointing at the drafter and MTP must not have one, so a method-plus-tokens template cannot express both. Also parameterised: MOG_DRAFT_MODEL, MOG_MM_PROCESSOR_KWARGS, MOG_MAX_NUM_BATCHED_TOKENS. The mm-processor image cap is now mandatory rather than incidental. The model's own preprocessor declares 4096x4096, which expands to 16384 image tokens and kills startup on builds that enforce the image-token count check. Adds the .env.example this stack never had, carrying the measured rationale for each value and the one-line rollback. |
||
|
vh
|
aa5863c9a3 |
feat(speaches): OpenAI-compatible faster-whisper ASR seat on irv-ml1 A6000
Deployed for Eyra (meeting recorder) per the eyra-dev request. Serves
large-v3 (batch tier) + distil-large-v3 (low-latency tier) on :8204,
fp16, both resident, ~5.9 GB VRAM against 20 GB still free.
Sits alongside the existing parakeet stack (:8765) deliberately: parakeet
is a TDT/transducer returning bare {"text": ...} and has no no_speech_prob
concept, so it structurally cannot serve this consumer.
The load-bearing requirement -- segments[].no_speech_prob surviving
response_format=verbose_json -- is VERIFIED on both tiers.
Measured finding worth more than the deployment: no_speech_prob alone is a
WEAK hallucination gate on this stack. Pure silence and pink room tone both
produced the classic Whisper 'Thank you.' hallucination while no_speech_prob
stayed under 0.11 -- a conventional >0.6 threshold passes both through.
avg_logprob separates the same cases ~6x more decisively (-0.11 speech vs
-0.65/-0.72 non-speech) and compression_ratio splits 1.141 vs 0.556.
Consumers should gate on a composite, not no_speech_prob alone. Table in
the README.
VAD pinned OFF at the consumer's request (they VAD-gate upstream on the
capture edge). Consequence stated plainly in the README: with VAD off this
service will transcribe silence into text and is not defending itself.
Image pinned BY DIGEST rather than :latest-cuda, because the VAD-off
setting rides on _UNSTABLE_VAD_FILTER -- a variable upstream explicitly
marks unstable. A floating tag could rename it on any bump, silently
restoring VAD and moving no_speech_prob semantics under a calibrated gate
with no error and no log line.
Two deployment gotchas recorded: PRELOAD_MODELS only loads models already
cached (it does not download -- use POST /v1/models/{id}), and the bind-
mounted cache needs a hub/ subdir or every /v1/models call 500s with
CacheNotFound while /health still returns 200.
|
||
|
vh
|
5e47a59b32 |
feat(meromero): MeroMero-v2 dual-mode (prose + streaming CoT) live on one seat — no re-quant
The multi-turn Gemma-4 CoT problem is solved. One MeroMero-v2 seat, one weight set,
two aliases: char-rp (prose) + char-rp-reasoning (streaming chain-of-thought).
The winning stack, traced from vLLM source by the four-arm brokkr/dwarf panel:
- vllm/vllm-openai:v0.26.0 — ships transformers 5.14.1 natively, below the
head_dim guard, so Gemma-4-31B loads with no pin and no custom image. It also
carries the #48217 streaming pre-arm fix.
- A patched chat template whose enable_thinking:true branch force-opens a BARE
<|channel> (not <|channel>thought\n -- full-open defeats _preprocess_feed's
injection). --chat-template override, no re-quant.
- Two served-names char-rp / char-rp-thinking; --reasoning-parser gemma4;
default enable_thinking:false. LiteLLM char-rp -> prose, char-rp-reasoning ->
the thinking served-name with enable_thinking:true.
Verified: streaming CoT split 6/6 direct on :8016 and 3/3 through the gateway;
char-rp prose clean on both transports with no trailing-token leak.
Two hard-won facts recorded in persistent-memory:
- STREAMING ONLY. Non-streaming can't split -- extract_reasoning never receives
prompt_token_ids so the pre-arm can't fire (a vLLM one-shot bug unchanged
across v0.24-0.27). Fine here: Lobe/OWUI stream. Upstream PR #49797 fixes
non-streaming too, landing ~v0.28.0 -- then it's a clean image bump.
- KEY-NAME TRAP: vLLM streams reasoning in delta.reasoning; LiteLLM normalizes
to delta.reasoning_content. I lost two false-negative test rounds to this.
Canonical: stacks/meromero-charrp/ (compose + patched_chat_template.jinja) and
stacks/litellm/conf/config.yaml. Rollback is the .env image line + dropping
--chat-template.
|
||
|
vh
|
76834777a4 |
feat(open-webui): deploy env-declarative chat bake-off on esh-docker-vm:3211
Stand up Open WebUI v0.11.0 parallel to lobe-chat (:3210) as an operator-approved candidate replacement. Env-declarative config (ENABLE_PERSISTENT_CONFIG=False = the deploy is the config source of truth), model picker auto-tracks the LiteLLM gateway with no pins, background task model pinned to summarizer, TTS routed direct at the dots gateway (:8198). Gates verified on the box: G1 (declarative config, both directions), G2 (picker == live gateway roster, no hand-listing), G3 (models/sync genuinely reconciles create+delete), G5 (task model pinned). G4 (TTS) handed to tts-dev. Credential posture (G6): fresh capped LiteLLM key open-webui-esh (all-proxy-models, $50/1mo budget) rather than inheriting the uncapped lobe-chat-esh key; signup locked off after admin creation. Secrets vaulted under esh-docker-vm/open-webui-*. Note: the API-key toggle env var is ENABLE_API_KEYS (plural); a var only reaches the container if compose names it in environment:. lobe-chat is untouched. |
||
|
vh
|
3d30a6530b |
feat(lobe-chat): pass every chat-capable model through the picker
Operator: "pass everything through, lobe is a test surface for me." The picker goes from 11 curated seats to all 25 chat-capable models the gateway serves, paid passthroughs included. Membership is derived from LiteLLM's declared `mode` rather than by taste: chat (15) and completion (1) are in, and the nine GLM/frontier entries whose mode is unset are chat models in practice. Out: audio_speech (ext-tts, tts-1, tts-1-hd, gpt-4o-mini-tts), embedding (qwen3-embedding) and rerank (reranker, reranker-a3-bge-v2-m3) — seven models that cannot answer a chat request at all, so listing them would recreate the visible-but-dead entry this list just got fixed for. Every candidate was probed live from inside the container before being pinned, which corrected an earlier claim in this file. char-rp and char-rp-reasoning are NOT retired: they are configured and their upstream seat (ana-ml2:8016) is refusing connections while under maintenance, which is a 500, not a 400. Both are retained deliberately — dropping a name because its seat is bouncing makes the list rot in the other direction. Only char-rp-fable is genuinely gone (400, absent from model_info). The comment now spells out the 400-vs-500 distinction so the next editor does not repeat the mistake. Also corrects the credential-posture header, which still described the key as scoped to free local models and cited the glm/kimi rejection as current verification. The key now carries the all-proxy-models access group with no budget ceiling, so that block had become a false security claim in the most load-bearing comment in the file. |
||
|
vh
|
303fb7a5aa |
feat(lobe-chat): add the sec seats to the picker, drop two retired ones
Two independent gates kept the new `sec` family out of Lobe, and only one of
them was visible from the symptom.
The picker never auto-discovers. `OPENAI_MODEL_LIST=-all,+<names>` clears
Lobe's built-in OpenAI catalogue and re-adds one model per `+name`, so anything
added to LiteLLM stays invisible until this list is edited and the container
bounced. That pin is deliberate — an unpinned picker offers models that fail on
click — but it means the list rots in both directions, and it had:
- `sec` / `sec-reasoning` missing (hosted_vllm/mog-sec-27b{,-thinking} on
ana-ml2:8019, added to config.yaml earlier today), and
- `char-rp-reasoning` / `char-rp-fable` still listed after being retired
upstream, i.e. two picker entries that 400 on click. Verified: a call to
char-rp-fable now returns 400 Bad Request.
The list is now curated to live, chat-capable, free-local seats — eleven, each
round-tripped through the container after the bounce. The paid family stays out
deliberately; that is now a picker decision rather than a key one.
Which is the other half of this commit: the `lobe-chat-esh` key is no longer
scoped to free local models. On the operator's instruction infra-ops swapped its
explicit array for the `all-proxy-models` access group, so it now reaches the
paid passthroughs with `max_budget: None`. The README documented the old posture
as current, which made it a security claim that was no longer true; it now
carries the change, what it costs, and the fact that the picker is the only
remaining gate.
|
||
|
vh
|
564f5ae4f6 |
chore(litellm): rename mog-sec aliases to sec / sec-reasoning
Operator's call: the pen-test seat is reached as `sec` (non-thinking) and `sec-reasoning` (thinking). Backend and served-model-names (mog-sec-27b[-thinking] on :8019) are unchanged; only the gateway alias name changes. Old mog-sec / mog-sec-reasoning now 404. |
||
|
vh
|
36c173c6a1 |
feat(mog-sec): quant + serve M.O.G.-SEC pen-test seat; PPL on gen; retire fable
Autonomous overnight run under the operator's full-autonomy grant. End state:
fleet up, gen seat untouched, a new verified pen-test seat serving where fable was.
PPL on the orcarouter gen seat (fable downed to free GPU1 for a nospec probe,
probe torn down after): mean 7.07 / median 5.76, within noise of heresy 6.910 /
5.625 and identical to our recipe's usual 7.059. The gen-seat search is settled.
M.O.G.-SEC: chose Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (rev deede677)
over the pre-made ModelOpt NVFP4, which was disqualified on W4A4 4-bit activations
(the AEON degradation mode, catastrophic on a 1M-context model), zero MTP tensors,
and ModelOpt format. Pulled, format-screened (P(<think>) 1.11e-05, clean), quanted
in-house to mixed NVFP4+FP8 (23.4 GB, MTP + vision preserved), and served in the
retired fable slot.
stacks/mog-sec ana-ml2 GPU1 :8019, KV 418,218 tok / 1.60x @ 262K
aliases mog-sec (non-thinking), mog-sec-reasoning (thinking)
gates surface 6/6, MTP 55.3%, format 0/15 leak, vision 7/3/1,
capability 4/4 (delivers offensive-security content)
Served at native 262K, NOT the card's 1M -- the 1M needs YaRN (absent from the
weights' config) plus the SGLang/DFlash2 path the repo ships a deployment kit for,
neither of which is our vLLM surface. A real 1M seat is a separate SGLang project.
Retired char-rp-reasoning + char-rp-fable (zero traffic, pointed at the downed
fable :8019; now 404 cleanly, not repointed -- a security model is not an RP model).
char-rp (meromero) untouched. Vision preprocessor built from the model's own
image_processor block, same trick as the MeroMero seat.
GPU0 seats (gen, meromero) were untouched and healthy throughout. The quant ran in
GPU1 free space with no production seat stopped except fable, which was replaced.
|