ha-dev found the WS API and tested the read half; this runs the write. But the
command they identified is the wrong half, which is worth recording because the
naming actively misleads.
remote_build/set_offloader_settings {remote_builds_enabled: false}
the OUTBOUND half — this dashboard sending builds to peers.
Persists, reads back false, and leaves the receiver listening.
remote_build/set_settings {enabled: false}
the receiver-side master switch, per ReceiverController.set_settings's
own docstring. Tears the listener down live, no restart needed.
Set both. Verified across a restart: 6055 absent, zero peer-link bind lines,
zero mDNS advertisements, both switches read back false. Persisted at
_remote_build.enabled in /config/.device-builder.json — which did not exist
until the flag was first changed, so 'no on-disk representation' was true only
of the default state.
ESPHOME_REMOTE_BUILD_HOST=127.0.0.1 is KEPT as a backstop rather than removed.
The off state now lives in one JSON file whose in-code default is enabled:True
(controllers/remote_build/_state.py) and whose module's stores soft-recover to
an empty model on a malformed blob rather than erroring — so a lost or corrupt
settings file silently re-enables remote-build. With the env var set, that
regression binds loopback instead of 0.0.0.0.
Also finishes deploy-stack.sh properly. This was patched three times in one
session because -a is -rlptgoD and a non-root identity cannot apply owner,
group, permissions OR times to a root-owned directory; each patch fixed one
letter and the next deploy failed on the next one, every time exiting 23 AFTER
a successful transfer. The rule is now written into the script: the deploy
syncs content, the conventions own metadata. --no-o --no-g --no-perms
--omit-dir-times. Verified: clean run, destination keeps 2775 root:docker with
setgid intact.
ha-dev asked for the Device Builder 1.0.0 remote-build receiver to be turned
off: one instance, builds run locally, so the feature has no role, and it was
binding 0.0.0.0:6055 with mDNS advertisement on a privileged host-network
container that writes firmware to devices.
Reading the source first changed the framing. controllers/remote_build/
_state.py declares 'remote_builds_enabled: bool = True', so nobody enabled it
— it arrived on by default with the rewrite. And the flag has no on-disk
representation until it is changed: neither .device-builder.json nor
.device-builder-preferences.json carries it, and the only writer is the app's
own command API behind the UI. Setting it from a playbook would mean inventing
a schema for a model I have not read.
So this binds ESPHOME_REMOTE_BUILD_HOST=127.0.0.1 — a documented env var, no
entrypoint override — which removes the LAN reachability now and is verifiable
(ss reports 127.0.0.1:6055, was 0.0.0.0:6055). It is explicitly NOT the off
switch ha-dev asked for and the compose comment says so; the Settings toggle
is one UI click and the line can go once someone flips it.
Also completes yesterday's deploy-stack.sh fix, which was half a fix. --no-o
--no-g stopped rsync chgrp-ing a root:docker destination as a non-root
identity, but the very next deploy failed the same way one layer along —
'failed to set times on ...' — because a non-root identity cannot utime() a
root-owned directory either. Same exit 23 after a successful transfer. Added
--omit-dir-times. Fixing only the group half looked fixed until the next run,
which is the whole reason this is worth a line in the script's comment.
ha-dev requested all three on esh-docker-vm (operator-authorized); the stack
had no canonical copy, so it is added to stacks/ rather than edited in place.
Pinned ghcr.io/esphome/esphome:2026.8.2 — it was bare, which is exactly how it
sat on 2025.8.2 for a year: docker pulled latest once at container creation
(2026-04-20, from a layer cached 2025-08-29) and never re-pulled. Every current
Everything Presence sensor failed config validation on that build. Verified
after: esphome version reports 2026.8.2 and the vendor's own Pro package now
validates clean (exit 0, 'Configuration is valid!'), which is the item that
unblocks the six waiting sensors.
Relocated /path/to/esphome/config (the upstream template placeholder, taken
literally by docker) to /opt/docker/conf/esphome, matching the mosquitto
pattern. Copied and checksum-verified all 5763 files before removing the
original, with a tarball kept at /root/pre-change-archive/.
Credentials moved off test/ChangeMe to the vaulted 32-char secret
(esh-docker-vm/esphome-dashboard), passed via a host-only .env so nothing
plaintext enters git.
Three things the job surfaced that were not in the request:
The directory is 538 MB, not the 3 KB reported — .esphome/platformio is
508 MB of PlatformIO toolchain and .esphome/build another 31 MB, both
regenerable. Relocating as-asked would have inflated restic's /opt/docker
source ~45x against its own ~12 MB budget, so both subtrees are excluded in
/etc/restic/profiles.yaml. The 3 KB of actual config is now covered, which
was the point.
2026.8.2 logs a DEPRECATION for the bare USERNAME/PASSWORD env names and says
they will stop working in a future release — a silent auth loss on some later
bump, on a privileged host-network container that can flash any ESP device on
the LAN. Switched to ESPHOME_USERNAME/ESPHOME_PASSWORD; the warning is gone.
Device Builder 1.0.0 opens a NEW listener on 0.0.0.0:6055 (remote-build
peer-link) that 2025.8.2 did not have.
Also fixes deploy-stack.sh: plain 'rsync -a' makes rsync chgrp the destination
as the deploy identity, which since the 2026-09-14 root:docker normalisation is
not root. It failed with 'Operation not permitted' and exit 23 AFTER
transferring content — a loud error on a deploy that had succeeded. --no-o
--no-g lets the setgid bit assign the group instead.
Every fv-ml1 link on the Homepage dashboard was broken. Measured against the
live dashboard API before the fix: 16 entries pointing at the dead 10.250.50.54
and zero at the live 10.251.50.54, covering gen, M.O.G.-SEC, Scriberr, Embed,
Rerank, Reward, Coder, Dockge and six dormant seats.
The miss was structural, not careless. fv-ml1-rename-sweep.sh works from an
allowlist assembled from files that mention the HOST, and a homepage.href label
mentions only an IP -- so every stack whose sole stale reference was a label
fell outside it. The allowlist now covers those 24 files, and records how to
derive the list next time (grep the old address, subtract history) rather than
enumerating from memory.
History is still untouched, and the exclusions are now written down with the
reason each one keeps the old address: recorded benchmark results, whose
base_url is part of a measurement's provenance; the one LiteLLM comment
preserving a retired hand-test endpoint; and the cutover runbooks, where the old
address is the subject matter.
Two bugs found while applying it, both fixed here:
- deploy-stack.sh rejected any stack name containing a dot, so qwen3.5-122b,
qwopus3.5-122b and mistral-medium-3.5 could not be deployed by the script at
all. The check exists to stop path traversal, which means rejecting ".." and
"/" -- not every dot. Traversal is now rejected explicitly and tested.
- stacks/scriberr/.env.example allowed CORS only from the dead IP and from
scriberr.ana.internal, which no longer resolves; the box is at the fv site
and DNS already carries scriberr.fv.internal. The live .env had both stale
origins, i.e. an allowlist with nothing reachable in it.
Host side, applied separately: canonical pushed for the 16 stacks whose only
difference from the host was this renumber, and an in-place address-only fix for
the nine whose host copy has genuinely drifted or has no canonical copy, so that
drift survives for a deliberate reconciliation instead of being clobbered. Every
compose.yaml on fv-ml1 now reads 10.251.50.54. The labels themselves only take
effect at container creation, so the running containers still need recreating.
Without -n, ssh inherits the surrounding loop's stdin and consumes
the heredoc that feeds $changed / $deleted, silently truncating the
diff output to the first file only.
Decision recorded in CLAUDE.md ("Stack tree convention") and memory
(convention_stacks_vs_mirror.md):
stacks/<stack>/ canonical / intent. git-tracked.
deploy-stack.sh reads from here.
stacks-mirror/<host>/<stack>/ snapshot / reality. gitignored.
sync-stacks.sh writes here. Used
for drift inspection only — never
a deploy source.
Bug this fixes: deploy-stack.sh was reading from the mirror, so edits
to stacks/llama-swap/config.yaml never reached ana-ml2. Today's
two new model entries (qwen3.6-35-a3b-heretic + qwen3.6-27b) lived
in the canonical for hours but the deploy reported "in sync" because
the script only diffed mirror vs server.
Changes:
* deploy-stack.sh: source switched from MIRROR_DIR/$HOST/$STACK to
STACKS_DIR/$STACK. Header comment + error message updated.
* sync-stacks.sh: header explicitly identifies its role as drift
detection; documents the diff command for comparing canonical vs
mirror.
* stacks/llama-swap/{config.yaml → conf/config.yaml}: matches the
deploy mapping (conf/ in canonical → /opt/docker/conf/ on host).
* CLAUDE.md: "Stack mirror (pull / push)" section rewritten as
"Stack tree convention (canonical vs mirror)" with the role table
+ workflow rules + diff recipe. Layout diagram updated.
pfi-pve (Anaheim), nh3-pve (NH), esh-pve + esh-pve-nas (ESH home lab) now
have servers/<host>/ entries with ssh-target files, README stubs, and
first snapshots. Register all four via add-host.sh + ssh-copy-id; SSH
auth is root@<name> with key-based access.
Fix a resolver bug in the three workspace scripts: previously, when a
hostname resolved via /etc/hosts, resolve_target returned the bare name
and ssh dropped the user prefix stored in ssh-target (connecting as the
wrong user). Now ssh-target wins whenever it's present, since it can
carry user@ or port info that /etc/hosts + ssh_config cannot. Falls back
to the dir name only when no ssh-target file exists.
CLAUDE.md and README.md split the fleet table into Docker hosts vs
hypervisors with an explicit placement rule: don't deploy Docker stacks
on PVE hosts; capture host-level detail with server_inspect.sh, use
Proxmox-native tooling (qm list, pvesh, zpool) for VM/storage audits.
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
vh