vh
838132cd6b
Session captured: the FV outbound-NAT root cause and its diagnostic signature, the fv-ml1 dead man's switch, fleet identity/group/path conventions and the root:docker normalization, nh3-dev's ts-input reachability fix, ESPHome modernisation and the kb KB-search tool, and the Hermes bearer rotation release. Six new detail files. Tried-and-abandoned gains three: probing OPNsense endpoints by POSTing at them (which rebooted the FV firewall), advertising a /32 from nh3-dev, and the nh3-scale remote-site masquerade rules that fired but were not the fix. Housekeeping: 8 Recent-decisions entries archived to archival-memory.md, and 21 oversized inline entries split into detail files per the two-tier rule -- they had been sitting fully inline in the index, which is what the split exists to prevent. Two pointers to a detail file archived this run were repointed at archival-memory.md. The index is 389 lines, still over the ~300 soft cap. The archival guards stop it there: only 4 further entries are old enough to move and every one carries an open deferred-work pointer. An over-cap file that keeps live decisions beats a scannable one that lost a deferred call.
4 lines
1.4 KiB
Markdown
4 lines
1.4 KiB
Markdown
# `[2026-09-11]` RECOVERY FOOT-GUN, will recur every colo power event: crowdsec crashes on the hard power-off and traefik's bou
|
|
|
|
⚠ **RECOVERY FOOT-GUN, will recur every colo power event: crowdsec crashes on the hard power-off and traefik's bouncer fail-CLOSES — empty-body 403 on EVERY HTTP service behind traefik (gitea, homepage, …) while the apps themselves are fine.** Signature (bifrost-dev reported it, gitea-shaped): HTTPS returns `403 content-length 0, no app body` on all routes, but git-over-SSH works (SSH bypasses traefik). Diagnosis: `gitea` direct on `localhost:3000` = 200 (app healthy), through traefik = 403; `crowdsec` container `Exited (255)`; `cscli decisions list` EMPTY (not an IP-ban). **The bouncer plugin does NOT self-recover from a startup-time LAPI-unreachable race** — even after crowdsec is healthy again, traefik keeps 403ing until traefik itself is restarted. **FIX: `docker start crowdsec` (its data/config are LOCAL volumes, comes up clean), wait for `cscli lapi status` = OK, THEN `docker restart traefik`** so the plugin re-inits against the live LAPI. Verified 403→200 on gitea API/web/PyPI-index from an off-box vantage. This unblocked bifrost-dev's 1.2.0 PyPI publish (+ worldtree/wyrd/ratatoskr) and any HTTP gitea access; heid's SSH pushes were never affected. → add to the recovery runbook: **crowdsec+traefik restart is a standard post-power-loss step.**
|