memory: snapshot — Anaheim tunnels on AES-128, ana-gw admin closed, Scriberr live, ESH DNS fixed
Rewrites the in-flight section, which had gone stale in one place (speaches is stopped, not live) and did not reflect a session in which everything opened was also closed. Three detail files carry the bulk: the FortiGate's public surface reduced to nothing including the ACME listener, Scriberr's deployment and the three upstream defects it required working around, and the ESH DNS repair that also made the IPv6 naming scheme real on three hosts. The tunnel entry in the decisions log described a cipher change as proposed and pending; it has since been attempted, found impossible in the form intended, and resolved differently, so the line now points at the outcome rather than the intention. New decisions cover the cipher adoption and the finding that the per-flow ceiling belongs to the UniFi gateway's software cipher rather than the firewall, the administrative closure, the transcription deployment, the DNS repair, the retirement of an ASR service whose only consumer was abandoned, and a database VM that until now had no fleet identity and no vaulted credential at all. Three approaches are recorded as abandoned: an AEAD cipher the far end cannot express, an upstream DNS setting the gateway accepts and ignores, and a scheme to claim unused delegated prefixes that founders not on the prefixes but on having to rebuild a firewall policy to use them.
This commit is contained in:
@@ -0,0 +1,50 @@
|
||||
# [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included
|
||||
|
||||
WAN admin was opened at the start of the session as a cutover contingency
|
||||
("so I don't have to drive down there"), then closed again on operator
|
||||
instruction once the AES-128 work landed. Net result: **the FortiGate's WAN
|
||||
address now exposes no TCP port at all.**
|
||||
|
||||
## Final state
|
||||
|
||||
External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two
|
||||
sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`.
|
||||
|
||||
**Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels
|
||||
drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) —
|
||||
both are recorded in auto-memory `reference_fortigate_ana_gw_access`.
|
||||
|
||||
## Port 80 was the FortiOS ACME listener, and I got it wrong first
|
||||
|
||||
`38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with
|
||||
`allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" —
|
||||
was **wrong**, and the reason is worth keeping:
|
||||
|
||||
> The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
|
||||
> matches inbound only**, so outbound SYN-ACKs were excluded *by construction*,
|
||||
> and concluding "the box sends no SYN-ACK" from that capture was unsound.
|
||||
|
||||
Re-run bidirectionally (`host … and tcp port 80`) it immediately showed
|
||||
`wan1 out 38.120.12.42.80 -> <scanner>: syn ack`. **Rule: to test whether a box
|
||||
*answers*, the filter must be bidirectional.**
|
||||
|
||||
The listener is opened by `config system acme / set interface "wan1"` and
|
||||
**bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It
|
||||
was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`,
|
||||
valid to 2026-10-27) is untouched and simply stops renewing, which is fine
|
||||
because WAN admin is closed and the box is being replaced.
|
||||
|
||||
## Retracted in the same pass: the "four all-port VIPs" alarm
|
||||
|
||||
Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS
|
||||
VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding
|
||||
on the VIP object — and only the first was checked. All 14 VIPs are scoped;
|
||||
`Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only.
|
||||
|
||||
Ground-truth external scan of all six public IPs is recorded in
|
||||
`reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443`
|
||||
(mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up
|
||||
candidates for the OPNsense translation, not exposure.
|
||||
|
||||
**Lesson, twice in one session: measure from outside instead of parsing config.**
|
||||
Both wrong answers came from a filter that answered a different question.
|
||||
@@ -0,0 +1,77 @@
|
||||
# [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live
|
||||
|
||||
Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup
|
||||
shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three
|
||||
wrong turns worth recording.
|
||||
|
||||
## Root cause
|
||||
|
||||
`esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises
|
||||
**itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over
|
||||
the DHCPv4-supplied resolver, so queries went to the UDM — which does not know
|
||||
`.internal` — and returned NXDOMAIN. AdGuard was never consulted.
|
||||
|
||||
Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver
|
||||
was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled
|
||||
entirely**, so every host there got the UDM and could never resolve `.internal`
|
||||
— esh-docker-vm was living proof.
|
||||
|
||||
## The three wrong turns
|
||||
|
||||
1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an
|
||||
explicit server is supplied**. Setting it bare looks like a no-op and invites
|
||||
the conclusion that the field is inert — which I drew, wrongly. Despite the
|
||||
`dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network.
|
||||
2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to
|
||||
AdGuard persists, reads back, and changes nothing. Proven with **fresh
|
||||
uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned
|
||||
real IPs. Reverted.
|
||||
3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the
|
||||
known `cmd/devmgr` behaviour.
|
||||
|
||||
Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were
|
||||
probed with a stdlib raw-socket Router Solicitation parsing option type 25
|
||||
(`rdisc6`/`tcpdump` were both absent; nothing was installed).
|
||||
|
||||
## What landed
|
||||
|
||||
RDNSS **redirected** rather than disabled — better than switching it off:
|
||||
|
||||
| VLAN | v4 | v6 RDNSS |
|
||||
|---|---|---|
|
||||
| `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
|
||||
| `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
|
||||
|
||||
The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over
|
||||
Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on
|
||||
SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client
|
||||
accepts it and never retries. A secondary that doesn't know your private zone is
|
||||
a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing
|
||||
camera DNS through AdGuard's filtering risks their cloud features.
|
||||
|
||||
## The naming scheme became real
|
||||
|
||||
The resolver address is the scheme's first live use, replacing a MAC-derived
|
||||
SLAAC address that would break on a NIC change. All three `esh-server` Linux
|
||||
hosts now carry `4411:B105` ("FOR ALL BIOS"):
|
||||
|
||||
```
|
||||
esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55
|
||||
esh-vm-db …:4411:b105:50:60
|
||||
```
|
||||
|
||||
Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals
|
||||
on re-delegation), backgrounds itself with a retry (SLAAC may not have landed;
|
||||
a blocking hook would stall bring-up on a headless box), and adds nothing to
|
||||
existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets
|
||||
`accept_ra=0` and would strand the host.
|
||||
|
||||
⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only
|
||||
despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores
|
||||
RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with
|
||||
`accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default
|
||||
route** — an IPv6 identity with no change to a hypervisor's routing. Expect this
|
||||
on every Proxmox node when its LAN gets v6.
|
||||
|
||||
Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits:
|
||||
auto-memory `reference_unifi_dns_rdnss_limits`.
|
||||
@@ -0,0 +1,46 @@
|
||||
# [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1
|
||||
|
||||
Self-hosted audio/video transcription + diarization. Operator chose GPU
|
||||
placement over ana-docker (8 cores shared with 50 containers, 37 GB disk)
|
||||
against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom.
|
||||
|
||||
**Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`,
|
||||
all seven backends up, zero failures: `whisperx pyannote sortformer parakeet
|
||||
canary voxtral openai`. ~30 GB of weights on `/tank`.
|
||||
|
||||
Stack: `stacks/scriberr/`. Full gotcha list in auto-memory
|
||||
`reference_scriberr_ana_ml2`.
|
||||
|
||||
## Three upstream bugs, none of them ours
|
||||
|
||||
**1. The Blackwell image does not exist.** Upstream's README documents
|
||||
`scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published
|
||||
`scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or
|
||||
silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9`
|
||||
(CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose
|
||||
back to the published image.
|
||||
|
||||
**2. It must run as uid 10001, not 1000** — and the error lies:
|
||||
`unable to open database file: out of memory (14)`. Error 14 is
|
||||
`SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates
|
||||
`appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to
|
||||
it, while the entrypoint's PUID remap covers only the data dirs.
|
||||
**Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the
|
||||
mount; fails on a plain named volume too → not the storage; the **published CPU
|
||||
image works at PUID=1000** because there `appuser` *is* 1000.
|
||||
Generalisable: *when a container "permission" bug appears, compare the uid the
|
||||
image was BUILT for against the uid you are RUNNING as.*
|
||||
|
||||
**3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env
|
||||
with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with
|
||||
`Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial
|
||||
failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet
|
||||
and Sortformer were silently absent. Occurrences 2 → 0 after the fix.
|
||||
|
||||
## Related
|
||||
|
||||
`speaches` on irv-ml1 **stopped** the same day (stack retained, one command to
|
||||
restart): Eyra was abandoned pre-implementation because Scriberr covers the need,
|
||||
leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and
|
||||
is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the
|
||||
~5.9 GB quoted — that figure is the loaded-model working set.
|
||||
Reference in New Issue
Block a user