From 054c09877781ea5124be0df8a4c098f81ea76607 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 4 Sep 2026 14:41:02 -0700 Subject: [PATCH] =?UTF-8?q?fix(esh):=20revert=20the=20forced=2010G=20?= =?UTF-8?q?=E2=80=94=20the=20link=20was=20flaking=20hard,=20operator=20cal?= =?UTF-8?q?led=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both ends are back on autoneg at 1000 Mbps and stable; UDM SFP+2 restored to autoneg True / setting_preference auto from the payload staged before the original change. ~50 s of renegotiation, nothing rebooted. The force failed progressively and I called it too early. Errors on the ESH-Media side went 200 at link-up, 221 at 42 minutes, 416 by four hours, with flapping the operator noticed before my soak had accumulated enough to raise it. I read a two-minute flat window as a plateau and said so; a marginal link declares itself over hours, not minutes. Do not re-force this port. The cable is being replaced at the weekend and a correctly-coded DAC negotiates 10G on its own. --- docs/runbooks/esh-media-dac-10g.md | 49 ++++++++++++++---------------- 1 file changed, 23 insertions(+), 26 deletions(-) diff --git a/docs/runbooks/esh-media-dac-10g.md b/docs/runbooks/esh-media-dac-10g.md index b86d0dd..bd08d7f 100644 --- a/docs/runbooks/esh-media-dac-10g.md +++ b/docs/runbooks/esh-media-dac-10g.md @@ -3,30 +3,27 @@ 2026-09-04. The link runs at 10G by an explicit rate override on the UDM side. **Read the pinned-port warning below before touching that port.** -> ## ⚠ CURRENT STATE: forced 10G, KNOWN MARGINAL, held deliberately +> ## CURRENT STATE: REVERTED to autoneg / 1G. The force is gone. > -> **Operator decision 2026-09-04: leave the override in place.** He has a -> replacement cable and will fish it through the garage at the weekend. Do not -> revert it and do not re-diagnose it — this is a held position, not an -> oversight. +> **2026-09-04 14:38 — the operator reported the 10G link "flaking hard" and I +> reverted it.** Both ends are back at 1000 Mbps, UDM SFP+2 is `autoneg True / +> setting_preference auto`, ESH-Media is reachable and stable. The revert took +> about 50 s of renegotiation and nothing rebooted. > -> The link is **not clean at 10G**: errors accumulate steadily (200 → 221 rx over -> 42 min, ~1 per 23 MB, roughly one bad frame in 15,000) and it dropped once, at -> 10:12, recovering on its own. Nothing rebooted. TCP rides over it; expect -> retransmits and jitter rather than visible corruption. +> **The forced-10G experiment failed, and the failure was progressive.** Error +> counts on the ESH-Media side over the life of the force: > -> When the cable is swapped: **remove the override** and let autoneg work. A -> properly-coded DAC negotiates 10G on its own — that is the whole point. - - link ESH-Media SFP+1 (port 11) <--DAC--> ESH-UDMPM SFP+2 (port 11, eth10) - cable OEM "SFP-H10GB-CU3M", serial CSC200803000004, 3 m passive twinax - before both ends 1000 Mbps, autoneg on - after both ends 10000 Mbps - -This link backhauls the USW Flex XG "ESH-Office-Vuong", the E7-ESH-Media access -point, and the Zigbee coordinator — the office switch and the AP both connect at -10G and were squeezing through a 1G uplink, so this was a real bottleneck rather -than a cosmetic one. +> 10:03 200 link-up burst +> 10:48 221 +21 in 42 min -- the first sign it was not plateauing +> 14:38 416 +195 over the next 4 h, plus user-visible flapping +> +> ⚠ **The lesson, and it is mine to own: I read a two-minute flat window as a +> plateau and reported it as one.** The counter simply had not moved yet. A +> marginal link declares itself over hours, not minutes — and the operator felt +> the flapping before the soak I left running had accumulated enough to call it. +> +> **Do not re-force this port.** The cable is being replaced at the weekend; a +> correctly-coded DAC negotiates 10G on its own, which is the actual fix. ## The diagnosis, and what made it certain @@ -78,12 +75,12 @@ pre-change `port_overrides` array was written to a revert payload before the forward one was sent. If a future attempt needs *both* ends forced, that is a different risk class and needs the operator's sign-off, not an agent's. -## ⚠⚠ THAT PORT IS NOW PINNED — a 1G module will not link +## The port is NOT pinned any more — the override was removed 2026-09-04 -UDM SFP+2 is `autoneg False, speed 10000`. Plug anything that is not a 10G -module into it and the link stays down with no obvious cause. If the DAC is ever -replaced with a properly-coded one, **remove the override** and let autoneg do -its job rather than leaving a pin nobody remembers. +While the force was in place, UDM SFP+2 was `autoneg False, speed 10000` and a +non-10G module would not have linked there. That is no longer true: the override +is gone and the port is back on autoneg. Recorded because the warning stood in +this runbook for four hours and someone may remember it. ## What the link bounce cost