fix(esh): revert the forced 10G — the link was flaking hard, operator called it

Both ends are back on autoneg at 1000 Mbps and stable; UDM SFP+2 restored to
autoneg True / setting_preference auto from the payload staged before the
original change. ~50 s of renegotiation, nothing rebooted.

The force failed progressively and I called it too early. Errors on the
ESH-Media side went 200 at link-up, 221 at 42 minutes, 416 by four hours, with
flapping the operator noticed before my soak had accumulated enough to raise
it. I read a two-minute flat window as a plateau and said so; a marginal link
declares itself over hours, not minutes.

Do not re-force this port. The cable is being replaced at the weekend and a
correctly-coded DAC negotiates 10G on its own.
This commit is contained in:
2026-09-04 14:41:02 -07:00
parent 514ce7acc8
commit 054c098777
+23 -26
View File
@@ -3,30 +3,27 @@
2026-09-04. The link runs at 10G by an explicit rate override on the UDM side.
**Read the pinned-port warning below before touching that port.**
> ## CURRENT STATE: forced 10G, KNOWN MARGINAL, held deliberately
> ## CURRENT STATE: REVERTED to autoneg / 1G. The force is gone.
>
> **Operator decision 2026-09-04: leave the override in place.** He has a
> replacement cable and will fish it through the garage at the weekend. Do not
> revert it and do not re-diagnose it — this is a held position, not an
> oversight.
> **2026-09-04 14:38 — the operator reported the 10G link "flaking hard" and I
> reverted it.** Both ends are back at 1000 Mbps, UDM SFP+2 is `autoneg True /
> setting_preference auto`, ESH-Media is reachable and stable. The revert took
> about 50 s of renegotiation and nothing rebooted.
>
> The link is **not clean at 10G**: errors accumulate steadily (200 → 221 rx over
> 42 min, ~1 per 23 MB, roughly one bad frame in 15,000) and it dropped once, at
> 10:12, recovering on its own. Nothing rebooted. TCP rides over it; expect
> retransmits and jitter rather than visible corruption.
> **The forced-10G experiment failed, and the failure was progressive.** Error
> counts on the ESH-Media side over the life of the force:
>
> When the cable is swapped: **remove the override** and let autoneg work. A
> properly-coded DAC negotiates 10G on its own — that is the whole point.
link ESH-Media SFP+1 (port 11) <--DAC--> ESH-UDMPM SFP+2 (port 11, eth10)
cable OEM "SFP-H10GB-CU3M", serial CSC200803000004, 3 m passive twinax
before both ends 1000 Mbps, autoneg on
after both ends 10000 Mbps
This link backhauls the USW Flex XG "ESH-Office-Vuong", the E7-ESH-Media access
point, and the Zigbee coordinator — the office switch and the AP both connect at
10G and were squeezing through a 1G uplink, so this was a real bottleneck rather
than a cosmetic one.
> 10:03 200 link-up burst
> 10:48 221 +21 in 42 min -- the first sign it was not plateauing
> 14:38 416 +195 over the next 4 h, plus user-visible flapping
>
> ⚠ **The lesson, and it is mine to own: I read a two-minute flat window as a
> plateau and reported it as one.** The counter simply had not moved yet. A
> marginal link declares itself over hours, not minutes — and the operator felt
> the flapping before the soak I left running had accumulated enough to call it.
>
> **Do not re-force this port.** The cable is being replaced at the weekend; a
> correctly-coded DAC negotiates 10G on its own, which is the actual fix.
## The diagnosis, and what made it certain
@@ -78,12 +75,12 @@ pre-change `port_overrides` array was written to a revert payload before the
forward one was sent. If a future attempt needs *both* ends forced, that is a
different risk class and needs the operator's sign-off, not an agent's.
## ⚠⚠ THAT PORT IS NOW PINNED — a 1G module will not link
## The port is NOT pinned any more — the override was removed 2026-09-04
UDM SFP+2 is `autoneg False, speed 10000`. Plug anything that is not a 10G
module into it and the link stays down with no obvious cause. If the DAC is ever
replaced with a properly-coded one, **remove the override** and let autoneg do
its job rather than leaving a pin nobody remembers.
While the force was in place, UDM SFP+2 was `autoneg False, speed 10000` and a
non-10G module would not have linked there. That is no longer true: the override
is gone and the port is back on autoneg. Recorded because the warning stood in
this runbook for four hours and someone may remember it.
## What the link bounce cost