esh-pfi-infrastructure/persistent-memory.d/2026-09-04-dac-forced-10g-failed.md

[2026-09-04] Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early

The ESH-Media ↔ UDM uplink is a 3 m OEM SFP-H10GB-CU3M twinax that negotiated 1000 Mbps with zero errors on both ends. Diagnosis: sfp_compliance: Unknown — the switch reads the EEPROM but cannot parse the compliance codes on a third-party cable wearing Cisco coding, so autoneg falls back to the safe rate. Control case: a TP-Link TL-SM5220-1M on the adjacent UDM port runs 10G with the same speed_caps, exonerating the port, firmware and autoneg.

Operator authorised forcing it. Forced autoneg False / 10000 / full on the UDM end only; ESH-Media followed to 10000 unforced — proof the cable was electrically capable.

⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG

10:03   200 errors   link-up burst
10:48   221          +21 in 42 min  — I reported this as "flat", it was not
14:38   416          +195 over 4 h, plus user-visible flapping the operator felt

A marginal link declares itself over hours, not minutes. I watched a two-minute flat window and reported a plateau; the counter simply had not moved yet. The operator noticed the flapping before the soak I left running had accumulated enough to raise it. Reverted 14:38, both ends back to autoneg/1000, stable.

⚠⚠ The correction that matters: a clean, zero-error link at 1G does NOT rule out a marginal cable. It only proves the cable is clean at 1G. Clean at 1G and marginal at 10G is exactly what a cheap 3 m twinax is — which means the platform's autoneg fallback was protecting something real, not being fussy about vendor coding. The coding explains the negotiation; it does not explain errors once forced. I conflated the two.

⚠ Do not re-force this port. The fix is the cable.

Method notes worth keeping

• Force the RECOVERABLE end. ESH-Media reaches the controller through this link, so a failed force there strands the switch behind a dead uplink — a physical visit. The UDM end is safe because the path to the controller (nh3-dev → 10.100.10.1 → 10.0.0.1, site VPN) does not cross it. Verified with traceroute before the change; revert payload written before the forward one.
• ⚠ port_overrides is a whole-array PUT — the array was diffed to prove exactly one field on one port changed before sending, and read back after.
• Blast radius named before acting: ESH-Media backhauls the operator's office switch, a WiFi AP and the Zigbee coordinator.

Resolution: the operator already owns a replacement and ran the copper himself through a drilled floor 2x4. Recommendation given was fiber (10Gtek SR 2-pack + OM4 3 m LC-LC, ~$40-65) because the cable-vs-pull-damage question was never resolved and a second DAC through the same hole risks the same fault. Runbook docs/runbooks/esh-media-dac-10g.md; commits b9988a7, 514ce7a, 054c098.