Re: "each packet was sent only one hop"
According to the FCC, the equivalent TTL was infinity. These packets are not standard TCP packets, I believe, they are proprietary Infinera packets, at least over the management channel, anyway.
Looks like they're 'standard' Ethernet frames, ie-
Currently, CenturyLink is in the process of updating its nodes’ ethernet policer to reduce the chance of the transmission of a malformed packet in the future. The improved ethernet policer quickly identifies and terminates invalid packets, preventing propagation into the network.
Shame the FCC report doesn't include the 'malformed' frame because presumably it was 'vaild' as far as being a correctly structured Ethernet frame. There's no TTL in Ethernet, so why broadcast storms happen. I'm guessing having a length of 64 bytes, it might have been a zero length frame, ie all header and no payload, so nodes <shrug> and pass them on.
Each time one of the bad packets hit a node, the node spammed *all* neighboring nodes with the same packets due to the broadcast address. That's the main problem, not the TTL, IMHO.
Yup. Ethernet working as intended. Just not the way Infinera/CenturyLink intended. Issue also seems to be-
As the supplier of these nodes, Infinera provides its customers – including CenturyLink in this case – with the proprietary management channel enabled by default. CenturyLink was aware of the channel but neither configured nor used it
And then why resource depletion on the switching card lead to LOS on customer circuits. Fix for that would I guess be to set process quotas and rate limit broadcast frames, so standard methods to reduce the impact of broadcast storms.
Also OOB access can be a real PITA for modern networks. So in the good'ol days, a modem hooked up to a craft/console port.. But that assumes you have a working phone line, and this outage affected phone calls because they were carried across CenturyLink's network. And same happens if you try using leased lines/xDSL if that supplier is also your wholesale customer. In an ideal world, have a good'ol fashioned order wire, or run dual networks so if one goes down, you should be able to access via the other. Biggest challenge is your OOB may not work anyway if the switching card's running flat out & out of resources.
(anyone who's ever done 'debug all' on a big-ish, busy Cisco knows what happens then...)