Once upon a time, we had to insert a new 48 ports card into an almost full C5500.
It was of course running production for many systems.
I was lifting the forest of already cabled cables when the network dude was slowly inserting the card.
Then there was a big ZAP and the whole switch went silent !
2 hours of unplugging the thing, unracking the switch and replacing it by a spare one, kindly provided by
our DC provider (Thanks, James !). Re-plugging all and reconfig of the switch and we were back in business.
The post-mortem analysis revealed that:
- the power from chassis to card was provided by a male connector
- since this was recycled HW, and apparently someone had unplugged it quite brutally in the past, said connector was bent
- when we slid the card in, the bent male connector created a short, killing the chassis and its 2 PSUs
I remember the then "CIO" (random idiot having the right nationality) requesting those LAN incident to stop, to which I replied:
"this would need to use only new kit for LAN and no longer recycled kit, therefore funding". Never got a reply.