the difference between an engineer and sn installer.
Quite.
One evening, some two decades back, we were upgrading the firmware on the FDDI hubs strewn around the site, a veterinarian pharmaceuticals research/production plant. Topo was a double ring, so one hub being rebooted should not have affected any of the other hubs. Still, it turned out it did as with a number of reboots a couple of the other hubs started complaining about the loss of one ring.
So, two of us grabbed the keys to the buildings and their data cabinets, a pair of walkie-talkies and flashlights, and set off into the night in the general direction of where the problem was. Now a break in a fiber could really be anywhere between hub A and hub B, although it probably wouldn't be somewhere halfway in a multicore run in an underground duct, as it would be very unlikely a disturbance would have taken out just a single strand. So we started with checking the patch panels and the leads to the hubs.
Nothing immediately obvious, so checking the connectors was next. And pulling out one of them the cause of the problem glaringly presented itself as the fiber just dropped out of the connector body.
It turned out there had been a couple of new CAT5 patches installed the day before, and apparently one of the installers had dropped a screwdriver or something on the hapless fiber connector below, somehow breaking the fiber in such a way that it came loose inside of the connector body after which that installer monkey just pushed it back in.
The next day facility services were instructed to put plexiglass covers over the fiber patches in all data cabinets.
Biting the hand that feeds IT
