Blamed for phone system hack, survived the bus
There was a time when I was nominally responsible for multiple site phone systems. When I started, these were Nortel PBXs, some in place for upwards of 20 years. No one had any Nortel training, and the documentation was... spotty. These were robust systems, but nothing lasts forever, and getting parts was becoming difficult. My CIO didn't want to spend his training budget on Nortel classes, so all I had to go on were notes by predecessors.
So, a monthly phone bill from one of the smaller sites suddenly rose from well under $1k to over $40k. Someone had called into the PBX remote service line, taken control and had made a lot of calls to Hong Kong and Singapore. I was promptly thrown under the bus for allowing this to happen. The CIO figured that making me the scapegoat would take the heat off him, and happily told the Finance VP that I was to blame.
Alas for him, I had my CV and my emails showing he'd denied me Nortel training (or even software manuals). I had had no Nortel experience prior, he'd picked me to expand the network and switch everyone to VoIP. That CIO was nothing if not a skilled politician. He switched gears instantly when he realized that he'd go down with me, and managed to use the hack to get the VoIP project budget increased. That kept me and my mates busy for years.
My fix for the Nortel hack problem was incredibly low tech, as the systems really didn't have security in mind when made. I disconnected every remote service line, marking it very clearly. The few times I had to access those beasts remotely, I'd call up the local tech (or power user) and have them plug that line in, then call to have it disconnected again afterwards. No, I never did go to Nortel class, but finally got a programming manual, which I gratefully binned along with the last Nortel PBX.
Biting the hand that feeds IT
