Had automation delete the entire Accounting department. Twice. In two days. And...
Reviewing my overnight security alerts before heading into work showed that every account in the Accounting Department was deleted at 7:15 AM. Odd. I went into work and asked my team members what happened. Blank stares as they scrambled to check the alerts they ignored. The security manager chuckled when he came in and said "Apparently we don't need an Accounting department."
The sysadmins, being sysadmins, recovered the accounts from the AD Dumpster, pronounced it a "glitch" that had never happened before so it would not happen again and they would do nothing else to investigate.
Yup, the next morning at 7:15 AM the entire Accounting Department got deleted again.
Now everyone is taking it seriously. Now.
At a meeting that afternoon they said the mass deletion was caused by a script they wrote to sync Oracle HR with AD, a script that ran at 7:15 AM. The script erroneously assumed that if a department did not have a manager then the department did not exist. So rather than just disabling the accounts it deleted them.
Turned out the Accounting Manager had gone on medical leave so he was removed as the department manager, leaving that position blank in Oracle HR.
Everyone in the meeting thought it was good they they had found the problem and prepared to leave because it really was no big deal, right?
Until I said "Wow, it's a good thing the CEO didn't go on medical leave, right? That script might have deleted every account in the company!"
Heads turned, faces got a shocked look, and that's when the sysadmin manager admitted that was exactly what would have happened.
Then they did a similar thing years later. They wrote a script to delete ex-employee Exchange mailboxes 90 days after the employees left the company. It worked great, until it didn't.
We were in a legal battle and there was a court order to preserve certain mailboxes for legal discovery. Some of the mailboxes were for departed senior managers. Their script dutifully deleted their mailboxes after 90 days because everyone forgot about it. Then the archive purge killed the backups some months later.
Then we were ordered to produce those mailboxes and Whoops! Their little automation script caused a massive legal problem with the court, you know, destruction of evidence...
Finally, after that disaster, senior managers ordered the sysadmins to put EVERY script they used into the Source Code system where they needed to check scripts out to use them or to modify them. The sysadmins whined and they got told that all code run on production systems was programming and now need explicit manager approval via a Change Control ticket, so they made it worse by complaining.