Re: Automation does have its place
A problem with automation scripts written by sysadmins is that they generally do not have a development background and do not consider enough corner cases and how things can go poorly nor do they write defensively Automation scripts should be considered as nothing less than production apps and subject to the same controls: peer review and source code check-in and check-out to name a few.
One day I woke up and reviewed alerts before work. I noted that the entire Accounting department had their accounts deleted. I thought to myself "Hmmm, I thought we needed an Accounting department."
Upon arrival at the office I asked what happened and was told it was "just a glitch" and the accounts had been retrieved from the AD Dumpster. It had never happened before so it would never happen again. Right.
As you've already guessed, it happened the next day. So now the sysadmins decided to investigate. They had written a script to sync the HR management software with the AD structure so AD reflected HR. Good idea. But the author failed to consider what would happen if a department manager went on leave.
The Accounting manager went on long-term sick leave and was removed from the HR org tree after one week per policy. When their AD import script saw no manager it branched to the cleanup section and because the the department apparently no longer existed, it deleted all of the active accounts instead of stopping for confirmation or just disabling the accounts. It actually was a cascading fault because one of the Accounting manager's subordinates supervised a smaller department and their accounts were wiped out as well when the subordinate had his account deleted by the script.
Just imagine if the CEO had gone on long-term leave...
Biting the hand that feeds IT
