Ah, pipes, pipes, pipes...
We had an obnoxious DB lead developer who enjoyed great praise from our customer. He requested to have sudo access to root instead of a subset of commands, as we UNIX overlords used to grant to everyone else. So as the customer is always right, we granted him the ability to sudo su -.
Then we started dealing with unexpected reboots in our production boxes, which in turn made the customer point the finger at us. I gathered the team, talked them calmly, and asked if anyone had slipped an involuntary reboot command somehow. None of the guys acknowledged the mistake, so I had to resort to looking over the secondary logging entries. The puzzling part were those reboots happening off our usual 9 to 5 schedule, but on the much more convenient India time our star DB developer lived in.
So there I was, sitting with a fellow admin looking at the secondary logs, and we saw how this guy was using a lot of the accounts of his DB team colleagues to log into the production boxes and then switching to his own account to ultimately switch to root, which made me mad as hell. I was red with anger, trying to gather as much evidence of this guy's wrongdoings as to warrant an instant layoff, and in the mare magnum of grep commands and redirecting stdout to new log files, I typed
# last | reboot
Which evidently took one of the nodes of the database production cluster down, cascading a myriad of messaging, pager and email alerts.
In the end, I chilled my head, wrote an email acknolwedging my mistake, while also detailing the reasons leading to it, and attaching a copious amount of evidence about the poor identity management procedures displayed by our star DB dev.
I got a slap on my wrist, he got stripped of his sudo privileges, and we all moved along.