back to article Funny, that: Handy script for wiping directories is capable of wreaking havoc beyond a miscreant's wildest dreams

Remember when September seemed so far away? Those of you still working from your bedroom since March should probably have changed your pyjamas by now. We'll wait. When you're ready, enjoy a tale from the Who, Me? vault courtesy of a reader who knows all about unplanned undergarment changes. "O" spent the early part of this …

        1. Blackjack Silver badge

          Re: capable hands of Windows Server 2003

          That was most likely to avoid another scandal like the Microsoft Plus CDs that had viruses on them.

          While I can't find the old news online, apparently a factory in Mexico didn't have an antivirus installed because "Windows 95 doesn't get viruses".

        2. C R Mudgeon

          Viruses and duplicators

          "It was ironic that even Microsoft couldn't guarantee their internal Windows machines didn't have latent virii."

          At a small but high-profile Amiga developer I once worked at, one release's master (floppy) disk was given a virus *by* the duplication house. As I recall, part of their QA was to compare a sample output copy against the original master -- for which they used an infected Amiga. I no longer recall whether the whole run was bad (vs. just those two diskettes), and if it was, whether any of them made it out into the wild.

          Somehow, I then became the mastering guy for all of our products, not just the one I was assigned to. I presume my level of outrage at the series of screwups that led to the above debacle is what landed me the task. My checklist included such items as: make N copies of the master, and keep some of them back; do a comparison ourselves; do it byte-wise, not merely a recursive file comparison -- I wrote a utility for that. Last but far from least: flip the $&(! write-protect tab before you send the master off to the duplicators.

          1. jake Silver badge

            Re: Viruses and duplicators

            Mass distribution of floppies infected with a virus happened several times over the years. There used to be a web site listing them, but today's useless search engines won't tell me anything about the history of viruses, they will only tell me where I can purchase an AV product for Windows. How fucking useless is that?

          2. Anonymous Coward
            Anonymous Coward

            Re: Viruses and duplicators

            Flip it? No, sir - you get a pair of cutters or similar and you remove that tab! Never write to that disk again!!

            Well, you can - just need to tape the hole over or something

        3. jake Silver badge

          Re: capable hands of Windows Server 2003

          To be fair, in the early days, CD mastering and pre-mastering software was fairly esoteric, and ran on proprietary hardware. This hardware usually ran whatever OS the lead engineer used/preferred at University ... AIX, HPUX, SunOS, BSD, and yes, Xenix.

          But WinDOS? Not so much. It was nowhere near stable enough for such critical tasks. (NT didn't exist yet, at least not in the early days.)

          1. Blackjack Silver badge

            Re: capable hands of Windows Server 2003

            Authoring tools to create master CDs that later were used to make the mass production copies, ran on Windows and DOS so that's how a CD could get infected by a Windows or DOS virus despite the other systems being different.

            The hilarious thing is that F-Prot for DOS was free and worked until Windows 98 SE to detect and erase viruses for Windows and DOS. And there was also Norton Antivirus Scanner that at least let you detect the things and was free to use and that one ran in Windows.

    1. Rob Daglish

      Re: capable hands of Windows Server 2003

      I'm sure I remember being told in the late 90s/early 2000s that Novell had offered to rebuild 1/4 of the MS internet servers with Intranetware and still run a better service than MS could manage...

  1. Anonymous Coward
    Anonymous Coward

    We've all been there

    We had a QA group that did something similar, a long test job that ran for hours, and cleaned up with something like "rm -rf $(LOGDIR}/".

    Inevitably some unexpected failure (is there any other sort?) in the test job resulted in this being run with LOGDIR unset. As a user in the same GID group as the staff. On a lab system which had the default automounter config which NFS-mounted all the user home directories, many of which seemed to have 775 permissions...

    The sysadmin finally twigged when he realised that the calls from people saying "some of my files have disappeared" were coming in in alphabetical order of username. A hasty shutdown of the home directory NFS system was followed by some forensic network access to find the guilty system.

    Fortunately the overnight backups & regular ZFS snapshots meant that the QA team responsible got away with an apology, and buying a few beers.

    1. John Brown (no body) Silver badge
      Thumb Up

      Re: We've all been there

      "the QA team responsible"

      Oh, the irony!

      1. jake Silver badge

        Re: We've all been there

        But wait. It gets better. Today's DevOps advocates are using stories like this to justify getting rid of QA entirely ...

    2. C R Mudgeon

      Re: We've all been there

      "... some unexpected failure (is there any other sort?) ..."

      Yeah. An expected failure is one that the code guards against. Case in point:

      if [ "x$LOGDIR" != "x" ]; ...

  2. Doctor_Wibble
    Unhappy

    Are we including sabotage?

    Of a completely unfortunate accidental sort of course.

    A departing employee completely accidentally left a floppy disk in their laptop, which was set as the boot drive and which by complete accident had its autoexec.bat consisting of just one command that would delete the contents of the C: drive if it was completely accidentally booted from.

    The great thing about booting from a floppy is that it is the sort of thing that you can hear and stop with a magic word and a mystical gesture before it does any of that completely accidental damage for which the git knew full well I would be the one blamed.

    .

    On the confessional front, have I deleted stuff I didn't mean to delete? Yep. On the other hand I still feel the stinging lesson of a disk failure that happened the day before the dvd blanks arrived upon which the much put-off backup of said disk was to be stuck.

    1. Boris the Cockroach Silver badge
      Unhappy

      Re: Are we including sabotage?

      Quote

      Yep. On the other hand I still feel the stinging lesson of a disk failure that happened the day before the dvd blanks arrived upon which the much put-off backup of said disk was to be stuck.

      OH fudge.. I knew there was something I had to do friday before beer time.....

    2. Anonymous Coward
      Anonymous Coward

      Re: Are we including sabotage?

      Departing employee should have included a line to re-write the autoexec.bat, or otherwise hide the evidence.

      I hope your company sent the disk to him with his last paycheck. I'd have included a note that the (completely accidental) effects of the disk resulted in the deletion of a perfectly glowing letter of recommendation for anyone enquiring as to his employment history.

      1. The Oncoming Scorn Silver badge
        Pint

        Re: Are we including sabotage?

        Del %0 always worked for me.

  3. Dan 55 Silver badge

    I'm writing a backup/restore script now, and completely paranoid that the backup won't backup or the restore won't restore or the person using it won't see the error message or someone will try to run it as a cron job with half the environment variables missing or something. There are more ifs with quotes and $? than there are lines that actually do stuff.

    1. Flocke Kroes Silver badge

      Have you considered...

      #! /bin/bash -xe

      # Prints each line of what will be executed after expansion but before execution

      # Exit on any error outside a condition

      # -x -e will not work: Everything after the second space is considered a single word

  4. Edwin

    There, but for the grace of God,

    go I...

    1. jake Silver badge

      Re: There, but for the grace of God,

      There is no god. You'll join the ranks eventually.

      1. Anonymous Coward
        Anonymous Coward

        Re: There, but for the grace of God,

        Oh, there is, but his grace for careless sysadmins is not boundless. In time thy faith shall be tested!

        (All those bearded guys in the Bible, at least some of them just *had* to be programmers, right?)

        1. A.P. Veening Silver badge

          Re: There, but for the grace of God,

          Definitely, look at those ten commandments, bashed into stone ;)

        2. DJO Silver badge

          Re: There, but for the grace of God,

          (All those bearded guys in the Bible, at least some of them just *had* to be programmers, right?)

          You're not the first to think that:

          Bearded Sky Man as a single parent and hobby programmer

  5. imanidiot Silver badge

    Philosophical questions

    If data is deleted, but no one is around to notice, was anything lost?

    I've sort of nearly done the same thing once, but only because someone was using a script they shouldn't have nor use.

  6. Anonymous Coward
    Anonymous Coward

    My contribution...

    I was doing a temp stint at a place that shall remain nameless to protect them from the hoardes of angry IT folks that would surely set fire to the HQ if said name were made known. Suffice it to say it was one of those huge juggernaughts that we all love to eviscerate with vitriol at every opportunity.

    I was in a server room tending to an old cluster that needed some TLC. Archaic disks that hadn't been backed up in aeons, software so old it had probably been given first drafts on clay tablets by Egyptian clerks wondering how to spell all the buzzword bingo bullshit, managed by monkies in feisty knickers.

    I had verified that the server was no longer connected to the internal network by order of the manager I had been assigned to. I made sure that it wasn't running any jobs that hadn't been marked as non critical, temporary, or otherwise able to be sacrificed without need for panick. At which point I start searching the disks for where the largest concentration of files (sizes, numbers, etc) were to be found.

    Imagine my surprise when the largest by an exponential margin turned out to be a personal directory full of porn. I dutifully made an offline copy for *cough* Reasons and began backing up the entire system to the specific NAS unit dedicated for just that cluster & purpose.

    I get done, detach the NAS, lock it in the drawer the manager indicated, and began cleaning up/out said server for repurposing to other tasks.

    I'm about halfway through when some guy I don't know barges into the server room in a wide-eyed frizzy-haired state & heads straight for me like a laser beam.

    "What have you done to my server?!" he roars as if he were Zeus & I'm about to get smote with lightning.

    I explain what I'd been brought in to do, show him the paperwork from my manager giving me authorization to do it, & explain that I've just given the machine a fresh, legally licensed copy of the OS (complete with drivers) to prep it for reuse elsewhere.

    "You can't DO that! That's MY server!" he roars again.

    Unimpressed I show him the paperwork that expressly says I most certainly can & have $Manager's orders to do so.

    Back & forth, back & forth, him roaring, me not giving a shit. I'd made *damn* sure I was on the machine $Manager specified to prevent me from fucking anything else up, so to have a different manager berating me for doing my job leaves me wanting to smack him upside the head with a NAS.

    He shouts he'll talk to $Manager, I nod & say to go ahead, & while he's off to go have some more shouting, I'll finish the job I want to get paid for.

    Turned out that Old Yeller had been some up-and-comer nepotistic bugger that had hit his Peter Principle limit. He was used to coming in to work, vanishing into his office, & surfing porn all day. Actual work? Don't make him laugh.

    My having taken down his personal porn server was seriously putting a crimp in his pseudo-productivity & he was Having Words with $Manager about why said interference Was Not Allowed.

    Except Old Yeller really should have talked to whomever he'd been related to first. Because that person no longer worked at the company, upper management was sick & fekkin' tired of the useless dolt, & this had been their shot across the bow to get him to actually DO something for a change.

    $Manager showed their paperwork authorizing said work & sent Old Yeller up the ladder. At some point Old Yeller was told to stuff it.

    $Manager thanked me for not giving in to the idiot, for having had the forethought to keep all the paperwork I'd needed to deflect said idiots anger, and for giving him back seriously needed resources.

    I was quite pleased with the trip to the pub for lunch & a pint on $Manager.

    I was even MORE pleased with the fat brown envelope he offered for his own copy of what I'd found on the machine.

    "It's all on that NAS you locked in the cabinet." made him grin like a shark swimming through a cloud of fresh chum.

    I was even MORE pleased when I uploaded the entire trove to my various torrent accounts (Demonoid FTW!) & watched my street cred go through the roof.

    Ahhhhhh... fun times!

    1. A.P. Veening Silver badge

      Re: My contribution...

      You sure earned your BOFH papers with that one and lucky to have a BMFH backing you.

    2. Anonymous IV
      Headmaster

      Re: My contribution...

      > one of those huge juggernauts that we all love to eviscerate with vitriol at every opportunity

      I'm all for colourful phraseology, but in the interests of factual accuracy I would point out that evisceration = disembowelling cannot commonly be done with vitriol = sulphuric acid...

      (I leave the matter of monkies in feisty knickers for the attention of another commentard.)

      1. This post has been deleted by its author

      2. jake Silver badge

        Re: My contribution...

        Those weren't feisty knickers, those were board shorts, and it wasn't the Monkees, it was the Beach Boys.

        Honestly, you British ...

      3. TSM

        Re: My contribution...

        > I would point out that evisceration = disembowelling cannot commonly be done with vitriol = sulphuric acid...

        I dont really see why not. I mean, yes, you'd have to be careful, and it would take a while. But you could get there in the end. Much sooner if you weren't worried about a bit of collateral damage, which if you're disembowelling someone is quite possibly the case.

    3. C R Mudgeon

      Re: My contribution...

      I wonder if it's even possible to express a concept like "extensible markup language" in hieroglyphics.

      1. jake Silver badge

        Re: My contribution...

        The beauty of hieroglyphics is that if a picture doesn't exist, even the illiterate can invent one. Thus we have emojis of poop. Joy.

        1. C R Mudgeon

          Re: My contribution...

          Fair point.

  7. Anonymous Coward
    Anonymous Coward

    Had the opposite problem

    We had a sysadmin who'd backup a folder with DB backups themselves to a locally attached drive via script. It was a secondary back up as these files were the crown jewels. It also only ran once a month.

    It worked fine for years until both she and I were off on leave, someone messed with the server and the drive letter of the target drive was changed to a network drive instead. This could only have been one of the other IT staff.. Anyway end result was a script frantically trying to copy a huge set of large files across a network that wasn't designed for it to a remote location with a PDC which was already low on space.

    The script would restart if it couldn't verify the files had copied successfully so it basically looped for 2 days, taking all of the remote sites offline.

  8. taxman
    Paris Hilton

    The Story of 'O'

    Doesn't quite follow the book or film that 'a friend of mine' has in their library.....

    Although being 'beaten into submission' does appear .....

  9. Version 1.0 Silver badge
    Joke

    "You've deleted my files!"

    PFY: No sir, I just emptied the Deleted Items folder as part of the system cleanup.

    PHB: But that's where I keep all my girlfriends pictures!

    PFY: Sorry sir, but I can't retrieve them.

    It's a joke these days but he was pissed when it happened.

    1. Throatwarbler Mangrove Silver badge
      Go

      Re: "You've deleted my files!"

      "No, son, you deleted your files. I just carried the bullet for a while."

    2. This post has been deleted by its author

  10. Eclectic Man Silver badge

    Outsourcing bid

    In a previous incarnation I was the security person on a bid to run the IT of an anonymous (as alway) Public Body. We went to the potential client's site and got a presentation with the other bidders' teams. The IT manager was asked how many incidents at each level they had in a typical week.

    The answer somewhat surprised me. In a typical week they had two level one incidents. That is, twice a week, the system was unusable by most staff and there was no workaround. Twice a week. Every week.

    After viewing the 'security' manual I hoped the staff were sensible enough to subvert and ignore it and do their work securely in spite of it.

    We ended up bidding for it and losing out to another well-known systems integrator / outsourcer. Who walked away after a month complaining that the client was impossible to work with.

    Some System Admin people make mistakes, some are just incompetent.

  11. Doctor Syntax Silver badge

    Makes a change from the Unix version.

  12. Imhotep Silver badge

    The Story Of O

    So O went into IT also? Interesting "Where are they now?" follow-up.

    1. Adrian 4 Silver badge

      Re: The Story Of O

      Seems a reasonable progression from her earlier career

  13. DS999 Silver badge
    WTF?

    Only an idiot

    Designs an automated file deleting script that doesn't output what it is deleting, and watch it work, if used on a production system.

  14. Tim99 Silver badge
    Facepalm

    I’ve done it.

    And so has everybody who has worked in IT for a few decades, particularly when one of the customer's senior manglement lusers insists on "doing it now".

    But my personal favourite was when I was asked to delete customer order data from the previous year in a live SQL database - I knew that a cascade delete would also run on the orderdetails table because I had designed the database. Obviously I checked that they had a backup (from the night before) and what was going to be deleted with a: SELECT COUNT(*) AS NumOrders FROM Orders WHERE OrderDate < "01/01/1999" - Then the phone rang as I was copying part of the line, just before I was about to paste it into a new command that I had started with DELETE FROM Orders - About fifteen minutes later I pasted what I thought I had stored in the paste buffer, just as the manglement luser came in and pestered. The paste added only the semicolon, but I still pressed [Enter] - The statement was therefore "DELETE FROM Orders;". Fortunately I was able to roll it back from the transaction log in spite of their system admin's new policy of saving storage space by truncating logs before they "got too big"...

    1. hoopsa

      Re: I’ve done it.

      The SQL Tool we used at work has an auto-commit function. I was often mocked for turning that off because 'I should have faith in my SQL'. But it always struck me as an accident waiting to happen and I preferred to commit as a separate step; that way I had the option of doing rollback as a separate step if something unfortunate had occurred.

      1. Down not across Silver badge

        Re: I’ve done it.

        It is always good to understand the RDBMS (or specific client) you are working with. Someone used to Oracle would be in for a shock if they had to work with for example Sybase where default is unchained mode. No rollback for you if you didn't explicitly begin transaction (or change chained mode for your session).

  15. Jou (Mxyzptlk)

    My temp cleanup...

    I made a (windows) temp cleanup tool quite a while ago. Deletes all but the 10 newest entries - some programs rely on things in TEMP during installation after a reboot.

    For known reasons I wrote it this way in the for /f loop:

    del /q "%TEMP%\..\temp\%%A"

    rd /s /q "%TEMP%\..\temp\%%A"

    Yes yes, trust environment variables.... NOT.

  16. petef

    Unix too

    In the early days of Unix on PCs (Interactive Unix, pre Linux) my team had 386 workstations. My colleague asked me to remove my user account from their machine to free up space I did that but left just a login with a home directory of root. That should have been that but the owner then decided to completely remove my account, blithely answering yes to questions such as remove home directory. The re-install involved a box of floppies.

  17. amanfromMars 1 Silver badge

    An AI Lionel's Roar ..... Quite Unlike Anything Ever Before

    Ever created what you thought was the neatest utility ever, only to realise that you have unleashed a data-destroying monster?

    Hmmm? And what of those others who would/could be able and emboldened to tell you the obverse/converse/reverse ...... they have created what they thought was a data-destroying monster only to realise what is unleashed too, is the neatest utility ever?

    If one has any sense at all, makes sure they be playing on your team, and for your side, for they can rightly be thought of as a Messi Equivalence.

  18. Anonymous Coward
    Anonymous Coward

    netbackup cathedral

    A while ago, I was on a Netbackup update to a newer solution.

    The situation was this:

    - dude was in charge for the last 15 years, alone and unmanaged

    - no documentation to speak of

    - he was using a method which is a NO-NO: the pre and post scripts installed on EVERY 300 clients

    - those pre post scripts were gigantic 5000+ shell lines, hostnames hardcoded, therefore a new system = update of the script

    - basically, the backup was triggered by Netbackup, who was doing only scheduling and receiving the data. All the rest was handled client side

    - he never used any Netbackup plugin AT ALL

    - he was doing insane things like shutting down DBs if a secret naming convention was used in the policy (remember scripts are on servers and policy is on Netbackup). So in case you setup the wrong naming convention, shit happens.

    Of course the dude left and our project was to cleanup this huge mess and upgrade Netbackup. Of cause, the guys knew an update in this state would not be possible without breaking everything. He even told me "good luck on his last day ! Nice."

    It took time but we finally did it. To hell with the cathedral !

  19. Anonymous Coward
    Anonymous Coward

    IFS

    Once got blamed by the client for half trashing a system. Detailed examination of the suspected script which I had not written, revealed that a certain programmer had used the Unix internal Field Separator reserved word environment variable IFS, thinking that he could shorten Image File Server to the same moniker. A nghtly clear down consisting of rm -r /$IFS wiped out a lot of the server before it was discovered. The programmer who had been given root access for emergency overnight support purposes then tried to prevent accidental over-writing of important data by a restore job by renaming tar to tar.dont. I think they took away his root access after that. Trying to panic hack failed jobs at 3am against a deadline causes chaos the day after.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021