back to article Funny, that: Handy script for wiping directories is capable of wreaking havoc beyond a miscreant's wildest dreams

Remember when September seemed so far away? Those of you still working from your bedroom since March should probably have changed your pyjamas by now. We'll wait. When you're ready, enjoy a tale from the Who, Me? vault courtesy of a reader who knows all about unplanned undergarment changes. "O" spent the early part of this …

  1. jake Silver badge

    ElReg asks ...

    ... "Remember when September seemed so far away?"

    Yeah, but then I'm old ... It's been September for nearly 10,000 days.

    Today is Monday, September the 9862nd 1993.

    1. fredds

      Re: ElReg asks ...

      Today is Pungenday, the 24th day of Bureaucracy in the YOLD 3186

    2. Rich 11

      Re: ElReg asks ...

      I'm older. Today is clock-tick 18,523,814,400.

  2. UCAP Silver badge

    My contribution ...

    Looking after my company's collection of Sun 3 workstations 30 years ago. Someone had been doing some development work that had resulted in /tmp filling up, with all of the assorted problems this gives. So one day when I came in (I was often the first in the office in those days) I logged on to the server using the root account and issued the immortal command rm -rf / tmp.

    Note significant space.

    It took me about 2 seconds to realise what I had done and hit <cntrl-C>, but by then most of /usr was toast.

    'fessed up to my manager and spent the rest of the morning re-installing from scratch. Fortunately I had done a backup just before the screw-up, so I could restore everyone's user directories with no loss.

    1. jake Silver badge
      Pint

      Re: My contribution ...

      Cheer up. I don't know of a single long-term un*x admin who won't (eventually, it might take a couple beers) admit to have done the same thing. Sometimes a couple of times.

      Always at the worst possible time, of course.

      1. Steve Davies 3 Silver badge
        Pirate

        Re: My contribution ...

        Yep been there done that.

        Two lessons were learned.

        1) before doing any rm -fr do a pwd just in case.

        2) become very anal about doing total system backups before any upgrade

        What can go wring will go wrong!

        1. A.P. Veening Silver badge
          Pint

          Re: My contribution ...

          What can go wring will go wrong!

          ROFLMAO! Have a ===>

          1. Anonymous Coward
            Coat

            Re: My contribution ...

            *wring* *wring*

            "Hello?"

            "It's gone what now?"

            1. A.P. Veening Silver badge

              Re: My contribution ...

              Do you mean Ring Ring?

              1. herman

                Re: My contribution ...

                Oh, god. The worst thing is that we thought the hair and clothes of the 70s were actually sexy...

          2. Efer Brick

            Re: My contribution ...

            As in wringing hands is despair?

            Lovely Freudian slip (where you say one thing and mean a mother)

        2. This post has been deleted by its author

          1. wyatt

            Re: My contribution ...

            Yep, this could have saved me a lot of trouble one night shift!

        3. William Towle
          Linux

          Re: My contribution ...

          > 1) before doing any rm -fr do a pwd just in case.

          > 2) become very anal about doing total system backups before any upgrade

          My personal favourite is to filter what goes into the history on systems where I tend to use the ^R search to repeat commands a lot.

          A few things such as a bare 'ls', 'df', or 'du' serve little useful purpose when regurgitated, and it becomes *very* difficult for a typoed search term to pull out a destructive `rm ...` when there aren't any there.

      2. fidodogbreath

        Re: My contribution ...

        Always at the worst possible time, of course.

        Is there a non-worst time to do that on a production server?

        1. jake Silver badge

          Re: My contribution ...

          ""Is there a non-worst time to do that on a production server?"

          Yes. During scheduled shutdown/maintenance windows come to mind. Immediately after a full system backup, of course.

          One that happened to a friend down at IBM Almaden ... Running late to get out the door (baseball game was due to start), he accidentally entered a similar command at approximately 5:04 PM local time on the 17th of October, 1989. About one millisecond later he realized what he had done. About one microsecond after that, the Magnitude 6.9 Loma Prieta earthquake hit, with the epicenter approximately 14 miles to the South South West.

          The SCSI drive, which, in his words, "was happily losing it's tiny little mind, and destroying mine alongside it" suffered a hard crash before the power went out. Seems that even high-end SCSI drives don't like imitating a pogo stick when the heads are moving around. DriveSavers in Marin managed to salvage most of the drive, thus saving a high-temp superconductor project over a year of data. Drivesavers didn't volunteer that the command had been run, so he didn't lose his job ... but his entire department got yelled at for not having a proper off-site backup strategy in place.

          1. This post has been deleted by its author

            1. Anonymous Coward
              Anonymous Coward

              Re: My contribution ...

              >"proper off-site backup"

              >And there's the words of wisdom to live by. What's more, use a differential backup, e.g. rdiff-backup, then you can rewind back to the moment before your files were encrypted by the ransomware.

              On a training course probably around 20 years ago in the centre of Manchester. During one of the lunches, it was mentioned about the city centre changing significantly and all the rebuilding work since the 1996 bombing and it turns out that the bomb warning gave the admin enough time to dial in and copy as much data as possible to the servers in other sites before it all went boom - including their original Manchester base.

            2. tiggity Silver badge

              Re: My contribution ...

              Proper off site backup strategies ....Which always depends on those in charges agreeing to it instead of saying too expensive....

          2. Anonymous Coward
            Anonymous Coward

            Re: My contribution ...

            I never understood the logic of firing someone who makes a mistake. If they repeatedly make the same mistake, sure, but all you're basically doing otherwise is replacing them with someone who hasn't learned not to make that mistake yet.

            And if the rest of the company doesn't put procedures in place to mitigate that mistake from happening in the future then nobody has learned anything and you're back to square one.

            1. veti Silver badge

              Re: My contribution ...

              If the perp doesn't fess up, how can the company know exactly what the mistake was?

              Making a mistake shouldn't be grounds for sacking, but keeping quiet about it should.

            2. This post has been deleted by its author

              1. fajensen
                Angel

                Re: My contribution ...

                It's always good to see someone having a shittier day! or Cats!!

          3. Anonymous Coward
            Anonymous Coward

            Re: My contribution ...

            Greatest excuse ever - "It was the earthquake that did it!"

            1. A.P. Veening Silver badge

              Re: My contribution ...

              How many people can say they were saved by an earthquake?

            2. Rich 11

              Re: My contribution ...

              "The god ate my homework."

          4. Cliffwilliams44 Silver badge

            Re: My contribution ...

            "proper off-site backup"

            That management had refused to pay for for years until after the incident I am sure.

      3. Anonymous Coward Silver badge
        Facepalm

        Re: My contribution ...

        "I don't know of a single long-term un*x admin who won't ... admit to have done the same thing"

        I have never done that - too many cautious tales from others (it's good to learn from other people's mistakes)

        I have however done a dd in the wrong direction.

      4. my cats breath smells like cat food

        Re: My contribution ...

        My contribution to the hall of shame involved accidentally recursively deleting /usr/bin, or most of it at least. We were a poor university in the 90s so only backups of user directories. Luckily the system was one of 4 HP 700 series running HP-UX we had scattered around the department so I managed to restore from one of the other boxen and all was good again. Pretty sure everything was restored, no one ever complained at least.

        It's amazing how sudo makes one think about what one is about to do, most days.

      5. Anonymous Coward
        Anonymous Coward

        Re: My contribution ...

        Oddly enough, I've never actually done the "rm / dirname" thing. I have, however, more than once, done "DELETE FROM table;" and forgotten the where clause. #confessions

    2. Doctor Syntax Silver badge

      Re: My contribution ...

      "most of /usr was toast.... everyone's user directories"

      The halcyon days when /usr was for user directories instead of being filled up with stuff that should be in /bin, /lib and the like!

      1. Jou (Mxyzptlk) Silver badge

        Re: My contribution ...

        I chrome that!

      2. UCAP Silver badge

        Re: My contribution ...

        Who said that user's directories were on /usr? They were actually on /user.

        However Sun's lovely installation system (booting the workstation from tape) did not give you the option to preserver existing partitions and the contents thereof; instead it worked on a simple "nuke-it-from-orbit" basis. Hence having to reinstall the OS meant that the user directories would have been automatically scrubbed.

        1. jake Silver badge

          Re: My contribution ...

          In the very early days (1970s), the user's home directories were on /usr ... Economy of typing further shortened this to /u on very early BSD. At some point, primarily due to splitting the filesystem over multiple disks, the user home directories were moved to /user, leaving /usr for shared read-only executables and their attendant tat.

          /user became /home much later ... and just to confuse things, MacOS (which is a bastardized BSD) uses /Users ...

          Yes, Sun's "stock" installation media wasn't exactly friendly. However, being based on BSD it was fairly easy to make your own to suit yourself. You could boot from tape, CD or floppy ... depending on your system ROM, of course. And again, changing the ROM wasn't really all that difficult. Personally, I think Sun's biggest sin in the early days was shipping each and every system with the same default root password ... They didn't even suggest changing it in the installation guides! And this for a machine that was designed from the ground up to connect to TehIntraWebTubes ... The 'N" in Sun stood for network.

    3. Anonymous Coward
      Anonymous Coward

      Re: My contribution ...

      been there done that

      restart in 10

      do my change

      check my change

      get distracted by someone.

      $%^& that switch i was just working on has gone off line, 3 mins its back minus my changes & the help desk ignores the monitoring tool showing my mistake and starts trying to understand why some people in the corner of 1 floor of some office are complaining their pc's and phones don't work, swiftly followed by it's all back now. I then have to do my change again before fessing up i caused the outage.

      One thing Juniper does much better, commit confirmed with roll back not needing a reboot.

      1. Jou (Mxyzptlk) Silver badge

        Re: My contribution ...

        > get distracted by someone

        That is, quite often, the biggest problem.

        1. The Oncoming Scorn Silver badge
          Pint

          Re: My contribution ...

          Hence my preference of automating tasks from batch files & scripts.

        2. Olivier2553

          Re: My contribution ...

          I am usually firm enough with my tone of voice that when I say that I am busy and come back in half an hour, people leave me to what I am doing.

          And when they come back, I explain to them what I was doing and why I could not be interrupted.

          And I found out that just saying "not now, come back in 30" does not distract me nor make me lose track of what I am doing. Probably because I did not give them time to present their request.

          1. Outski

            Re: My contribution ...

            I try to do the same, and with colleagues, it tends to work. Unfortunately, these days, my "office" colleagues are my family, and the tone of voice required doesn't go down so well with MrsO.

            1. jake Silver badge

              Re: My contribution ...

              I've got two desks up here in the office ... One is for Ranch business, the other is my Consulting business. Everybody knows to leave me alone when I'm sitting at the Consulting desk.

              Household business takes place down in the kitchen.

      2. Anonymous Coward
        Anonymous Coward

        Re: My contribution ...

        Much better than skipping the "reload in 10", getting distracted by someone, and boring a router that's a 3 hour drive away.

        ...or so I've heard.

        More offices need a convention like a sign, or a glowing red light to indicate "I'm busy doing something that requires full concentration, and screwing up can cost the company a shitton of money, so don't bother me unless the building is on fire".

        A graduated scale could work too, indicating what level of disturbance is ok at the moment. Levels could include:

        * I forgot how to print on A4 instead of letter (reverse per local custom)

        * can you reset my password?

        * where we going for lunch?

        * the boss's computer locked up and he has a presentation in 10 minutes.

        * building's on fire!

        * where are we going for drinks after work? (Essentially a NMI).

        1. veti Silver badge

          Re: My contribution ...

          Whenever such systems have been tried, they are roundly and routinely abused by basically everyone. There's the poser who claims to be mission-critically busy for weeks on end, and the chancer who sets that status when they want some, ahem, uninterrupted Internet time.

          And, of course, the huge majority of people who completely forget/ignore the whole thing - who neither set their own status, nor pay a smidgen of attention to anyone else's.

      3. wyatt

        Re: My contribution ...

        I have memories of Nortel Passports where if you didn't commit a change and it rolled it back, it'd drop all of it's ATM connections. Always interesting to see, it Christmas tree'd whilst doing it.

    4. Gerhard den Hollander

      Re: My contribution ...

      Had a variation of that happen to me ages ago.

      cd /tmp

      rm -rf *

      Noticed some files refused to go

      Realised those needed superuser permissions

      su -

      rm -rf *

      Realised that this was taking rather longer than expected

      Noticed the - after the su....

      Realised the backup tape was fortunately still in the tapedrive but ejected, which indicates last night's backup had run.....

      Managed to stop the Rm in time to keep the system alive enough I could restore from tape

    5. The Bloke next door

      Re: My contribution ...

      For a moment there, I thought I knew you....

      When I first joined Sun in '87 the sales database in the local offices was Ingres. During a demonstration of how to run a task on a remote machine via the new "on" command, the office became noticeable busier with increasingly raised voices.

      Upon asking what the issue was, it transpired that the Ingres database had disappeared, cue some detective work. It appeared that "on" mounted the whole directory containing the programme being run on the remote machine under /tmp. Read /write too..by anyone.

      At the remote site, fed up with people putting files in /tmp, a colleague had a cron job that periodically scanned /tmp and removed extraneous files (and directories).

      Hindsight is a wonderful gift.

  3. DJV Silver badge

    capable hands of Windows Server 2003

    Yeah, I've seen those "capable hands" running at a small software house where I was temping back in 2005. I don't know who set up the server but they had to reboot it at least twice a week because it would just curl up and die for no logical reason. What was worse was that it had done so since it had first been installed. No one ever investigated any further - they'd just reboot it without question.

    Near the end of that temping stint (about 5 weeks), they did offer me a full time job there. Having been there long enough to see first hand how the way the place was (mis)managed - and I don't just mean the IT side of it - I politely declined. They went bust a couple of years later. No surprise.

    1. Blackjack Silver badge

      Re: capable hands of Windows Server 2003

      Wasn't in the 2000s when it was discovered or at least rumored that Microsoft used Linux for their own servers? I remember that being a thing people said a lot back then but I do not know if it was true.

      1. MacroRodent
        Devil

        Re: capable hands of Windows Server 2003

        Maybe you are remembering the time Microsoft bought the Hotmail service around that time. It ran on FreeBSD, and continued to run that way for a long while even after the purchase, which was a bit embarrassing for MS, when this tidbit leaked out, as it was in the middle of the "Open Source is Cancer" era of Microsoft. (demon icon for FreeBSD, but the real one looks much friendlier.)

        1. A.P. Veening Silver badge

          Re: capable hands of Windows Server 2003

          The real beauty there is the attempt by Microsoft to replace their AS/400s with NT servers.

      2. Pirate Dave Silver badge
        Pirate

        Re: capable hands of Windows Server 2003

        If memory serves, at one point Microsoft used Xenix to write the final "gold" copy of their Windows install disks before sending them off to the duplicator house. It was ironic that even Microsoft couldn't guarantee their internal Windows machines didn't have latent virii.

        1. Blackjack Silver badge

          Re: capable hands of Windows Server 2003

          That was most likely to avoid another scandal like the Microsoft Plus CDs that had viruses on them.

          While I can't find the old news online, apparently a factory in Mexico didn't have an antivirus installed because "Windows 95 doesn't get viruses".

        2. C R Mudgeon Bronze badge

          Viruses and duplicators

          "It was ironic that even Microsoft couldn't guarantee their internal Windows machines didn't have latent virii."

          At a small but high-profile Amiga developer I once worked at, one release's master (floppy) disk was given a virus *by* the duplication house. As I recall, part of their QA was to compare a sample output copy against the original master -- for which they used an infected Amiga. I no longer recall whether the whole run was bad (vs. just those two diskettes), and if it was, whether any of them made it out into the wild.

          Somehow, I then became the mastering guy for all of our products, not just the one I was assigned to. I presume my level of outrage at the series of screwups that led to the above debacle is what landed me the task. My checklist included such items as: make N copies of the master, and keep some of them back; do a comparison ourselves; do it byte-wise, not merely a recursive file comparison -- I wrote a utility for that. Last but far from least: flip the $&(! write-protect tab before you send the master off to the duplicators.

          1. jake Silver badge

            Re: Viruses and duplicators

            Mass distribution of floppies infected with a virus happened several times over the years. There used to be a web site listing them, but today's useless search engines won't tell me anything about the history of viruses, they will only tell me where I can purchase an AV product for Windows. How fucking useless is that?

          2. Anonymous Coward
            Anonymous Coward

            Re: Viruses and duplicators

            Flip it? No, sir - you get a pair of cutters or similar and you remove that tab! Never write to that disk again!!

            Well, you can - just need to tape the hole over or something

        3. jake Silver badge

          Re: capable hands of Windows Server 2003

          To be fair, in the early days, CD mastering and pre-mastering software was fairly esoteric, and ran on proprietary hardware. This hardware usually ran whatever OS the lead engineer used/preferred at University ... AIX, HPUX, SunOS, BSD, and yes, Xenix.

          But WinDOS? Not so much. It was nowhere near stable enough for such critical tasks. (NT didn't exist yet, at least not in the early days.)

          1. Blackjack Silver badge

            Re: capable hands of Windows Server 2003

            Authoring tools to create master CDs that later were used to make the mass production copies, ran on Windows and DOS so that's how a CD could get infected by a Windows or DOS virus despite the other systems being different.

            The hilarious thing is that F-Prot for DOS was free and worked until Windows 98 SE to detect and erase viruses for Windows and DOS. And there was also Norton Antivirus Scanner that at least let you detect the things and was free to use and that one ran in Windows.

    2. Rob Daglish

      Re: capable hands of Windows Server 2003

      I'm sure I remember being told in the late 90s/early 2000s that Novell had offered to rebuild 1/4 of the MS internet servers with Intranetware and still run a better service than MS could manage...

  4. Anonymous Coward
    Anonymous Coward

    We've all been there

    We had a QA group that did something similar, a long test job that ran for hours, and cleaned up with something like "rm -rf $(LOGDIR}/".

    Inevitably some unexpected failure (is there any other sort?) in the test job resulted in this being run with LOGDIR unset. As a user in the same GID group as the staff. On a lab system which had the default automounter config which NFS-mounted all the user home directories, many of which seemed to have 775 permissions...

    The sysadmin finally twigged when he realised that the calls from people saying "some of my files have disappeared" were coming in in alphabetical order of username. A hasty shutdown of the home directory NFS system was followed by some forensic network access to find the guilty system.

    Fortunately the overnight backups & regular ZFS snapshots meant that the QA team responsible got away with an apology, and buying a few beers.

    1. John Brown (no body) Silver badge
      Thumb Up

      Re: We've all been there

      "the QA team responsible"

      Oh, the irony!

      1. jake Silver badge

        Re: We've all been there

        But wait. It gets better. Today's DevOps advocates are using stories like this to justify getting rid of QA entirely ...

    2. C R Mudgeon Bronze badge

      Re: We've all been there

      "... some unexpected failure (is there any other sort?) ..."

      Yeah. An expected failure is one that the code guards against. Case in point:

      if [ "x$LOGDIR" != "x" ]; ...

  5. Doctor_Wibble
    Unhappy

    Are we including sabotage?

    Of a completely unfortunate accidental sort of course.

    A departing employee completely accidentally left a floppy disk in their laptop, which was set as the boot drive and which by complete accident had its autoexec.bat consisting of just one command that would delete the contents of the C: drive if it was completely accidentally booted from.

    The great thing about booting from a floppy is that it is the sort of thing that you can hear and stop with a magic word and a mystical gesture before it does any of that completely accidental damage for which the git knew full well I would be the one blamed.

    .

    On the confessional front, have I deleted stuff I didn't mean to delete? Yep. On the other hand I still feel the stinging lesson of a disk failure that happened the day before the dvd blanks arrived upon which the much put-off backup of said disk was to be stuck.

    1. Boris the Cockroach Silver badge
      Unhappy

      Re: Are we including sabotage?

      Quote

      Yep. On the other hand I still feel the stinging lesson of a disk failure that happened the day before the dvd blanks arrived upon which the much put-off backup of said disk was to be stuck.

      OH fudge.. I knew there was something I had to do friday before beer time.....

    2. Anonymous Coward
      Anonymous Coward

      Re: Are we including sabotage?

      Departing employee should have included a line to re-write the autoexec.bat, or otherwise hide the evidence.

      I hope your company sent the disk to him with his last paycheck. I'd have included a note that the (completely accidental) effects of the disk resulted in the deletion of a perfectly glowing letter of recommendation for anyone enquiring as to his employment history.

      1. The Oncoming Scorn Silver badge
        Pint

        Re: Are we including sabotage?

        Del %0 always worked for me.

  6. Dan 55 Silver badge

    I'm writing a backup/restore script now, and completely paranoid that the backup won't backup or the restore won't restore or the person using it won't see the error message or someone will try to run it as a cron job with half the environment variables missing or something. There are more ifs with quotes and $? than there are lines that actually do stuff.

    1. Flocke Kroes Silver badge

      Have you considered...

      #! /bin/bash -xe

      # Prints each line of what will be executed after expansion but before execution

      # Exit on any error outside a condition

      # -x -e will not work: Everything after the second space is considered a single word

  7. Edwin

    There, but for the grace of God,

    go I...

    1. jake Silver badge

      Re: There, but for the grace of God,

      There is no god. You'll join the ranks eventually.

      1. Anonymous Coward
        Anonymous Coward

        Re: There, but for the grace of God,

        Oh, there is, but his grace for careless sysadmins is not boundless. In time thy faith shall be tested!

        (All those bearded guys in the Bible, at least some of them just *had* to be programmers, right?)

        1. A.P. Veening Silver badge

          Re: There, but for the grace of God,

          Definitely, look at those ten commandments, bashed into stone ;)

        2. DJO Silver badge

          Re: There, but for the grace of God,

          (All those bearded guys in the Bible, at least some of them just *had* to be programmers, right?)

          You're not the first to think that:

          Bearded Sky Man as a single parent and hobby programmer

  8. imanidiot Silver badge

    Philosophical questions

    If data is deleted, but no one is around to notice, was anything lost?

    I've sort of nearly done the same thing once, but only because someone was using a script they shouldn't have nor use.

  9. Anonymous Coward
    Anonymous Coward

    My contribution...

    I was doing a temp stint at a place that shall remain nameless to protect them from the hoardes of angry IT folks that would surely set fire to the HQ if said name were made known. Suffice it to say it was one of those huge juggernaughts that we all love to eviscerate with vitriol at every opportunity.

    I was in a server room tending to an old cluster that needed some TLC. Archaic disks that hadn't been backed up in aeons, software so old it had probably been given first drafts on clay tablets by Egyptian clerks wondering how to spell all the buzzword bingo bullshit, managed by monkies in feisty knickers.

    I had verified that the server was no longer connected to the internal network by order of the manager I had been assigned to. I made sure that it wasn't running any jobs that hadn't been marked as non critical, temporary, or otherwise able to be sacrificed without need for panick. At which point I start searching the disks for where the largest concentration of files (sizes, numbers, etc) were to be found.

    Imagine my surprise when the largest by an exponential margin turned out to be a personal directory full of porn. I dutifully made an offline copy for *cough* Reasons and began backing up the entire system to the specific NAS unit dedicated for just that cluster & purpose.

    I get done, detach the NAS, lock it in the drawer the manager indicated, and began cleaning up/out said server for repurposing to other tasks.

    I'm about halfway through when some guy I don't know barges into the server room in a wide-eyed frizzy-haired state & heads straight for me like a laser beam.

    "What have you done to my server?!" he roars as if he were Zeus & I'm about to get smote with lightning.

    I explain what I'd been brought in to do, show him the paperwork from my manager giving me authorization to do it, & explain that I've just given the machine a fresh, legally licensed copy of the OS (complete with drivers) to prep it for reuse elsewhere.

    "You can't DO that! That's MY server!" he roars again.

    Unimpressed I show him the paperwork that expressly says I most certainly can & have $Manager's orders to do so.

    Back & forth, back & forth, him roaring, me not giving a shit. I'd made *damn* sure I was on the machine $Manager specified to prevent me from fucking anything else up, so to have a different manager berating me for doing my job leaves me wanting to smack him upside the head with a NAS.

    He shouts he'll talk to $Manager, I nod & say to go ahead, & while he's off to go have some more shouting, I'll finish the job I want to get paid for.

    Turned out that Old Yeller had been some up-and-comer nepotistic bugger that had hit his Peter Principle limit. He was used to coming in to work, vanishing into his office, & surfing porn all day. Actual work? Don't make him laugh.

    My having taken down his personal porn server was seriously putting a crimp in his pseudo-productivity & he was Having Words with $Manager about why said interference Was Not Allowed.

    Except Old Yeller really should have talked to whomever he'd been related to first. Because that person no longer worked at the company, upper management was sick & fekkin' tired of the useless dolt, & this had been their shot across the bow to get him to actually DO something for a change.

    $Manager showed their paperwork authorizing said work & sent Old Yeller up the ladder. At some point Old Yeller was told to stuff it.

    $Manager thanked me for not giving in to the idiot, for having had the forethought to keep all the paperwork I'd needed to deflect said idiots anger, and for giving him back seriously needed resources.

    I was quite pleased with the trip to the pub for lunch & a pint on $Manager.

    I was even MORE pleased with the fat brown envelope he offered for his own copy of what I'd found on the machine.

    "It's all on that NAS you locked in the cabinet." made him grin like a shark swimming through a cloud of fresh chum.

    I was even MORE pleased when I uploaded the entire trove to my various torrent accounts (Demonoid FTW!) & watched my street cred go through the roof.

    Ahhhhhh... fun times!

    1. A.P. Veening Silver badge

      Re: My contribution...

      You sure earned your BOFH papers with that one and lucky to have a BMFH backing you.

    2. Anonymous IV
      Headmaster

      Re: My contribution...

      > one of those huge juggernauts that we all love to eviscerate with vitriol at every opportunity

      I'm all for colourful phraseology, but in the interests of factual accuracy I would point out that evisceration = disembowelling cannot commonly be done with vitriol = sulphuric acid...

      (I leave the matter of monkies in feisty knickers for the attention of another commentard.)

      1. This post has been deleted by its author

      2. jake Silver badge

        Re: My contribution...

        Those weren't feisty knickers, those were board shorts, and it wasn't the Monkees, it was the Beach Boys.

        Honestly, you British ...

      3. TSM

        Re: My contribution...

        > I would point out that evisceration = disembowelling cannot commonly be done with vitriol = sulphuric acid...

        I dont really see why not. I mean, yes, you'd have to be careful, and it would take a while. But you could get there in the end. Much sooner if you weren't worried about a bit of collateral damage, which if you're disembowelling someone is quite possibly the case.

    3. C R Mudgeon Bronze badge

      Re: My contribution...

      I wonder if it's even possible to express a concept like "extensible markup language" in hieroglyphics.

      1. jake Silver badge

        Re: My contribution...

        The beauty of hieroglyphics is that if a picture doesn't exist, even the illiterate can invent one. Thus we have emojis of poop. Joy.

        1. C R Mudgeon Bronze badge

          Re: My contribution...

          Fair point.

  10. Anonymous Coward
    Anonymous Coward

    Had the opposite problem

    We had a sysadmin who'd backup a folder with DB backups themselves to a locally attached drive via script. It was a secondary back up as these files were the crown jewels. It also only ran once a month.

    It worked fine for years until both she and I were off on leave, someone messed with the server and the drive letter of the target drive was changed to a network drive instead. This could only have been one of the other IT staff.. Anyway end result was a script frantically trying to copy a huge set of large files across a network that wasn't designed for it to a remote location with a PDC which was already low on space.

    The script would restart if it couldn't verify the files had copied successfully so it basically looped for 2 days, taking all of the remote sites offline.

  11. taxman
    Paris Hilton

    The Story of 'O'

    Doesn't quite follow the book or film that 'a friend of mine' has in their library.....

    Although being 'beaten into submission' does appear .....

  12. Version 1.0 Silver badge
    Joke

    "You've deleted my files!"

    PFY: No sir, I just emptied the Deleted Items folder as part of the system cleanup.

    PHB: But that's where I keep all my girlfriends pictures!

    PFY: Sorry sir, but I can't retrieve them.

    It's a joke these days but he was pissed when it happened.

    1. Throatwarbler Mangrove Silver badge
      Go

      Re: "You've deleted my files!"

      "No, son, you deleted your files. I just carried the bullet for a while."

    2. This post has been deleted by its author

  13. Eclectic Man Silver badge

    Outsourcing bid

    In a previous incarnation I was the security person on a bid to run the IT of an anonymous (as alway) Public Body. We went to the potential client's site and got a presentation with the other bidders' teams. The IT manager was asked how many incidents at each level they had in a typical week.

    The answer somewhat surprised me. In a typical week they had two level one incidents. That is, twice a week, the system was unusable by most staff and there was no workaround. Twice a week. Every week.

    After viewing the 'security' manual I hoped the staff were sensible enough to subvert and ignore it and do their work securely in spite of it.

    We ended up bidding for it and losing out to another well-known systems integrator / outsourcer. Who walked away after a month complaining that the client was impossible to work with.

    Some System Admin people make mistakes, some are just incompetent.

  14. Doctor Syntax Silver badge

    Makes a change from the Unix version.

  15. Imhotep

    The Story Of O

    So O went into IT also? Interesting "Where are they now?" follow-up.

    1. Adrian 4

      Re: The Story Of O

      Seems a reasonable progression from her earlier career

  16. DS999 Silver badge
    WTF?

    Only an idiot

    Designs an automated file deleting script that doesn't output what it is deleting, and watch it work, if used on a production system.

  17. Tim99 Silver badge
    Facepalm

    I’ve done it.

    And so has everybody who has worked in IT for a few decades, particularly when one of the customer's senior manglement lusers insists on "doing it now".

    But my personal favourite was when I was asked to delete customer order data from the previous year in a live SQL database - I knew that a cascade delete would also run on the orderdetails table because I had designed the database. Obviously I checked that they had a backup (from the night before) and what was going to be deleted with a: SELECT COUNT(*) AS NumOrders FROM Orders WHERE OrderDate < "01/01/1999" - Then the phone rang as I was copying part of the line, just before I was about to paste it into a new command that I had started with DELETE FROM Orders - About fifteen minutes later I pasted what I thought I had stored in the paste buffer, just as the manglement luser came in and pestered. The paste added only the semicolon, but I still pressed [Enter] - The statement was therefore "DELETE FROM Orders;". Fortunately I was able to roll it back from the transaction log in spite of their system admin's new policy of saving storage space by truncating logs before they "got too big"...

    1. hoopsa

      Re: I’ve done it.

      The SQL Tool we used at work has an auto-commit function. I was often mocked for turning that off because 'I should have faith in my SQL'. But it always struck me as an accident waiting to happen and I preferred to commit as a separate step; that way I had the option of doing rollback as a separate step if something unfortunate had occurred.

      1. Down not across

        Re: I’ve done it.

        It is always good to understand the RDBMS (or specific client) you are working with. Someone used to Oracle would be in for a shock if they had to work with for example Sybase where default is unchained mode. No rollback for you if you didn't explicitly begin transaction (or change chained mode for your session).

  18. Jou (Mxyzptlk) Silver badge

    My temp cleanup...

    I made a (windows) temp cleanup tool quite a while ago. Deletes all but the 10 newest entries - some programs rely on things in TEMP during installation after a reboot.

    For known reasons I wrote it this way in the for /f loop:

    del /q "%TEMP%\..\temp\%%A"

    rd /s /q "%TEMP%\..\temp\%%A"

    Yes yes, trust environment variables.... NOT.

  19. petef

    Unix too

    In the early days of Unix on PCs (Interactive Unix, pre Linux) my team had 386 workstations. My colleague asked me to remove my user account from their machine to free up space I did that but left just a login with a home directory of root. That should have been that but the owner then decided to completely remove my account, blithely answering yes to questions such as remove home directory. The re-install involved a box of floppies.

  20. amanfromMars 1 Silver badge

    An AI Lionel's Roar ..... Quite Unlike Anything Ever Before

    Ever created what you thought was the neatest utility ever, only to realise that you have unleashed a data-destroying monster?

    Hmmm? And what of those others who would/could be able and emboldened to tell you the obverse/converse/reverse ...... they have created what they thought was a data-destroying monster only to realise what is unleashed too, is the neatest utility ever?

    If one has any sense at all, makes sure they be playing on your team, and for your side, for they can rightly be thought of as a Messi Equivalence.

  21. Anonymous Coward
    Anonymous Coward

    netbackup cathedral

    A while ago, I was on a Netbackup update to a newer solution.

    The situation was this:

    - dude was in charge for the last 15 years, alone and unmanaged

    - no documentation to speak of

    - he was using a method which is a NO-NO: the pre and post scripts installed on EVERY 300 clients

    - those pre post scripts were gigantic 5000+ shell lines, hostnames hardcoded, therefore a new system = update of the script

    - basically, the backup was triggered by Netbackup, who was doing only scheduling and receiving the data. All the rest was handled client side

    - he never used any Netbackup plugin AT ALL

    - he was doing insane things like shutting down DBs if a secret naming convention was used in the policy (remember scripts are on servers and policy is on Netbackup). So in case you setup the wrong naming convention, shit happens.

    Of course the dude left and our project was to cleanup this huge mess and upgrade Netbackup. Of cause, the guys knew an update in this state would not be possible without breaking everything. He even told me "good luck on his last day ! Nice."

    It took time but we finally did it. To hell with the cathedral !

  22. Anonymous Coward
    Anonymous Coward

    IFS

    Once got blamed by the client for half trashing a system. Detailed examination of the suspected script which I had not written, revealed that a certain programmer had used the Unix internal Field Separator reserved word environment variable IFS, thinking that he could shorten Image File Server to the same moniker. A nghtly clear down consisting of rm -r /$IFS wiped out a lot of the server before it was discovered. The programmer who had been given root access for emergency overnight support purposes then tried to prevent accidental over-writing of important data by a restore job by renaming tar to tar.dont. I think they took away his root access after that. Trying to panic hack failed jobs at 3am against a deadline causes chaos the day after.

  23. Anonymous Coward
    Childcatcher

    The brute force and ignorance approach

    This didn't happen to me, but I know the person it happened to rather well. I'm not sure quite what went wrong, but the end result was that a script got run in the process of a backup which at some point did something like rm -rf /$DIR_TO_REMOVE and, of course, DIR_TO_REMOVE was not set, and the -u option was probably not thought of (and probably did not exist in the shell then).

    This happened on what I am sure was an 11/750 running BSD 4.2 or 4.3 with one or more Fuji Eagles. These were ... quite slow machines. And the person concerned was in the machine room next to the machine, because that's where you had to be to change tapes during backups.

    With great presence of mind they realised what they needed to do: pull the power from the machine or, more precisely, the disk, immediately (it may be that in fact all they did was toggle the write-protect switch on the drive, but I like to imagine the fantastic noise old machines made when they lost power and the drives spun down), so that most of the actual writes to the disk did not happen.

    Of course the machine would not boot and there was then a saga involving working out how to cold-boot an 11/750, which I am fairly sure involved understanding how to get it to come up off its DECTape (TU58) drive and then getting enough of BSD to run (from tape? or a spare drive? not sure). But once that was done, of course almost everything was still there on the disk: except that (after suitable incantations of fsck and mucking around with clri & fsdb) a lot of files had lost their names and now lived in /lost+found in the way that important files usually do.

    All this was done from a paper console. I have to confess that when I've used a paper console I've been glad that vi would fall-back to open mode (ie to ex): the person who did this despised, and still does despise, vi and will use nothing but ed (and, of course, emacs, I mean, everyone uses emacs, right?): this attitude is probably why they succeeded in putting the machine back together at all: it's rather unlikely that something as vast and overcomplicated as vi would have been present on whatever minimal Unix they got up to recover the system.

    1. Olivier2553

      Re: The brute force and ignorance approach

      Knowing a bit of ed is always useful, even if once in a decade.

      Plus there is not much to know anyway, it pretty much consists in 5 commands, it will not clutter your memory.

    2. Down not across

      Re: The brute force and ignorance approach

      and will use nothing but ed (and, of course, emacs, I mean, everyone uses emacs, right?): this attitude is probably why they succeeded in putting the machine back together at all: it's rather unlikely that something as vast and overcomplicated as vi would have been present on whatever minimal Unix they got up to recover the system.

      Erm... I wouldn't put emacs and minimal in the same sentence.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like