back to article Techie basks in praise for restoring workforce email (by stopping his scripting sh!tshow)

Ho ho ho! It may be Christmas Eve but Who, Me? stops for no festive season. This time, our weekly reader's column of tech support problems of their own making comes from "Harold", who was a senior managed compute engineer for a major provider of stuff. "I was tasked with a simple project to extract a full list of users in all …

  1. Anonymous Coward
    Anonymous Coward

    We've all been there, I managed to kill a worldwide instance of Oracle with a silly little error on a recurring query. I would tell more or email it but I would be found out.

  2. defiler

    Beware recursion

    It has the ability (and knack) to make fools of us all...

    1. Ryan 7

      Re: Beware recursion

      You should also be careful of recursion.

      1. Alan Brown Silver badge

        Re: Beware recursion

        beware recursion and directory ".."

    2. Jim 59

      Re: Beware recursion

      "...when trying to make it recursive."

      Madness. The guy should never have attempted anything recursive, or with the potential to become recursive, on a live server. And probably not even on his own laptop. Absolutely bonkers.

  3. Will Godfrey Silver badge
    Happy

    As I was once told by a wise old man

    There is a limit to everything.

    ...

    Make sure that includes recursion.

  4. Major N

    I used to run a site using Server Side includes to look up template files to keep the look across the site (circa 2002 or so). My host had default 404 etc error pages which were write-locked but for some reason I could save over them in place, so I added SSI Includees to make the error pages match those of the rest of the site.

    Later on, I rewrote the site in PHP, and since the SSI error pages wouldn't use PHP, and I had learned more in those days, I used a .htaccess page to point the server at new, PHP error pages, retired the old SSI based templates, and never looked back.

    Unfortunately, one day, my host deleted the .htaccess file, so the server failed back on the old error pages, which still had SSI include links to template files that no longer existed. They also had the server set up so that on a 404 page not found during an SSI Include request, it would instead include the 404 page.

    So we had a 404 page that would try to include a template file, which because it didn't exist, would instead include the 404 page in line, and so on... 404s all the way down, until the server ran out of resources and fell over. For all paying users on the node.

  5. Anonymous Coward
    Anonymous Coward

    It's also a good idea to run scripts where you definitely only want to READ data from a non-administrative command prompt. Speaking from experience obviously... :o)

  6. Andy Non Silver badge

    Accidental DOS on my employer

    While working as a software developer for a large well know credit reference company I had a phone call one day from the central security team on another site asking me what I was doing. Apparently a particular part of their website was under a massive DOS attack and they'd tracked it back to my computer. All I was doing was accessing the site as an ordinary user, nothing fancy. Not even actively, it was just open in my browser. They asked me to stop whatever it was I was doing, which just entailed me closing the browser window and everything apparently went back to normal. I have no idea what caused the DOS but I can only surmise that there must have been a bug either with the website or with the network setup which triggered when a particular part of the site was accessed from an IP address within the company causing something to get stuck in a high priority loop. Explanations on a postcard to...

    1. Waseem Alkurdi
      Joke

      Re: Accidental DOS on my employer

      You accidentally caused a legacy Microsoft operating system on your former employer?

      1. Jim 59

        Re: Accidental DOS on my employer

        Should have CP/M'd 'em.

  7. Roq D. Kasba

    I learnt to test my WHERE clauses on a DELETE with a SELECT first

    The hard way.

    1. defiler

      Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first

      I tend to favour select count(id), so I just get the number of records. Usually that's enough to give me a good indication of whether I'm about to set fire to the week.

      1. Waseem Alkurdi

        Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first

        On a similar note:

        find -name 'file' -delete

        This command sucks a{ss,rse}.

        Deletes any file that matches the criteria, no confirmation, nor a list of files it blew away.

        1. Anonymous Coward
          Anonymous Coward

          Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first

          There's nothing wrong with the -delete option, you should first run your find with a -print (actually -depth -print since -delete implies -depth for obvious reasons) to get the list of files it will delete and sanity check it before you use -delete. Same thing as with any potentially dangerous command like 'rm -rf' or 'mkfs', you need to be SURE you are doing what you want to do and shouldn't rely on software crutches to do it for you. Failure to do so runs the risk of being El Reg's next "Who, Me?" story :)

          Unrelated, but the code for find's -delete option was based on a patch I sent to the Bugtraq security list about 20 years ago. How time flies!

          1. Waseem Alkurdi

            Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first

            Unrelated, but the code for find's -delete option was based on a patch I sent to the Bugtraq security list about 20 years ago.

            Ouch. Sorry!

            (Followed by military salute of respect to senior officer)

            1. Peter Gathercole Silver badge
              Coat

              Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first

              Grrr. Creaping featureism!

              The 'correct' use should be:

              find <dir> -name -exec rm {} \;

              (the starting directory should be required, I don't know when GNU made it optional). I admit that it spawns more processes, but if you're really worried about this, then:

              find <dir> -name -print | xargs rm

              These work on pretty much all unix-like operating systems whereas the GNU ones don't, which is why they trip off the fingers when I'm working.

              Mine is the copy with the SVID in the pocket!

              1. GrumpenKraut
                Headmaster

                Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first

                > find <dir> -name -print | xargs rm

                You surely meant

                find <dir> -name -print0 | xargs -0 rm

                Right?

                1. Anonymous Coward
                  Anonymous Coward

                  Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first

                  You should have read for context and thought about why I wrote this patch for the Bugtraq security list!

                  It originated from a discussion regarding attacks where someone who had write access one of the directories being cleaned up (often directories where everyone has write access, like /tmp) could fool that command into deleting ANY file on the system. Both of the variants posted above are vulnerable to this, so I posted a find patch I'd written with the -delete builtin which was not vulnerable.

                  It isn't creeping featurism if it solves an actual problem.

                  1. vulture65537

                    Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first

                    > fool that command into deleting ANY file

                    I think you mean all files.

                    http://www.zen19351.zen.co.uk/article_series/find_xargs_rm.html

                2. Peter Gathercole Silver badge

                  Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first @GrumpenKraut

                  No. You will find that -print0 is not in the SVID version of find. There is a typo, though, as I missed out the filename!

                  I take your point about print0, but even though whitespace is actually allowed in filenames in pretty much all UNIX-like filesystems.

                  If I ruled the world, the set of characters that would be illegal in filenames would be much larger! I really hate when filesystems are shared between UNIX-like OS's and Windows (Samba, NFS etc.), and I see filenames created by Windows including spaces, asterisks and other characters. And that is not to mention when different multi-byte character encoding systems were used, prior to UTF8 becoming dominant.

                  But then, many people think I'm far too old to make relevant comment about modern computing.

                  1. GrumpenKraut

                    Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first @GrumpenKraut

                    > But then, many people think I'm far too old to make relevant comment about modern computing.

                    Welcome to the club...

          2. phuzz Silver badge

            Re: I learnt to test my WHERE clauses on a DELETE with a SELECT first

            If i'm typing a particularly long command that will delete or otherwise change a lot of files, I'll either not add the sudo rm at the start, or type zsudo in case I accidentally hit the enter key before I'm ready.

            I'm sure we've all deleted a parent directory when we were intending to delete a subdir.

  8. Nunyabiznes
    Joke

    MS update

    Did the OP ever work for MS, say in the Win7 update group this spring?

  9. Paul Johnston
    Alert

    Recursion is difficult

    I once advised a Professor that rather than staying with his very old Sparc box running Solaris under x86 was a better idea.

    He bought a new machine and was not very happy as he said it was far slower than his old machine.

    I said I couldn't see why but he was the respected professor of computer science.

    A couple of days later he admitted there was an infinite loop in his Prolog code and once this was sorted it really was very fast.

    1. Anonymous Coward
      Anonymous Coward

      Re: Recursion is difficult

      You have to be careful with infinite loops, who knows where it will end?

      1. Ken Shabby
        Devil

        Re: Recursion is difficult

        See previous post

      2. harmjschoonhoven

        Re: Recursion is difficult

        Some recursive functions are easy to understand, but hard to execute like the Ackermann function.

        Wilhelm Ackermann was a pupil of David Hilbert who was a pupil of Ferdinand von Lindemann who was a pupil of Felix Klein who was a pupil of .... Probably going all the way down to Noah who had just to count to two.

      3. Anonymous Coward
        Anonymous Coward

        Re: Recursion is difficult

        Hopefully at 1 Infinite Loop

      4. Anonymous Coward
        Anonymous Coward

        Re: Recursion is difficult

        It doesn't end. It's recursion all the way down.

    2. EVP

      Re: Recursion is difficult

      I usually recurse when I do the same stupid mistake again. It’s not difficult at all.

  10. Anonymous Coward
    Anonymous Coward

    Holiday articles

    "Ho ho ho! It may be Christmas Eve but Who, Me? stops for no festive season."

    Seeing as how this is in the Data Centre portion of El Reg, I'm assuming that the people responsible for pushing out the Christmas Eve article are getting double time for both work and travel, plus time in lieu, plus meal allowance and travel reimbursement and a minimum callout time of four hours...

    ...and that you've written a script to publish the article automatically while you're downing a few lagers at the pub.

    If any of the El Reg ink stained wretches are at the office, (oh heck, even if you're not) Merry Christmas to you all!

    1. Waseem Alkurdi
      Devil

      Re: Holiday articles

      "Double time for work and travel time, no minimum, travel allowance, nor TIL," the Head counter-offers.

      - The Head, BOFH 2002 Episode 26, 12/27/2002.

  11. J. Cook Silver badge
    Coat

    This is the loop that never ends;

    it goes on and on, my friends...

    Some coder started running it, having forgetting to fuzz,

    and now it will continue, simply because...

    This is the loop that never ends;

    it goes on and on, my friends...

    Some coder started running it, having forgetting to fuzz,

    and now it will continue, simply because...

    [loops ad nauseum]

  12. agurney

    Re: Recursion is difficult

    It was a dark and stormy night, and the Captain said to Antonio "Antonio tell us a tale"; so, Antonio told us a tale, and it went like this:

    "It was a dark and stormy night, and the Captain said to Antonio "Antonio tell us a tale"; so, Antonio told us a tale, and it went like this:" ..

    1. Ken Shabby
      Angel

      Re: Recursion is difficult

      To understand recursion, you must first understand recursion

      1. Anonymous Coward
        Anonymous Coward

        Re: Recursion is difficult

        are we back to this again? Do I have to repeat myself?

        1. Ken Shabby
          Go

          Re: Recursion is difficult

          See following post

          1. Sgt_Oddball

            Re: Recursion is difficult

            Stop me if you heard this one before....

            1. Dave559 Silver badge

              Re: Recursion is difficult

              There is clearly a challenge here to see how far the new nested comment layout can be recursed before we break it… ;-)

              1. The First Dave

                Re: Recursion is difficult

                Please reply if you can read this...

                1. Flakk

                  Re: Recursion is difficult

                  Please reply if you can read this...

                  1. Anonymous Coward
                    Anonymous Coward

                    Re: Recursion is difficult

                    I cannot read this comment.

                    1. The First Dave

                      Re: Recursion is difficult

                      Don't worry, I can still read it

                      1. Simon Reed
                        Windows

                        Re: Recursion is difficult

                        The version I heard, and have often repeated, goes:

                        "It was a wild and stormy night, and the Captain said to the First Mate,"Jack, tell us a story". And the First Mate began.

                        "It was a wild and stormy night, and the Captain said to the First Mate,"Jack, tell us a story". And the First Mate began.

                        "It was a wild and stormy night

                        And so on ad duffyouupeum.

  13. swschrad

    also, pick your log disk carefully

    several decades ago, almost four, I had to break a disk cluster on a VMS system. this is a job to beware. but it needed to be done to advance reliability. so, since there were more users than space on any single disk, had to divide them. didn't work well to split halfway down the first letter of last names, so I had to count instances and fork them left and right. test went fine. so one night, kicked off the script.

    first AM when users started logging in, they couldn't. I couldn't remotely. so I used the admin credentials.

    logging went to the system disk because, space.

    until there was none.

    deleted the log, drove in, job done, lesson learned. should have flipped the logging over the VAXcluster to another machine. learn, grasshoppers.

    1. Joe W Silver badge
      Pint

      Re: also, pick your log disk carefully

      Yeah, that one bit me as well. Meh, you live and learn... Had to buy beer for the group that week.

  14. Anonymous Coward
    Anonymous Coward

    Cisco debug commands

    The ultimate way of wrecking your day, a command where if you forget an option it overloads the cpu.

    Happened a good 10 years ago, a colleague was doing some checks, ran a debug but hit enter before the last option, the core switch went into a loop, we realised what had been done, a quick glance towards the computer room, and without a word both of us got up walked down to the core and rebooted it.

    Apparently causing a mass panic when someone looked for us and found two empty desks.....

    Neither of us work for that company now though.....

  15. Martin-73 Silver badge

    Oddly I've had the opposite experience.

    Inasmuch that I cocked up, but admitted my mistake when it became clear I'd caused it. (AFTER fixing the issue).

    I managed to cock up a 7 way lightswitch arrangement such that one switch acted as a 2 seperate circuits when one of the intermediate (en_US '4 way') switches was thrown one particular way. Very confusing till I spotted the mistake in wiring, and recognized my connection style on the switch.

  16. OssianScotland
    Facepalm

    In a similar vein...

    For my sins, I am an MCT, and once ran a (successful) course on SQL Server Querying for a company.

    As with other Microsoft courses, it was all run in a virtual environment, on laptops I brought in, but using a database (AdventureWorks, for those who have met it) which had nothing to do with the actual line of business.

    We finished a few hours early, so the delegates asked if they could put their new skills into practice on the real databases. Since they had a nice training room, with plenty of PCs, I didn't see any problem with this, and the IT manager couldn't see any issues either, so they all set up, practicing joins, sub queries, grouping, and all the other fun you can have with SQL.....

    ...for about 10 minutes, when the DBA came running in and demanded, in no uncertain terms, to know what the f@&k we were doing to his production servers, which had slowed to a crawl under the extra load.

  17. Anonymous South African Coward Bronze badge

    Dead, see Parrot

    .

    .

    .

    .

    .

    Parrot, see Dead

    1. Ken Shabby
      Linux

      Nah, it is looping for the fjords.

  18. commonsense

    Root cause analysis?

    Do you not do any problem management? Did nobody ask for root cause?

    1. NotBob

      Re: Root cause analysis?

      Anyone wanting a root cause analysis from the BOFH will either be found as the root cause or be found in the next episode...

  19. Anonymous Coward
    Anonymous Coward

    I once crashed BT's public X25 packet network, UK wide

    With an accidental DOS from an app i left running overnight. I guess this was late 80's.

    It was in Beta at the time and since it was overnight it probably only had 7 users on it.

    And hopefully that helped them make it more robust, so all to the good.

  20. This post has been deleted by its author

  21. The Vociferous Time Waster

    Powershell is dangerous in the hands of some...

    At a place I worked as a contractor there was a low level server ops guy, let's call him John Smith, and he wrote a very simple script to change passwords. I don't quite know how his script was more efficient that the one line of powershell that it ran but then he was a very low level person, the type who tend to do the stuff that nobody can be bothered to script because they can't script clicking on the next box.

    So the script that "John" wrote basically took a username and changed the password. I would hazard a guess that it basically did:

    Get-ADUser $user | Set-ADAccountPassword -NewPassword "Changeme123!"

    OK, so there were some other options in there as well but you get the gist.

    Anyway, "John' runs this script using his privileged account (let's him do desktop stuff like change passwords) and it seems to hang. He goes off to lunch.

    At this point it's worth noting that in the line of powershell above if $user is null then Get-ADUser will get every user object in the domain and pipe all of them to the password change applet. It hadn't hung, it was just a little busy.

    While "John" was off at lunch the rest of the IT department realised that they were unable to reach stuff, many had locked their desktops while eating their own supermarket "own brand" sandwiches at their desk and when they tried to unlock their desktops found that their passwords were no longer valid.

    Later in the incident room it became apparent that "John" was responsible however he not only managed to retain his job but eventually managed to move sideways (and slightly down) into the desktop team.

  22. I Am Spartacus
    Linux

    To understand recursion

    "To understand recursion, you must first understand recursion."

  23. Graham Butler

    I took out a Sun Solaris server with a looping bash script. Thankfully it was development, not production. I stupidly appended a line to the logfile I was grepping with the result of finding an entry (rather than a separate file) and, more to the point, used the exact phrase I was grepping FOR. Something akin to '<search term> found on line xxxx".

    It took about 5 seconds to blow up like the Hulk and take the box down.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like