back to article Confessions at a Christmas do: 'That time I took down an entire neighbourhood'

Ah, Christmas dos. Remember them? A chance to humiliate oneself in front of colleagues and make the odd drunken confession or two. Welcome to the tenth whoopsie of our Twelve Borks of Christmas (12BoC) series* Register reader "James" (a pseudonym generated by Santa's Little Regomiser) recalls one that was blurted out over past …

  1. oiseau
    Facepalm

    Hello:

    In order to boot the device off the UBR, the command 'reboot cable modem [MAC address]' is used ...

    That particular UBR software was obviously written by a certified DH who could not bother to do it properly.

    An incomplete instruction should result in a non-executable instruction and a potentially dangerous one shoud print a Are you sure reply requiring a y/N from the operator.

    eg:

    ...

    [root@cable_DH~]# reboot cable modem

    Usage:

    admin cable modem [options] [MAC address]

    Options:

    -list - Display cable modem data

    - off - Switch off

    - reboot - Reboot

    - ...

    ... or something to that effect.

    You know, basic stuff.

    Cheers,

    O.

    1. Steve K

      Or a specific "-all" switch

      Agreed - or a specific "-all" switch

      Fail-safe, rather than epic fail....

      1. chuBb.

        Re: Or a specific "-all" switch

        Bah the more critical the bit of kit the terser command...

        Darwinism in the workplace where rtfm is an aptitude test, and aptitude dictates access to production

        1. Anonymous Coward
          Devil

          Re: Or a specific "-all" switch

          Knowing the terse commands was a badge of honor when I started in IT. N00bs who actually RTFM were suspect.

          Of course that was 40 years ago and I've changed my views as I gained experience. Too bad some people still haven't.

          1. chuBb.

            Re: Or a specific "-all" switch

            I still regularly dump strings looking for undocumented switches of smartarsery

      2. Blackjack Silver badge

        Re: Or a specific "-all" switch

        Maybe the dos in the article was Microsoft DOS? That would explain things.

      3. jake Silver badge

        Re: Or a specific "-all" switch

        Exactly. Never allow code that makes changes to assume it knows what the user intends. The command must be explicit, or throw an error code & message.

    2. chivo243 Silver badge

      I would have thought an authentication would have been required for such an operation at such a scale

      1. The commentard formerly known as Mister_C
        Trollface

        That's why the team all have full administrator rights.

        1. Yes Me Silver badge

          full administrator rights

          I thought SOP was to have the system password pinned to the wall, in case some fool logged the console out.

          1. jake Silver badge

            Re: full administrator rights

            Don't be silly. That's only for the management over-ride password.

            The actual root password is on the postit stuck to the bottom of the keyboard.

    3. Anonymous Coward
      Anonymous Coward

      I know an ex-game server admin , who after getting annoyed by a good sniper, wrote

      "!sm ban all snipers" into the console.

      And the admin mod thought he ment "ban all"

      Boy that was fun trying to clear the server ban list of all the players on the server who'd just been banned..

    4. Giles C Silver badge

      Nope

      The most dangerous command on a Cisco switch is debug.

      There isn’t fortunately a shortcut for debug all, but the command gets more specific the longer the string gets.

      I.e. debug ip Bgp

      Will dump all bgp messages to the console

      Debug ip bgp events

      Will only give a subset

      Depending on how busy the system is this can render it inorperable as the cli won’t accept any new commands.

      Most systems work similar to this and that is why the admin commands need to be restricted to those who understand the implications of what they are doing.

      It is quite easy to bring a router down with the wrong debug command.

      1. Down not across

        Yes, I have mentioned it before we had few people accidentally type "debug all" on a core router instead of the CPE they were intending to do it on. Funnily enough, none of them ever did it again. Repeat offence probably would have been career limiting.

        1. J. Cook Silver badge

          had one of the router jockeys at the ISP I was working for accidently changed the default route on an access router (lots and lots and lots and lots of connections!) to a customer's T-1 line instead of the OC-48 uplink to the core router. (We also had a different router jockey do a 'debug all' and instantly crash the core router he was working on as well.)

    5. Terry 6 Silver badge

      Even I. Not a true techie, who hasn't written any decent code for a number of decades, wouldn't have written an instruction that allowed a general command to run just by missing a limiting input from a specific one. At the very least I'd mandate an input variable at the end of the line- maybe "all" if that's what was needed.ELSE Print "error".

      It doesn't take much planning to realise that from time to time someone is going to hit [return] before reaching the end. Because it's something that people just do

    6. Xalran

      On some equipments you have global commands that are just circuit/subscriber/L2VPN/L2VPN/VLAN specific commands... without specifying the circuit/subscriber/L2VPN/L2VPN/VLAN.

      So yes when dealing with these sort of equipment you can end up resetting a whole bunch of circuit/subscriber/L2VPN/L2VPN/VLAN by forgetting to specify the one you want to work on.

      1. Evilgoat76

        Huawei

        A lot of the commands on our Huawei and Versa kit assume if you don't specify a port/board/card you mean all of that particular resource. I have a dim recollection the XOS switches do too.

    7. storner
      Devil

      Rubbish. Pure Unix tradition is to keep commands short - that's why the "delete" command is "rm".

      For the equivalent of this story, "rm" should delete all files. Why bother with this asterisk thing? That'll teach the newbies a lesson!

      1. Terry 6 Silver badge

        On the contrary, you prove the point..

        That's a highly specific short command, used with inclusive arguments. And is the very opposite of a long command with an optional argument at the end, that will screw up a a system if the end argument is omitted or mangled.

    8. DS999 Silver badge

      Aliases

      I always used aliases for the reboot command on critical systems - it is too easy for people to type reboot in the wrong window or whatever. I'd alias the reboot command in the root profile to something like echo "use reboot-`servername`". Then reboot-`servername` would be aliased to /sbin/reboot.

      While I agree that the command syntax is stupid, if that's what the vendor delivers you it is up to you to find to make it safe.

  2. Xalran

    I will have to tell some day about problems ( in some cases really really really weird ones ) being solved by rebooting cards on backbone routers or PE routers... When it wasn't rebooting the whole thing.

    Or the Funky BGP issue that took us ( I wasn't alone on that one ) month to pinpoint and many more months to solve.

  3. don't you hate it when you lose your account

    Said it before

    And I'll say it again.

    I hate networking :)

    Reading the entrails of a freshly killed chicken is way easier, and you get a roast dinner.

    1. SImon Hobson Bronze badge

      Re: Said it before

      Maybe so - but it keeps many of us in paid work, or at least it used to for me before I changed career.

  4. jake Silver badge

    Neighborhood? How about half a city?

    When we were decommissioning the old Fabian Avenue telco Central Office (now home to the Charleston Village condominiums), I was given the task of making sure the electrical power to the site was off. Not just at the breaker down in the street box, but the physical breaker at the Colorado Avenue Sub Station in Palo Alto was to be pulled, thus making certain all power was deactivated until we could make certain everything was isolated.

    This would involve taking out the entire Charleston Gardens section of Palo Alto for an hour or so mid-morning, mid-week when it would cause as little disruption as possible. (Charleston Gardens is a mostly residential section of Palo Alto, bordered by Middlefield, San Antonio and Charleston roads, if you care.) The neighbors were notified the week before, both by snail-mail and people physically knocking on doors to explain and hand out the small paper notice explaining what and why.

    Come the morning of the Great Event, I was selected to physically make sure the power was off at the sub-station. Cell phones being a fad of the future, I drove down the Frontage road (West Bayshore) and arrived at the the sub station promptly at 10AM. Conversation went something like this:

    Me: I'm here to see that the power to the Fabian project is off.

    Site Engineer: We're all ready for you, I'll get to it in a second ... or you can just throw that switch (points).

    Me: OK (throws switch).

    Engineer: NOT THAT SWITCH!

    Most of South Palo Alto: WTF‽‽‽‽‽

    Management swallowed the story that the main breaker tripping was an un-foreseen knock-on effect of the smaller section being taken down ... The Engineer and I made a bee-line for Fred's as soon as he cleared up the problem, which took into the late afternoon. We're still friends.

  5. Genesisjoec

    What in the...

    What manufacturer allowed the execution of an incomplete command! Especially one with such severe consequences..

    1. Anonymous Coward
      Anonymous Coward

      Re: What in the...

      Probably most of them. The problem is the coder *assumes* everyone else *knows* what can happen under certain circumstances, and doesn't even think that their respective product might be being used by someone who hasn't really got that much experience with the product.

      I still see it even now. I recently identified an issue being caused by bad input, and when I raised it (along with a single line fix to prevent it from re-occurring) I was told "Well, they should know!" and the fix was denied.

      <sigh>

      1. heyrick Silver badge

        Re: What in the...

        Good, a paper trail. Which means they'll be accepting liability if it all goes to hell because a problem that was clearly pointed out was refused.

        1. A.P. Veening Silver badge

          Re: What in the...

          Or if you wish to prevent it, you forward it to their legal department with a short description and estimate of the potential damage, thus making it a career limiting move for that developer.

      2. Terry 6 Silver badge

        Re: What in the...

        It's not really about user ignorance in this case, just normal fat thumb/ motor memory malfunction, i.e. hitting [return} before typing the defining input. A known common danger that should not be allowed to lead to dire consequences.

        1. Down not across

          Re: What in the...

          You mean like delete from table <return> where col .... oh crap... ?

          1. Claptrap314 Silver badge

            Re: What in the...

            START TRANSACTION ;

            DELETE FROM table <return>

            I fail to see the problem...

            1. Down not across

              Re: What in the...

              You obviously have not worked with multi-vendor environment where some flavours of RDBMS behave differently and it is easy to forget/omit things. In a predominantly Oracle shop with say few Sybase systems, there is a good chance for someone forgetting to explicitly start transaction with BEGIN TRAN given they are likely used to Oracle's implicit transactions.

              Kinda besides the point.. for some situation yes there are mitigating options ava

              ilable, and for some not.

              1. Charlie van Becelaere

                Re: What in the...

                Upvote for the typo

                graphy.

                1. jake Silver badge

                  Re: What in the...

                  Demonstrating that syntactic white

                  space is inherently evil would be my guess ... causes no end of con

                  fusion.

      3. Terry 6 Silver badge

        Re: What in the...

        In a sense that's not even the real problem - whether the user should know is not the main route to the issue. Any user even if it's Linus T or Bill G will sometimes make an error. An error of recall, or of physical movement. At which point robust software would say "no" rather than "OK boss, I'll delete my entire record and close down the city".

  6. kpanchev

    I still remember when I was a software developer, one of the first tests we ran on any software was the piano test - pretend you are playing a very lively melody on the keyboard and hit as many random keys as you can... if the software managed to survive without deleting the whole database or crashing the OS (Win95 at the time), it was deemed worthy for further testing...

  7. TWB

    Sounds like...

    ... a proper 'Who, me?." this story.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like