back to article Why have just one firewall when you can fire all the walls?

To quote the ancient philosophers: "Monday Monday, dah dah dah, can't trust that day." And so it is, dear reader, that we find ourselves yet again betrayed by the beginning of the working week and its requirement to spend the next five days exchanging your labor for currency (except of course for our American readers – Happy …

  1. Anonymous Coward
    Anonymous Coward

    I know of someone who did something similar on a mobile phone network.

    Whilst anyone can (and ultimately will) make a mistake at some point in their career, the thing that caused his 'fast exit, stage right' was that there was no authorized change request covering the work. The icing on the cake was there had been an email sent to everyone reminding people that no changes were to be made without a change request barely a week before.

    1. John Riddoch

      Yeah, you can get away with major outages caused by an approved change record with a minor slap on the wrist and the agony of a post incident/change review, but a minor outage without a change record often results in a swift exit from the company.

    2. Korev Silver badge
      Flame

      I know of someone who did something similar on a mobile phone network.

      You mean he was firewalled?

      See icon -->

      1. The Oncoming Scorn Silver badge
        Pint

        Its Always DNS

        Short version, PM wants another tick box on his project & despite reiterating many many many times no changes to certain critical areas where I work should be made without full verification that it was OK to proceed, he stuck in a change request an hour or two after the last time I told him not to make any changes.

        Friday afternoon, printing goes down around midday in three of those critcal areas, after my return from lunch & walking into this mess, a quick investigation reveals DNS is pointing to the "New queue\Hostname\IP" not the legacy.

        So I message him & ask is he aware of ANY DNS changes pushed today, no he them proceeds to ask what steps I have taken & after 20 mins of teams chat, admits he did put in a request on Thursday for DNS changes, "BUT theres NO way those changes WOULD have ANY effect on certain critical areas printing", then he like the rest of the crew out east decide that with the time difference etc its time to call POETS Day.

        Eventually after many breakdowns of communications including command centre thinking I'm working on undoing the DNS screwup, I finally track down someone capable of reversing the DNS changes, verify print operation & leave at 8pm for my local bar.

        I got a recognition award for my efforts this morning (No points exchangeable for gift cards) from him this morning for staying late & fixing his screw up.

    3. John Brown (no body) Silver badge

      "I know of someone who did something similar on a mobile phone network."

      Was this Australia, quite recently?

  2. SVD_NL Bronze badge

    "Thankfully one of Charles's ancestors in the monkey colony had anticipated just such an undesirable mutation, and had permanently and irrevocably white-listed the policy management server"

    The true hero of this story!

    Undoubtedly this firewall rule is written in blood. (or a lot of overtime coffee, at the very least!)

    1. stiine Silver badge

      I would guess it was written in shoe leather and time.

    2. bemusedHorseman
      Megaphone

      First rule of development:

      Remember kids, you can break anything, as long as you don't break rollback!

  3. Serg
    Facepalm

    Can tell it's a Monday!

    "requiremnet" indeed...

    1. Will Godfrey Silver badge
      Meh

      Re: Can tell it's a Monday!

      Oh come on! That's the the commonest mistake in the world. I find the fatser you type the moer of them you craete :P

      P.S. I still upvoted you for spotting it so quickly - want a job proof reading?

      1. Pascal Monett Silver badge

        In this case, a simple spell checker would have caught the mistake.

        So, are The Vultures using one over on that side of the pond now ?

        1. TonyJ

          Weirdly I always used to struggle to type administrator properly. It was usually mangled to some variation of adminini... or such.

          The other favourite of mine to mangle was GPUPDATE... and even now, typing this, it almost became GPUDPATE. A weirdly incorrect muscle memory.

          1. stiine Silver badge

            That one still bites me.

        2. old_n_grey
          FAIL

          "In this case, a simple spell checker would have caught the mistake."

          Unless, like my manager when I first moved from accountancy to IT, you simply add the misspelled word to your dictionary!

        3. PRR Silver badge

          > are The Vultures using {spell check} over on that side of the pond now ?

          Won't be a USA Vulture: "....spend the next five days exchanging your labour for currency..." most of us in USA won't labor on Thursday 23 Nov, the US Thanksgiving. (Canada did theirs a couple weeks back, to beat the snow...)

        4. John Brown (no body) Silver badge

          "In this case, a simple spell checker would have caught the mistake."

          Sometimes, when in a hurry, I've been known to accidentality hit "add to dictionary" instead of the correct spelling. If I'm in such a hurry that I don't notice what I've done, just noted that the red line went away, the next time I make the same typo, the spilling chucker isn't going to highlight it because I already told it that was correct :-)

          But that's why journalists used to have Editors and proof-readers. From comments made by El Reg authors here, it get the sense that at least some of them write and hit publish, especially with "breaking" type articles, with no checking involved other than their own self-checking, which is hard. Proof-reading your own work, it's easy to see what you think you wrote rather than what you actually wrote :-)

          I wonder if there's an industry rag for journalists with their own "Who, Me?" column? :-)

          1. Grogan Silver badge

            LOL, I can't say I've had that happen, but I gave up on adding all the slang words I use long ago :-)

        5. Grogan Silver badge

          The stupid things underline so many words that aren't in their dictionaries that I tend to end up ignoring them and missing a real typo :-)

          P.S. One thing that happens a lot to me is U.S. dictionaries and me typing Canadian/British spellings of words, one thing that has helped to condition me to ignore spell checking.

        6. Peter Ford

          I can't spell 'contanier', which is a bugger when a lot of our stuff is on Docker...

    2. Howard Sway Silver badge

      Re: Can tell it's a Monday!

      They even spelt "labour" properly in the same sentence!

    3. diodesign (Written by Reg staff) Silver badge

      Oops

      Yeah, it's fixed. Don't forget to email corrections@theregister.com if you spot something wrong.

      We should run articles through spellcheck, but sometimes we forget.

      C.

  4. Kildare

    We've all been there.

    Let he who has never sawn off the branch he was switting on, cast the first stone - I think that's mixing metaphors but you know what I mean

    1. jmch Silver badge
      Flame

      Re: We've all been there.

      We'll burn that bridge when we come to it!

      1. NXM Silver badge

        Re: We've all been there.

        Yes, we've all passed a lot of water under that bridge.

    2. Sam not the Viking Silver badge

      Re: We've all been there.

      I understand that the way to avoid blame is to save someone else from their 'oh-no' moment. No noise, no fuss. Just a knowing nod of complicity. In this way, if anything ever goes awry for you, you will always have a cooperative supporter. Enough to divert finger-pointing by those out-of-the-loop.

      Retelling this story on behalf of a friend.

      1. FrogsAndChips Silver badge

        Re: We've all been there.

        I've met coworkers who would have been happy to stab you in the back even after you'd saved their asses.

        1. heyrick Silver badge

          Re: We've all been there.

          I've met co-workers who'd look you right in the eyes then stab you in the front. Some workplaces are toxic to the point where one wonders how any actual work gets done.

    3. Martin
      Happy

      Re: We've all been there.

      Switting on?

      1. collinsl Bronze badge

        Re: We've all been there.

        Yes, Switting, the well known act of sawing through a branch you are sitting on. Shortened from the Old English SawSitting

        /s

        1. Doctor Syntax Silver badge

          Re: We've all been there.

          Avoid sitting on saws. They make a big impression.

          1. The Oncoming Scorn Silver badge
            Alert

            Re: We've all been there.

            The first cut is the deepest.

      2. Kildare

        Re: We've all been there.

        It's Monday! (See above) :)

      3. Doctor Syntax Silver badge

        Re: We've all been there.

        As monkeys are involved it's clearly a combination of sitting and swinging and as blaming is involved somebody's going to have to swing for it.

        1. John Brown (no body) Silver badge

          Re: We've all been there.

          Certainly someone will be flinging shit!

  5. Michael H.F. Wilkinson Silver badge

    Didn't affect an entire network, but I do recall a scary experience with a computer that was controlling a 1.5m diameter IR telescope high up in the mountains of Switzerland. We we testing a new IR spectrograph, and one of the instructions I got from the engineer was that I should not move the telescope below -10 degrees declination, or else the liquid nitrogen and liquid helium might get poured out of the system, and various things might fail dramatically. The software controlling the telescope was, lets say, "interesting" in that an English language user interface was a late addition (afterthought is the correct phrase). It was very basic: it would prompt you for the coordinates of the object of interest, show the coordinates on the screen, and ask for confirmation by asking "Is this OK?".

    At one point, I noticed I made a typo in the coordinates of the object of interest, entering -16 deg declination rather than -6 deg. At the prompt "Is this OK?" I dutifully entered "N" for no, just as I had successfully entered "Y" for yes previously. I was horrified to see the cheerful response I had seen so often before "Then I go!", and could hear the telescope motors start humming. There was no way to stop this before it pointed to this low position in the sky. Apparently, the user interface would consider any character input as a thumbs up, except for Cntrl-D (Unix EOF). We rushed upstairs expect all kinds of damage caused by this action. Luckily, the spectrograph survived this abuse, and worked fine for the rest of the session. I did suggest to our Italian hosts that they might want to update their UI and manuals.

    1. Korev Silver badge
      Boffin

      > I should not move the telescope below -10 degrees declination, or else the liquid nitrogen and liquid helium might get poured out of the system, and various things might fail dramatically.

      Does that include the humans that got in the way?

      1. LogicGate Silver badge

        Only if they yell "I AM INVINCIBLE!" first

    2. Pascal Monett Silver badge

      "could hear the telescope motors start humming"

      So, that is proof that 1) the programmer didn't bother with actually checking inputs, and 2) zero quality control or even proper testing.

      I would venture that this would be grounds for a serious dress-down, if not a pink slip outright. When you're writing code for expensive equipment, the least you can do is make sure your code is not going to risk breaking anything and that means testing, testing and testing some more.

      Because if the telescope had been damaged, it could be proven that the code did wrong and that company would have been obliged to send a new telescope free of charge.

      If that had happened, I'd wager the CEO would have suddenly been very interested in testing procedures . . .

      1. Michael H.F. Wilkinson Silver badge

        Re: "could hear the telescope motors start humming"

        Note that the spectrograph was ours, bolted on the back of the scope, and the telescope software had nothing to do with the limits on the spectrograph, and there was no way for the coders to know that this limit applied.

        1. Doctor Syntax Silver badge

          Re: "could hear the telescope motors start humming"

          Even so, it's abysmal coding around an "Are you sure" prompt.

          I remember a similar condition applied to an X-ray detector in an electron microscope - the detector liquid notrogne supply shouldn't be allowed to boil dry. The thought occurred that if exposure to room temperature would destroy it did that mean it had ben manufactured under cryogenic conditions?

          1. LogicGate Silver badge

            Re: "could hear the telescope motors start humming"

            Well.. I have seen "Rugged" embedded PC's sold with a 0.1 g rating.

            Strangely the manufacturer does not exist anymore.

          2. matthewdjb

            Re: "could hear the telescope motors start humming"

            Absolutely. The "do it" should only have been Y. Everything else is No.

          3. Solviva

            Re: "could hear the telescope motors start humming"

            My communal laundry web booking system - if you want to cancel a booking you get a confirmation box "Are you sure you want to cancel?" Where the choices are OK and Cancel.

            1. Yes Me Silver badge

              Re: "could hear the telescope motors start humming"

              Sounds like a case where you should press the Any key.

      2. druck Silver badge
        Facepalm

        Re: "could hear the telescope motors start humming"

        So, that is proof that 1) the programmer didn't bother with actually checking inputs, and 2) zero quality control or even proper testing.

        Sounds very much like software written by scientists rather than engineers.

        1. John 110
          Boffin

          Re: "could hear the telescope motors start humming"

          "Sounds very much like software written by scientists rather than engineers."

          Sounds very much like software lashed together at the last minute and in somebody's "spare" time by scientists rather than engineers.

          There, fixed that for you.

        2. Killfalcon Silver badge

          Re: "could hear the telescope motors start humming"

          A grad student, 10 years ago, who now works on a different continent.

    3. imanidiot Silver badge

      Holy hell, with consequences that high how is anything OTHER than "Y" not taken as cancel/nope/oh-shit-no?

      1. A.P. Veening Silver badge

        Holy hell, with consequences that high how is anything OTHER than "Y" not taken as cancel/nope/oh-shit-no?

        As it is Switzerland, "S", "J" and "O" might also be acceptable as positive confirmation.

        1. Dr Paul Taylor

          In Greece, Ν for ναι means yes, but that's ok because you can check from the Unicode that it's not N. OK could mean yes too, but Ο for όχι would mean no.

          1. collinsl Bronze badge

            We should fit all keyboards with a tick and a cross button to resolve this thorny issue once and for all.

            1. GroovyLama

              We can put it next to the "Any" key!

              1. LogicGate Silver badge

                ...or replace the caps-lock.

                I honestly have never ever needed to use caps-lock, but I have had to clean up after an unscheduled caps-lock hundreds of times...

                1. John Brown (no body) Silver badge

                  I NEED MY CAPS LOCK KEY, YOU'LL HAVE TO PRISE IT FROM MY COLD DEAD HANDS.

                  Signed

                  Bombastic Bob

                  :-)

                2. usbac Silver badge

                  At a previous job, I was always fighting having everything in our ERP system entered in all caps. When I addressed the issue with our customer service department (the department where this was a problem), I kept getting told that the system requires everything to be in all caps. I kept telling them "No! It absolutely does not". They kept doing it any way.

                  So, finally one evening I went around with some pliers and wire cutters, and removed all of the caps lock keys from their keyboards (wire cutters to make sure there wasn't any part of the key-switch sticking out that they can press on). There was a loud uproar the next day, with demands for new keyboards, and threats to "ruin" their keyboards, so that they would need to be replaced. I told them that any replacements will not have a caps lock key on them.

              2. OhForF' Silver badge
                Trollface

                Press any key to continue or any other key to abort. Continue?

            2. Elongated Muskrat Silver badge
              Trollface

              Press the X button to cancel your voting selection.

            3. matthewdjb

              Do you work for the SAPGui team?

            4. John Brown (no body) Silver badge

              "We should fit all keyboards with a tick and a cross button to resolve this thorny issue once and for all."

              "After much user testing, thought and consideration, we have decided to deprecate the "tick and a cross button" with two separate buttons, one with a tick and another with a cross."

            5. Killfalcon Silver badge

              Sony tried that, for the playstation!

              In Japan, the X is traditionally 'no' and a circle is 'yes'.

              For reasons, nearly all English localisations of Japanese games flipped this around, with Circle being the cancel button and X being select/accept, so it didn't even help in the end.

        2. Fred Daggy Silver badge

          And if English is used in the interface, the Y will NOT be in the expected position. For QWERTZ rules the keyboard landscape there. Hope the bespoke interface doesn't expect US keyboard ... or shudder French-French.

          1. heyrick Silver badge
            Thumb Up

            "or shudder French-French"

            Icon, for obvious reasons.

            I have to buy AZERTY keyboards as I live in France (and ordering in UK layout is rather more expensive, moreso after you-know-what). I also mail order keyboard stickers so I can 'fix' the keyboard back to a sane layout where you don't need a three finger salute just for the @ symbol, or shift to get to numbers, or an entire key wasted on a superscript 2 (seriously, WTF is with that?).

            France, of course, being France, took it to the next level. Go look up the BÉPO layout. Thankfully it currently seems more an academic endeavour then a layout that people are actually using. I get what they're trying to do, but oh my god there're four f'king Es right next to each other...my head hurts.

            1. Scott 26

              > four f'king Es

              probably for the fork handles (four candles)

            2. I could be a dog really Bronze badge
              Headmaster

              Donning my pedant's hat, no there aren't four Es.

              There is one E (E), one E-grave (È), one E-acute (É), and one E-circumflex (Ê). These are different characters, and it's understandable that in a language where their usage is high, having a multiple-key entry method for the ones with diacriticals would be "irritating". On a Mac, the keystrokes are Option-\,E; Option-E,E; and Option-I,E to get the E with grave, acute, and circumflex respectively.

              At times during the years of doing It support I've had to deal with a number of different keyboard layouts, and even the more common French layout (at least on the Macs I dealt with, and IIRC as it was long time ago now) was annoying to use as it needed a shift to get numbers.

              And back when we were Apple dealers, we used to have access to all the different localised versions of Mac OS - this being when it came on a few floppies ! It was fun to try out the different versions - German extended the menubar somewhat due to the long words, Japanese had a lot of "squiggles" (and Mount Fuji replaced the Apple logo heading the "Apple" menu), but all had some of the menu items in English as they didn't have localised words for everything.

        3. Anonymous Coward
          Anonymous Coward

          For an English prompt? Hell nah.

    4. Terje

      I had to think a while on how the declination rather then resulting altitude would be a problem, but figured out the spectrograph must have been at nazmyth focus and rotating with the dec axis?

      1. Michael H.F. Wilkinson Silver badge

        It was mounted at Cassegrain focus, but declination -10 could push it beyond the safe levels when the object was not due south, as I recall. Apparently, the engineers left quite a safety margin (thank goodness).

        1. Terje

          Ahh, that would explain it, better make sure we never try to target an object that may end up in a position we potentially can't handle!

    5. heyrick Silver badge

      "There was no way to stop this before it pointed to this low position in the sky."

      It defies belief that there isn't a set (not one, multiple) big red abort buttons that when pressed will bring the entire machine to a stop.

      Aren't they mandatory on big equipment in Switzerland?

      [place where I work (France) has plenty of them and they even get tested!]

      1. Anonymous Coward
        Anonymous Coward

        At work, a colleague was telling me how he caused a bit of a stir when he went to witness some testing of a piece of (rather expensive) equipment we were having customised for us.

        Normal operation was for a controlled shutdown as it had things like magnetic bearings that "don't really like" having the power yanked while the machine is still turning. But he insisted on seeing the emergency stop tested ... and it ... did absolutely nothing ! The engineers at the vendor were suitably embarrassed and by the next day some quick redesign and wiring had been done to the red button actually worked - I think it triggered a "very fast stop & shutdown" via the controller rather than killing the power, but that was adequate for the intended application.

    6. This post has been deleted by its author

  6. Bebu Silver badge
    Windows

    As soon as...

    Windows loomed on the horizon in "antivirus/security package that included a firewall" it was pretty much an iceberg looking for a Titanic :)

    In a very similar environment and possibly in the same era, faced with a similar requirement I went for a custom linux/ebtables/iptables (perhaps ipchains) screening bridge between the polloi and *the asset*.

    Basically a discarded PC and two network adaptors and custom 2.4 kernel + ebtables userland. [std Python "Four Yorkshireman" sketch ... "you try and tell the young people today that"...]

    Worked a treat until it wasn't needed a few years later.

    1. SVD_NL Bronze badge

      Re: As soon as...

      I've never been a fan of firewalling all PCs seperately for rules like this.

      Basic security rules like blocking certain services? sure.

      Network flow rules? sounds like a nightmare!

      L3+ managable switches, or VLAN segmentation is a much easier to control solution, with the added benefit that you enforce compliance for all devices.

      (Or back then, a network bridge like you set up)

      1. tip pc Silver badge

        Re: As soon as...

        l3 switches with access list on the vlan that the machine of interest is connected to.

        1. stiine Silver badge

          Re: As soon as...

          What a recent development.

      2. John Brown (no body) Silver badge

        Re: As soon as...

        "(Or back then, a network bridge like you set up)"

        Sounds like an ideal situation for a "data diode" as per the El Reg article the other day. PC controls microscope and can send data out to the network, but nothing comes into it.

    2. Martin
      Happy

      Re: As soon as...

      ...it was pretty much an iceberg looking for a Titanic...

      That is a wonderful turn of phrase - I'm nicking that! Thanks.

      1. Terje
      2. The Oncoming Scorn Silver badge
        Pint

        Re: As soon as...

        So am I.

  7. Anonymous Coward
    Anonymous Coward

    ah the old "Invert Selection" cockup.

    Happened here once with SCCM ... lotta laptops had to be tracked down and repaired.

  8. Martin
    Happy

    ...we find ourselves yet again betrayed by the beginning of the working week and its requirement to spend the next five days exchanging your labour for currency...

    Speak for yourself - I've retired.

    (Where's the smug git icon?)

    1. Victor Ludorum
      Unhappy

      Given the corrent state of the economy, maybe the phrase should be exchanging your labour for cost of living vouchers

      1. Anonymous Coward
        Anonymous Coward

        That's basically what money is though.

  9. GroovyLama

    I remember we once had an odd set-up where we had applications deployed across multiple servers for High Availability, with the Load Balancer checking a status file on each server to see which apps were available for live traffic. This was in the early days of cloud and containerisation, and hadn't moved to the newer solutions yet.

    A colleague in a timezone ahead of me had conducted a review and was preparing for shutdown of some apps, and arranged for it to be done. Problem was, the Load Balancer (network team responsibility, we were apps team) was configured incorrectly, so for some of the applications it was checking the wrong status file for availability. This meant that when he shut down the application instances we no longer needed, the Load Balancer took that to mean some other apps were also not required, so closed off access to them.

    I was receiving calls from the client about losing access before I'd even reached the office on that one! Luckily it was an easy fix - start up the redundant apps again and all access restored.

    Thorough review of the Load Balancer config later and all such inconsistencies were fixed up. Some valuable lessons were learnt through that on making sure that health checks are accurately checking the right source!

  10. stiine Silver badge
    Flame

    documentation

    "Charles noted that the documentation was not exactly helpful: "Basically it was a list of the available variables and functions without any explanations or examples.""

    Recent documentation is only an aide-mémoire and its been this way since the late 1990s.

  11. JulieM Silver badge

    The Bacon Saver™

    On any machine to which you don't have immediate physical access, you need a cron job that, once an hour, inserts a firewall rule right at the top that allows access on port 22 from the static IP of your home broadband.

    The time between an ill-judged edit and the door back in reopening is still going to be the longest up to 59 minutes of your life, but at least you know there is a way back in.

    1. Claptrap314 Silver badge

      Re: The Bacon Saver™

      I have JUST NOW implemented code (in AWS) to open up access to my current IP address when I click on the app. Of course, the non-standard base64 encoding by AWS & Google were only the start of the fun & games to make this SAML implementation happen.

  12. derrr

    Start with nothing, and add bits...

    1. This post has been deleted by its author

  13. Anonymous Coward
    Anonymous Coward

    Am I being daft but why would you create a rule on every single workstation rather than creating the rule on the device you're actually trying to isolate and protect?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like