Why have just one firewall when you can fire all the walls? To quote the ancient philosophers: "Monday Monday, dah dah dah, can't trust that day." And so it is, dear reader, that we find ourselves yet again betrayed by the beginning of the working week and its requirement to spend the next five days exchanging your labor for currency (except of course for our American readers – Happy …
I know of someone who did something similar on a mobile phone network.
Whilst anyone can (and ultimately will) make a mistake at some point in their career, the thing that caused his 'fast exit, stage right' was that there was no authorized change request covering the work. The icing on the cake was there had been an email sent to everyone reminding people that no changes were to be made without a change request barely a week before.
Short version, PM wants another tick box on his project & despite reiterating many many many times no changes to certain critical areas where I work should be made without full verification that it was OK to proceed, he stuck in a change request an hour or two after the last time I told him not to make any changes.
Friday afternoon, printing goes down around midday in three of those critcal areas, after my return from lunch & walking into this mess, a quick investigation reveals DNS is pointing to the "New queue\Hostname\IP" not the legacy.
So I message him & ask is he aware of ANY DNS changes pushed today, no he them proceeds to ask what steps I have taken & after 20 mins of teams chat, admits he did put in a request on Thursday for DNS changes, "BUT theres NO way those changes WOULD have ANY effect on certain critical areas printing", then he like the rest of the crew out east decide that with the time difference etc its time to call POETS Day.
Eventually after many breakdowns of communications including command centre thinking I'm working on undoing the DNS screwup, I finally track down someone capable of reversing the DNS changes, verify print operation & leave at 8pm for my local bar.
I got a recognition award for my efforts this morning (No points exchangeable for gift cards) from him this morning for staying late & fixing his screw up.
"I know of someone who did something similar on a mobile phone network."
Was this Australia, quite recently?
"Thankfully one of Charles's ancestors in the monkey colony had anticipated just such an undesirable mutation, and had permanently and irrevocably white-listed the policy management server"
The true hero of this story!
Undoubtedly this firewall rule is written in blood. (or a lot of overtime coffee, at the very least!)
> are The Vultures using {spell check} over on that side of the pond now ?
Won't be a USA Vulture: "....spend the next five days exchanging your labour for currency..." most of us in USA won't labor on Thursday 23 Nov, the US Thanksgiving. (Canada did theirs a couple weeks back, to beat the snow...)
"In this case, a simple spell checker would have caught the mistake."
Sometimes, when in a hurry, I've been known to accidentality hit "add to dictionary" instead of the correct spelling. If I'm in such a hurry that I don't notice what I've done, just noted that the red line went away, the next time I make the same typo, the spilling chucker isn't going to highlight it because I already told it that was correct :-)
But that's why journalists used to have Editors and proof-readers. From comments made by El Reg authors here, it get the sense that at least some of them write and hit publish, especially with "breaking" type articles, with no checking involved other than their own self-checking, which is hard. Proof-reading your own work, it's easy to see what you think you wrote rather than what you actually wrote :-)
I wonder if there's an industry rag for journalists with their own "Who, Me?" column? :-)
The stupid things underline so many words that aren't in their dictionaries that I tend to end up ignoring them and missing a real typo :-)
P.S. One thing that happens a lot to me is U.S. dictionaries and me typing Canadian/British spellings of words, one thing that has helped to condition me to ignore spell checking.
I understand that the way to avoid blame is to save someone else from their 'oh-no' moment. No noise, no fuss. Just a knowing nod of complicity. In this way, if anything ever goes awry for you, you will always have a cooperative supporter. Enough to divert finger-pointing by those out-of-the-loop.
Retelling this story on behalf of a friend.
Didn't affect an entire network, but I do recall a scary experience with a computer that was controlling a 1.5m diameter IR telescope high up in the mountains of Switzerland. We we testing a new IR spectrograph, and one of the instructions I got from the engineer was that I should not move the telescope below -10 degrees declination, or else the liquid nitrogen and liquid helium might get poured out of the system, and various things might fail dramatically. The software controlling the telescope was, lets say, "interesting" in that an English language user interface was a late addition (afterthought is the correct phrase). It was very basic: it would prompt you for the coordinates of the object of interest, show the coordinates on the screen, and ask for confirmation by asking "Is this OK?".
At one point, I noticed I made a typo in the coordinates of the object of interest, entering -16 deg declination rather than -6 deg. At the prompt "Is this OK?" I dutifully entered "N" for no, just as I had successfully entered "Y" for yes previously. I was horrified to see the cheerful response I had seen so often before "Then I go!", and could hear the telescope motors start humming. There was no way to stop this before it pointed to this low position in the sky. Apparently, the user interface would consider any character input as a thumbs up, except for Cntrl-D (Unix EOF). We rushed upstairs expect all kinds of damage caused by this action. Luckily, the spectrograph survived this abuse, and worked fine for the rest of the session. I did suggest to our Italian hosts that they might want to update their UI and manuals.
So, that is proof that 1) the programmer didn't bother with actually checking inputs, and 2) zero quality control or even proper testing.
I would venture that this would be grounds for a serious dress-down, if not a pink slip outright. When you're writing code for expensive equipment, the least you can do is make sure your code is not going to risk breaking anything and that means testing, testing and testing some more.
Because if the telescope had been damaged, it could be proven that the code did wrong and that company would have been obliged to send a new telescope free of charge.
If that had happened, I'd wager the CEO would have suddenly been very interested in testing procedures . . .
Even so, it's abysmal coding around an "Are you sure" prompt.
I remember a similar condition applied to an X-ray detector in an electron microscope - the detector liquid notrogne supply shouldn't be allowed to boil dry. The thought occurred that if exposure to room temperature would destroy it did that mean it had ben manufactured under cryogenic conditions?
"Sounds very much like software written by scientists rather than engineers."
Sounds very much like software lashed together at the last minute and in somebody's "spare" time by scientists rather than engineers.
There, fixed that for you.
At a previous job, I was always fighting having everything in our ERP system entered in all caps. When I addressed the issue with our customer service department (the department where this was a problem), I kept getting told that the system requires everything to be in all caps. I kept telling them "No! It absolutely does not". They kept doing it any way.
So, finally one evening I went around with some pliers and wire cutters, and removed all of the caps lock keys from their keyboards (wire cutters to make sure there wasn't any part of the key-switch sticking out that they can press on). There was a loud uproar the next day, with demands for new keyboards, and threats to "ruin" their keyboards, so that they would need to be replaced. I told them that any replacements will not have a caps lock key on them.
"or shudder French-French"
Icon, for obvious reasons.
I have to buy AZERTY keyboards as I live in France (and ordering in UK layout is rather more expensive, moreso after you-know-what). I also mail order keyboard stickers so I can 'fix' the keyboard back to a sane layout where you don't need a three finger salute just for the @ symbol, or shift to get to numbers, or an entire key wasted on a superscript 2 (seriously, WTF is with that?).
France, of course, being France, took it to the next level. Go look up the BÉPO layout. Thankfully it currently seems more an academic endeavour then a layout that people are actually using. I get what they're trying to do, but oh my god there're four f'king Es right next to each other...my head hurts.
Donning my pedant's hat, no there aren't four Es.
There is one E (E), one E-grave (È), one E-acute (É), and one E-circumflex (Ê). These are different characters, and it's understandable that in a language where their usage is high, having a multiple-key entry method for the ones with diacriticals would be "irritating". On a Mac, the keystrokes are Option-\,E; Option-E,E; and Option-I,E to get the E with grave, acute, and circumflex respectively.
At times during the years of doing It support I've had to deal with a number of different keyboard layouts, and even the more common French layout (at least on the Macs I dealt with, and IIRC as it was long time ago now) was annoying to use as it needed a shift to get numbers.
And back when we were Apple dealers, we used to have access to all the different localised versions of Mac OS - this being when it came on a few floppies ! It was fun to try out the different versions - German extended the menubar somewhat due to the long words, Japanese had a lot of "squiggles" (and Mount Fuji replaced the Apple logo heading the "Apple" menu), but all had some of the menu items in English as they didn't have localised words for everything.
"There was no way to stop this before it pointed to this low position in the sky."
It defies belief that there isn't a set (not one, multiple) big red abort buttons that when pressed will bring the entire machine to a stop.
Aren't they mandatory on big equipment in Switzerland?
[place where I work (France) has plenty of them and they even get tested!]
At work, a colleague was telling me how he caused a bit of a stir when he went to witness some testing of a piece of (rather expensive) equipment we were having customised for us.
Normal operation was for a controlled shutdown as it had things like magnetic bearings that "don't really like" having the power yanked while the machine is still turning. But he insisted on seeing the emergency stop tested ... and it ... did absolutely nothing ! The engineers at the vendor were suitably embarrassed and by the next day some quick redesign and wiring had been done to the red button actually worked - I think it triggered a "very fast stop & shutdown" via the controller rather than killing the power, but that was adequate for the intended application.
This post has been deleted by its author
Windows loomed on the horizon in "antivirus/security package that included a firewall" it was pretty much an iceberg looking for a Titanic :)
In a very similar environment and possibly in the same era, faced with a similar requirement I went for a custom linux/ebtables/iptables (perhaps ipchains) screening bridge between the polloi and *the asset*.
Basically a discarded PC and two network adaptors and custom 2.4 kernel + ebtables userland. [std Python "Four Yorkshireman" sketch ... "you try and tell the young people today that"...]
Worked a treat until it wasn't needed a few years later.
I've never been a fan of firewalling all PCs seperately for rules like this.
Basic security rules like blocking certain services? sure.
Network flow rules? sounds like a nightmare!
L3+ managable switches, or VLAN segmentation is a much easier to control solution, with the added benefit that you enforce compliance for all devices.
(Or back then, a network bridge like you set up)
I remember we once had an odd set-up where we had applications deployed across multiple servers for High Availability, with the Load Balancer checking a status file on each server to see which apps were available for live traffic. This was in the early days of cloud and containerisation, and hadn't moved to the newer solutions yet.
A colleague in a timezone ahead of me had conducted a review and was preparing for shutdown of some apps, and arranged for it to be done. Problem was, the Load Balancer (network team responsibility, we were apps team) was configured incorrectly, so for some of the applications it was checking the wrong status file for availability. This meant that when he shut down the application instances we no longer needed, the Load Balancer took that to mean some other apps were also not required, so closed off access to them.
I was receiving calls from the client about losing access before I'd even reached the office on that one! Luckily it was an easy fix - start up the redundant apps again and all access restored.
Thorough review of the Load Balancer config later and all such inconsistencies were fixed up. Some valuable lessons were learnt through that on making sure that health checks are accurately checking the right source!
On any machine to which you don't have immediate physical access, you need a cron job that, once an hour, inserts a firewall rule right at the top that allows access on port 22 from the static IP of your home broadband.
The time between an ill-judged edit and the door back in reopening is still going to be the longest up to 59 minutes of your life, but at least you know there is a way back in.
This post has been deleted by its author
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
