CIO made a dangerous mistake and ordered his security team to implement it Welcome to another instalment of Who, Me? It's The Register's reader-contributed column that shares your missives about massive mistakes, and how you managed to move on after them. This week, meet a reader who asked to be Regomized as "FireBug," a name that makes sense because the story he sent concerns a firewall he worked on …
Even when the order comes in a meeting full of dissenters and after I have insisted on the CYA email, they never seem to get the boot for this type of issue.
Hence I decide that the company is no longer worth getting out of bed for and go to pastures new. And in my next interview I do explain clearly why I left the last company, and it has never seemed to be held against me.
Happened to me twice, once, there was a vulnerability found on Cisco ASA vpns, and one of the security people found a fix that had been out on a blog.
They wanted me to run it on the production firewall, I put my foot down and said I wasn’t installing an unauthorised fix on a production firewall hosting 40 vpns.
The department director sent me an email saying he authorised the change, I applied said change and my vpn went down 20 seconds later….
When I got to the office the firewall was stuck in a boot loop, it took an entire weekend to get it working again.
second time someone doing a software deployment forgot to request a firewall change and wanted me to do an urgent change. I stuck out for an authorisation from their director (who I knew well) before making the change. He was glad I did and I thing the testers got a telling off for not testing before going to production- this didn’t break anything but proves you must CYA…
Yeah, that'll indeed bruise an ego, especially one who doesn't know he doesn't know but thinks he does. Ergo, this was obviously the nicest decision they could take.
That said, it's nice when you get a manager who knows the nuts and bolts. There's less waffling about with business-speak in meetings and when you explain why something can't be done that way, there is agreement without much discussion (that's supposing that you know what you're talking about). Plus, if you're doing things right, you get his entire support when things still go pear-shaped.
That is very much the difference between manglement and management.
I'm fortunate enough that at the current place my bosses are the latter, and leave me to get on with my job.
Certainly had the opposite at prior places - one particular mangler had a reputation to the extent that even basic instructions from him were insisted upon via email by a good proportion of the 20 or so of us that had to put up with him as a boss.
One of the best managers I ever had was decidedly non-technical (he was from marketing).
He would not lose face by telling his peers that he'd check with his guys (usually, me) and come back, rather than being browbeaten into committimg to... unwise.. ideas.
A truly excellent manager knows when they are out of their depth; in IT, knowledge ages very fast (and badly) - won't I know it!
There was a Lotus Notes running on duck.domain.local, Server 2012 R2. The replacement on Server 2022 got installed, and a few days later a duck.domain.local DNS CName was created to point the new server. However, before the old server got fully decomissioned, the IT responsible customer employee, who said he will decommission the old one, died. Nine month later a call arrived, "why is there a DNS CName entry for duck.domain.local? It is pointing to the wrong server.". A few mails with a clear "Cover You Bases" style communication, the CName was cleared because "he gave the order to do so", the old one re-registered his name, and then a few hours later hell broke loose. Luckily the fix was easy, reinstate the CName, and shut down the still running old "duck.domain.local" Lotus Notes, and everything was fine.
I can think of at least three reasons why that was a terrible configuration from a Domino pov off the top of my head: no site doc; no clustering; CNAME is dumbass in Domino... If I scratched my head a bit, I could probably think of more.
At the root of it, though, is that it looks like it was designed and configured by someone who should never have been let loose in the Domino Directory. If that was the deceased, well, I'm sorry for his family, but it doesn't excuse a terrible configuration.
wanted me to implement a particular feature in his software. Note that he had no software skills, only marketing and sales. I explained to him that it wouldn't be a good idea as there would be specific negative consequences. I forget the details now. However, he insisted so I did as he directed. A few months later the fault I predicted came home to roost and the guy had me remove the feature. He was annoyed with me because I didn't dissuade him from his folly forcefully enough! So my fault!
Similar experience, but with a user. The details are lost due to the mists of time but it was along the lines of:
User: Can you get the system to do such-and-such?
Me: No, I can't do that.
User: Oh come on, you can do it for me?
Me: No, I can't do it because the system won't allow me to do it because of X, Y and Z.
User: But <<your boss*>> said it could be done.
Me: Ah... well, uhm, I'm afraid he's wrong. It can't be done for the reasons already stated.
*You know the kind - he "knew a bit about computers" (mid-90s) therefore, he knew "everything".
Why do all those newfangled systems today no longer allow what could be done back in the mid-90s?
Because in the process of making everything "smart" the folks specifying and coding it have made ever-more-basic assumptions about how the workflows "should" operate, how the user "should" want to do things and how the admins "should" need to configure it.
Rather than give a wide menu of possibilities from which to build a configuration that fits its intended purpose on THIS site in THIS corporate culture.
And the customers manglement go along with it because that way they can employ more (cheap) trained monkeys and fewer (expensive) knowledgeable and professional sysadmins.
Nothing can get in or out!! Superb! Nobody can find anything! A bit like hiding demolition plans at the bottom of a locked filing cabinet in a toilet with a sign on the door saying "Beware of the Leopard".
I assume Firebird told the CIO "I don't like saying 'I told you so', but I told you so," or at least was sorely tempted to do so.
I installed a new Firewall at one employer. The IT Manager wanted it set up so that only the DC could talk to outside DNS servers, so I set it up that way. 2 months later senior management decided not to renew my contract, despite the IT manager fighting my corner.
A couple of months later, they had a lightning strike which took out the VMware cluster, but the firewall was merrily chugging away...
So, no VMware = no DC = no DNS.
I got a call from the IT manager and had to talk him through going through the Firewall settings and finding the rule that blocked anything other than the DC talking to an external DNS service...
The company was unbelievable. The IT Manager left a few months later, they gave the position to his newly qualified apprentice. The wife of the CEO moved into his old office and threw out all of the folders in the office...
A few weeks later, the software supplier rang the old IT Manager privately, because the apprentice had contacted them to buy Windows licenses. He said, I thought you had purchased Datacenter licenses? Yes, he had, but the CEO's wife had thrown out all the software licenses for all products, because they were taking up space in her new office!
...is decide what the job is, decide who should do the job, let them get on with doing the job and hold them to account for doing the job right.
What they actually do do, all to often, is direct someone to follow a set of instructions and stand over them while they do it.
I used to work for Verizon. They wanted all their customer's search results changed to Bing. Not some simple URL change, but a backend modification to the site and apps. An expensive project, that took months. We advised against it, and did some comparisons of search results to point out how much worse Bing would be, but they insisted on moving forward. I don't know why, but I'm willing to bet Bing wrote a big check.
A few days after the switchover, search just stopped returning results. We investigated, and found Bing's API just wasn't returning anything back. Sent a bug report to Bing and told Verizon exec's nothing we could do. Took about 6 hours before Bing started working again. Same thing repeated roughly every couple weeks. Besides the outages, customers were complaining about the poor quality search results as well.
A month after the switchover, we were asked how much it would cost to switch back off of Bing. We told them it would cost just as much as it did to implement Bing in the first place. They opted to leave Bing in place.
I had to visit a very large US customer about a problem on the mainframe. The team were depressed because the new big boss had told them to move off the mainframe on to commodity hardware. Commodity hardware means if you need more capacity you go down to your local Walmart and buy it. The big boss said it would save big bucks once the transition was done.
As part of my problem resolution I had to report to the upper management chain. I also spoke about my views on their change to commodity hardware.
I made comments like "you are very brave" (as in Yes Minister), and "I do not think you understand the capacity you need".
Someone kindly took me aside and explained how their estimates were right. They looked at the amount of traffic into the mainframe on the busiest day, and used that as their base line.
My comments like "on the mainframe 90% of the traffic between applications and the data base are cross memory and is not measured" fell on deaf ears. (Mouth engaged, brain in neutral, ears turned off)
They got a taste of things to come when they ran tests with live data and found the network could not cope, fixing the network then showed they didn't have enough commodity hardware.
Overall the project cost more than the mainframe - but they could not go back a) the mainframe people had all left, b) it would have embarrassed the people who wanted the new scheme. c) they would have all this kit they didn't need.
I used to meet up with someone at a conference, and hear all the news.
Frankly, breaking-out of that proprietary platform lock-in was probably still a good move. The longer companies continue to update their in-house software on the mainframe, the more expensive it gets to transition off. Even if the first go-round was more expensive than planned, commodity hardware quickly got much faster and much cheaper over the years, and generally required less labor than IBM.
Companies that never made the jump, still running business-critical functions on their mainframes, sure wish they had done such a transition long ago.
This is, quite simply, not true. For applications that actually need the kind of guaranteed uptime and throughput mainframes can provide, there's no cheaper alternative. Yes, other solutions can and do work, they just end up being "mainframes by another name", with similar - and often higher - running costs. The money to switch infrastructure is basically waste.
It's perhaps worth noting that even most of the "business consultancy" companies have realized this by now and don't recommend even trying* beyond reasonable modernization (as one should do for any application)...
* Full disclosure: I work for a company that still uses mainframes extensively and recently had one of those companies review their "long-term strategy", with (to the mainframe engineering team, myself included) entirely unsurprising results.
For applications that actually need the kind of guaranteed uptime and throughput mainframes can provide, there's no cheaper alternative.
I've seen z/OS and VSE crash plenty. Linux on x86 servers can easily run longer...
Mainframes do have slightly better RAS than a single x86 server, but not better than even a small cluster of them. IBM spends on lot on marketing, and the best they can come up with is that the "Mainframe" might be 99.99999% reliable, while a single x86 server (at 1% of the price) is only just 99.9999% reliable. While hand-waving away that it is the hardware is reliable... NOT their proprietary operating systems that run on it. (NOTE: Not hyperbole, actual IBM claims: https://www.itjungle.com/2024/10/21/ibm-nears-the-end-of-the-road-for-server-reliability-improvements/)
Mainframe IO is literally two Intel PCIe NICs in an enclosure... Or a couple Fibre Channel cards for SAN. You can *easily* have far more IO in a rack full of x86 servers than on a single 48U rack IBM mainframe.
Disclosure: I'm a senior z/VM & Linux on Z (s390x) admin. I manage several mainframes, and support several sysprogs...
My above comment on IO might be misleading to non IBM pros.
Those "two Intel PCIe NICs" or "couple Fibre Channel" I referred to are each one "card" and IBM mainframes accept about a dozen cards in each I/O cage. Larger IBM systems may have more I/O cages, but it's common for a single rack mainframe to only have one I/O cage.
In the old days, when x86 servers were single core, and NICs didn't have ToE, mainframes certainly could push more I/O than commodity servers. But now? Not so much.
"We should really get more memory for the server."
Nothing.
Then the 6 monthly OS and RDBMS upgrade over the weekend (those were the days when upgrades came on CDs).
On Monday explain to IT manager what and why thrashing is and how it relates to the problems we're having now.
In such circumstances it's surprising how quickly decisions can be made and fortunately the field techs arrived pretty quickly.
And usually in those instances, money somehow magically appears to solve the emergency...
Let me guess: the memory that was "too expensive" ahead of time was then purchased faster than you can say "priority shipping" and the field techs arrived to install it, thus averting disaster.
It's amazing how some managers seem to bumble their way from emergency to emergency, and somehow can tap dance around ever needing to explain why they seem to have so much experience solving crises...
In 1990, RAM was about a hundred bucks per meg. By ~'92 it was hovering between $92 and $95 per meg, where it stayed until about 1997, when the price started to plummet. By late '98 or so you could get SIMMs for about $5/meg new, half that used.
The HDD world was even more dramatic ... In January of 1990, a client paid $1199 for a 40 Meg Western Digital HDD (with a one year warranty). In November of 1994, he paid $849 for a 1 Gig Seagate (five year warranty). By July of '98, a WD 8.4G was $382. That's roughly a drop from $36/meg to 85¢/meg to 5.23¢/meg.
People who weren't in the industry as adults during the '90s have no clue how fast things were moving ...
Have been reading Wikipedia entries about the linage of computers which (essentially) started with the Manchester Baby. Fascinated to note that there were several cases where in the process of building one computer they had already decided what to change for the next to make it <several> times faster. Of course, that computer was a year or more away so carry on with the current one and put it to work while you wait for the next.
It must have been a really exciting time to work in computers. I mean, we got a pale shadow of that in the 1980s where each year there were genuine improvements but increasing software complexity somewhat hid them from end users! And now, I'm stuggling to justify upgrading 5+ year old laptops and desktops because actually they are good enough still!
M.
I have a hand-written receipt[0] dated early December of 1977 for "8ea 16 Kbit Mostek MK4116 DRAM, new, in factory tube, with seals" ... for the low, low price of $336, plus tax. So I guess one question to the "42" answer is "What was the price in US dollars for 1 (one) 16K bit DRAM in late 1977?" ...
By way of reference, $42 in '77 would be worth about $225 today.
[0] From the late, very much lamented Haltec, no less ...
Just for reference, in my records I bought in 1992 4Mb of 70ns EDO RAM for what would have been, at the time, €154.36.
I'll leave the financial wizards to calculate the price per GB at the time and the equivalent price today.
Also, following my records, in 2014 I bought 16Gb of DDR3 1866Mhz CL9 for €167.95 (value at that time).
I'm sorry, but I think this indeed proves that RAM prices have dropped so far that it's a wonder people still make them.
Printer memory was particularly obscene, especially back when it required proprietary daughtercards instead of standardized modules.
I remember always wanting to get a 512K upgrade board for some HP laser printer, which would've brought the memory up to a whopping 1MB but, crucially, would've been enough to do full-page bitmap prints at full resolution. I could never justify it, though, as the expansion board cost SEVERAL hundred dollars.
well ok... about 1997.
Heard a tale of a mangler who ordered the minion to do everything his way, said minion says "no bosss, you do it like that it will fail and be scrap"........ mangler was most insistant. sadly minion didnt think to get the order in writing. needless to say said item went to the great scrap heap in the sky while the mangler screamed at the minion for fubar'ing the job.
minion lost his job as a result.
A few months later though........ minion has his revenge at an unfair dismissal hearing as one of the other minions heard the whole thing....
I never worked there due to the number of people who came and went for the guy. then he went bust and there was much rejoycing.
Had a boss who was actually a half-decent guy, but he did do some questionable things - like shut down the machine room aircon (I can't for the life of me remember why), Trouble was, he had no idea how to turn it back on again (we'd just moved to a new DC and no-one knew the building that well. The aircon was a massive fan that blew cold air into the room next door). We had about 40 minuttes before machines started to go down due to heat, and the DC engineer was 30 minutes away. Close call.
The reason we couldn't restart the aircon was because of power off button had a sprung loaded rotating locking ring (you'd never know just by looking at it). All the engineer did was rotate the ring to allow the power button to reset and start the aircon.
For anyone wondering, this was back in the 80's and the 'machine room' was not purpose built - it was re-purposed.
You had aircon? When I started at one job, the computer room was on the top floor, south facing and no aircon. In summer, the first person in the office opened the windows in the server room and at the other end of the building in order to get some cooler air flowing into the server room!
My first week, I told them we needed AC, they said no, no money (they went out and put AC in the CEO's office instead). I bought a thermometer with remote censor. It read a peak of over 60°C in the rack! Not the CPU temperature, the ambient temperature between the devices in the rack! Surprisingly, despite my dire warnings, the whole lot survived the summer, apart from one old server, which crashed.
Compressed air to remove 12 years worth of dust from the server brought it back to life again, somehow.
I left pretty soon after that.
OP here. When they first moved in, they hadn't even tested the aircon. They'd started installing the hardware (ICL 2966, a few rows of FDS640s, magtape drives, EDS200s), powered it on, thought "oh, it's getting a bit warm, we'd better turn on the aircon". Queue lots of little brown bits of rust being spewed forth all over the place.
Still, the place before had the machine room on the top floor. The building had a flat roof. Which leaked. I remember putting buckets on top of mag tape drives to catch the water! They had proper aircon though - which was great until one unit burst a pipe....
Oh, and the ground floor had an industrial unit - heavy engineering of some sort - so it wasn't uncommon to feel the buidling shake. Great location for a machine room full of spinning disks.
Fun days. I miss them.
Similar tale but instead of an aircon, I got a bathroom extractor fan with self-closing cover hooked to a thermostat. When the cover stuck, the motor superheated and the whole thing caught fire, it created giant plastic flakes that filled the server room. The G5 HP kit ran the fans like a 747 at takeoff, pulled all the flakes in, then packed them tight inside before thermal shutdown. There was a 10ft scorch mark up the side of the building to boot.
Bless it, the kit was vacuumed out and then booted with no issues. The room was cleaned. The CEO agreed to a real server room aircon unit, which was the first our HVAC contractor had ever installed and immediately malfunctioned to 15c and 86% humidity (felt like a tropical rainforest) before getting it right. The HP servers and MSA soldiered on.
That's why I choose HP kit still.
They tried, but I ignored them.
MD: Why are we paying for an ADSL line.
Me: It's our backup in case the fibre link goes down.
MD: Cancel it.
I ignored him and he got fired a while later for other reasons.
Chairman of the board: We don't need to be paying for this ADSL cancel it.
I ignored him too.
Six months later workman puts a JCB through our fibre connection, we are without it for 4 days. I had the entire network redirected through the ADSL within an hour of the fibre being cut and we ran on limited bandwidth for 4 days, but weren't totally dead in the water which was the alternative. Chairman actually thanked me for ignoring his instruction to cancel the ADSL.
Back in the late 90s, I was doing desktop support for a small startup. Back then, ThinkPad 600e ruled the laptop roost - they were tough as old boots and we could manage them quite effectively.
Boss man wanted a little Pony Vaio, probably so he could show off to his mates. It was an expensive piece of crap which could only run Win98, so absolutely not supportable in an NT environment. I politely said no, not a good idea. He went to one of my people and got them to order 3.
He got one, and somehow ended up with a laptop bag bigger than anyone else. No end of ongoing support problems (which he carefully never asked me about) later and he got a "sideways" pro/demotion to pre-sales. Needless to say, "because of his new role", he felt a ThinkPad might be more suitable.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
