Engineer crashed mega-corp's electricity billing portal, was promoted Welcome again to “Who, me?”, The Register’s confessional column in which techies unburden their souls by revealing that they have broken stuff. This week meet “Matt” who told us he works for a company that makes “email tracking software for corporate communications.” Said software uses a web server “to detect message opens and …
One previous company I worked for had a rather bad habit of promoting people into positions where they couldn't do much damage. One such person, promoted to the position of project manager, was renowned for having -never- brought a single project to a successful conclusion.
They would never have dreamed of promoting someone who was -good- at their job. After all, who else would do that job?
...we call it (I live in a whole other Country) "Falling Upwards". The American military has another name for that...
A certain former US Senator was once a Marine Pilot... involved in some accidental missile firing on top of a Carrier deck. That missile hit another airplane and it went as well as you'd expect.
What's that expression? "Polishing some leather chair in the Pentagon with your *ss?" or "Pushing pencils in the Pentagon?" Anyway, since the guy was somebody's son, and the circumstances were not clear, (I wouldn't recall the story) they put him in a position where he couldn't detonate any amount of explosive ordnance, by the distance of a single button press.
On the other hand, the man on the article was working with insufficient information.
This post has been deleted by its author
A certain former US Senator was once a Marine Pilot... involved in some accidental missile firing on top of a Carrier deck. That missile hit another airplane and it went as well as you'd expect.
A little fact checking here.. The errant missile wasn't launched from his plane. It was from one on the other side of the carrier and due to static as the techs connected the missile to the fire control system. And the senator is still an active senator. Oh, and he was Navy not Marine.
Years ago I was recommended to a high-ranking manager as a replacement for a staff member working on a (needlessly) complex system.
I swept in and began suggesting immediate fixes and planning to eliminate various systemic problems, including a number of landmines buried in the recompilation process I thought that a more jaundiced eye than mine might view as a job security perimeter.
Some days later I found that I had been replaced by the original staff member, though no-one told me about it and I found out when our different administration practices collided at full speed and their respective boilers burst spectacularly.
I never understood why I wasn't popular as my work habits had always stood me in good stead in my previous position, the one from which I had received such a glowing write-up. And yes, I considered the "give him a glowing reference or we'll have to keep him" scenario, but had evidence to suggest this was not the case and the recommendation was genuine.
Some years later I had moved to a different department and gained some distance and perspective. I suddenly realized that the manager in charge of Project Limpsalong had made his career from leaping into forest fires and noisily directing crack teams of firefighters to Put Things Right.
In suggesting that we could remove the ignition sources I was threatening his visibility and his promotion prospects.
Unfortunately, even had I realized this in time I would have been doomed. I can't have the power to fix a broken process and just sit on my hands, belting it with a Brummy Screwdriver every time it stalls. It's a pride in work thing.
Oh well.
"Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?"
Someone who makes a screwup of that magnitude is someone who has a reasonable chance of remembering it. It's a learning experience. Why waste it?
"Someone who makes a screwup of that magnitude is someone who has a reasonable chance of remembering it."
You assume they even realised they'd done it.
Salestwits can do even more damage than that to a company in a heartbeat (and usually "they're our most sucessful guy", leaving a trail of peopel swearing never to do business with XYZ again due to overselling and general pushyness(*)), as can incompetent management allowing tech staff to refuse to fix their "rather unique" interpretations of standards (eg: some vendor implementations of SNMP) which don't play nice with anything else and have a similar long term effect on repeat sales (along with being struck off candidate lists for other outfits who've been related the horror stories and rely on ABC standard being implemented the same way as everyone else does it)
(*) I was related the story of one salesman who would oversell product capabilities to management then let the customers get so angry and goad them with silence until the tech staff would swear at him in meetings - at which point he'd pick up his stuff and walk out, using the outburst as justification that the customer was being unreasonable and he couldn't work with them. It took the employre more than 20 years to realise the tens (maybe hundreds) of millions of dollars in damage he'd done to the company's reputation and long-term sales figures.
"Promoted for failure?"
Not really accurate. The failure was not giving all the required data for the install to proceed. In fact, not simply "not giving" but actively hiding it. Sadly, that's an all too common occurrence. Job is specced up and when you get on site, it's never as simple as the spec. appears, often made more difficult by not being able to speak to the relevant people for clarification. I deal with "project managers" often enough to know that many are little more than jumped up marketing/sales types who've been trained to tick boxes and have little idea of what it's like on the ground. The good ones always seem to work for other companies.
"I deal with "project managers" often enough to know that many are little more than jumped up marketing/sales types"
They're the best sort if you're a vendor. You can make much more money out of them as long as you document all the ways they've screwed up and the customer will have to pay to get things done the way you wanted to do it.
The client even learned something – Matt said “additional protocols were put into place by the client to prevent this from occurring again.”Matt’s company learned something too. “We only install our software on dedicated systems for production environments,” Matt told us. “Oh, and I still have a job - as the senior tech engineer.”
This sounds like quite a grown up company. They didn't hang the techie out to dry for a mistake (which was probably worsened by the Company's procedures) and they change their way of working to lower the chance of it happening again. That sounds good to me.
Well, it it was working remotely it was up to the local techie to know what IP:port could be bound without issues - still, you should be careful when you don't have a clear picture of someone else's system. It was a shared mistake, with little consequences. Yet, it's good they learned that deploying into a complex network requires some planning and proper knowledge of the overall system.
That said, I never like too much to bind to 0.0.0.0 (or its IPV6 equivalent), because if a new network interface is added for whatever reason (now quite easy with VMs), applications may become bounded where they shouldn't.
At least this application allowed to control binding - there are some that bound to everything without even asking you.
No, not grown up... just political...
Hauling a techie over the coals for this would have involved an investigation that was likely to identify and knock over other cans-o-worms that could do with being left alone.
Just lucky it was a big enough problem to have got away with it, and of course knowing how to reverse the change.
I agree. However, theres a lot of apps that only allow access from the subnet you bound the service to. Either that or its a free for all on 0.0.0.0.
Its a piss poor security mechanism put in place to save time on an actual security mechanism. Either that or the dev building the services hasnt a clue about networking.
A lot of hacked together trashy services written in nodejs using the built in webserver are guilty of this.
the customers change control process should have stopped this from happening.
The project should have had to have done change control on that shared server and explained what was to be done and how.
The OS's management people should have then reviewed the install process and then either performed the install or provided appropriate access to perform the install.
Any danger of the existing webservice stopping should have been spotted and appropriate action advised BEFORE the change started.
Me personally, i would not have let a third party install anything on my important webserver. For what they needed i would have insisted a new server be spun up or vm. In fact i'd have a pre determined process for adding new web servers. In addition i'd have reverse proxies and load balancers and ensure new URL's are either on our existing webservers or if it needed its own software stack installing it'd be on a new dedicated os, isolating something new and different from the existing.
looks like the customer is at fault here for not having rigorous enough processes to ensure the integrity of their service.
Sadly, a "just get it done" attitude from higher ups increasingly takes precdence over engineering best practices. As lomg as you strictly adhere to ITIL forms, there's no patience for attention to substance. I assume most enterprise shops are now hopelessly borked because of this. Many vendors, Microsoft leading the pack, insist on their pound of flesh for dev and qa systems, resulting in the home team bean counters refusal to approve them. So things are usually brokem before a vendor shows up. Add to this the prevailing "Training? Yeah, we'll get to that. Just ask the consultants. How many tickets did we close today?" attitude of many managers, and you've got a prescription for vendor-inspired mayhem like happened here..
I did a recent stint where change meetings were the boogeyman, directly causing my part of the world to go T.I.T.S.U.P. a couple of times.
And, really, you never know what kind of vendor you will get. I have worked with more than one vendor which told me they would not support the product if we did not do certain things their way, and this usually happened on the day of installation even with several calls and emails beforehand supposedly detailing the process and our requirements.
From turning off all workstation firewalls*, to blank SQL sa passswords, to, yes, full take-over of IIS installations in bindings -- as happened here -- or putting an application in the default website rather than its own. As well, the customer had no means to stand up another server just for the application so we would have to go with it, at least for a short time.
* still forced by a major medical software vendor for one of its Borged products which I will not name, but it does rhyme with Henry Schein.
Working on a site that takes perimeter control seriously we have a very simple form for vendors to complete before installing anything which may need a route through the firewall. In the past 5 years I cannot remember a single vendor who completed it in advance despite frequent reminders. We were not in the position where we could refuse permission for the install to take place, there were always somebody else's critical project with ridiculous timescales. I would normally end up sat with a firewall engineer looking to see what ports and protocols were being bounced from the new server IP address then negotiating with the vendor about what was allowed through.
The vendor default position is normally cant we jut open all ports to that IP address and they are seriously shocked when this is not allowed
"Matt’s company learned something too. “We only install our software on dedicated systems for production environments,” Matt told us."
About ten years too late, by the sounds of it, but welcome to common sense.
Literally, what was their main web portal doing running the email tracking too? That's just stupid.
Kind of forgiveable in a mom-n-pop kind of place, but the second you're into a "real server" then you should be virtualising out to individual VM's with stated purposes.
Case in point: When I arrived, my workplace had four physical servers. One of those ran finance and, for some unfathomable reason, file shares and print server (no finance integration, including talk-home software), and a myriad other things.
Replace with the same number of physical servers, virtualised everything, and now run 30+ virtual machines on pretty much the same hardware. Ironically, not only is everything faster, it's more energy efficient, much more resilient, everything is replicated to more places, and you can safely assume that one server does one job (including the hypervisors which do nothing but... VM hypervising).
There's no way that you should be doing ANYTHING else on a public-facing web server machine. Hell, it shouldn't even be in the same VLAN / network.
And if you've not pushed your public-facing stuff through an IDS/IPS reverse proxy (also a separate machine), then you're just opening yourself to attack.
Exactly this.
Apart from anything else, why would you run a public-facing server on the same network as internal systems? If stuff inside your corporate network needs to be accessible from the public internet, it should be done via tightly controlled ports through a DMZ, tied down as much as humanly possible.
Do you want to get hacked? Because that's how you get hacked...
Client was silent on critical network details and little things like multi-tenancy
:-) Their Pleasures to Ensure and Endure to Glorious Release with Immaculate Relief in the Powerful Energy of Ecstasy. .....with Heavenly Travelling Servers of the COSMIC Trail Trialling Futures for Media Presentation of an Unfolding Augmented Virtual Reality ProgramMING ...... which Commands RePlacement of Presents with Search and Research into the Tale.
And when the SourceRootRoute is Hellishly Heavenly, is the posit here IT be Almighty with Endless Almighty Forces just doing their Sensational Protection Thing in Lead of Attending Sources.
Be You They? In The Singularity is Such Easily Possible.
One small quantum step for Mankind, one giant quantum leap for All Kinds into what Treasured Temptations have to Offer and Realise/BeTrue.
That's noteworthy news, El Reg. Even an Exclusive.
is the ability to segment off parts from other parts so external technicians can work on it. Too many systems I encounter house multiple functions from several systems on the same box, which make it a nightmare to pass over to someone else without risking compromising the entire lot. Architects who used to be Developers rarely thinks of this, those that were Sysadmins notice a little better.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
