Techies tossed appliance that had no power cord, but turned out to power their company Welcome to another week of nimble newsifying from The Register, which as always kicks off the working week with a fresh instalment of Who, Me? It's the reader-contributed column in which you admit to mistakes that almost trashed your career. This week, meet a reader we'll Regomize as "Steve" who sent us a story that took place …
So Steve identified a presumably undocumented single point of failure, caused an outage that presumably didn't kill anyone, and learned an important lesson meaning he'd almost certainly do things differently from now on, making him a more valuable employee......
And they fired him. Bravo.
Unless this was genuinely a pattern of reckless behaviour that had already been called out, I just don't get some companies' attitudes to throwing techies under the bus like this, especially where their own line management is complicit.
It's more likely to drive people into covering things up when they make mistakes, rather than being open and honest.
There are perhaps better ways that it could have been handled - such as giving notice that there may disruption, yanking kit out on the weekend instead, additional monitoring/alerting for critical services, etc. But eventually *someone* was going to come along and unplug that box, whether they've done any 'due diligence' or not.
But, as you say, the response that ensued was the telling part. Had Steve not ultimately lost a job over it, he'd have done well to find another job elsewhere and steer clear of such a toxic culture.
It is sadly all too common to throw people under the bus rather than address the challenges and issues head on, but there are people in senior leadership positions who will take the latter path. People who genuinely value you for more than just how you contribute to profits; people who wouldn't even countenance using you as a scapegoat and would instead throw themselves under the bus first; people who would offer themselves up to the chopping block ahead of you when it comes to redundancies. I've worked for a few people like that and it's genuinely transformative.
No job/workplace is perfect, but good leadership makes all the difference.
>There are perhaps better ways that it could have been handled - such as giving notice that there may disruption, yanking kit out on the weekend instead, additional monitoring/alerting for critical services, etc. But eventually *someone* was going to come along and unplug that box, whether they've done any 'due diligence' or not.
>But, as you say, the response that ensued was the telling part. Had Steve not ultimately lost a job over it, he'd have done well to find another job elsewhere and steer clear of such a toxic culture.
Steve (and his boss) were performing unapproved change work in a datacentre without informing anyone of what they were doing, when they were doing it and why. This is no longer the wild west of the 1990s - the story happened this decade - and you can absolutely guarantee that they broke a shedload of company rules designed to prevent people doing cowboy stuff like they did.
As he was contacted by his boss Steve would have every reason to believe that it was approved, especially with a solid reason of reducing costs by eliminating redundant rack space. If anyone were to get fired it should have been the boss.
OTOH it does seem a bit extreme to remove an unidentified bit of kit without documenting it. Maybe they were a bit stir-crazy after lockdown.
Exactly. And why were there no labels on the Ethernet cables?
A one-time colleague told me of a supervisor she worked under at Bell Labs. His standing rule was that both ends of an Ethernet cable had to have a label stating where it went (that is, what was at the other end). He enforced it by immediately disconnecting any unlabeled cable he noticed, and if anything broke it was considered the fault of whoever connected it in the first place.
I am working in a substantial high-availability environment involving Etherchannel port groups and left/right multipath FC fabrics (so lots of cables) where the main kit was installed back in 2010/11, and although most of the individual items of kit have been replaced, the 'shape' has remained pretty constant. Most of the cabling was installed with spare capacity, and the majority of it is still the original cables, just having been re-used as they've become free (although this re-use has not always been done with neatness in mind).
When I started sometime after the initial installation, I looked at the environment and saw identifiable unique matching labels at both ends of the cables, and concluded that it would be easy to work in. What I didn't realise was that although the cables were labelled, nobody had maintained a database of what each cables were used for or what they were connected to!
So even though the cables were labelled, re-working any links still means tracing cables through the knot of cable spaghetti. Rather disheartening, really. Nearly got it right, but failed at the last hurdle.
This environment has been scheduled to be decommissioned for about the last 7 years (but each time it's been postponed), so nobody can stomach de-tangling and the required outage for an environment that has, officially, not much time left.
Yes, there should have been some documentation for the entire installation with a binder and a log sheet of any work done. This should be a printed copy of what is in a digital achive which is accessible company wide and whose URL is referenced in the printed copy. Presumably this device was energized by POE? My company has been called in on existing systems with little to no documentation so many times... Was the original installation not done per some specification? I never understand this. Would you buy a car if no specs or maintenance data were avaialable? Would you not have some service record for your car?
> Steve (and his boss) were performing unapproved change work in a datacentre without informing anyone of what they were doing, when they were doing it and why. This is no longer the wild west of the 1990s - the story happened this decade - and you can absolutely guarantee that they broke a shedload of company rules designed to prevent people doing cowboy stuff like they did.
The box in question did not even have a power cable plugged in. If it was PoE powered you would almost certainly see blinkenlights on the front or hear/see fans whirring - both of which would give pause for thought when it comes to unplugging it.
How on earth is any reasonable person supposed to assume that an unpowered piece of equipment in a rack actually provides a critical path for internet connectivity? Don't forget that the entire escapade was at the suggestion of his boss, with the express purpose of removing unused kit (and presumably save on rack space costs as a result).
The *real* problem in this situation is the management/leadership culture that was exposed. Questions could - and should - have been asked as to why they were making unscheduled & unnanounced changes. And questions could - and should - have been asked as to why the internet connection was solely reliant on a box that wasn't running and didn't even have power connected. But the response was instead to fire Steve, and... honestly... in many ways that did him a favour by getting him out of such a crappy environment.
> The box in question did not even have a power cable plugged in. If it was PoE powered you would almost certainly see blinkenlights on the front or hear/see fans whirring - both of which would give pause for thought when it comes to unplugging it.
It could have some passive optic failover inside of it that would have no indication of it being active in use.
But yes, it should be labelled and documented.
Passive pass through was my first thought as well. Rack mounted DC kit is unlikely to be PoE so basically this box was redundant* but nobody had arranged the outage to directly connect the up & down stream devices possibly due to all the WFH going on at the time.
Either way, they were in sparking spurs mode.
* there's a small chance that it was a fully configured and deliberately powered off as a final get out of jail when SHTF.
One specific Riverbed Steelhead kit my previous company used comes to mind. The box has this strange LAN splitter connected to it, one end going into a switch and another into WAN, and the box itself has a second LAN cable going into the switch. If the box received power then the splitter would switch data to go into the box and cut off the link directly going into the switch, which will then do it's job. If the box is powered down however the splitter would somehow pass data onto the LAN cable instead and allow for a data path direct to the switch.
I'm going to take a wild guess and say the box that they unplugged was a Steelhead or similar caching device.
The device was clearly an optical tap, the mythical and *very* *expensive* security hardware snake-oil salesmen like to sell to ignorant CISOs. This kind of device is really impressive, it duplicates all the traffic and makes pretty dashboards, it gives a sense of omniscient power to the customer. Except the volume of intercepted traffic is way to high to do anything useful with it except as one-shots, and interpreting the pretty dashboards requires expensive work-hours of security experts. So the middle-boxes get installed, demoed a few times, and then are mostly shutdown till some crisis justifies the expert work-time.
Because in shutdown mode they are a pass-through they can be installed without any coordination, the CISO does not need to consult with any technical expertise that would blow holes into the seller speech. Access the datacenter, unplug the main output fiber, plug it into the middle box, pretty easy to do causing just a short traffic drop that won’t run any alarm bells. The inevitable outage occurs in the future way after the snake-oil salesman has been paid, either because of a pass-through hardware failure, or because some network reorganisation stumbles on the undocumented “critical” SPOF box. But the security “expert” who recommended the purchase does not care, keeping traffic flowing is not his problem, it worked at first so it should work forever maintenance is not something security experts worry about.
This was my thought, not having that level of technical knowledge, but plenty of organisational team management experience, risk assessment etc.
If a bit of kit is system critical 1) Everyone with any responsibility for that location should bloody well know it's there and 2) It needs a sign saying "here be dragons" or something to that effect.
That neither of these was in place suggests company wide failings..
"That neither of these was in place suggests company wide failings.."
That was my thought on reading the story. I'm a keener on documentation and procedure coming from aerospace and a licensed pilot. With an airplane, the fastest way to drop it's value in half, other than crashing, is to lose the logs. With rockets, it can be important to keep running documentation on everything that has been upgraded, replaced and modified. I also started a checklist that went at the front of the printed/written logbook to list anything with a downcheck since not everything can have a lockout or non-op tag affixed although it was helpful to put one at the propellant loading ports and power connections. Even with a small team, a person could get hit by a bus and if they were the only one that knew of a defect, bad things would happen in their absence.
I started putting labels on my wall warts and plugs so when I was crawling under the desk and needed to know what was what, I could just look at a label. Same for networking cables, etc. I just picked up yet another P-touch for $1 at an estate sale the last weekend which brings me up to 5-6 of them. It was just before the sale was closing and the leftovers were going to be binned so that's a point where I can negotiate for awesome deals on a big pile O stuff. Also got a NIB Dick Smith negative ion generator kit from the '80's for $1, a big box of Radio Shack project boxes and an old HP 200 vacuum tube oscillator. I wish I had showed up earlier as everything was higildy pigildy and I'm sure I missed more good stuff for lack of time to look. At a sale last year I was looking at a paper shredder that I decided not to get........ about $80k in cash and PM's under the shredded paper in the bottom. Grr. The auction staff found it during the clear out and gave it to the surprised family members who had no idea their recently deceased parent had that stash. I made sure to list where my box of collectable currency and bullion to include with my personal papers. It would be too easy for it to be thrown out given where I've put it if somebody was just doing a quick triage of what's in the box where it lives. I've not found anything like that at an estate sale, but I'm always looking.
Personally, I would have noticed the activity lights on the ethernet ports and took a pause.
Sometimes you gotta unplug something and see who screams, but never just rip out an unknown.
I learned that the hard way when I took down and entire automobile manufacturing plant one time. Luckily, "just following orders" saved my bacon.
I haven't worked in networks for a little while, but back when I did it was pretty common for IDS/IPS devices to "fail-to-wire" ie if they failed in some way they just acted like wire. So probably it was an IxS of some kind and either wasn't being used or nobody could be bothered with the recabling costs to join the two ends together so they just left it there connecting two network cables together.
I headed up IT a global transportation company across the 1990s. I would have fired them before they exited the CoLo.
However, they wouldn't have had access in the first place. And they would have been legacy people that I hadn't hired because I didn't hire blithering idiots like this.
ZERO defense for these two, and it's telling about those here trying too do so.
I assume you've got a set of assumptions about the circumstances here. We all must to some extent because the only facts are those in the article, and theoretically any of those could be lies too. However, the defenses and attacks I'm seeing are fundamentally differences in accepted facts like did anyone above the boss authorize this decommissioning operation? I've seen comments deciding that's a clear yes and a clear no, and my own defense is a more moderate I don't know on the boss, but the employee covered in the article had sufficient reason to believe that it was because his boss said so and would be the path by which the information would be delivered.
When you confidently assert what you would have done, consider whether your answer would be different if the unstated circumstances were the opposite of what you're assuming. If you would fire both of them no matter whether there was authorization from above, whether anything was documented, whether there was system monitoring, whether there was a logical purpose for that equipment, fair enough. I would not agree, because people decommissioning equipment with permission and on orders removing something that shouldn't have been there is enough for me to take their side if that's how the facts ended up. If you're making that decision based on guesses, you should at least tell us what guesses you're basing it on.
During '00 and '01 I was wearing the hat of Incumbent NOC manger AND the hat of acting NOC sysadmmin supervisor.
When I did anything sysadmin related, I made my "in case of bus hits this is the management replacement" approve my orders, ev en though he was my direct report (supervisor of performance and monitoring), and I invol ved my boss too (regional technical manager). This was a sanity check, as, while i was very good at my stuff, I am only human.
What on earth makes you think that good people would want to work for you these days?
Why wouldn't you have investigated why it got to that state, why someone - presumably under your watch - left a ticking timebomb single-point-of-failure in a rack without a big fat "here be dragons" sticker on it?
Why would you immediately jump to firing someone who was doing something with good intentions and trying to save the company some £££? Have you never made a mistake?
Judgemental asshole.
"Good intentions" have started many a war. And yes, as an exec over data centers supporting billions of revenue (this story is about a financial company, regardless of revenue,) I was judgemental. That was my job, and These two were reckless beyond any measurement.
If your fee fees got hurt because I would not have two (ok, no idea about Steve's level of experience, but certainly his boss) idiots playing with critical corporate assets, then you're not living in reality and haven't had much responsibility placed on you.
And if they were there under the instruction of their boss, the CTO, or the CIO?
What we can say with complete certainty is that the senior management was lax and shit. You said it yourself, they should not have had access.
I'm in financial services, albeit at the millions of revenue rather than billions end of the scale, but my chairman used to be a COO at a billions/systemically important institution, and I know a few CIOs, CTOs and CROs (risk, he's in insurance and IT is a risk) including European head of one that kind of literally is the system in terms of money supply in the USA.
None of them would have sacked before investigation, they might have sacked people, but it would be the people who created such a lax access environment, not the poor schmucks who think that's normal. Rot comes from the top.
There are unlabelled boxes in my organisation's datacenters. And they'll stay unlabelled. I know of them, but don't know what they look like or where they're installed. I don't even want to know. I actually don't 'Need to Know'. There's a reason for why they are unlabelled, and if I say anything more, some scary guys will start saying not so nice things to me.
Any large organisation that is considered an asset to their country most likely have them. Many smaller organisations also have them. Financial organisations definitely have them.
> Why wouldn't you have investigated why it got to that state, why someone
He already answered that for you,
> I would have fired them before they exited the CoLo.
> However, they wouldn't have had access in the first place. And they would have been legacy people that I hadn't hired
-- because regime change, where every fault is the fault of the people who came before. *Especially* the experienced ones who are still there and understand the reasons for things to be the way they are - get rid of them as quickly as possible, by any excuse available!!
With a previous work hat on, where I worked we had an internet connection from what we would these days call an alt-net. It had started as a modest sized outfit, been acquired (and renamed), been acquired again (and renamed again), and after several iterations became part of a very large well known UK telecoms provider.
But as hinted at, at each acquisition, some of the people who understood our bit of the network left - either made redundant or saw the writing on the wall and left themselves. And at each iteration, knowledge of our bit of the legacy network got less and less.
I recall at one point we wanted to up the speed of the connection - and knew that the hardware (radio link to a local POP) could support it. But their systems by this time had no knowledge of what we had - in fact for billing purposes we had a fictitious combination of products just to make billing possible. IIRC it was at this point that I looked at the packets coming from their equipment (LLDP, CDP, and the like - but I don't recall the details now) and was able to tell them "we're connected to port A of switch B, and then onto port X of router Y". They were then able to identify what our connection was.
As an aside, with a later upgrade we got transferred to fibre, but then that got turned off when the government of the day royally f***ed altnets by changing the business rates system so they had to pay business rates on what the network could potentially support rather than what it actually did. Think pricing a duct as though it was full of fibre cables, and all those cables were fully utilised by paying customers - vs. pricing it on the basis that there's actually only a single customer at the end of it. This put altnets at a massive disadvantage to the incumbent (BT OpenRetch) who had much higher utilisation factors in their plant.
I've met people who manage the way you do -- fire from the hip before investigating and understanding the situation. Fortunately I met them at conferences and seminars and never actually worked for one (although a few bosses came close). I doubt I'd work for them for long. Life is too short to suffer the insufferable.
The word "presumably" is doing a lot of work there.
Another way of looking at it is that Steve and his boss found something they didn't know about, failed to make any checks and removed it without recording what they were doing, thereby causing the company huge losses.
Sure, Steve may have been a better employee afterwards, but the starting point sounds pretty low.
If taking out one seemingly minor box took the entire company down, there are bigger issues at stake.
As other people have said, why wasn't it clearly labelled if it was so important? Which moron had "architected" it such that there was a clear SPOF? Or which bean counter had nixed the resilient approach that an architect had come up with?
If anything there should be more chaos monkeys like Steve in organisations, because then things will be architected and built properly, instead of corners being cut all over the place.
Your average enterprise IT deployment is full of bits like a Brazilian favela that's been built with badly mixed concrete and is now held up with sticks and duct tape. Stuff that wouldn't pass building regulations in a million years. Yet even though western society has relatively stringent building regulations for physical buildings, we outsource design and build of critical IT systems to the cheapest, and don't apply any engineering rigour to any of it. Sacking people like Steve who are trying to make do with the sticks and duct tape doesn't fix the underlying problem.
One place I worked there was a v tall chap (6' 8" in fact) who was helping me sort out an issue in the server room when he leant back and his head hit the emergency power down button on the wall.
There were a few people ranting and raving about it but a) it was an honest mistake, b) it should have had a plastic cover over it to prevent such problems and c) 2 x small servers didn't turn off as somebody had wired them into a power circuit outside of the server room so we actually found out something useful (and dangerous had it not then been fixed).
After a bit of huffing and puffing from the powers that be plus a stern "don't do it again" they left him alone but IMO it was never his "fault" and looking to cast blame as the 1st action was asinine anyway.
I do have a few questions about that design..
Why is the emergency power-down button high enough that it could require a ladder for shorter people to hit easily?
In one of the companies I worked for, they gently taped down the plastic cover on the "emergency shutdown" button, because people passing by kept pushing it to see what happened. And put a huge sign saying "Emergency use only: This Is NOT An Exit" on the fire exit door, because several employees would use it to leave at the end of their day, setting off the fire alarm. A very 'sick' workplace, mostly due to management style at the C-level trickling down to the rest.
Many years ago I worked for a UK breakdown company who had a base in Leeds.
Big posh glass building had been SYSTIMES UK HQ, (Google it).
On the first day there the head of IT proudly showed me the server room, a glass box in the middle of a glass building, with a big green push button to open the electrically operated door.
On the way out of the glass box he asked me to open the door, but be careful to press the big green button, not the big red one right next to it.
The big rid one cut all power to the computers and flooded the room with halon gas...
(yes the red one had been hit several times in the past)
You are presuming that Steve in this story actually had to make do with sticks and duct tape. When it's highly likely that he DID have far better tools and materials available to him if he and his boss had bothered following proper protocols. I get the idea of "this doesn't seem to be doing anything, let's rip it out" but that's just not how it works in doing any sort of work in IT. Just ripping it out is the "big clubs, thick eyebrows, neanderthal" way of doing things. The only PROPER decision at that point is to leave it alone, take extensive photographs, do whatever other work you can, leave and then finally figure out what that thing is and whether you can remove it. If, after going through the proper process, it's decided it can go, you get another trip to the CoLo (hurray) and if THEN the SHTF you have the defence of: "we went through the proper process and no-one could tell us it was a problem".
This is largely how it is in the rest of the world. Houses of corrugated rooves - that leak when it rains - held up by a few sticks (not boards), and they have one electrical outlet for their 1-2 room (9-12 sq m) building, and when I went to visit they splurged and turned on the overhead fan for me.
That there are house / building regulations is a government action, because it's better that peoples' homes not fall on them. However, if you want the same for software (where peoples' homes don't fall on them), then it will take government action, and regulation, to enforce minimums to that effect.
Generally, software people are against government mandates, regulations, and interference -- so it comes down to, "Be careful what you wish for."
> but the starting point sounds pretty low
Could also be a reflection on the size of company and how IT was run, when I moved from a medium sized place where I was the sole IT person to a big multinational the contrast was huge - I am amazed that I never trashed the old company with the way of working I followed (although I did come close on a couple of occasions !)
"Another way of looking at it is that Steve and his boss found something they didn't know about, failed to make any checks and removed it without recording what they were doing"
It also sounds like they might not have known if it was worth some good coin before tossing it on the scrap heap. I've got a side job coming up at the estate of a person that collected electronic test gear. From the photos I've seen, there's a good $100k+ of equipment in $1k lumps and a load of things that might be worth $10ea of anybody's money. The person that might get the contract to do the liquidation doesn't know anything about the gear, just that there's lots and lots of if stacked high and deep in this person's house (they passed away). My job will be to triage values and make a list of the more esoteric pieces to be able to set up sales in bulk to rental houses and other companies that might want to take advantage of some good deals. Besides knowing what the gear is, I also know what parts might need to go with it to make a sale. A Booton microwattmeter is much more valuable if the amplified probe comes with the box. An IFR 7550 spectrum analyzer with a tracking generator option is more than one without. It's also a good idea to make sure the optional battery hasn't leaked all over the inside and to get it out if there is one since it's likely dead anyway.
"There were different levels of lockdown at different times during 2020/1 - with different rules for different areas too."
In AU they varied from which state and which region you were in.
In Melbourne and to a slightly lesser extent Sydney there were quite stringent restrictions but AFAIK a site visit as described here would have always been permissible subject to masking etc.
In my area there was an exemption for travelling between your two households which, my being in the process of selling and relocating, meant I was driving 300km each way every second weekend from early 2020 through 2022 (not once questioned.)
Very strange times but probably couldn't hold a candle to the strangeness we now encounter daily.
WA and especially Perth was it's own little bubble of normality whilst entry in and out of the state via Aircraft, Train or Car was severely restricted with a 2 week isolation stay as your prize.
It was weird going to work and the shops as normal whilst everywhere else was in full lockdown.
Well, ok, we don't know the "rules" that were in place at the time, but they were largely arbitrary, based on governments response to the public vibes.
So, he may not have broken the law, but he still made an unnecessary journey during a quarantine period.
After all if quarantine has been properly enacted, COVID would have been eradicated within 3 weeks.
Some exceptions to 100% quarantine were inevitable. This one wasn't.
I know it's Who Me ? and not On Call.
But here in France depending on the companies lockdown could be purely theorical.
For example, as I was On Call, I had a nice sheet of paper signed by the Head HR and the local CEO that got me free out of jail ( described as : Allowed to travel any where in France and outside, at any time, by any mean of transport available. On the other hand I was not allowed to enter our office building (access badges had been deactivated).
So technically the only place I wasn't allowed to be during the lockdowns was... at my desk.
Now I didn't use that paper, as the 1 hour per day self signed paper was more than enough for me to fetch food and stuff.
"For example, as I was On Call, I had a nice sheet of paper signed by the Head HR and the local CEO that got me free out of jail"
I had my disguise all ready. I adopted the persona of an AT&T wireless technician. I had vinyl stickers made for my tool boxes and picked up some nice AT&T logo'd shirts from eBay. Making an ID card and business cards was simple. With so many people out of the office, any Brown Shirt trying to check my bone fides would have difficulty so they would have to go by evidence at hand. A clipboard with a "work order" to put me anywhere I wanted to go was easy enough. I didn't do the big work that required a truck, just software and diagnostics to combat why I wasn't driving a company logo'd vehicle. Toss in a Panasonic Tough Book and Fanny's your auntie.
"He went on an unauthorized, and not known about visit to the data center that wasn't in any way time critical."
Unauthorized by whom? Because it was authorized by his boss who is the person who generally communicates authorization for projects assigned to someone, and you have absolutely no information suggesting that the boss didn't receive authorization and a command to do this.
And not time sensitive? It sounds like it could be. Depending on the colo contract, a lot of them sell space in long-term contracts, meaning that if that was coming up for renewal, the situation is to reduce space now or pay for it for some large amount of time to come. That's the definition of time sensitive.
"datacenter that is by definition well ventilated"
No? All server rooms I have been in have had air cooling (not air conditioning) but no forced exchange of air to the outside.
And levels of lockdown differed wildly between countries, AFAIR we never had to prove reason for travel, but as many WFH, it made for nice quiet roads for MC trips. Often for work.
AFAIR we never had to prove reason for travel,
Our UK manufacturing plant remained open* and certainly at least one manager was stopped at a police checkpoint and asked why he was travelling (this was during the more severe restrictions). A quick explanation and he was allowed on his way.
*At least one of our customers was providing parts from our products to seven different Covid related projects, including CPAP machines and lab incubators.
"All server rooms I have been in have had air cooling (not air conditioning) but no forced exchange of air to the outside."
Have you any idea how much heat servers produce? You have to have air conditioning, it's an absolute must. Even here in North where we can get cold air from outside in winter.
"Have you any idea how much heat servers produce?"
Yes, easy to look at the power measurement on the UPS output. All of it has to be removed.
"You have to have air conditioning"
No, just cooling.
AC also means also adjusting humidity, and increasing temperature. The cooling units may look like AC units, and be called AC by laypersons, but are not.
That does depend on much…
During the first lockdown (England) I had a whole building (50 desks) recabled. This far out I forget the details but we used a couple of local electrical contractors, it was their only job for several weeks and as there would only be the same 3~4 people in the office social distancing rules were easily complied with.
However, as the business had been able to get itself categorised as essential (it provided services to adults with learning disabilities), this would have probably helped, if any one did investigate.
1. Unauthorized entry to the data center.
2. Removed equipment simply because they didn't know what it was. (Talk about hubris...)
3. They didn't document the equipment/cable configuration in any way before removing it.
4. Took down the company for several hours.
I learned at a very young age that even unpowered equipment can play an active role in a network. In my case, a long time ago, when dinosaurs roamed the Earth, the building I worked in had a defunct ADP alarm system installed by some prior tenant. If you unplugged its phone line, even though it was an unpowered box, it took down a chunk of the building's telephone system. A few years go by, and the telephone company was in the building doing work for another tenant, and I asked one of the telephone company techs about it. He went to the building's telephone cabinet, looked around, and unplugged an unlabeled phone cable from a small, unlabeled box. I didn't see him do anything else. After that, we could unplug that ADP box and toss it. Don't know & don't care why it did what it did, but never had an issue with the phones during the half dozen years after that I worked there.
Additional lesson - don't be afraid to ask another tech, especially if the unknown looks like it might be closer to their area of expertise. (In my case, telephone lines in the age of coax ethernet.)
> ...unplugged an unlabeled phone cable from a small, unlabeled box. I didn't see him do anything else. After that, we could unplug that ADP box and toss it.
I was buying phone patch panels and got something like that. A separate loop for "alarm". If the alarm wants to sound it disconnects the whole house/office to get unimpeded access to the outside phone line. I suspect he unplugged from ADP box and stuck the plug in another hole, bypassing the ADP box. Experienced wire-men can do that "like magic".
My present alarm is 130 pounds of furry barkers. I do have a cutout in the line cuz the old inside wires will have very different problems than the old outside wires (still copper), and it is a good diagnostic to check for voltage and tone each side separately.
Although you might debate whether the Internet set-up was bad and it needed some re-architecturing anyway, the reasons for firing were also:
Neither of us were allowed to visit the datacenter without approval from the very top, let alone ripping out servers without raising a change.
If you do this in my organization, you're fired as soon as they find out. Even if you don't disrupt anything. Even if it is just once. And if people shows up with this on his CV, they won't get an interview.
It seem as that there are as few people here whose fee fees are hurt because these two jackasses (again, no idea about Steve's experience, so will placed it on his boss,) were acting like school kids digging through an abandoned house and paid the price.
I'm not a fan of generational blame placing, but find it difficult to believe that anyone who hasbworked in IT more than 10 years would believe this was completely rational and above reproach. This seriously is about as awful behaviour as it gets.
The fact that you're referring to people's feelings as "fee fees" just backs up your pathetically childish black-and-white view of the world.
It doesn't take much humanity to consider that your fellow humans are fallible, and we put systems in place to cover for that. If the systems don't work for whatever reason - including if there's a culture of bypassing heavyweight SOPs because they're not working for the business on a day-to-day basis - then that's not necessarily the fault of an individual.
You also need to consider that people learn from their mistakes. If you only ever hired extremely competent, very expensive, senior staff, who had made their learning mistakes elsewhere, then that's your prerogative - but it's not a luxury most businesses have.
Should Steve and his boss have been subject to some form of disciplinary process - yes. Should they have been fired on the spot as you claim - no. Should Steve have been fired - probably not, because he was there at his boss's request, and he's just made a mistake which he's undoubtedly learned from and will now never make again. By firing him you're immediately losing some of the little positive value gained from the outage they caused.
Generally speaking if Steve worked for you, and this happened on your watch, and you enabled it (implicitly or explicitly, culturally or process-based), YOU should be the one carrying the can for it, not one of your underlings. That's what you get paid the megabucks for. If your first reaction is to sack your people, then you're a pretty toxic manager.
Sorry if I've hurt your "fee fees".
Tosser.
I'm still not entirely certain what that actually means, because my first reading, admittedly one I'm not very certain of, was that this was a new policy after the failure rather than the status quo beforehand. If that was the rule all the way through, that just brings up more questions, questions like why did they have access if they were explicitly not allowed to go there, who was informed or approved the cost-saving plan, and all sorts of other things the article doesn't say. Sure, if we assume that the answer was they were explicitly forbidden from entering it, nobody asked for permission, they cheerfully ignored the rule, then yes, the firing decision was completely right, but that's assuming an unlikely set of circumstances based on no evidence at all, the same way that it would be justified to fire them if they entered the datacenter by breaking the window but we have no reason to think they did.
If, for example, the rule that you're not allowed to go in there was not stated beforehand but someone hastily added it later, then I do not think it is justified to fire someone for breaking it. If this plan had been run by others who had approved, it wouldn't be justified. I don't know whether any of those happened, but neither do you have proof of any of the conditions that justify your opinion.
I had a digital camera (Fujifilm DX-10 Smart Media) in the '90s. By today's standards, it was a toy (256 colour 1024x768 resolution), but I used it to record several things for the company I was contracting with (including when a floor standing Magstar 3575 tape library was delivered with damaged packaging and 2" dents in the external case, and rails for the robot gripper that were no longer parallel!)
The company itself also had one or more Sony Mavica cameras, writing to 3.5" floppies.
Let's see what failures of judgement happened here.
1) changes were done on a whim without clearing it with the Director of IT.
2) Changes were done on a piece of equipment that was not identified nor documented.
3) Changes were done at a time when EVERYONE was working remotely and not done during a time after hours when most people are not working
4) Before deciding to remove a piece of equipment, no effort was made to label the cables, then take a photo, so that if there was a problem, the actions could be quickly reversed
Now, people do make mistakes, but what happened here was a serious lack of judgement that is required for people responsible for the literal core of a business. IT systems are inherently fragile, there's nothing about just pulling out equipment that you have no idea what it's for, nor precautions not taken to quickly reverse actions, that will IMPROVE the situation.
It's not a toxic corporate culture, it's removal of people from important duties who lack the required JUDGEMENT. Getting fired INSURED that the lession was taught, not a funny laugh and "oops my bad". Many people keep doing bad habits, until consequences occur that are irreversible.
When you try to discover why one oughtn't it is usually a mystery — of the religious flavour like the trinity rather than those of the more mundane Agatha Christie variety.
Always worthwhile if and when when the whole shemozzle has to be powered off to "inadvertently" leave the holy artifact unplugged and await the apocalypse or second coming. Surprising how many of canonical relics fail to power up when the supply is restored.
Only slight less mysterious is a perfectly ordinary IPv4 address in a zone file stating "this address us never to be used."
As far as could tell no one knew why that particular address was excommunicated and anathematized. My guess was when the site had its own MS Active Directory/Domain Controller &c ... the address was used by one of the servers. The whole lot was taken behind the barn a few years before my advent as I presume were the staff as none were extant.
It was already bound to happen, and unfortunately Steve was the chosen one.
Honestly, the person who set that shit up should've been fired. And they should've, at the very least, documented it or have put it inside a basement far away with signs saying "Beware of the leopard".
The chances of up-to-date and accessible/discoverable documentation for a piece of kit that's *powered off* in a large organisation is relatively small. Documentation should be kept up to date. But that doesn't happen as diligently as it should.
It's generally a pretty safe assumption that if something that appears to be an active piece of kit doesn't have power then it's not "active".
I've never come across a piece of powered hardware yet that effectively acts as a patch panel passing network connectivity through even when it's disconnected from power. Power cord plugged in and device "off" is a different matter. But no power cord at all - the mind boggles at what this piece of hardware was doing and why it had been designed that way.
From only a limited sample, but my PoE devices have lights, even when connected to the PoE router but not actively drawing power.
Ok, a managed PoE router may have been told to shut down power delivery to that port, to extinguish all the LEDs, but still keeping the port active for traffic - this is getting needlessly complicated![1] And if they have infrastructure that is complicated but no fallback for this bit of exotic kit...
Hmm, Steve may have got the boot but hopefully IT were also given a stern talking to.
[1] or it is one of the sneaky spy devices jake mentioned
Plenty of nics with a watchdog which will close a relay to bypass the card if it isn't regularly poked from software.
That would make the device a massive, and expensive network coupler, presumably coupling two cables from a patch panel which would be better off just directly connected...
We know it looked distinctly out of service (not powered up) and yet wasn't prominently labelled to warn others that this was very unusual kit (a sticker over the strangely unused power socket , with a note explaining why - which may, in a bad world, be a reference like "read doc#17, p.472").
Heck, I just wrote a sticky label to go next to an LED because I couldn't figure out or remember why it was on when the rest of that computer appeared to be off - and I only built that machine from parts last November!
Not leaving on-site labels for anything that is very out of the ordinary is shoddy. Yes, including kit that has been decommissioned but left in the rack: masking tape and a sharpie, at the minimum.
THAT is the sort of IMMEDIATE documentation that was missing!
> "Not leaving on-site labels for anything that is very out of the ordinary is shoddy. Yes, including kit that has been decommissioned but left in the rack: masking tape and a sharpie, at the minimum.
THAT is the sort of IMMEDIATE documentation that was missing!"
Amen.
Like code, I think the same should apply to hardware -- What [the hardware] does should be self-explanatory, and if it can't be, then document it!
I always wrote on the kit itself with a sharpie: a few times I've gone to a cabinet and found all the sticky labels on the floor...
Sharpie is still not foolproof, I guess, and harder to change when a change is required.
A device that was actually out-of-use would usually have the word DEAD in red sharpie on it if I'd been there. If not, I'd be doing a proper check to see what it was...
Honestly, the person who set that shit up should've been fired. And they should've, at the very least, documented it or have put it inside a basement far away with signs saying "Beware of the leopard".
I see your HHGTTG and raise you a Pratchett...
If you put a big red button in a distant cave with a sign saying "End of the world switch, do not touch!" the paint wouldn't even have time to dry.
This post has been deleted by its author
The problem is when you can't find *anyone* who understands it. Or the people who *do* understand it have all either left, retired, or "moved on" (metaphorically speaking).
Sometimes the only way to know what some unknown software service, hardware, etc, does is to "pull the plug" and see who starts shouting. You can try to do that in as controlled a way as possible, but you can't let perfect be the enemy of the good - because sometimes there isn't a "good" way to go about it.
> "pull the plug" and see who starts shouting
We did that once. Proper checks revealed nothing, so we pulled the plug to see who complained.......
Nothing happened. So we eventually went home.
Then at 6am the plant nightshift couldn't clock out, because some clowns had unplugged the box that controlled the timecard punch clock.
Facilities Management had a habit of visiting little used buildings unannounced and powering down 'unused' cabinets to save power, on the basis that someone would eventually complain if it was actually wanted
That was until the day that they couldn't get out of a building as the control panel was alarming. Turns out that cabinet contained an X.25 PAD connecting the door/alarm control unit back to head office
Yeah. We had a locked room in a building that was scheduled for closure. The people 'managing' the closure and moving of staff asked around and did not see anyone using it for a while so assumed that it wasn't important. Building closure scheduled for 6 months.
Oh! But then a certain TLA team pointed out that that was their secure room. (And I mean secure. No external walls, no windows, no naughty electro-magnetic emanations, a very sturdy door, and if you don't know who I am talking about ... You don't need to.)
We were still in that building 2 years later.
> > Leave it alone
> The problem is when you can't find *anyone* who understands it. Or the people who *do* understand it have all either left, retired, or "moved on" (metaphorically speaking).
> > Leave it alone
Sometimes your role is to be the one who comes to understand things. That isn't ignoring the things, and it may be that pulling the plug and seeing who comes hollering is part of gleaning understanding. But.. in due time, with due notice (as much as possible), and with understanding of the boss. :-)
The UK's Cyber Security people have just issued a statement telling organisations to be prepared to work without IT and use 'pen and paper' to run things to avoid shutdowns such as happened to Marks & Spencer and Jaguar Land Rover recently.
https://www.bbc.co.uk/news/articles/ced61xv967lo#:~:text=People%20should%20plan%20for%20potential,the%20ready%20as%20a%20precaution.
"People should plan for potential cyber-attacks by going back to pen and paper, according to the latest advice.
The government has written to chief executives across the country strongly recommending that they should have physical copies of their plans at the ready as a precaution."
I was once running data centre cables from network box to patch panel to structured cabling to patch panel to patch panel to structured cabling, etc, to get back to the core switch I needed. Having got there I found that the port I was allocated already had a cable in it. I contacted network management and was told to remove it and connect my cable. They would trace the erring connection and reconnect it properly. I found out later somebody had 'noticed the wiring was untidy, and bunched everything up to the first row of ports.' Presumably at least most connections had stayed within the correct VLAN, or it would have been noticed somewhat faster.
> I found out later somebody had 'noticed the wiring was untidy, and bunched everything up to the first row of ports.'
802.1x is very handy at automagically dealing with this kind of issue - either by shutting down the errant ports or noting what MACs are present and VLANning accordingly
The worrying thing there is if you're the person who's supposed to understand it...
Some years ago, the opposite happened to me: A major broadcaster was replacing _all_ its internal and external analogue (and digital, of various flavours) circuits with shiny new digital circuits. We spent ages (years) planning this, because of the requirement never to go off air... All sites had main and standby systems, usually in different rooms, different power circuits, different entry points for the signalling etc and because of this we could set things up on the reserve circuits, switch to them, make the main circuits live, test, and switch back - no interruption.
Excellent. Over a whole country, no interruption visible to the viewers.
But... most rack mounted video equipment in those days used either analogue or digital inputs and outputs from the back of the rack, controls and indicators on the front. The cabling was loomed into great thick (but very tidy, when installed) bundles down the back of the racks and under the floor. Planning had revealed that there were whole racks which would become redundant, but whose space would be occupied by new kit. For very rare cases where equipment had to be moved within a rack, it was usually possible to move it up or down in the rack, or remove it, with a bit of jiggling, through the back - without interrupting its operation.
So, one apparatus room at a time, things were changed. Until we arrived at the one that spoiled our error free record: some bozo had wired a single unit in place with its output going round the wrong side of the rack frame. This unit carried the output from the station; it had to stay live. It also had to be temporarily removed from the rack so other stuff could go in its place; and there was no physical space to get it out the back because of the wiring routing... so this single wire ruined our two-year-without-interruption by seventeen seconds as it was disconnected, pulled out, and reconnected.
Given the time in the morning, I do wonder if anyone noticed, but still, it gripes!
My Operations teamlead calls this "fear-driven Operations" and will have nothing of it.
If you don't know what it is, ask everyone who might know.
If nobody knows, raise a change to decom it.
Disconnect network cables and wait 6 weeks.
If no complaints, power it off but leave it in the rack for 4 weeks.
Still no complaints? Throw it in the bin.
Even with over 20 years experience, I always asked for a second set of eyes on something before pulling the trigger, even if I was the senior guy in the project. It was a good practice, one it helped junior tech learn something, or in some cases point out something they don't understand. Two, it also showed our manager that we worked as a team, and were diligent in checking all the boxes before proceeding.
"We better do more research" was a commonly used phrase. I used to substitute the Jaws quote, We're gonna need a bigger boat, the team liked the analogy.
Steve's boss suggested they go. It's not wholly unreasonable for Steve to assume that it's a sanctioned trip. Combined with some COVID lockdown stir-crazy-jumping-at-the-chance-to-see-a-different-four-walls mania I can see how he might have not asked as many questions as he perhaps he should have done.
If anyone should be carrying the can for the escapade, it's Steve's boss, not Steve.
How can something that's powered down "provide a vital network connection"? Maybe when powered down it somehow acted as an RJ45 coupler? Because I don't care how "exotic" it is, it can't be routing or firewalling if it doesn't have power!
Or was it some device small enough to operate via PoE, and they wrongly assumed because it wasn't plugged in it wasn't running but didn't bother to check for any LEDs?
I have seen a couple boxen that were PoE when at idle, with no LEDs or other indicators that they were "live". When the proper wake signal was received over the LAN, they quietly did their jobs, still with no power indicators. The only time they had lit LEDs was when they were being configured with a (dumb) terminal plugged directly into 'em.
Yes, they were (and still are, in at least two cases I am aware of) clandestine security devices.
Someone should tell them that security by obscurity is no security at all ...
I've come across devices which have some form of relay (Might not be mechanical but you get the idea) where if the device has no power, the traffic by-passes the equipment. The equipment has to have successfully booted for it to flip the relay for the traffic to actually be processed.
Seen those as well, as a failsafe for when the device is unwell (the relay only flipping after the device has not only powered up but also POSTed).
But if you've gone to the trouble of removing the power cable then you could also replace the box with a plain old coupler. Seems odd to leave it the way it was - and anything odd should have a clear label attached!
The most common occurrences are in physical IPS or IDS units where there is mechanical shunt on the interfaces where if the box loses power or crashes it is hardware bypassed.
This is where dropping the circuit would cause a huge impact. However what should happen is that you then fix the problem.
Nor excusing Steve’s actions but a box of thsi type should by physically labelled with a warning message
But why leave it in that state if you've made such state permanent by pulling the power cable? I don't buy the "no maintenance window" excuse, or that if that was the case that anyone would leave it in a state like that without sticking a label on it or the RJ45 cables saying something like "change #12345 waiting for maintenance window, do not touch".
You could unplug those cables and connect to a coupler in about 2.3 seconds. No one would even notice a blip in the internet connection that short so long as you do it when most people are out of the office. If you're big enough you can't get away with a couple seconds interruption, you have redundancy in your internet connection and it has a bandwidth higher than copper can handle.
>"How can something that's powered down "provide a vital network connection"?
In a career far away, selling a product long since discontinued, we had a product that performed in-line Intrusion Detection. This was before Let's Encrypt took over the world and the bulk of network traffic was not encrypted.
To minimize a single point of failure, the network card was a special pass-through type which would "fail closed" (ie: the circuit is closed and passing traffic) when the power was shut off.
Basically, if the power supply died or the box was rebooted, it would not also take down the network. Sure, no IDS was happening, but it didn't take the world off-line either.
The only way this would be a single point of failure is if someone unplugged the network cables from the box....which is the crux of this story. Had our hero made note of which wire was which, they could have simply bypassed the missing equipment by looping one wire back to the switch.
Imagine you want the ability to passively sniff all the dark fibre coming into a data-centre and maybe even have the ability to modify some of it in transit. Also imagine you won't want most of the staff to know, so it can't appear on any diagrams, make any policy or routing decisions, or have any possibility of being responsible for an outage that some lowly DCO or even most members of a NOC might end up investigating at 3am.
15 years or so ago you'd get some screamingly expensive 100gbe fibre cards with the in and out ports optically linked with an honest-to-god mirror on a spring . Completely passive, low loss optical pass-through. One booted and the system was verified as working, it'd keep hitting a hardware watchdog which used a powered solenoid to move the mirror out of the way and let the host see/mangle the data. If it all went wrong - anything from power loss, system crash or even poor performance, then the watchdog wouldn't get hit and the mirror would spring back into place making the device networkingly invisible again.
Of little use to most people and unnecessary for nearly all applications. But I can absolutely see situations where they'd be kept powered off but in situ, ready to be remotely booted.
This is from the "stupid tired operator tricks" files ...
Several billion years ago, as the Internet measures time, call it 1986ish, I was burning in some two dozen full racks of T-carrier gear.
After the end of the ten day burn-in period, one of the final tests was to physically pull the plugs on the redundant power supplies, and then re-insert the plug, before moving on to the next supply. Lather, rinse, repeat, first with the supplies plugged into "power A", and then the supplies plugged into "Power B". A Sun workstation logged the relevant voltages & currents, to be printed out as part of the complete verification package for each machine. I got to the end of the long line of plugs, absentmindedly noted that this plug was different from the rest & unplugged it ... only to take down the Sun box, and completely trash the drive holding the data. It's the only time I ever lost data on a CDC Wren SCSI drive ... and it invalidated the ten day burn-in results for about two million dollars worth of "must ship" gear at quarter end.
I shouldered the blame, as I had pointed out to my Boss that having the Sun plugged into the test power bus was probably not a very good idea. All I could do is claim exhaustion, working a couple weeks of fourteen hour days because Sales had over-sold production capability and for some reason TheBoard decided we had to make the projected sales figures. Fortunately, my Boss managed to cover my ass & I kept the job. Daft thing is that I wasn't even part of the QA group that managed the burn-in, I was only roped in to help because of a lack of hands ...
I had a manager who occasionally went all OCD about cabling in the office Comms room. On one memorable occasion we were alerted to a number of system outages and went down to the basement to find him pulling power and ethernet cables out of the racks to "tidy them up". He had successfully unplugged a VPN concentrator, an AD DC, and a number of backbone switches by the time we found him and stopped him.
To gain access to the DC and have the confidence and forthrightness to embark on a tidy up involving removal of kit Steve & his boss must have had the right authority in the business to do so.
if i had come across an unfamiliar, undocumented unpowered (as it had no power cable) box in a dc i'd have removed it too.
my 1st suspicion would be that someone has attempted to infiltrate our systems, especially if i worked in an insurance company.
not sure how the actions could have resulted in dismissal, especially in what should have been a regulated environment.
i would have traced the cables though and checked the switch to see if it detailed what it was meant to be & checked our records to see what was meant to have been patched into the switch ports but we all know how inaccurate documentation can be.
had i done the checking while the thing was still connected i'd have noted the switch interface as being up & traffic passing, i'd have still disconnected it though because as a network guy with admin access to switches i should know what it was.
If a server guy had removed it because they didn't know what it was i still can't conceive as to why they'd get sacked.
it's possible the device was a network tap with passthrough capability for when it fails or disconnected from a mains supply as in this case, but it could be poe.
https://www.amazon.co.uk/Dualcomm-1000Base-T-Gigabit-Ethernet-Network/dp/B004EWVFAY?th=1
should have been quite obvious what it was especially as lights should be blinking
used to use them at a previous job but failed to see the point as no one looked at the data.
When I worked for BT, I was aware that access to the data centres was strictly controlled. You either worked there or needed an appointment to get in. Many housed UK Critical National Infrastructure, HMG systems and other services that would get the CEO a personal telephone call from the Prime Minister in the event of unscheduled downtime, and would be an item on international news bulletins (such as in the event that Heathrow Airport was closed due to, say the system managing air freight crashing, for example). Even the CEO could not get in without an appointment.
So how on Earth did 'Steve' and his manager get into a site where they needed high level permission in advance? I reckon the data centre manager has some questions to answer on this.
Why do you assume any of the following:
1. that this facility was anything like those. There are lots of colos that don't have critical infrastructure security levels where the client provides people with credentials to access their area. There's still security, but generally not make an appointment weeks in advance level.
2. that the people entering didn't have the required credentials. They hosted servers there, these people can recognize and probably installed them, this sounds like the kind of people who would have access.
3. that they didn't follow the procedures. Even if assumption 1 is correct and thus that they couldn't have had general access credentials, they clearly planned to visit and decommission. They would have had the time to make that appointment. If they were not allowed to do that, it's equally if not more likely that the person who has to answer for it is the one who signed off their appointment or the one who didn't need to but should have been gating that access. Assuming, of course, that there was any policy requiring that be gated.
So that proves it is a critical infrastructure-level facility, does it? Which part of the sentence says that? That's a statement of a corporate policy, no more and no less. It does not tell you anything about what precautions the datacenter concerned should or did have. Nor does it indicate that the people who entered didn't have the needed credentials to do so; with approval, they were allowed to go there, so they would need the means to enter under those cases. True, a company could implement that by locking up the credentials and only handing them out when the visit was approved, but we have no evidence saying they did. If they were trusted not to misuse the credentials but did so, that's not on the datacenter's operators. I have credentials to access plenty of systems that I am not supposed to access except in case of emergency, but I nonetheless have them because I might need them at short notice in that emergency. I am trusted not to misuse them and, if I do so anyway, I will face consequences. If I did misuse them, nobody's going to wonder how I possibly came by them.
You're trying to use that sentence as evidence against the datacenter's manager. It's flimsy enough against Steve given the lack of detail. It's entirely useless against that manager. You have no evidence that they simply walked in and no evidence that the security protocols you've used for one DC should apply to this unidentified other DC. Datacenter managers are not responsible for enforcing client internal policies. Unless you can demonstrate that they failed to follow datacenter policies, your accusation is without merit. The sentence you quoted proves none of that.
The data centre manager is responsible for access to the site, and therefore the access Steve and his manager obtained without authorisation. The data centre manager is responsible for ensuring the rules are enforced. I never claimed that the centre Steve and his boss visited housed CNI, I was merely using an example from my personal experience working at BT. I thought that was clear.
See the posting below titled "I'm Steve" where he explains in point 3 that the policy for high level authorisation for access had only been made a few weeks prior to his visit, and without a major announcement.
There are datacenter rules and there are client rules. The datacenter's manager is only responsible for the former. In a lot of cases, that rule is something like "The people entering have a card for access, are on the list of approved people, and have identification as being those people". Whether or not the company wants them to enter today is the company's problem. Each datacenter has its own requirements, sometimes more severe, especially for critical ones, and sometimes even less.
For example, long ago I worked at a place which had access to various servers. In their case, the requirement for access was just the first part, having a card that opened the door. If someone misused one, they'd find out and deal with it later. The reason is that they didn't run anything critical at all on servers they managed and they didn't house those servers next to others that did. The cards were limited and could only give access to locations they managed, but otherwise, there was little security, in my opinion insufficient security. That was not the fault of the datacenter manager. If fault was involved, it was that of the client who didn't request more and who handed out credentials very freely. For context, I was a programmer there, so I didn't need access credentials and didn't get them until I helped fix something once. They decided that was good enough reason to give me access credentials, credentials I used a grand total of zero times.
All I have to say is that this section is getting the honest ones.
In the last few itterations of "On Call" we've had several cases of people who were fired on the spot or shortly after, for being perfect.
Pointing out the boss' mistake, being too good, going too fast had all shaken my firm beliefs that bosses are idiots but still use common sense, and that whoever publishes stories here NEVER sugarcoats anything.
At least here, we have a bona fide "I would have fired them too" element that one can stand behind without proof of membership in the 1%. Which is likely there simply because the author deserves kudos for honesty. Whew !
So the powered down device was unknowingly functioning like a hub? That's pretty silly of them. (That isn't doing any kind of "security" filtering if it's a non-powered device... that's bullshit)
I'm not sure I'd blame "Steve" for that. As for the unauthorized data center visit, that should be on Steve's boss.
Long story short(message lost), the IT director is ultimately responsible for having created/allowed the environment and circumstances which paved the way this event to become essentially inevitable. Lack of procedures, no accountability through validation, and failure to maintain proper documentation(let alone document control) doesn’t bode well for the company’s management as a whole- there’s a saying which is appropriate here: “when the fish rots, it starts with the head”. The “operational responsibility” falls upon the highest ranking supervisor on site(or in the immediate area of work), so they should have stepped up, taking the consequences as their own, rather than throwing his subordinate under the bus. It is deplorable that someone so lacking in moral integrity would be allowed to oversee a data center, let alone a McDonald’s.
I want to know about 'the boss', surely if anyone loses a position over this, it is the senior role holder?
As for the device, if it was powered off, in a datacentre and for an insurance company (if they're big enough to have colo space in a DC then they must be throwing some transactions through that link), then it wasn't doing anything. There is no way a *powered off* device can be doing any 'exotic security' stuff at DC speed transactions. I would suggest it more likely that the device is a leftover device that has a fail open state, by leaving the thing hooked up without power, you have an open gate, this could be on one or two (or more) links, think something like a Riverbed WAN optimizer. In this scenario, the only reason the link dropped is because there is no longer a path for data to flow. Correctly recording the data links previously in place would have allowed to the onsite folks to either remove the loop or connect the two cables to relevant LAN ports to recreate the path without having something taking up rentable space in a colo.
Of course I agree that it should not have been thrown away without at least noting which things connected to what, but I have definitely ripped out kit that seems to be doing nothing (though making some record of how to put it back if needed). As someone else noted, the SPOF has been found, the event should be recorded and the techie has a great lesson going forward (and an answer to a few interview questions).
I don't care about the covid stuff, different countries, states and even cities had their own sets of rules, so I don't care about that, but if the visit was unplanned and outside change control, again, the boss is the one who should have taken that bullet.
One other red flag. What did they intend doing with the unused servers they removed? Since this was an unauthorised, unofficial project, was Steve's boss's plan to sell them on to his scrap-dealer mate down the golf club?
If so, we can add theft to the charge sheet along with knocking out connectivity, breaking lockdown rules and generally being an arse.
And, if they planned to use the heavy metal bits to bash people over the head, they might be existing or aspiring murderers. There's no suggestion that they did, but it's not technically contradicted by the article so we can accuse them of it. Going down the list of criminal offenses, I think I can accuse them of any number of crimes if I get to invent and then assume any facts required.
Undocumented equipment/unlabled/configuration in the colo?
Two people who got into a secure colo without permission?
Sacking the senior bloke?
That sounds like a well paid IT Director covering his big fat well paid hairy ass.
Side note - got an Engineer into a colo without permission not so long ago, confusion over two colo's close together, but the engineer was well known to the staff so they let him in - he presses button and server comes up :-) )
Wow, lots of assumptions in this lot so I'll try and clear a few things up.
1. The device was some kind of security inspection device with a fail open interface making it one of the worlds largest RJ45 connector.
No lights, no PoE, just a slab of tin taking up 4u.
2. We came to learn it had been "decommissioned" about 3 years previously but the staff involved had long since left the company and there was zero documentation. Outsourcing had near enough wiped out any local knowledge.
3. Although this particular visit was unauthorised, we had made many previous trips to the data centers that were fully approved. I had no reason to believe that this was any different and management had only implemented this rule in the previous weeks without making a big announcement about the policy change.
4. Yes there was a lack of change control but again I wasn't ever the one tasked with raising it in the past.
5. Management had made the goal quite clear that DC cost needed to be severely reduced by a certain deadline so there was an amount of pressure to hit this deadline.
6. No COVID rules were broken.
7. We had no plans to steal the equipment as suggested by one reader. All equipment was piled ready to be recycled by a reputable firm.
8. The company was a shower of crap to work for and I was quite happy to be the fall guy.
9. Hindsight is a wonderful thing and taking note of the patch panel numbers would definitely had been a good idea.
1The story was meant to provide a bit of amusement, but I suppose also a cautionary tale too.
One quibble left is that the "Who, Me?" stories are ones where you get away with it... but checking the terms, I see the word "usually". So you're good. ish. And so is your ex-ex-boss.
"Who, Me? It's the reader-contributed column in which you admit to mistakes that almost trashed your career."
"Who, Me? is a weekly column in which our readers confess to catastrophes they created in the pursuit of IT excellence - and usually managed to get away with."
I wonder if you e-mail to whom@reg instead of whome, it is for stories from people who did something so terrible that they and their name are erased from corporate memory?
The description of the unplugged device in the article makes me think that it was (or incorporated as many inline traffic analysing appliances do) a TAP that had been powered down and was therefore in its passive pass-through state. I’ve come across these a few times. It’s possible, depending on exactly what it is, that it was intentionally left in bypass and is only connected to power when it needs to be used.
Big whoopsie, but also a lesson learned I hope, not to disconnect equipment without carefully examining the layer 1 documentation, or if the documentation is poor, following the cables and leaving it alone if you can’t.
"he jumped at the chance to leave the house, drive for a few hours, and revisit the real world."
Riiight. Someone missed the commute and being in the office rather than working from home.
I smell a story planted by a middle manager anxious to propagate the "commute!=hell on wheels, office>home" agenda.
Someone can enjoy working from home and still chafe a little at never leaving it. I'm guessing you are now or at least have experienced remote working, but do you spend every minute there? I did during the pandemic because that was the point of isolation, but on occasion I would have enjoyed if I could have done something, not necessarily work, outside of there, especially after it had dragged on for some time. When I was remote working after those restrictions ended (I'm now in a hybrid schedule by company command), I didn't spend all my time at home. I still went out to forestall cabin fever.
Some of my colleagues even now tell me that they much prefer to come in to the office rather than work from home.
I don't understand this attitude (even though my home setup is not as good and my family causes much distraction, I'd rather WFH than WFO), but it definitely exists. I'm sure it was even more of a thing when people were getting fed up with being in lockdown.
Where do I begin?
If visiting the colo DC required senior management approval, how were they able to raise an access request and be allowed, presumably unescorted, to their data hall and racks? Either something doesn't quite add up or they urgently need to find a new colo.
As for removing equipment without approved change and no rollback plan, I'm not surprised he lost his job. The 'it wasn't labelled it was an accident waiting to happen' defence doesn't cut it I'm afraid.
I worked for a bank based in a Northern city some years back and one of my responsibilities was looking after a system that allowed some of our wealthier customers to buy and sell shares online. I implemented a new version of the system based on a UNIX HA cluster running on shiny new servers and using SAN for storage rather than the old SCSI array. The new system went in and was up and running with no issues. That was up until the day that we lost all connectivity to the system, even remote console access disappeared. Cue a stroll over from the office building to the data halls to see what was going on. After going through the rigmarole of getting an access pass for the computer hall I went in to log into the console of the servers only to find the floor tiles in front of the servers up and the data centre manager and his PFY ripping out cables. I asked what they were doing and was told they were removing the old system.
Unfortunately this was not the case and they had disconnected and were pulling out all the copper and fibre connecting the shiny new system to the storage and the network. The only saving grace was that as a somewhat anally retentive pedant, I had labelled both ends of every cable with the source and target port details and recovery only took a couple of hours with just 2 fibre cables having to be replaced because they had been damaged.
The lesson I learned from this was never install a replacement system right next to the system you're replacing.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
