Dead LAN's hand: IT staff 'locked out' of data center's core switch after the only bloke who could log into it dies An IT department is pulling its hair out this month after realizing a coworker who died last year was the only person who could log into a crucial network switch. This is according to Dylan, a sysadmin at a small US healthcare company, who today told El Reg a story of how he and his colleagues ended up locked out of the …
The external IT Auditor is an external organisation. They would only ever approach our offices if both Clerk and Deputy Clerk fell under the same bus at the same time. And even if all the parish councillors went under the bus as well resulting in no executive authority, things would devolve up to the district council. If the district council was wiped off the earth, there's be other more important things to worry about.
I had configs and layout documentation stored on a hidden server on the network. I backed everything up to it. Then I archived it and encrypted it so it could be stored in plain site on several servers, on tape backup, and store the tape in a bank vault. The network didn't change much, so this was a viable option. But when the network went down, the other admins had no clue how stuff was configured because they weren't doing the backups like they were supposed to. Some managers were let go. I figured out what the problem was and reloaded the router with the config off my hidden server. When I left, I still don't think they found the server.
The backup server is a Raspberry Pi v1.0. It's in a small, non-descript case with a label that has the IP address and a message that reads "Critical network monitoring equipment. Do not disconnect."
It sounds like he was a bad admin. But he may have seen the writing on the wall about getting fired and purposely felt like making things as difficult as possible for his successor, not knowing that it would be his successor in a literal sense. People seem to be speaking ill enough of the dead in the article that it didn't sound like he was very cooperative.
And I can't deny that in a crap job situation many of us would not have outright sabotaged anything, but would have felt this way, right or wrong. In the end, it was management's fault for letting things get this way, and possibly for both either maintaining an incompetent employee or so neglecting a once-useful employee that he ended up indifferent and apathetic.
The deceased either knew the writing was on the wall so came up with a scenario that meant if he got fired he'd have some satisfaction from the shit-storm he'd leave behind or knew someone didn't like him and was trying to make it impossible for them to get rid of him.
I've worked with people in the past who were actually very good at their jobs but were constantly being bad mouthed by management. This was often when a PHB thought they knew everything (often loudly) and the colleagues were constantly being forced to prove them otherwise to the PHB's boss.
I've worked with people in the past who were actually very good at their jobs but were constantly being bad mouthed by management. This was often when a PHB thought they knew everything (often loudly) and the colleagues were constantly being forced to prove them otherwise to the PHB's boss.,
I had fun with that once or twice. The boss had been badmouthing me to suppliers, customers and contractors.
One day there was a fault with the internet (telco end, not ours) and he came in while I was talking to one of said customers and mentioned it. I just nonchalantly said "Well, you know much more about networking than I do, I'll go off clock and watch in awe as you fix it".
Both the customer and I found it quite a joy to be sharing a moment together watching him get more and more flustered as he proclaimed his knowldge, went in with a 'fix', only to have that fail so go with another thing. He'd heard me talk of a bad LAN card flooding the network with "martian packets" (an early outage before I got some resiliency built in) so decided one of the machines was causing that and unplugged them one by one, claimed bad cabling so opened up packets of patch cables (all the shop ones were made by me, the ones he opened were on the shelf IE saleable stock), and various other things..
What made it more enjoyable for me is it was a "brief" (as in 3-4 hours) but major outage. The customer worked a couple of blocks away and had everything with the one telco so had lost phones and internet, and had come to me in person to see if I was aware what was up. As our net and phone line was also out, I'd already phoned the telco to make sure they were aware of a fault when the boss came on the scene. Given we could log into the router and it had a "disconnected" status icon, I was fairly certain the internal LAN was fine and the other side was dead, especially as someone a couple of blocks away had the same problem :)
Love being able to show that kind of manager up when the time comes.
for a small/medium size organization to protect against this kind of attack. You need a complex mixture of technology, policies, procedures and auditing to make sure this doesn't happen. Instead, the organization can outsource it completely and it's no longer a problem.
Instead, the organization can outsource it completely and it's no longer a problem.
Of course then you have an entirely new set of problems starting with GDPR compliance, data leakage, cloud providers deciding you're last in line for a fix, connection troubles, being held for ransom with rate hikes or mandated software "upgrades", deciding your app isn't worth supporting any longer, etc.
When just making sure the CEO had a copy of all passwords available would have fixed this at no extra charge.
But don't let that all get in the way of your cloudy sales pitch!
When just making sure the CEO had a copy of all passwords available would have fixed this at no extra charge.
Or better, the CEO's secretary, as that is a person that can usually be entrusted, but would never think of abusing the power that has been let with them.
As opposed of a CEO who is more likely to think it is their company and they can and know how to play with the password.
Or better, the CEO's secretary, as that is a person that can usually be entrusted, but would never think of abusing the power that has been let with them.
So you've seen this too. It's an interesting phenomenon, some form of "absolute power corrupts" I gather.
One organization I worked with was effectively run by a very competent secretary (that was also a force to be reckoned with if you did wrong). Her "boss" was a miserable twat....
They run the place.
Free hint to all consultants: ALWAYS ask the secretary about the Boss's computer knowledge. You can save a lot of time and trouble for a lot of people over the long haul.
I know of several CEO-types of Fortune-500s who make a big show of "checking the computer", even though their network cable was "accidentally" never installed.
I can't count the number of times I've swapped the Boss's top of the line CPU, gathering dust and spiderwebs under his credenza/return, artfully changing screensavers every couple minutes, for his secretary's underpowered kit ... without the Boss noticing.
My late wife was (officially) the Financial Controller of a small craft brewery, but as the two Partners were complete imbeciles, who squandered their father's legacy, she actually ran the company. Then she was struck down by an incurable disease, and after she died, the company folded.
This post has been deleted by its author
You don't deserve the up votes in the same way I don't deserve the down votes.
Are you trying to tell us the CEO should login three times a day just to make sure those passwords have not been changed ? It happened to me when organizations asked me to help and provided me with two or three written down passwords that were no longer valid. By the way I'm not a cloud stuff salesman, I'm currently configuring telecom equipment but I've learned there is more to security than TACACS/RADIUS and a password kept in an envelope in CEO's office. Clue for you, do a Google search for privilege separation and privileged access management and start from here.
As someone mentioned in a post below, you're just changing the nature of the problem but your rogue admin scenario is gone. Yes, the outsourcer might have a rogue employee too but your company can hope to be compensated financially for the down time. As for compliance you'll have to do it anyway. Outsourcing or not, you're the owner of the data so act on securing it wherever it is stored. And it is better and cheaper to focus on securing data instead of caring for every switch and wiring closet door in addition to data. My career too is threatened but there's nothing I can do, developers have stolen the show. I would really like to adapt to the new reality but time is a serious constraint.
No, that just means that you've traded one problem (rogue employee) for another problem (rogue employee of outsourcer) but I agree with your statement "You need a complex mixture of technology, policies, procedures and auditing to make sure this doesn't happen.", but you left out the most important part: good people. If you don't have good people, you can buy tech, etc, to no avail.
>for a small/medium size organization to protect against this kind of attack. ... Instead, the organization can outsource it completely and it's no longer a problem.
It is still a problem, I've come across many small/meduim businesses that prove the rule that businesses like to do business with similar sized organisations. So the small business has outsourced their IT to a 1~2 man operation (1 tech who actually knows the passwords and where they are kept & 1 sales/consultant who instructs the tech but doesn't actually know how to access the tech's folders).
This is exactly why companies need oversight, you need to badger colleagues to document and record, get organised people. I hate this idea that just because you're asked to document your work and record things, that you're getting the boot next week. No one is trying to steal your job! We need the knowledge so that we can keep you and your job. You vanish on holiday for 2 weeks and no one knows what you do you put the whole company a risk and everyone's job.
Get your head out of your arse and get stuff written down, and sensitive stuff stored securely, even if it's just a print out given to the IT manager or even get Finance to put it in their safe. On the other hand the more you hoard knowledge, the more complaints HR may receive about you and then you're considered a risk for not complying with regs, this means they may find some way to demote or remove you for not complying.
You may not be going full on "DevOps" but the TPJ has a lot of sensible ideas about ensuring work is documented, critical people who cause bottlenecks are utilised correctly and knowledge is recorded so others can step in when needed to keep companies in crisis moving along.
And do you really want Imminent Failure to reveal Special Forces AIMissions for Universal Resolution with AISolutions for Systems in a Failure withTotal Collapse of Non-Cooperative Operating Systems ......with Exclusive Elite Executive Officer Suites in Stasis?
Yes would be the Correct Answer there whenever ready for Everything Quite Wonderfully So Easily Different.
And here be but one Blank Canvas upon which such Futures are Wrought and Writ with Almighty 0Day Provider Protection Testing Vulnerabilities to Distractions and/or Destruction with Enigmatic Ethereal Exploitation of Earthly Resources with Depleted Intelligence Sources.
What do you Think? AI Fact and/or Pump Fiction? :-)>
And just whenever Theresa was praying for it not to get any worse ........ An AI LifeRaft just Simply Appears out of Nowhere.
That Project is Magic. Pure and Simple.
That may be true, but companies will, at times, ask for documentation under the theory that I'll write down everything that a replacement working with a lot less knowledge and for a smaller bill can simply pick up. That isn't a reason to refuse to document, and I have never done that and would not suggest that anyone else do so. Still, some people don't understand that the hundreds of pages of documentation and procedures, while as organized and clear as I can make them, are long and require thorough reading to understand. I've been praised frequently on the quality and quantity of my documentation, but it has not prevented others from contacting me after I've left to ask questions that were answered in my documentation with more information and clarity.
If the admin struggled with the basics of FTP and ESX, then there is a good chance that they struggled with networks too, so I’d expect it to be a factory config plus the minimum required.
As there is only one admin fo the whole environment then the environment can’t be that large either, so reverse engineering all of it should be fairly trivial. Sounds like a nice short term contract for someone in the local area.
Fully agree that an audit and some guidance on how to do things properly is in order.
> so I’d expect it to be a factory config plus the minimum required.
I would hope that one of the "about 15 different" ways to get in, was to try the default password...
One client I work with, I ensure they maintain all records they have of the various passwords used by their ex-IT guy and his email account. The chap left 4 years ago and even now they encounter situations where this list of passwords has proved helpful, in part because people do stuff and rather than go through the account/contact change process, simply leave everything as they found it. Unfortunately, they also tend not to communicate which set of credentials actually got them into a particular account/system etc.
When I was writing the Dartmouth Time Sharing System I realized that I was the only one who knew how to bring the system up. So one Saturday morning during experimental time sharing (when the sysprogs could hack away, cause crashes etc. - not a critical time for anyone) I made myself unavailable for bringing up the system. It took them 2 hours to bring up the system - there were really sharp people there - and then I had their attention so they listened when I explained the details of bringing up and running the system.
I maintain a website for the local square dancing federation. I made it quite clear that there should be people backups for me. The master password for the web site is known to several officers of the federation and to some others who are actually skilled to maintain the website. There is complete documentation for procedures for maintaining the website stored on the website which anyone is free to download (without the master password).
This same federation lost the entire subscriber database when the person managing it was secretive and stored everything on her personal computer. She died in a car accident and nobody could get into her password-protected computer.
I have seen in several companies, the "wilfull employee" and how much they sabotage attempts to discipline them. And how bad management are at dealing with them. The various futile attempts to enforce documentation and redundancy rules, the meetings to convince them to change their ways and the powerlessness of the hierarchy to enforce the rules.
I had one subordinate who refused to show me anything she was doing, and would lock her terminal every time I approached. If I asked her to do something she didn't want to do, she would claim she was working on something far more important and then refuse to elaborate (what she was doing was stuffing around with linux settings she was unqualified and unauthorised to change, resulting in several outages). But as a team leader, I had no authority to discipline her and our manager's nickname was Homer (fat, bald and really "yellow"), so no resolution there. When I would email her with work (or put it on her desk), she would simply ignore it or place it back on mine (one charming instance, I gave her a print out of a Priority 1 issue to resolve, went to the computer room for an hour and came back to find the printout back on my desk and the issue still outstanding)
I would love to hear anyone who has actually gotten a rogue employee either fired or buttoned down.
I ran across that four times in the 9-5 portion of my career. I handled it by carefully documenting everything, and once I had my ducks in a row, I went over "Homer's" head. I got fired for this effrontery once (turned out that Homer was the owner's nephew). The other three times, I was given Homer's job. Two of those three times it would have been easier getting fired. Choose your battles wisely.
We had a particularly inept senior developer who had done the rounds of the development teams. I ran the tech support team and was told in a management meeting that it was my turn to have him. I responded that's fine you'll have my resignation letter on your desk by the end of the day.
We had a restricted access office with access to hardware consoles for mainframes, the x.25 network management station etc, we were just introducing UNIX and had root / sysadmin access to every server in the organisation.
I had spent 3 years turning around a failing team and was delivering to all SLAs with a 5 9's availability record and was not prepared to inflict this asshat on a team who had worked so hard to turn things around.
My intransigence ended up starting a wave of revolt amongst the other team managers and no technical manager was willing to have him back due to the damage he did to service delivery and team morale. This lead (eventually) to a disciplinary process and the exit of the guy from the organisation voluntarily. He beet being fired by about 2 hours as he resigned just before his disciplinary hearing.
This tactic worked because I actually was willing to resign over the principle.
The system is still working. Therefore, the sadly departed admin likely wasn't as bad as he is made out to be.
Lots of system problems are due to 'finger trouble'. Not having the password effectively prevents that, with the result that the system is still working, long after the admin moved on to the great cloud network in the sky.
So, why exactly are they annoyed with him?
On crucial core network kit (at small firms), I often print the password and attach it to the device (along with IP address, SSID, etc). Normally the back/underside. Anyone with enough physical access to see it could do a factory reset (or just unplug it) anyway.
I know that I won't be around forever, and I also know that there's unlikely to be a formal, structured, handover when that time comes. I want to make it as easy as possible for my successor because when the shoe has been on the other foot I have appreciated such acts too.
There was a joke about a prison van colliding with a concrete truck and ended with the police looking for a gang of hardened criminal. Took my fancy so I used concrete truck instead of bus after that.
I was looking after a small school with about 300 devices and any time I changed a password, I email it to the principal and the IT Co-ordinator (aka ITC) and to my account which was accessible to the agent I contracted for/to and also update the electronic documentation. The first 3 or 4 times I would mention that I had changed the password and emailed the new password to specific staff just in case I got hit by a concrete truck on the way home. The prick principal laid a complaint against me because I was going round making suicidal comments. Just couldn't win at that site.
Are all well and good, so long as the person tasked with keeping them knows if they're accurate or up to date.
I've taken over IT networks, been handed a "handover package" which comprised of nothing more than "we have switches, servers and computers." or being shockingly out of date.
In the face of an 'inaccessible switch, no known config, mission critical' situation am I the only one to think of the bleedin' obvious? As in: check the configs of all the *other* devices plugged into that switch and thus just reverse-engineer what it was doing? Switches don't work in isolation. It's not that hard. Been there, done that. Doing it again at the moment, as there's one on a network I've just 'inherited' :-(
And this is the reason why there used to be a backdoor by using the direct serial connection.
I the "trunk" line feeding the switch is on a single VLAN and not showing up in a traceroute, it can be thought to be a dumb switch. Most likely it has a 802.1q trunk line and may require additional network tracing. This tracing may include visiting all devices connecting to the switch and seeing the settings there.
Ah, the good old days of tracing a ethernet cable through the network floor......
Ok couple of things, first one; almost all the managed switches I came accross (used in businesses) has a local console port and anyone with the right tools can reset a lost password without loosing any configuration information. I guess people are getting too comfortable with click here and click there user interface. Start typing. Second, the guys at the company saying for a coworker "not a good engineer" yet "STRUGGLING TO REPLACE AN EXPIRED ESXI CERT" at the time. Well I guess the whole IT department needs to go considering what happened and what is happening.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
