back to article Dead LAN's hand: IT staff 'locked out' of data center's core switch after the only bloke who could log into it dies

An IT department is pulling its hair out this month after realizing a coworker who died last year was the only person who could log into a crucial network switch. This is according to Dylan, a sysadmin at a small US healthcare company, who today told El Reg a story of how he and his colleagues ended up locked out of the …


        1. hplasm

          Or travelling together on the same GoodieCycle...

        2. J.G.Harston Silver badge

          The external IT Auditor is an external organisation. They would only ever approach our offices if both Clerk and Deputy Clerk fell under the same bus at the same time. And even if all the parish councillors went under the bus as well resulting in no executive authority, things would devolve up to the district council. If the district council was wiped off the earth, there's be other more important things to worry about.

  1. Maelstorm Bronze badge

    I had configs and layout documentation stored on a hidden server on the network. I backed everything up to it. Then I archived it and encrypted it so it could be stored in plain site on several servers, on tape backup, and store the tape in a bank vault. The network didn't change much, so this was a viable option. But when the network went down, the other admins had no clue how stuff was configured because they weren't doing the backups like they were supposed to. Some managers were let go. I figured out what the problem was and reloaded the router with the config off my hidden server. When I left, I still don't think they found the server.

    The backup server is a Raspberry Pi v1.0. It's in a small, non-descript case with a label that has the IP address and a message that reads "Critical network monitoring equipment. Do not disconnect."

    1. tellytart

      Hmm, wonder if I worked the same place as you! I wondered what a RPi was doing in one of the racks on the lower ground floor apps room with a label strikingly similar to yours!

    2. Anonymous Coward
      Anonymous Coward

      Congratulations, you are part of the problem.

      Running an organisation's IT/network is a -team- sport.

      1. jake Silver badge

        Horse crap.

        IT isn't any kind of sport. It's a job.

        Oh, wait ... you're one of those new-agers who expects a trophy for participating, and do-overs when you make a mistake, right?

        1. Kiwi

          Re: Horse crap.

          and do-overs when you make a mistake

          Isn't that the purpose of backups? :)

          1. jake Silver badge

            Re: Horse crap.

            No. Backups aren't a do-over. Backups are a roll-back mechanism.

  2. Unicornpiss

    Maybe he wanted it that way?

    It sounds like he was a bad admin. But he may have seen the writing on the wall about getting fired and purposely felt like making things as difficult as possible for his successor, not knowing that it would be his successor in a literal sense. People seem to be speaking ill enough of the dead in the article that it didn't sound like he was very cooperative.

    And I can't deny that in a crap job situation many of us would not have outright sabotaged anything, but would have felt this way, right or wrong. In the end, it was management's fault for letting things get this way, and possibly for both either maintaining an incompetent employee or so neglecting a once-useful employee that he ended up indifferent and apathetic.

  3. NateGee

    I smell a rat

    The deceased either knew the writing was on the wall so came up with a scenario that meant if he got fired he'd have some satisfaction from the shit-storm he'd leave behind or knew someone didn't like him and was trying to make it impossible for them to get rid of him.

    I've worked with people in the past who were actually very good at their jobs but were constantly being bad mouthed by management. This was often when a PHB thought they knew everything (often loudly) and the colleagues were constantly being forced to prove them otherwise to the PHB's boss.

    1. Kiwi

      Re: I smell a rat

      I've worked with people in the past who were actually very good at their jobs but were constantly being bad mouthed by management. This was often when a PHB thought they knew everything (often loudly) and the colleagues were constantly being forced to prove them otherwise to the PHB's boss.,

      I had fun with that once or twice. The boss had been badmouthing me to suppliers, customers and contractors.

      One day there was a fault with the internet (telco end, not ours) and he came in while I was talking to one of said customers and mentioned it. I just nonchalantly said "Well, you know much more about networking than I do, I'll go off clock and watch in awe as you fix it".

      Both the customer and I found it quite a joy to be sharing a moment together watching him get more and more flustered as he proclaimed his knowldge, went in with a 'fix', only to have that fail so go with another thing. He'd heard me talk of a bad LAN card flooding the network with "martian packets" (an early outage before I got some resiliency built in) so decided one of the machines was causing that and unplugged them one by one, claimed bad cabling so opened up packets of patch cables (all the shop ones were made by me, the ones he opened were on the shelf IE saleable stock), and various other things..

      What made it more enjoyable for me is it was a "brief" (as in 3-4 hours) but major outage. The customer worked a couple of blocks away and had everything with the one telco so had lost phones and internet, and had come to me in person to see if I was aware what was up. As our net and phone line was also out, I'd already phoned the telco to make sure they were aware of a fault when the boss came on the scene. Given we could log into the router and it had a "disconnected" status icon, I was fairly certain the internal LAN was fine and the other side was dead, especially as someone a couple of blocks away had the same problem :)

      Love being able to show that kind of manager up when the time comes.

  4. Anonymous Coward
    Anonymous Coward

    It can become cost prohibitive

    for a small/medium size organization to protect against this kind of attack. You need a complex mixture of technology, policies, procedures and auditing to make sure this doesn't happen. Instead, the organization can outsource it completely and it's no longer a problem.

    1. whitepines

      Re: It can become cost prohibitive

      Instead, the organization can outsource it completely and it's no longer a problem.

      Of course then you have an entirely new set of problems starting with GDPR compliance, data leakage, cloud providers deciding you're last in line for a fix, connection troubles, being held for ransom with rate hikes or mandated software "upgrades", deciding your app isn't worth supporting any longer, etc.

      When just making sure the CEO had a copy of all passwords available would have fixed this at no extra charge.

      But don't let that all get in the way of your cloudy sales pitch!

      1. Olivier2553

        Re: It can become cost prohibitive

        When just making sure the CEO had a copy of all passwords available would have fixed this at no extra charge.

        Or better, the CEO's secretary, as that is a person that can usually be entrusted, but would never think of abusing the power that has been let with them.

        As opposed of a CEO who is more likely to think it is their company and they can and know how to play with the password.

        1. whitepines

          Re: It can become cost prohibitive

          Or better, the CEO's secretary, as that is a person that can usually be entrusted, but would never think of abusing the power that has been let with them.

          So you've seen this too. It's an interesting phenomenon, some form of "absolute power corrupts" I gather.

          One organization I worked with was effectively run by a very competent secretary (that was also a force to be reckoned with if you did wrong). Her "boss" was a miserable twat....

          1. jake Silver badge

            A good secretary is like an NCO in the army of your choice (was: Re: It can become cost prohibitive)

            They run the place.

            Free hint to all consultants: ALWAYS ask the secretary about the Boss's computer knowledge. You can save a lot of time and trouble for a lot of people over the long haul.

            I know of several CEO-types of Fortune-500s who make a big show of "checking the computer", even though their network cable was "accidentally" never installed.

            I can't count the number of times I've swapped the Boss's top of the line CPU, gathering dust and spiderwebs under his credenza/return, artfully changing screensavers every couple minutes, for his secretary's underpowered kit ... without the Boss noticing.

            1. Anonymous Coward
              Anonymous Coward

              Re: A good secretary is like an NCO in the army of your choice (was: It can become cost prohibitive)

              My late wife was (officially) the Financial Controller of a small craft brewery, but as the two Partners were complete imbeciles, who squandered their father's legacy, she actually ran the company. Then she was struck down by an incurable disease, and after she died, the company folded.

          2. This post has been deleted by its author

        2. Is It Me

          Re: It can become cost prohibitive

          When I was sole IT guy for a company I gave the HR guy a sealed envelope with the relevant passwords in and instructions to only pass it on if I was dead/injured etc.

      2. Anonymous Coward
        Anonymous Coward

        @whitepines - Re: It can become cost prohibitive

        You don't deserve the up votes in the same way I don't deserve the down votes.

        Are you trying to tell us the CEO should login three times a day just to make sure those passwords have not been changed ? It happened to me when organizations asked me to help and provided me with two or three written down passwords that were no longer valid. By the way I'm not a cloud stuff salesman, I'm currently configuring telecom equipment but I've learned there is more to security than TACACS/RADIUS and a password kept in an envelope in CEO's office. Clue for you, do a Google search for privilege separation and privileged access management and start from here.

        As someone mentioned in a post below, you're just changing the nature of the problem but your rogue admin scenario is gone. Yes, the outsourcer might have a rogue employee too but your company can hope to be compensated financially for the down time. As for compliance you'll have to do it anyway. Outsourcing or not, you're the owner of the data so act on securing it wherever it is stored. And it is better and cheaper to focus on securing data instead of caring for every switch and wiring closet door in addition to data. My career too is threatened but there's nothing I can do, developers have stolen the show. I would really like to adapt to the new reality but time is a serious constraint.

    2. stiine Silver badge

      Re: It can become cost prohibitive

      No, that just means that you've traded one problem (rogue employee) for another problem (rogue employee of outsourcer) but I agree with your statement "You need a complex mixture of technology, policies, procedures and auditing to make sure this doesn't happen.", but you left out the most important part: good people. If you don't have good people, you can buy tech, etc, to no avail.

      1. Anonymous Coward
        Anonymous Coward

        @stiine - Re: It can become cost prohibitive

        Just assume you'll never have the good people you'd like and use this assumption as a foundation for building the security in IT systems. Be trustful but put safeguards in place.

    3. Roland6 Silver badge

      Re: It can become cost prohibitive

      >for a small/medium size organization to protect against this kind of attack. ... Instead, the organization can outsource it completely and it's no longer a problem.

      It is still a problem, I've come across many small/meduim businesses that prove the rule that businesses like to do business with similar sized organisations. So the small business has outsourced their IT to a 1~2 man operation (1 tech who actually knows the passwords and where they are kept & 1 sales/consultant who instructs the tech but doesn't actually know how to access the tech's folders).

  5. FuzzyWuzzys

    Read The Phoenix Project!

    This is exactly why companies need oversight, you need to badger colleagues to document and record, get organised people. I hate this idea that just because you're asked to document your work and record things, that you're getting the boot next week. No one is trying to steal your job! We need the knowledge so that we can keep you and your job. You vanish on holiday for 2 weeks and no one knows what you do you put the whole company a risk and everyone's job.

    Get your head out of your arse and get stuff written down, and sensitive stuff stored securely, even if it's just a print out given to the IT manager or even get Finance to put it in their safe. On the other hand the more you hoard knowledge, the more complaints HR may receive about you and then you're considered a risk for not complying with regs, this means they may find some way to demote or remove you for not complying.

    You may not be going full on "DevOps" but the TPJ has a lot of sensible ideas about ensuring work is documented, critical people who cause bottlenecks are utilised correctly and knowledge is recorded so others can step in when needed to keep companies in crisis moving along.

    1. amanfromMars 1 Silver badge

      Re: Read The Phoenix Project!

      And do you really want Imminent Failure to reveal Special Forces AIMissions for Universal Resolution with AISolutions for Systems in a Failure withTotal Collapse of Non-Cooperative Operating Systems ......with Exclusive Elite Executive Officer Suites in Stasis?

      Yes would be the Correct Answer there whenever ready for Everything Quite Wonderfully So Easily Different.

      And here be but one Blank Canvas upon which such Futures are Wrought and Writ with Almighty 0Day Provider Protection Testing Vulnerabilities to Distractions and/or Destruction with Enigmatic Ethereal Exploitation of Earthly Resources with Depleted Intelligence Sources.

      What do you Think? AI Fact and/or Pump Fiction? :-)>

      And just whenever Theresa was praying for it not to get any worse ........ An AI LifeRaft just Simply Appears out of Nowhere.

      That Project is Magic. Pure and Simple.

    2. doublelayer Silver badge

      Re: Read The Phoenix Project!

      That may be true, but companies will, at times, ask for documentation under the theory that I'll write down everything that a replacement working with a lot less knowledge and for a smaller bill can simply pick up. That isn't a reason to refuse to document, and I have never done that and would not suggest that anyone else do so. Still, some people don't understand that the hundreds of pages of documentation and procedures, while as organized and clear as I can make them, are long and require thorough reading to understand. I've been praised frequently on the quality and quantity of my documentation, but it has not prevented others from contacting me after I've left to ask questions that were answered in my documentation with more information and clarity.

  6. chivo243 Silver badge

    Where are they?

    Documentation, drawings, config backups, DR plans? I would be ripping heads off and shitting down necks of Nick's bosses right about now...

    There must be some documentation on Nick's workstation, something surely?

  7. StuntMisanthrope

    Rest in profile.

    Be excellent and careful. #tombstone

  8. Dwarf

    If the admin struggled with the basics of FTP and ESX, then there is a good chance that they struggled with networks too, so I’d expect it to be a factory config plus the minimum required.

    As there is only one admin fo the whole environment then the environment can’t be that large either, so reverse engineering all of it should be fairly trivial. Sounds like a nice short term contract for someone in the local area.

    Fully agree that an audit and some guidance on how to do things properly is in order.

    1. Roland6 Silver badge

      > so I’d expect it to be a factory config plus the minimum required.

      I would hope that one of the "about 15 different" ways to get in, was to try the default password...

      One client I work with, I ensure they maintain all records they have of the various passwords used by their ex-IT guy and his email account. The chap left 4 years ago and even now they encounter situations where this list of passwords has proved helpful, in part because people do stuff and rather than go through the account/contact change process, simply leave everything as they found it. Unfortunately, they also tend not to communicate which set of credentials actually got them into a particular account/system etc.

      1. swm

        When I was writing the Dartmouth Time Sharing System I realized that I was the only one who knew how to bring the system up. So one Saturday morning during experimental time sharing (when the sysprogs could hack away, cause crashes etc. - not a critical time for anyone) I made myself unavailable for bringing up the system. It took them 2 hours to bring up the system - there were really sharp people there - and then I had their attention so they listened when I explained the details of bringing up and running the system.

        I maintain a website for the local square dancing federation. I made it quite clear that there should be people backups for me. The master password for the web site is known to several officers of the federation and to some others who are actually skilled to maintain the website. There is complete documentation for procedures for maintaining the website stored on the website which anyone is free to download (without the master password).

        This same federation lost the entire subscriber database when the person managing it was secretive and stored everything on her personal computer. She died in a car accident and nobody could get into her password-protected computer.

        1. Roland6 Silver badge

          >when the person managing it was secretive and stored everything on her personal computer. She died in a car accident and nobody could get into her password-protected computer.

          Must have had full disk encryption enabled, so filesystem couldn't be read by another OS.

  9. OzBob

    Was his name Terry Childs Jr?

    I have seen in several companies, the "wilfull employee" and how much they sabotage attempts to discipline them. And how bad management are at dealing with them. The various futile attempts to enforce documentation and redundancy rules, the meetings to convince them to change their ways and the powerlessness of the hierarchy to enforce the rules.

    I had one subordinate who refused to show me anything she was doing, and would lock her terminal every time I approached. If I asked her to do something she didn't want to do, she would claim she was working on something far more important and then refuse to elaborate (what she was doing was stuffing around with linux settings she was unqualified and unauthorised to change, resulting in several outages). But as a team leader, I had no authority to discipline her and our manager's nickname was Homer (fat, bald and really "yellow"), so no resolution there. When I would email her with work (or put it on her desk), she would simply ignore it or place it back on mine (one charming instance, I gave her a print out of a Priority 1 issue to resolve, went to the computer room for an hour and came back to find the printout back on my desk and the issue still outstanding)

    I would love to hear anyone who has actually gotten a rogue employee either fired or buttoned down.

    1. jake Silver badge

      Re: Was his name Terry Childs Jr?

      I ran across that four times in the 9-5 portion of my career. I handled it by carefully documenting everything, and once I had my ducks in a row, I went over "Homer's" head. I got fired for this effrontery once (turned out that Homer was the owner's nephew). The other three times, I was given Homer's job. Two of those three times it would have been easier getting fired. Choose your battles wisely.

    2. Anonymous Coward
      Anonymous Coward

      Re: Was his name Terry Childs Jr?

      We had a particularly inept senior developer who had done the rounds of the development teams. I ran the tech support team and was told in a management meeting that it was my turn to have him. I responded that's fine you'll have my resignation letter on your desk by the end of the day.

      We had a restricted access office with access to hardware consoles for mainframes, the x.25 network management station etc, we were just introducing UNIX and had root / sysadmin access to every server in the organisation.

      I had spent 3 years turning around a failing team and was delivering to all SLAs with a 5 9's availability record and was not prepared to inflict this asshat on a team who had worked so hard to turn things around.

      My intransigence ended up starting a wave of revolt amongst the other team managers and no technical manager was willing to have him back due to the damage he did to service delivery and team morale. This lead (eventually) to a disciplinary process and the exit of the guy from the organisation voluntarily. He beet being fired by about 2 hours as he resigned just before his disciplinary hearing.

      This tactic worked because I actually was willing to resign over the principle.

  10. herman Silver badge

    The system is still working...

    The system is still working. Therefore, the sadly departed admin likely wasn't as bad as he is made out to be.

    Lots of system problems are due to 'finger trouble'. Not having the password effectively prevents that, with the result that the system is still working, long after the admin moved on to the great cloud network in the sky.

    So, why exactly are they annoyed with him?

    1. Martin Summers

      Re: The system is still working...

      Read the article mate, also don't work in IT please. Thanks.

    2. hmv

      Re: The system is still working...

      Thanks a lot.

      My desk now has a new dent in it.

  11. Anonymous Coward Silver badge


    On crucial core network kit (at small firms), I often print the password and attach it to the device (along with IP address, SSID, etc). Normally the back/underside. Anyone with enough physical access to see it could do a factory reset (or just unplug it) anyway.

    I know that I won't be around forever, and I also know that there's unlikely to be a formal, structured, handover when that time comes. I want to make it as easy as possible for my successor because when the shoe has been on the other foot I have appreciated such acts too.

  12. TrumpSlurp the Troll

    Not fired just died

    For all those commentards who have said that he was fired, the article says they were thinking of firing him when he died unexpectedly.

  13. JJKing

    Some people; no humour.

    There was a joke about a prison van colliding with a concrete truck and ended with the police looking for a gang of hardened criminal. Took my fancy so I used concrete truck instead of bus after that.

    I was looking after a small school with about 300 devices and any time I changed a password, I email it to the principal and the IT Co-ordinator (aka ITC) and to my account which was accessible to the agent I contracted for/to and also update the electronic documentation. The first 3 or 4 times I would mention that I had changed the password and emailed the new password to specific staff just in case I got hit by a concrete truck on the way home. The prick principal laid a complaint against me because I was going round making suicidal comments. Just couldn't win at that site.

  14. user0

    Was it running RIP or OSPF? Must have been RIP

    I'll get my coat.

  15. Sequin

    Was he also the only person to have the password to the laptop holding all of the bitcoin?

  16. andy gibson

    Handover documents

    Are all well and good, so long as the person tasked with keeping them knows if they're accurate or up to date.

    I've taken over IT networks, been handed a "handover package" which comprised of nothing more than "we have switches, servers and computers." or being shockingly out of date.

  17. Anonymous Coward
    Anonymous Coward

    Maybe check the configs on the kit plugged into it?

    In the face of an 'inaccessible switch, no known config, mission critical' situation am I the only one to think of the bleedin' obvious? As in: check the configs of all the *other* devices plugged into that switch and thus just reverse-engineer what it was doing? Switches don't work in isolation. It's not that hard. Been there, done that. Doing it again at the moment, as there's one on a network I've just 'inherited' :-(

  18. gnarlymarley

    serial connection hackable

    And this is the reason why there used to be a backdoor by using the direct serial connection.

    I the "trunk" line feeding the switch is on a single VLAN and not showing up in a traceroute, it can be thought to be a dumb switch. Most likely it has a 802.1q trunk line and may require additional network tracing. This tracing may include visiting all devices connecting to the switch and seeing the settings there.

    Ah, the good old days of tracing a ethernet cable through the network floor......

  19. Decox

    Easy fix

    They do know that you can just boot into recovery and just pull the running config.txt over to a machine and take out the encrypted login yeah? I'm a Cisco certified network admin and do work on government contracts, not that hard to do guys.

    1. Anonymous Coward Silver badge

      Re: Easy fix

      How do you boot into recovery without interrupting service?

      1. dnicholas

        Re: Easy fix

        At night. Order pizza and tell everyone to **** off.

        That's how I do it

  20. OneITGuy

    Ok couple of things, first one; almost all the managed switches I came accross (used in businesses) has a local console port and anyone with the right tools can reset a lost password without loosing any configuration information. I guess people are getting too comfortable with click here and click there user interface. Start typing. Second, the guys at the company saying for a coworker "not a good engineer" yet "STRUGGLING TO REPLACE AN EXPIRED ESXI CERT" at the time. Well I guess the whole IT department needs to go considering what happened and what is happening.


POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like