back to article Dead LAN's hand: IT staff 'locked out' of data center's core switch after the only bloke who could log into it dies

An IT department is pulling its hair out this month after realizing a coworker who died last year was the only person who could log into a crucial network switch. This is according to Dylan, a sysadmin at a small US healthcare company, who today told El Reg a story of how he and his colleagues ended up locked out of the …


  1. Kijoma

    They could of course attack other monitoring routers set as bridges to log what vlan's and traffic eminate from this things ports and build up a picture of it's config ? Use the Mikrotik ones as they are cheap enough unless you have a cisco fetish.

  2. W60

    rubbish statement - takes two to tango

    "No clue about VLANs, no clue about if it has STP, or trunking, or anything."

    Yes you do - if you have access to the attached switches/devices to the core then you have the other half of the config. Totally agree with the previous comments on the mgmt side should ensure this situation doesn't happen, and backups (all things fail human and silicon) but as a network guy with 20yrs +, STP is a mutual thing as should be the vlan and trunking (packet sniffer will show up most of this). You will need down time for the replacement but hunt for the clues first on the wire and attached devices first if you really are situation. You should not get in to this situation and is bad working practise from all parties, it is a fire fight but not all is lost (if you do not know what your core switch is doing for your critical traffic with out seeing the config please find the door....oh you cant do that with out being provided a map I see your problem!)

    1. Roland6 Silver badge

      Re: rubbish statement - takes two to tango

      >You will need down time for the replacement but hunt for the clues first on the wire and attached devices first if you really are situation.

      This is what Dylan should be doing now, aided by Dell support!

      Such preparation means that the most can be obtained from the scheduled downtime in April.

      Perhaps Dylan has received guarantees from Dell that with the network down they will be able to gain access to the switch without losing the configuration. But even then with senior management visibility and attention, Dylan would be well advised to do the leg work and document the As-Is network configuration; in part to provide basic information necessary to start configuring a replacement device - something I would be doing in any case - in part because it is highly likely the existing network gear is running old software..

  3. Anonymous Coward
    Anonymous Coward

    No Postit on the poor chap's monitor?

  4. Christian Berger

    Uhm... wait for the downtime...

    ... shut it down, pull the CF-card and read the configuration if it's a larger device.

    Alternatively boot into "factory config", log in with the default credentials and read the starup config.

  5. Anonymous Coward
    Anonymous Coward

    Not quite the same, but I got a query this morning from a guy asking if I happened to have a copy of an old Cisco IOS because a customer had an ancient switch that held a vital part of their network together that had gone down

    (as it happened we did have something suitable on an equally ancient XP PC with 20Gb drive running as a TFTP server... plenty of space left on the drive, so we never bothered deleting old files, and no big deal if it went BANG! tomorrow)

  6. darksurf

    Surely this is a joke

    Almost ALL Enterprise grade manageable switches and routers have a password recovery method upon booting the device and having physical access. Dell is no exception.

    How they can not recover this device is beyond me. This place sounds sketchy with untrained technicians running everything.

    1. Kiwi

      Re: Surely this is a joke

      How they can not recover this device is beyond me. This place sounds sketchy with untrained technicians running everything.

      Might be something hidden in the bits about "critical infrastructure" and requiring scheduled downtime a month or two in advance?

      Sure, it would be nice if every manager and the like knew exactly how every bit of network kit was run, and they knew how to build to best practices etc (including not relying a single piece of kit that cannot be shut down for a few minutes without a quick way to bring things back up - they do have a 4 hour replacement contract after all).

      But then if managers could do our jobs, we would not have jobs now would we? If the boss could build the network, why hire someone else? And if you hire someone who passed enough of your checks to be allowed to build your network, there's a fair bet he could at least fake it well enough to appear that he could be trusted to do his job.

      Sure, it's a simple matter of rebooting the switch. But sometimes, that is not so simple. I used to run a couple of seperate lots of web/email server (with mirroring) in different locations, and if I had to reboot either routers or servers I could change the DNS, wait a bit (set to 15min TTL IIRC), then reboot knowing full well that if anything failed to come up no one but me would know. Not every one has this set up though.

  7. dnicholas

    Well at least...

    ... he changed the default username and password.

    That's several steps above some of the messes I've had to sort out.

  8. MachDiamond Silver badge

    Managers are shite

    Until you get one that isn't. I've worked for myself most of of the time but there have been some employee jobs mixed in. It was years and years until I finally got a manager that knew his job and did it well. They're a pretty rare breed and this situation is due to a poor manager.

    Systems change all of the time. When I worked in aerospace, stuff changed at least once a week. This forced me to come up with a way of documenting the hardware I was working on in a way that was easy to update. I also would document from 3 different approaches that each worked the best depending on what made the most sense when troubleshooting. That spilled over to how I managed the software on my computer. Since I was the lead avionics person, being the only avionics person, all of the e-CAD was on my computer along with licenses for Solidworks and other other SimWare. On my desk was a folder that contained all of the serial numbers, logons and passwords so if I were to get hit by a bus somebody could pull up whatever they needed that wasn't checked in at the time as a .pdf or other portable format. The folder also contained a whole bunch of other inane company procedure crap so it wasn't obvious that there was a page of credentials in there. Security wasn't a huge issue since there were only about 9 of us in the design office. When we had a proper engineering manager, he got a copy of my notes to keep and made everybody do the same thing. With just 9 people, nearly all of us were a one person departments. A big part of my job became documentation management since we started doing some government contract work and The Man is all about paper. Yes, it took away time from my working on electronics, but it also helped the company win a million dollar aerospace prize after we had a fire and had other people come in to help recover in one long night. A whole system was rewired with a single error while I slept.

    I still got grief for spending time documenting stuff and trying to weed out the endless useless files in SVN that were automatically checked in since nobody ever received any training on which bits go in and which get tossed out.

    If there is something in a company that is massively critical, it can't bottleneck through one person. Even worse if the person is a complete tosser. There has to be somebody in management that skips the 3 martini lunches and thinks about "What happens if this breaks?" What happens if the power goes out? What happens if this person leaves with no notice/dies/gets sick/defects to a competitor? If it's a janitor, easy, hire a new janitor. If it's an EVP, what procedure needs to be in place to do a secure exit process? If it's the guy in IT that has all the passwords, hire in a third party analyst to figure out what passwords that person better hand over to keep their job and good reference.

  9. dnicholas


POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like