back to article Dead LAN's hand: IT staff 'locked out' of data center's core switch after the only bloke who could log into it dies

An IT department is pulling its hair out this month after realizing a coworker who died last year was the only person who could log into a crucial network switch. This is according to Dylan, a sysadmin at a small US healthcare company, who today told El Reg a story of how he and his colleagues ended up locked out of the …

Page:

  1. Notas Badoff

    All right, what's all this, then?!

    So they needed a network audit before firing him. And they really needed an audit after he 'left'. And now they need a miracle? Oh, and an audit.

    1. Yet Another Anonymous coward Silver badge

      Re: All right, what's all this, then?!

      "talk -ouija" isn't an option ?

      1. Unicornpiss

        Re: All right, what's all this, then?!

        Perhaps they should consult with Oracle. After all, Oracle's some of Oracle's old icons were a ripoff from Ouija's pointer or "planchette" as I think it's called.

        1. Mage Silver badge
          Coat

          Re: Oracle's old icons

          What icons? A Planchette has none. Original had a hole for a pencil for 'automatic writing', later version was a pointer for the Ouija board, which might only have decorations, no icons, as it's Yes, No and alphabet. Released originally as a game and a game company still has the name as a trademark.

          So what are these Oracle icons?

    2. Mark 85

      Re: All right, what's all this, then?!

      I don't think an audit will solve the problem... It might have prevented it but it's too late now. Maybe the answers are on the mysterious #4 server as I've known at least one admin who kept a server "hidden" for emergencies. Turned out to have config info, some nasty info on manglement, and a pile of server and network scripts.

    3. Anonymous Coward
      Anonymous Coward

      Re: Speed impacted my children's education

      Selling audits is so much fun.

    4. Anonymous Coward
      Anonymous Coward

      Re: All right, what's all this, then?!

      I'm the AC who mentioned Radius/TACACS farther down in the comments as being helpful. Here's a link with brief explanations of AAA, RADIUS and TACACS+. http://www.pearsonitcertification.com/articles/article.aspx?p=2449614

      I have no ties to Pearson and am not able to speak to the efficacy their certification course ware.

      also see:

      http://wiki.freeradius.org/guide/Getting%20Started

      http://networkradius.com/doc/FreeRADIUS%20Technical%20Guide.pdf

  2. Anonymous Coward
    Anonymous Coward

    Sounds like poor management allowed this to happen.

    1. jake Silver badge

      ITYM

      "It was clearly bad management which allowed this to happen."

    2. smudge

      Indeed. "Over-reliance on key personnel" is considered in even the most basic of risk assessments.

      1. doublelayer Silver badge

        That is true, but it really depends how large the place was. For example, I am mostly a developer, but I volunteer some system administration for a local charity that I appreciate. They used to have an administrator, but they left and they don't have that many systems. When I arrived to look over the systems and start my work, I found the following:

        1. A server that contains a domain controller and shared network folders.

        2. A UPS for aforementioned server. Not plugged in to the wall socket or, thankfully, the server.

        3. A backup system that seemed to be set up properly. It used removable disks that were swapped out every week, when there was an administrator. Since that admin left, they had one disk inside the system that contained the most recent backup and two disks that contained backups from eight months previous.

        4. A firewall that nobody had the access codes to. Nobody knew what this firewall was or wasn't doing and I just wanted to get rid of it once I felt confident to rebuild the network.

        This is what happens when there is only one person working on the system and the company lacks the ability to manage that person. The charity is small, the director is nontechnical, and the system was consequently chaotic. There wasn't a clear person at fault, but we could all agree that there was a problem.

        1. Olivier2553

          Similar situation.

          I volunteered to take care of the network in my condo because the existing situation was a mess.

          Well, not completely a mess, at the origins, it had been set-up properly, 2 WiFi per floor, one in each wing of the building, a WiFi bridge over to the other building, using Linksys and Ubiquity (I mean, not the cheapest possible hardware), under DD-WRT... So a good built. But obviously, the company managing the system was not the one that once installed it, so when the network hanged, the only solution was a power-off to reboot the system (and because there are 15 WiFi, it is faster to turn off the central power of the building! I joke you not. Just hope nobody is in the lift during these 5 seconds!).

          So, that was the situation when I moved in 2 years ago, so I volunteered to look after the system, I changed the main router/authentication platform and now I am left with a bunch of WiFi under DD-WRT without knowing the password.

          If the few hacks I found online are not working, I will be left with the solution of a reset-reconfigure, but I am not really looking forward to that.

          1. doublelayer Silver badge

            I would suggest, if possible, that you get them to buy new hardware and set up a replacement in parallel. Otherwise, I hope you are good at network administration. I have tried long enough to get a multiple-AP network going with multiple openwrt devices, and I found it to be a terribly long and painful process involving far too much fiddling with DHCP. I'll be the first to say that my network admin experience is suboptimal, but there is still a lot of complexity and ways for that to completely fail.

          2. Anonymous Coward
            Anonymous Coward

            Ubiquiti

            Having had to adopt a Ubiquiti wifi network left in a similar state, I'd say you're not in too bad a place. Not sure what you're using as a controller (an old laptop running debian will do it fine) but, as long as you use the same SSID and passwords, you can wipe and reset the whole system pretty easily - start with one AP (reset, then adopt) and then roll it out from there when you're happy with the config. The Ubiquit forums are your friend in this situation.

          3. M.V. Lipvig Silver badge
            Trollface

            "I volunteered to take care of the network in my condo "

            Found the problem.

    3. Anonymous Coward
      Anonymous Coward

      Happens in every IT department I've worked in. There are always staff members who see knowledge as power and always managers who allow it to persist.

      I have audited my own current IT department (I work in infosec) and have pointed this issue out repeatedly over the past couple of years, it's in almost every report I write but still isn't addressed. They state that since they have sharepoint up and running this isn't a problem, but as always there's bugger all on there from the staff members in question. It's all full of content from the team-minded people, the ones who play well with others etc.

      Thing is one staff member in particular has been suspended several times in the past few years for not following change management processes and causing the network to drop, screwing DNS, breaking the web filtering etc. He's one of those who is a full believer in "knowledge is power" and has an inept manager who won't challenge him. I am desperate for that staffer to end up at a disciplinary so I can grill him but until then my hands are tied.

      It's bloody frustrating, not just for me but all the IT staff who have to put up with this crap.

      1. Anonymous Coward
        Anonymous Coward

        There are always staff members who see knowledge as power

        Ah, I think that's where I've gone wrong then. I've always sought to be professional and leave documentation so that someone coming along after me has something to work from. Of course, that makes me dispensable ...

        Also, I happen to know that at my last job my carefully engineered and fully documented network was ripped out by a "I don't understand it so I'm ripping it out" imbecile who only thinks he has a clue about networking (barely understands the basics of IPv4 addressing). Amusingly, I'd had no outages on the DNS we hosted for something like 600 customer domains for several years thanks to proper engineering with suitable redundancy - and he killed about 100 that were left on our servers when he killed the master and didn't know he needed to promote a slave before the zones timed out ! Prior to that we'd had a policy in the technical section of doing detailed network diagrams when doing a customer install - IP addresses, WiFi details, all the useful stuff that an engineer could do with to look after the network. Same person demanded all that useful stuff be removed because it didn't make for a "pretty drawing" to hand over to the customer (we kept our hand drawn originals !)

        At my previous job to that, I had a database of what was patched to where, a database of all IP allocations, detailed network diagram, where all the data sockets were located, the patch leads were colour coded by function, etc. Before long all the patch leads were replaced with black (and didn't use the management bars well), the database was tossed, etc, etc ...

        It's "slightly disheartening" to know that you've left a situation where any half-capable person can come in, look at the documentation, and take over without too much hassle - only to find they've replaced you with a monkey (in at least one case, that's being unkind to primates !) who's set out to wreck things.

      2. Kiwi
        Devil

        I am desperate for that staffer to end up at a disciplinary so I can grill him but until then my hands are tied.

        Er, you don't read much of the BOFH do you?

        1. jake Silver badge

          "you don't read much of the BOFH do you?"

          No, I don't. It was tired & derivative of itself before it left Usenet.

          But thanks for asking.

  3. stu_san
    Joke

    Old cartoon

    I recall an old cartoon (from Datamation? no idea) of a mother and children who are mourning next to a grave in the rain. A portly gentleman is leaning close and saying, "I know this is a bad time, but do you remember him saying anything about source code?"

    1. jake Silver badge

      Re: Old cartoon

      I think I saw that one in Infoworld. Probably alongside one of Metcalfe's columns poo-pooing Linux & FOSS back in the mid-late '90s.

    2. Anonymous Coward
      Anonymous Coward

      Re: Old cartoon

      There is a picture in reddit https://i.redd.it/ejlc2pmrd9n21.jpg where the joke is that the angry girl is going to shoot the author's chromebook and the script has him say "don't do that that is the only copy of my book". Most commenters note that the author is probaby writing in G Suite and only needs to get a new chromebook if she shoots. :-)

      Maybe time to start clouudifying configs. :-)

  4. Anonymous Coward
    Anonymous Coward

    There's help out there ...

    If it wasn't for the fact that this switch was so integral to their network, a simple solution might be to expose the thing to the internet and offer a bounty to the first enterprising hacker who could get in, reset the login credentials and then report in with the solution.

    1. jake Silver badge

      Re: There's help out there ...

      Reading between the lines, chances are good that this has already occurred ... Well, the "cracker got in" bit, anyway.

    2. Terry Barnes

      Re: There's help out there ...

      I don’t know, I think you’d be inviting someone malicious to hold the config to ransom.

      1. jmch Silver badge

        Re: There's help out there ...

        "I don’t know, I think you’d be inviting someone malicious to hold the config to ransom."

        How is that a bad thing? They don't have the config anyway, I'm pretty sure they would be willing to pay a reasonable amount to get it back.

        1. ChrisBedford

          Re: There's help out there ...

          Yeah, but given the predominant mentality out there, the chances are good that someone even more malicious gets in first and without backing it up screws the entire config so nothing works.

        2. Terry Barnes

          Re: There's help out there ...

          Malicious as in - pay us or we delete it and hard reset the switch back to factory defaults. No IT problem is solved by inviting criminals to solve it for you.

  5. Kicker of Metaphorical Cats

    BOB

    The password is BOB. Unless he was dyslexic, in which case it will be BOB.

    Seriously, to blame a lame engineer for this stuff is ridiculous. The guys boss should be on the chopping block for allowing his network to be run that way. Managers get a pass when someone goes rogue, but not when they ignore an ongoing problem with a critical process.

    1. Anonymous Coward
      Coat

      Re: BOB

      The only bad admin that changes the default password form root/calvin?

      1. stiine Silver badge

        Re: BOB

        And I change them both, so neither root nor calvin are valid.

      2. Anonymous Coward
        Windows

        Re: BOB

        That's iDRACs you insensitive clod.

      3. bpfh

        Re: BOB

        Which just reminded me of a few years ago where scott/tiger got me into more databases than it should, but I blame that on the consulting firms that had excellent sales in managing to sell Big Red To small businesses where Access would have done...

    2. jmch Silver badge

      Re: BOB

      "Managers get a pass when someone goes rogue, but not when they ignore an ongoing problem with a critical process."

      Well, yes agreed, but also, say a manager asks sysadmin for full audit of network including providing backup configs and passwords, the manager is still dependent on the sysadmin's honesty and competence, either of which could be lacking. How would the manager even notice if anything related to this particular switch was missing, given that it was literally months before anyone even noticed the switch was there?

      1. phuzz Silver badge

        Re: BOB

        Not to mention the process usually goes

        "Bob, I need a full network inventory...oh wait, the sales department are complaining that their VPN isn't working, can you fix that first..."

        Aaaaaaaaaaaaaaaaand the audit (and any other documentation) never gets done because it's always less important than keeping everything running.

        We have a core switch that only one person can log into, not only because they are the only person with a password, but also because it's broken and will only respond to packets from certain MAC addresses.

        1. nerdbert
          Pint

          Re: BOB

          Exactly this.

          Manager: So we need to document this flow. And we to fix these things ASAP.

          Me: You can have one or the other.

          Manager: Why? Why can't we have both?

          Me: Because you insisted on a three hour meeting, face to face in a conference room to go over this. Now I have time to do one or the other.

          Manager: Hmmmmm... Ok, just fix the things. Then we'll have to have another meeting to go over the plan to document everything.

          Me: *sigh* Sure thing, boss. Just put the meeting notice on my calendar. (Screaming inwardly to myself, "And you've still learned NOTHING!")

          Just a lightly edited extract of a real situation. Much of the problem is that for managers their work product is meetings, so they think things are getting done if you have a meeting about a subject rather than actually working on the problem.

          1. swm

            Re: BOB

            I once had a boss who wanted a task done in three months. He then asked for another task taking two days. I said, "OK, but the three month task is now three months and two days."

            He had an interesting look on his face.

      2. Anonymous Coward
        Anonymous Coward

        Re: BOB

        This touches on something that's been really pissing me off lately.

        I consider myself a fairly competent IT person. I've worked as a systems admin, databases admin, software developer, and a network engineer for two ISP's.

        Lately, I've been applying for a bunch of IT manager positions. I'm not having much luck because I'm told I need x years of management experience. They don't seem to give a shit about my 26 years of technical experience!! No, I could have a shit technical background, but if I had 5 years of experience as some PHB, I would be in.

        I (being a little pissed about getting turned down without even an interview) even mentioned to a couple of companies that maybe if they hired managers with real technical skills, it may make things run better at their company?

        1. RFC822

          Re: BOB

          Why do you think that 26 years technical experience qualifies you to be a manager? Do you think that 26 years management experience would qualify somebody for a technical position?

          1. jake Silver badge

            Re: BOB

            What RFC822 said.

            Free advice for new sysadmins: Take as many business related courses as you can stomach. Haul your ass to your nearest post-secondary school that offers night courses and talk to a career counselor. Tell 'em that you are a techie, but are interested in management. You want to take courses that can be applied to a future MBA (should you want to go that route later).

            If you already hold a four year degree, and you can code fluently in one or more upper level languages, chances are you can snooze through an MBA in two years (or less, if the classes line up right). Lest you think getting an MBA is difficult, think about all the feckless idiots you know who hold one ;-)

            I realize that not all of us are cut out for management ... the objective isn't necessarily to become a manager, but rather to learn their lingo. It's amazing how fast long-closed doors open once you learn to talk to Moneybags in his/her own language. On top of that, an MBA will better prepare you for when the time comes to strike out on your own and become a consultant.

    3. Tom 7

      Re: BOB

      That would involve Manglement having a clue! Have you ever tried to get them to understand what their own job is let alone yours!

      1. Anonymous Coward
        Anonymous Coward

        Re: BOB

        I would suggest the password is just the letter a

    4. Anonymous Coward
      Anonymous Coward

      Re: BOB

      If Nick is as bad as they say, no one tried "password"?

  6. Zarno

    In a prior life, I was tasked with the boots-on-ground closet to closet network hardware audit of an office complex.

    Serial numbers, model numbers, port count, etc etc. One of the duties was pulling configs from everything that could give them up.

    Was it password protected without the password being known by anyone? Hold this button on the panel, connect via terminal emulator over the serial port, enter this obscure command, and then set this password after hoovering up the config.

    Mostly CISCO gear, but I'd assume with hardware access anything can be changed.

    Dell might just be evil though?

    1. Crazy Operations Guy

      That is something you can do on Dell networking equipment. The problem is that they can't afford to take it down for the few minutes it would take to do that and why they are screwed until April when they have a scheduled maintenance period.

      From the sounds of it, this data-center is mission-critical and even a minute of downtime would be quite costly, especially since nothing is broken.

      1. Zarno

        I remember not having to do a reboot or introduce downtime, but IIRC it did flag physical access in the logs.

      2. Yet Another Anonymous coward Silver badge

        You are doubly screwed if for whatever reason it doesn't come back up - and a replacement is 4-6 weeks delivery

        1. pavel.petrman

          The article says they have a 4 hour replacement on it from Dell. This to me says something about it being a critical piece of network equipment, because why else would a company with _this_ level of management incompetence buy a 4 hour replacement.

          1. jmch Silver badge
            Happy

            "why else would a company with _this_ level of management incompetence buy a 4 hour replacement?"

            Erm... you're kind of answering your own question there :)

      3. jake Silver badge

        "especially since nothing is broken."

        How do you know nothing is broken if you can't log into it? Just because it is routing traffic properly? How do you know that on top of it's normal duties, it isn't logging all LAN traffic and shipping it off to your competitor? Or that it hasn't otherwise been rooted, given the lack of clues possessed by the former admin?

        To me, the box is a corporate hazard and needs to be airgapped, preferably last December!

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like