All right, what's all this, then?!
So they needed a network audit before firing him. And they really needed an audit after he 'left'. And now they need a miracle? Oh, and an audit.
An IT department is pulling its hair out this month after realizing a coworker who died last year was the only person who could log into a crucial network switch. This is according to Dylan, a sysadmin at a small US healthcare company, who today told El Reg a story of how he and his colleagues ended up locked out of the …
What icons? A Planchette has none. Original had a hole for a pencil for 'automatic writing', later version was a pointer for the Ouija board, which might only have decorations, no icons, as it's Yes, No and alphabet. Released originally as a game and a game company still has the name as a trademark.
So what are these Oracle icons?
I don't think an audit will solve the problem... It might have prevented it but it's too late now. Maybe the answers are on the mysterious #4 server as I've known at least one admin who kept a server "hidden" for emergencies. Turned out to have config info, some nasty info on manglement, and a pile of server and network scripts.
I'm the AC who mentioned Radius/TACACS farther down in the comments as being helpful. Here's a link with brief explanations of AAA, RADIUS and TACACS+. http://www.pearsonitcertification.com/articles/article.aspx?p=2449614
I have no ties to Pearson and am not able to speak to the efficacy their certification course ware.
also see:
http://wiki.freeradius.org/guide/Getting%20Started
http://networkradius.com/doc/FreeRADIUS%20Technical%20Guide.pdf
That is true, but it really depends how large the place was. For example, I am mostly a developer, but I volunteer some system administration for a local charity that I appreciate. They used to have an administrator, but they left and they don't have that many systems. When I arrived to look over the systems and start my work, I found the following:
1. A server that contains a domain controller and shared network folders.
2. A UPS for aforementioned server. Not plugged in to the wall socket or, thankfully, the server.
3. A backup system that seemed to be set up properly. It used removable disks that were swapped out every week, when there was an administrator. Since that admin left, they had one disk inside the system that contained the most recent backup and two disks that contained backups from eight months previous.
4. A firewall that nobody had the access codes to. Nobody knew what this firewall was or wasn't doing and I just wanted to get rid of it once I felt confident to rebuild the network.
This is what happens when there is only one person working on the system and the company lacks the ability to manage that person. The charity is small, the director is nontechnical, and the system was consequently chaotic. There wasn't a clear person at fault, but we could all agree that there was a problem.
Similar situation.
I volunteered to take care of the network in my condo because the existing situation was a mess.
Well, not completely a mess, at the origins, it had been set-up properly, 2 WiFi per floor, one in each wing of the building, a WiFi bridge over to the other building, using Linksys and Ubiquity (I mean, not the cheapest possible hardware), under DD-WRT... So a good built. But obviously, the company managing the system was not the one that once installed it, so when the network hanged, the only solution was a power-off to reboot the system (and because there are 15 WiFi, it is faster to turn off the central power of the building! I joke you not. Just hope nobody is in the lift during these 5 seconds!).
So, that was the situation when I moved in 2 years ago, so I volunteered to look after the system, I changed the main router/authentication platform and now I am left with a bunch of WiFi under DD-WRT without knowing the password.
If the few hacks I found online are not working, I will be left with the solution of a reset-reconfigure, but I am not really looking forward to that.
I would suggest, if possible, that you get them to buy new hardware and set up a replacement in parallel. Otherwise, I hope you are good at network administration. I have tried long enough to get a multiple-AP network going with multiple openwrt devices, and I found it to be a terribly long and painful process involving far too much fiddling with DHCP. I'll be the first to say that my network admin experience is suboptimal, but there is still a lot of complexity and ways for that to completely fail.
Having had to adopt a Ubiquiti wifi network left in a similar state, I'd say you're not in too bad a place. Not sure what you're using as a controller (an old laptop running debian will do it fine) but, as long as you use the same SSID and passwords, you can wipe and reset the whole system pretty easily - start with one AP (reset, then adopt) and then roll it out from there when you're happy with the config. The Ubiquit forums are your friend in this situation.
Happens in every IT department I've worked in. There are always staff members who see knowledge as power and always managers who allow it to persist.
I have audited my own current IT department (I work in infosec) and have pointed this issue out repeatedly over the past couple of years, it's in almost every report I write but still isn't addressed. They state that since they have sharepoint up and running this isn't a problem, but as always there's bugger all on there from the staff members in question. It's all full of content from the team-minded people, the ones who play well with others etc.
Thing is one staff member in particular has been suspended several times in the past few years for not following change management processes and causing the network to drop, screwing DNS, breaking the web filtering etc. He's one of those who is a full believer in "knowledge is power" and has an inept manager who won't challenge him. I am desperate for that staffer to end up at a disciplinary so I can grill him but until then my hands are tied.
It's bloody frustrating, not just for me but all the IT staff who have to put up with this crap.
There are always staff members who see knowledge as power
Ah, I think that's where I've gone wrong then. I've always sought to be professional and leave documentation so that someone coming along after me has something to work from. Of course, that makes me dispensable ...
Also, I happen to know that at my last job my carefully engineered and fully documented network was ripped out by a "I don't understand it so I'm ripping it out" imbecile who only thinks he has a clue about networking (barely understands the basics of IPv4 addressing). Amusingly, I'd had no outages on the DNS we hosted for something like 600 customer domains for several years thanks to proper engineering with suitable redundancy - and he killed about 100 that were left on our servers when he killed the master and didn't know he needed to promote a slave before the zones timed out ! Prior to that we'd had a policy in the technical section of doing detailed network diagrams when doing a customer install - IP addresses, WiFi details, all the useful stuff that an engineer could do with to look after the network. Same person demanded all that useful stuff be removed because it didn't make for a "pretty drawing" to hand over to the customer (we kept our hand drawn originals !)
At my previous job to that, I had a database of what was patched to where, a database of all IP allocations, detailed network diagram, where all the data sockets were located, the patch leads were colour coded by function, etc. Before long all the patch leads were replaced with black (and didn't use the management bars well), the database was tossed, etc, etc ...
It's "slightly disheartening" to know that you've left a situation where any half-capable person can come in, look at the documentation, and take over without too much hassle - only to find they've replaced you with a monkey (in at least one case, that's being unkind to primates !) who's set out to wreck things.
There is a picture in reddit https://i.redd.it/ejlc2pmrd9n21.jpg where the joke is that the angry girl is going to shoot the author's chromebook and the script has him say "don't do that that is the only copy of my book". Most commenters note that the author is probaby writing in G Suite and only needs to get a new chromebook if she shoots. :-)
Maybe time to start clouudifying configs. :-)
If it wasn't for the fact that this switch was so integral to their network, a simple solution might be to expose the thing to the internet and offer a bounty to the first enterprising hacker who could get in, reset the login credentials and then report in with the solution.
The password is BOB. Unless he was dyslexic, in which case it will be BOB.
Seriously, to blame a lame engineer for this stuff is ridiculous. The guys boss should be on the chopping block for allowing his network to be run that way. Managers get a pass when someone goes rogue, but not when they ignore an ongoing problem with a critical process.
"Managers get a pass when someone goes rogue, but not when they ignore an ongoing problem with a critical process."
Well, yes agreed, but also, say a manager asks sysadmin for full audit of network including providing backup configs and passwords, the manager is still dependent on the sysadmin's honesty and competence, either of which could be lacking. How would the manager even notice if anything related to this particular switch was missing, given that it was literally months before anyone even noticed the switch was there?
Not to mention the process usually goes
"Bob, I need a full network inventory...oh wait, the sales department are complaining that their VPN isn't working, can you fix that first..."
Aaaaaaaaaaaaaaaaand the audit (and any other documentation) never gets done because it's always less important than keeping everything running.
We have a core switch that only one person can log into, not only because they are the only person with a password, but also because it's broken and will only respond to packets from certain MAC addresses.
Exactly this.
Manager: So we need to document this flow. And we to fix these things ASAP.
Me: You can have one or the other.
Manager: Why? Why can't we have both?
Me: Because you insisted on a three hour meeting, face to face in a conference room to go over this. Now I have time to do one or the other.
Manager: Hmmmmm... Ok, just fix the things. Then we'll have to have another meeting to go over the plan to document everything.
Me: *sigh* Sure thing, boss. Just put the meeting notice on my calendar. (Screaming inwardly to myself, "And you've still learned NOTHING!")
Just a lightly edited extract of a real situation. Much of the problem is that for managers their work product is meetings, so they think things are getting done if you have a meeting about a subject rather than actually working on the problem.
This touches on something that's been really pissing me off lately.
I consider myself a fairly competent IT person. I've worked as a systems admin, databases admin, software developer, and a network engineer for two ISP's.
Lately, I've been applying for a bunch of IT manager positions. I'm not having much luck because I'm told I need x years of management experience. They don't seem to give a shit about my 26 years of technical experience!! No, I could have a shit technical background, but if I had 5 years of experience as some PHB, I would be in.
I (being a little pissed about getting turned down without even an interview) even mentioned to a couple of companies that maybe if they hired managers with real technical skills, it may make things run better at their company?
What RFC822 said.
Free advice for new sysadmins: Take as many business related courses as you can stomach. Haul your ass to your nearest post-secondary school that offers night courses and talk to a career counselor. Tell 'em that you are a techie, but are interested in management. You want to take courses that can be applied to a future MBA (should you want to go that route later).
If you already hold a four year degree, and you can code fluently in one or more upper level languages, chances are you can snooze through an MBA in two years (or less, if the classes line up right). Lest you think getting an MBA is difficult, think about all the feckless idiots you know who hold one ;-)
I realize that not all of us are cut out for management ... the objective isn't necessarily to become a manager, but rather to learn their lingo. It's amazing how fast long-closed doors open once you learn to talk to Moneybags in his/her own language. On top of that, an MBA will better prepare you for when the time comes to strike out on your own and become a consultant.
In a prior life, I was tasked with the boots-on-ground closet to closet network hardware audit of an office complex.
Serial numbers, model numbers, port count, etc etc. One of the duties was pulling configs from everything that could give them up.
Was it password protected without the password being known by anyone? Hold this button on the panel, connect via terminal emulator over the serial port, enter this obscure command, and then set this password after hoovering up the config.
Mostly CISCO gear, but I'd assume with hardware access anything can be changed.
Dell might just be evil though?
That is something you can do on Dell networking equipment. The problem is that they can't afford to take it down for the few minutes it would take to do that and why they are screwed until April when they have a scheduled maintenance period.
From the sounds of it, this data-center is mission-critical and even a minute of downtime would be quite costly, especially since nothing is broken.
How do you know nothing is broken if you can't log into it? Just because it is routing traffic properly? How do you know that on top of it's normal duties, it isn't logging all LAN traffic and shipping it off to your competitor? Or that it hasn't otherwise been rooted, given the lack of clues possessed by the former admin?
To me, the box is a corporate hazard and needs to be airgapped, preferably last December!