back to article YOU are the threat: True confessions of real-life sysadmins

Some sysadmins will go to extremes to secure a network, viewing it (wrongly) as their property. For proof, look no further than Terry Childs, the City of San Francisco sysadmin who lost his job and subsequently refused to give over the system's virtual keys to his superiors in 2008. It took just under a million dollars, …

  1. Michael H.F. Wilkinson Silver badge
    Joke

    Simon will be recharging the cattle-prod

    "Some sysadmins will go to extremes to secure a network, viewing it (wrongly) as their property."

    What do you mean WRONGLY?

    >KZZZEERRT!!!<

  2. Lee D Silver badge

    I've generally found that, given the amount of trust in IT people, they are in the higher tier of people who actually can be trusted with such data and control. I work in schools and, technically, I have more access to more information, with more "potential" for mischief than anyone else - even the head, governors or bursar combined.

    Yet you find that, aside from laziness or incompetence, actual malicious intent is incredibly, extremely rare; almost non-existent.

    That said, in job interviews, I'm often asked in true cliche: What is my biggest weakness?

    My answer is truthful... it is MY network. I might be running it for YOU and your business and your users, but it's MY network. That's a weakness, yes, as I get protective over my network, access to it, and what changes are made with it. But it's also what keeps "OUR" network running and safe.

    If I implement a rule (as I have just done) banning USB sticks, then USB sticks are banned. I don't do such things lightly, or for no reason, or because I like to punish the users. I do it to save the school from legislative issues, or network compromise, or some other requirement that are more important than you needing to put in that £2 USB stick you got from some exhibition to transfer your stuff home because you're too lazy to email or work out how to use Google Drive or similar.

    Your sysadmin is protective of your network. It *is* his baby is his eyes. That's a good thing, and a bad thing at the same time, depending on your sysadmin. But if your sysadmin is any good, then let them do that. Let it be their domain, quite literally. Complain when what your business needs isn't present, by all means, but accept that your quick-fix solution is not necessarily the solution the sysadmin needs you to have.

    It's like leaving your house with a house-sitter and then complaining that they fixed the gutters, cleared the drains, set all the clocks to the right time, etc. Let it be their house for a while (if not in law, then at least in practice) if they are going to look after it more by it being so. The worst thing in the IT world is complacency because they're not allowed to fix things properly, so they lose interest in fixing things at all.

    1. Groaning Ninny

      *MY* network? Not really.

      I hope you continue to work well in your job, and that your honesty doesn't cause you to fail in any future job interviews. I'd certainly mark you down if you gave that sort of answer to me. It may well be *my* network, but I am really just custodian of it for the people who matter. The security changes I'm implementing are done with the consent of my users, after informing/consulting as appropriate. Maybe the difference is that I work with people who are prepared to listen, or maybe it's that they've learned that it's worth listening, and that I'll listen to them.

      If my house sitter decided to change all my bulbs for very low energy (dimmer than I had and still want) then I'd complain. If they changed the locks and security on my door to include procedures that are unacceptable then I'd seek recompense, and never ask them back again.

      1. K

        Re: *MY* network? Not really.

        " I am really just custodian of it for the people who matter. The security changes I'm implementing are done with the consent of my users"

        Give that Ninny a prize!

        You've hit the nail on the head, the key here is custodian, guardian and sentinel..

        The company I work for started off with just 3-4 of us, I was the original back-end developer and systems administrator, ultimately sole responsibility for all infrastructure was my responsibility, including all changes and purchases, so being naive, it was my network.

        But as time has rolled on the company has increased to 120 staff, the infrastructure is still my responsibility, but I "matured" and realised it wasn't mine, I simply had oversight of it, and it was even more difficult to reliquish some of that responsibility and trust others with it, but now I have a team of Administrators and sharing this responsibility is the best thing I could have done...

      2. Anonymous Coward
        WTF?

        Re: *MY* network? Not really.

        "I hope you continue to work well in your job, and that your honesty doesn't cause you to fail in any future job interviews. I'd certainly mark you down if you gave that sort of answer to me."

        I would never give that kind of answer but I'm really assured that *every* recommendation I give is writen and all the people in the board are awere of it. If they decided they know better how to manage the IT department, in the end I have something to recall them of it and say "Told you so..." .

        "It may well be *my* network, but I am really just custodian of it for the people who matter. The security changes I'm implementing are done with the consent of my users, after informing/consulting as appropriate."

        Consent?? You dont deal directly with the beancounters do you? They are "IT masters!"...

        Informing? Yes, that's correct.

        1. Cynic_999

          Re: *MY* network? Not really.

          "I would never give that kind of answer but I'm really assured that *every* recommendation I give is writen and all the people in the board are awere of it. If they decided they know better how to manage the IT department, in the end I have something to recall them of it and say "Told you so..." .

          It's really great when your only consideration is for one aspect of the company. You can then afford to adopt such an "I told you so" approach if everything possible is not done to advantage that single part and something goes wrong as a result. If you are running the entire company however, you would soon realise that many compromises have to be made, and some things have to be run at sub-optimum levels so that other things work better. Crossing the road carries risks, so if your only concern was safety you would prohibit people from ever crossing a road, and if they ignore you because and someone got run over while going to the shop to buy your food, you can sit high on your horse and say, "I told you so ...."

          1. Anonymous Coward
            Anonymous Coward

            Re: *MY* network? Not really.

            My recommendations go well over the entire aspects of the business because, well, that's my job. But if I'm asked about the idea of certain people (finantial, HR...) becoming local admins in their windows boxes, for example (just because they need that important software callled iTunes be available), should I simply say yes and aceept the "compromise"? I have responsability over the security and stability of the network and everything related with it. The everything "writen" it's because boards and bosses love to have a scapegoat whenever is needed.

    2. DropBear

      It's like leaving your house with a house-sitter and then complaining that they fixed the gutters, cleared the drains, set all the clocks to the right time, etc.

      No, it's like expressing your displeasure about the house-sitter boarding up the main door considering he has no real need to leave and that the small inconvenience of you not being able to get back in is far outweighed by the greatly enhanced protection against a potential zombie apocalypse...

      1. wolfetone Silver badge

        I always see the systems I look after as children, and I'm the legal guardian of them. People entrusted them to me, expecting me to look after them and protect them. Once they outlive their usefulness - get too old (like we all do) they get thrown out and left to fend for themselves.

        So really, if you look after a network or any sort of system, it is YOUR network and you do with it what you see fit. I admired the Childs story back in the day because he did his job properly to a point. I bet you can count on the one hand how many times that network was compromised compared to other networks where configurations and access were much more freely accessible.

        1. JEDIDIAH
          Linux

          Other professionals have to worry about personal liability for their mistakes.

          The problem in San Francisco is that management beyond Childs was incompetent. They should have had an exit strategy for ANY employee and made proactive steps to make sure they would not be in precisely the sort of position they ended up in.

          Other professions have standards and licensing to the point where they will tell upper management to "go pound sand" because they have a license to look after.

      2. Apdsmith

        While I agree with what you're saying, isn't part of being a sysadmin being mindful of the business requirements - keeping the systems fit for purpose has to include being able to trade, after all.

    3. Anonymous Coward
      Anonymous Coward

      Malicious people are out there

      Yet you find that, aside from laziness or incompetence, actual malicious intent is incredibly, extremely rare; almost non-existent.

      Is this your gut feeling, or is it an evidence-based opinion?

      I worked for a company where the CTO decided he was going to snoop in on my telephone calls. He took two private telephone calls I made to my partner at home and shared them amongst his friends. He then proceeded (along with his friends) to blackmail me out of my job and a 5 figure sum of money. When I left I signed an agreement preventing me from disclosing details in public (for reasons I won't go into but are probably obvious).

      To be clear, reasonable personal use of your work phone was permitted within the company.

      This person is now a CTO in a company that provides telecoms services to other businesses. It's clear to me that if he feels he can profit from listening in on others private conversations, he will do so.

      I've also worked at a company where a sys-admin took great delight in walking around the office with a t-shirt that said "I read your email ;-)". I kid you not!

      So please don't tell me this is extremely rare or almost non-existent.I see no reason why a CTO or sys admin should be considered more trustworthy than any other person - and there's plenty of dishonest people out there.

      1. Anonymous Coward
        Anonymous Coward

        Re: Malicious people are out there

        I worked for a company where the CTO decided he was going to snoop in on my telephone calls. ... shared them amongst his friends ... blackmail me out of my job and a 5 figure sum of money ... I signed an agreement preventing me from disclosing details in public.

        This person is now a CTO in a company that provides telecoms services to other businesses. It's clear to me that if he feels he can profit from listening in on others private conversations, he will do so.

        So because you didn't have the balls to stand up to his criminal acts, he is now in a more powerful position, able to perform more such criminal acts, against even more innocent people.

        I hope you're proud of yourself there, champ.

        1. Anonymous Coward
          Anonymous Coward

          Re: Malicious people are out there

          So because you didn't have the balls to stand up to his criminal acts, he is now in a more powerful position, able to perform more such criminal acts, against even more innocent people.

          I guess you've never been the victim of blackmail.

          I hope you're proud of yourself there, champ.

          Thanks for condemning me without having the slightest understanding of the situation. You seem to be under the impression that I did nothing about his 'criminal' acts. Maybe you should check what the CPS's conditions are about deciding when they will and won't prosecute someone? Many guilty people are able to avoid justice despite the victims doing all they can to try and seek it.

          1. Anonymous Coward
            Anonymous Coward

            Re: Malicious people are out there

            I guess you've never been the victim of blackmail.

            Sounds like you're not the type to employ violence, and the person who did this knew that.

      2. Anonymous Coward
        Anonymous Coward

        Re: Malicious people are out there

        took two private telephone calls I made to my partner at home and shared them amongst his friends. He then proceeded (along with his friends) to blackmail me out of my job

        Blackmail is never acceptable, and I sympathise with you that he was able to get some compromising info on you such that you didn't feel you could report him to the police, but you surely bear some responsibility here?

        Even when reasonable personal use of your work phone was permitted it's likely that there is some small print saying that such calls can be recorded or reviewed. I work for a respectable company, and I still wouldn't use company resources for any communications which would reflect badly on me if they were known. It's a company network, I work on the basis that the company has full access, whether or not they openly say so.

      3. amanfromMars 1 Silver badge

        Malicious people are out there but there be no safe secure place in virtual cyberspace to hide*

        When I left I signed an agreement preventing me from disclosing details in public (for reasons I won't go into but are probably obvious). ... AC

        Ye olde thirty pieces of silver doing their right dodgy crooked thing, AC, or was it the clumsy blunt threat of physical violence doing its crazy thing? They be the obviously probable reasons which are oft paraded and offered to buy an unpleasant silent compromise and continuing sub-prime third party privacy for a proprietary protocol breach.

        * And that really fcuks up abused and abusive sysadmin and there aint no easy solution for resolution of their problems ..... which are all most probably revolving around trying to prevent greater intelligence revealing arrogant ignorant actions and wannabe universal emperor plans. Oh dear, what a shame .... not.

        Plonkers are as plonkers do ..... and the future requires not their leads and leaderships.

        1. Anonymous Coward
          Anonymous Coward

          Re: Malicious people are out there but there be no safe secure place in virtual cyberspace to hide*

          Ye olde thirty pieces of silver doing their right dodgy crooked thing, AC, or was it the clumsy blunt threat of physical violence doing its crazy thing?

          A threat that they would have me arrested and prosecuted. They showed me the evidence that they had fabricated which would be used for their proof of my wrongdoing. There was never a direct physical threat, but I did believe I was at risk of physical violence. I must be one of a very small number of people in history who has signed a compromise agreement (under duress) that gave me less than I was contractually entitled to.

      4. SoaG

        Re: "I read your email ;-)"

        Did he actually read the email? Or was it a simple and effective way to educate users (at least a little bit) about data security?

    4. Anonymous Coward
      Anonymous Coward

      @Lee

      My answer is truthful... it is MY network. I might be running it for YOU and your business and your users, but it's MY network.

      Good luck to you, but I'd be terminating your interview at this point. It's the employers network, not yours. Not mine.

      I own the car, not the mechanic that does the service, repair, or upgrades.

      (This analogy is intended as a slur against neither mechanics, nor admins, as both require enviable skills and experience to do right)

      If I implement a rule (as I have just done) banning USB sticks, then USB sticks are banned. I don't do such things lightly, or for no reason, or because I like to punish the users.

      Try this where I work and you'd be out the door the same day. Don't misunderstand, USB sticks are already banned, but in large corporates decisions on removing a technology require extensive forward planning and are never cost free - you'd make recommendations, but ultimately even the CTO doesn't make all the decisions - the business has priorities for its budget that may not include additional network security and associated redevelopment costs for processes or systems built within the current specification.

      My BOFH decided he wanted to ban internal FTP. His reasoning is sound in my view, however, he's just about to be overruled on that from further up the pay grades as the cost to the business of using SFTP, SCP, DTU et al is simply higher than the budget allows. His recommendations have been taken on board but it has been decided that his preference will be set aside for the time being.

      I realise I'll get downvoted to hell for this, but what can you do. Network admins don't get to determine the toolset or security policy anymore than the doorman downstairs gets to determine the corporate dress code. Staff are staff: they ALL dance to the corporate tune. Anyone wanting to dance to their own beat is free to start their own company and then it really will be their network.

      1. zen1

        @ AC: Re: @Lee

        ..."My BOFH decided he wanted to ban internal FTP...."

        I guess I'm lucky because both Security and myself decide on the policies that will be implemented on our network. As for the "It's my network" argument. I view it as a stewardship. It's ours to care for while we're there, but it never belongs to us. Quite honestly, if it were my network and I had a fraction of their money, I would have purchased the RIGHT equipment the first time, before something blows up and causing a system wide domino effect type of outage. But that's not the case so I will manage and care for the equipment I'm responsible for, the best I can.

        I am responsible for the firewalls, the content filters and proxies for my employer. They have the expectations that the equipment will as close to every time, all the time, as possible. Since my performance review is on the line, plus pride in the level and quality of services I provide, I take it damn personally when a piece of equipment fails because of hardware or software problems.So much so that I look like a doting parent taking care of deranged child... That has a few extra arms, heads kidneys and maybe a multiple personality or two...

    5. Cynic_999

      I see what you are saying, but to use your own analogy, would you really be happy with a babysitter who painted your living-room purple and replaced all the meat in your refrigerator with vegetables and nuts because in her opinion purple was a far better colour than magnolia, and eating meat was unhealthy or unethical?

      Sure, you can recommend certain things to your boss and explain why you think they are necessary, but at the end of the day if the boss says that he wants people to be able to transfer data via USB sticks, you will have to actively facilitate it.

      1. JEDIDIAH
        Mushroom

        Simply astounding.

        The stupidity of comparing a trained network administrator with a nanny is just mind bogglingly stupid.

        That analogy only (maybe) works if the nanny also is the legal guardian of the parents as well. The IT staff may well claim ownership of the IT infastructure because they are the only ones that know how it works and no one else is even capable of supervising them.

        Nothing like a "nanny".

        More like the nursing home that will wipe your butt when you are old and senile and no longer capable of taking care of yourself.

  3. Anonymous Coward
    Anonymous Coward

    Joe

    Can we get Joe to take down those bastards calling from "Microsoft" and delete ALL the numbers from their database.

    Let's shoot for the moon: If they have computerised door locks, can we lock everyone out of the building?

    Maybe even cause an explosion, a la Skyfall.

    (Note for the NSA/GCHQ/whoever: That was a joke. I'm making fun of what Hollywood thinks computers and hackers can do.)

    1. DropBear

      Re: Joe

      It wouldn't work. As any fule kno, any true hacker brought in to help could reverse any and all that in thirty seconds, tops...

      1. Cynic_999

        Re: Joe

        "It wouldn't work. As any fule kno, any true hacker brought in to help could reverse any and all that in thirty seconds, tops..." ... by furiously typing in reams of machine code interspersed with cryptic Unix commands to patch running programs as they read and understand megabytes of raw hex values cascading down 15 different screens at the rate of 100's of lines per second ...

        Hollywood has taught me that most code is green, but malware always executes in red.

    2. Cpt Blue Bear

      Re: Joe

      "Let's shoot for the moon: If they have computerised door locks, can we lock everyone out of the building?"

      Not out. In...

  4. Ben Liddicott

    You can't defend against your bodyguards...

    ...as Mrs Ghandi learned.

    Your only option is to pick trustworthy guards... and be the sort of person they are willing to be loyal to.

  5. Anonymous Coward
    Anonymous Coward

    Anyone will compromise anything given the right incentives. For some the incentive is gain, money, cars, holidays, women, whatever. For others it is loss, threatening their family, privacy (everyone has secrets), or some other such angle. Fear and greed, and it will be forever thus.

    While I’m not an admin, I’m perfectly capable of extracting much confidential data from my employer; The keys to the kingdom, as it were. My principles aren’t for sale for any amount of money, but I prioritise the health and safety of my family above any career, employer, or indeed anyone else that isn’t my family. Would I “compromise critical life support systems” to protect my family from harm? Absolutely, and I rather suspect you’d all do the same.

    1. Anonymous Coward
      Anonymous Coward

      Old phrase....

      You'd kill for your wife but die for your kids....

    2. Anonymous Coward
      Anonymous Coward

      Re: anyone/anything - I rather suspect you’d all do the same.

      I'm not sure what I'd do, and I hope I never get given an ugly choice of the type(s) mooted here. But it's worth noting that some people would refuse and do refuse under extreme bribery or coercion. I mean, what about those people that Amnesty International campaign for? They have typically paid various prices for not being quiet or not staying in line.

      Why is it that Reg commentards should be so uniquely lacking *as compared to* the rest of the human race?

      Would I “compromise critical life support systems” to protect my family from harm? Well now - just how many tens, hundreds, thousands, (etc) of deaths of other people should my family be worth? I think if I want to test (however hypothetically) my behaviour under coercion, I need something a little more specific to be going on with. Are we talking about an 10% increase in the failure chance of some really old sick guy's respirator, or 10% of all airliners suddenly dropping out of the sky?

      1. Anonymous Coward
        Anonymous Coward

        Re: anyone/anything - I rather suspect you’d all do the same. @AC

        Would I “compromise critical life support systems” to protect my family from harm? Well now - just how many tens, hundreds, thousands, (etc) of deaths of other people should my family be worth?

        How many deaths are my family worth? To me? It would be worth all of them. And I'm not sorry if that offends anyone. We could be talking about one death, one thousand, one million, or everyone else. It would be worth all of them to me.

        Thankfully, it's a purely hypothetical situation, with the sole purpose of realising that once you can be made to do something in extreme circumstances, there are many less severe variations in which you can be made to do something less severe. Everyone can be bribed or coerced to act against their principles or preferences - perhaps not for direct personal gain, but there's always a way.

        Segmenting admin power, particularly in larger companies where more than one admin exists, can reduce your "administrator risk" more effectively than just about anything else you can do. One account does not need to rule them all.

  6. chivo243 Silver badge
    Headmaster

    Do this

    Open your directory service, what ever platform it runs on, AD, OD or whatever is in place. Have a look at the names, and how many are no longer employed? I am regularly asking HR if so and so is still employed? NO? Why weren't we notified?

    1. Chris King

      Exit Procedures and the "V'Ger Rule"

      I've ended up writing "Exit Procedures" for previous employers just because they didn't consider this.. and invariably used myself as the first example.

      No matter what the reason for my departure **, I like to make sure that my now ex-employer can't claim I left back-doors into systems. and that everything will continue to run without the presence of my user account. If I'm gone, I'm TOTALLY gone, end of story.

      (** I've never been sacked from an IT gig, only from a summer job as a waiter. One of my "mates" phoned up the restaurant and posed as an angry customer who claimed that I'd told him to f**k off)

      As I said on the Internet Storm Center a few years back...

      When I design or build a system, I make absolutely sure that it's designed to what I call the "V'Ger Rule". If you've seen "Star Trek: The Motion Picture", you'll understand.

      Put simply, the "V'Ger Rule" states:

      "A System must continue to operate in a correct and safe manner in the absence of its Creator".

      Or, put another way:

      1. No blowing up any spaceships ;

      2. No joyriding in Carbon Units ;

      3. Fat, balding starship captains are to be shot on sight, especially ones that follow the "If you can't eat it, drink it, steal it, spend it or have sex with it, blow it up" mantra.

      1. Anonymous Coward
        Anonymous Coward

        Re: Exit Procedures and the "V'Ger Rule"

        One of my "mates" phoned up the restaurant and posed as an angry customer who claimed that I'd told him to f**k off

        Ah, this is obviously some strange usage of the word 'mate' that I wasn't previously aware of.

        1. Jes.e

          Re: Exit Procedures and the "V'Ger Rule"

          "Ah, this is obviously some strange usage of the word 'mate' that I wasn't previously aware of."

          Downvotes to a HHGTTG reference.. here!?!

          What is wrong with you people???

    2. Blake Davis

      Re: Do this

      Payroll sends the IT dept a list monthly of terminated employees just for this situation. HR occasionally forgets to let us know about a termination, but they never forget to stop the pay checks!

      1. Anonymous Coward
        Anonymous Coward

        Re: Do this

        What about automatically disabling the account if it's not used for 30 days and then doing an automatic cleanup after 90 days? Zeeze...

        1. Peter2 Silver badge

          Re: Do this

          >"What about automatically disabling the account if it's not used for 30 days and then doing an automatic cleanup after 90 days? Zeeze..."

          I take it that you haven't heard of "pregnancy" or the common employment terms of "maternity leave", "long term sickness", "suspension" (ie; garden leave) and the army of related issues where simply deleting an account because a user hasn't used it for 3 months can cause the company serious problems?

          "No, your honour, and esteemed members of the jury. We had fully intended to allow employee X to resume their duties after their time away from the workplace. The person standing in for them was only working on a temporary basis and we hadn't already made a decision to dismiss employee X and replace them with this temp..., no the fact that employee X's computer account had been deleted is a total coincidence and this entire court case is a terrible misunderstanding! No, your honour, we don't think you look stupid and we aren't trying to insult your intelligence..."

          Motto of the story. IT has one job- HR has another.

          1. JEDIDIAH
            Thumb Down

            Re: Do this

            > I take it that you haven't heard of "pregnancy"

            Revoked access can always be granted again. It's not like this is a one way process. You are only removing them from some database. You aren't actually killing them.

            The worst that can happen is that you inconvenience someone that's been away from the office from a long time.

  7. Anonymous Coward
    Anonymous Coward

    Fine line

    There is a fine line to walk. Most people waver from one side to the other at times, but most try to walk along it.

    To me, the network is mine, in as much as it is my responsibility to keep it running well. This means I will make "executive" decisions at times which users dislike, but are for their own good. I lock things down, refuse to let them have software XYZ, etc. I will normally provide an alternative, or at least a good reason, and that is always open to debate with me and my manager.

    The problem I currently have is that the network is not mine. My boss is a techy person himself, and he makes the decisions. I am just the most recent "child minder". As an example, when I started there was no AV solution in place. I was repeatedly told "we've never had a virus in the X years we have been running, so we obviously don't need an AV". Which was fine, until we got a virus.

    We now have a system in place, but I'm still ignored on other fronts, like security (which is practically non-existent, everyone uses the same password, even on external sites, "so we can access other people's accounts if they are off"). They do not change, even when members of staff move on.

    For me, I keep plugging away in the vain hope that I will be listened to some day. I also ensure that these conversations are logged in my email, to cover my back. In the end, though, I will be the one who has to clear up the mess when something goes wrong...

  8. Conrad Longmore

    One trick I heard of..

    One trick I heard of (and I cannot remember where I heard it, it may be apocryphal) was that a large organisation wanted to fire a sysadmin, but they needed a few hours to make sure that all the passwords could be changed and accounts disabled.

    So, they made up an excuse to get the employee on a LONG flight to another location (I think this was in the US) where they would be completely out of contact with everything and everyone. When they got to the other end, they were met by management and HR and then terminated.

    I don't know if this story is even true, but it does demonstrate the lengths you might have to go to if you need to fire a potentially rogue sysadmin. Alternatively giving them a large pile of cash on a smooth transition might also work..

    1. Paul Crawford Silver badge

      Re: One trick I heard of..

      They were kind. The alternative punishment/time-waste is to send them to a meeting to suffer hours of "death by powerpoint"!

      But seriously, the problem in some cases is they only have one admin, or only one that every looks after XYZ systems, so on antagonistic exit (or a bus accident, etc) they find they can't do anything due to a lack of passwords or alternative admin accounts.

      Businesses, particularly those with only one admin person, should have a policy of root passwords being written down and kept in a safe and regularly tested to ensure they still allow access, and that password changes are recorded and done for good reason[1].

      [1] Changing periodically to me is dumb, it just promotes writing stuff down in insure places. For example, changing once per year would give a hacker a mean time of 6 months to do stuff. just how long do you need to set up shadow accounts, email redirects, etc?

      However, if you think a compromise might have occurred, or someone leaves, then changing is essential.

      1. Anonymous Coward
        Anonymous Coward

        Re: One trick I heard of..

        I recall leaving one company for a competitor and the company I left changed their primary system root password. Fair enough.

        I was then asked to go back in to the old company to help them out of a bind (before starting at the new company however) and had to be given the password again.

        It was something along the lines of

        <xxx>Tr41T0R

        Where xxx represented three letters of the company I was moving to. Nice! :)

      2. Number6

        Re: One trick I heard of..

        Businesses, particularly those with only one admin person, should have a policy of root passwords being written down and kept in a safe and regularly tested to ensure they still allow access, and that password changes are recorded and done for good reason[1].

        The one I've seen for small companies is for the critical passwords to be written down, sealed in an envelope with a couple of signatures (sysadmin and manager) across the seal. If any sort of access is needed in the absence of the sysadmin then it can be done, but then the passwords need to be changed and a new envelope created. It's more a way of ensuring access if the sysadmin wants to check out the underside of a bus, or similar, but doesn't protect against a malicious sysadmin.

        1. Pascal Monett Silver badge

          Sealed envelopes are impressive and all that, to be sure, but I doubt they'll be of any use against a rogue BOFH that went and changed the passwords without creating a new envelope.

          Said envelopes are only an insurance if the passwords are regularly checked and validated.

          And even that doesn't guard against an additional shadow account created with the same credentials and abilities, but using a different password that only the BOFH knows.

          Not unless there is a log of some sort that the sysadmin cannot touch that records all instances of password change.

          I'll believe in it when you show me an example of something that runs under sysadmin supervision that the sysadmin cannot touch or prevent its functioning.

    2. Number6

      Re: One trick I heard of..

      It wouldn't work now unless you made sure to pick a flight without on-board wifi...

    3. JEDIDIAH
      Linux

      Re: One trick I heard of..

      If I were terminated and then stranded, I wouldn't have to ever work anymore. Whatever company that pulled that stunt on me would be paying for my retirement.

  9. Yugguy

    IDIOTS

    "what's 5 years in the big house"

    Proof positive the average IT nerd knows eff all about the real world and the kind of people they are likely to meet in prison.

    1. Anonymous Coward
      Anonymous Coward

      Re: IDIOTS

      Proof positive the average IT nerd knows eff all about the real world and the kind of people they are likely to meet in prison.

      Mostly grandmothers who haven't paid their TV/Council tax these days, isn't it?

      5 years in a US jail? No ta. 5 years in a UK country club? For £20M? If I didn't have a family to support, then I'd swap 5 years free time for that.

      You just find the meanest, toughest guy in there and pay him £1M to guarantee nobody touches you. Then you go to the library, get some books, and catch up on some reading. Maybe do what everyone else does and hit the gym or do a law degree?

      1. Yugguy

        Re: IDIOTS

        Then the crown finds that 20m, freezes it, and you spend the 5 years playing mummy and daddy with said large gentleman.

  10. Suricou Raven

    So in short, you've four motivations to look out for.

    1. Wealth/bribery. How much would it cost for the admin to sacrifice their career and self-respect?

    2. Coercion: Especially important in national security - I wouldn't put it past quite a few countries to hold children hostage to get passwords off their father.

    3. Principles: Would they sacrifice their career or even their freedom for what they believe to be the greater good, Snowden style?

    4. Resentful stupidity: The company just screwed them over on health insurance, but the CEO has a private jet. Time for some self-destructive vengence.

    1. Paul Crawford Silver badge

      Re: So in short, you've four motivations to look out for.

      Everyone has their price, its just a shame its so low in some cases.

      Really you need to plan for people making mistakes or doing the wrong thing, and have arrangements to detect and correct that as far as possible. Often that costs money or causes inconvenience though so its not done...

    2. Yugguy

      Re: So in short, you've four motivations to look out for.

      I don't work any more in a Government secure data arena so I'm not too worried about coercion.

      But out of those four only my family's safety would have made me break trust.

    3. Anonymous Vulture

      Re: So in short, you've four motivations to look out for.

      The four methods of compromise are actually represented by the acronym MICE - Money, Ideology, Conscience and Ego. Same methods used in every conflict throughout human history to compromise an individual and entice, persuade, or force them to do your bidding.

      Money - Clear enough. Pay them what they want and they do what you want.

      Ideology - They think that you believe what they believe and will work towards a mutual goal as a result.

      Conscience - You can either persuade them that what you are asking them to do is morally right, or you can appeal to their conscience more directly by threatening loved ones.

      Ego - Oddly enough, this may be the underlying weakness in most IT personnel. Do something just to prove you can.

  11. system11

    Can I hire Joe to get rid of these PPI refund plague calling systems?

    1. Anonymous Coward
      Anonymous Coward

      > Can I hire Joe to get rid of these PPI refund plague calling systems?

      It is cheaper to get blacklisted by the cold callers. Always press 5. Always tell the salesperson you are interested. Always ask if he will wait a minute while you turn off the vegetables which are boiling over. Never, ever, put down the receiver. Even just pressing 5 when you hear a recorded message will achieve redundancy for the salesperson who takes over the call, in the end. Call centre operatives can not afford to waste their time on anything like the scale that they waste yours. Their business model relies on the honesty of those who show no interest. Don't help them to meet their call targets.

  12. TheWeddingPhotographer

    There is a conundrum here

    The issue is that employers who employ System Admins often don't know best practice or worst practice. They often don't know the first thing about networking.

    So what happens - the keys are handed over to the admin, and then the admin generally is the one setting (or driving the set-up of any rules / policies)

    In a large team, this is usually mitigated, as said admin is working with a team of peers, and has a supervisor who knows what's what

    In anything other than a large team, the admin has a huge amount of responsibility that generally is not policed. it is that sort of admin that poses the most risk

  13. Anonymous Coward
    Anonymous Coward

    Frustration

    I was taken on to assist their onsite IT admin (not a particularly large company). All was fine for a few months, I ironed out the niggles the staff had yet the original IT guy didn't bother with. When they looked at getting a new server, I suggested a couple but they went with his choice. ok. no problem.

    As soon as the system was up and running I heard the night before that I would be terminated in the morning. I'm not malicious by nature, but as far as I was concerned, this was taking the p*ss. So all I did was change the password on the encypted backups so they would carry on as before, but god help them if they do a test restore (or worse).

    I do recall thinking I should put all the key services under my own account and then that account gets changes the services would fail, but I thought that was a little too much.

    Only saving grace was my informant told me that the IT guy was getting a £2k / month pay cut

    1. Anonymous Coward
      Anonymous Coward

      Re: Frustration

      My very first IT job was in support for a small firm and did a sterling job. I was on a 3 month probation period, at the end of which I would be paid £11k...cool :)

      After the three months were up they decided to offer me £6.5k - which I wasn't very happy about. One of the managers I tried to get to admit that they had offered me £11k denied even being in the meeting!

      During my time there I had

      - rebuilt their network - removing a lot of bottlenecks

      - rebuilt their Lotus Office server (about 6 times trying many different permutations) until I stumbled upon the correct sequence and fixed it - saving the company several thousand pounds of consultants fees

      - built a BBS (this was just before the internet really took off by the way) in 4 different languages (using Wildcat) and depending on which language they chose they got a background flag of the associated country.

      They really stiffed me, so I decided to go elsewhere. Then, for no reason I can fathom, one of the managers threatened me by saying that he played golf with someone high up in the company I was going to and if I didn't toe the line until I'd worked off my notice that he would put the boot in.

      Well, I was royally pissed off by that, so after I left (I was still the last one out of the office as I had all the keys and codes!!) I decided to make all four BBS lines make outbound calls to foreign climes and set things up so that they didn't time out. I have no idea what their phone bill would have been by the time they figured it out, but it couldn't have been less than a couple of £k.

      On top of that, a week later their office was burgled and ransacked - which I had NOTHING to do with. The odd thing is that even though I had recently left, was obviously disgruntled and had previously held all the keys to the office and the codes (which I don't think they changed) I still never got a visit from the Bill. Very odd that was.

      1. Anonymous Coward
        Anonymous Coward

        Re: Frustration

        Disgusting.

  14. Anonymous Coward
    Anonymous Coward

    Who will save the systems from the men and women who save the systems from you?

    We're called Backup Administrators. As in, we're the Admin's for your network capable Backup Software.

    And yes, we have unlogged root on all of your servers already (but you won't be able to tell, even if you look hard at the audit logs).

  15. Anonymous Coward
    Anonymous Coward

    A lot of the issues could probably be dealt with by treating IT staff like human beings who are an essential part of the organisations past present and future well being as opposed to a "resource" that costs too much money and doesn't play well with others.

    1. Anonymous Coward
      Anonymous Coward

      Yes I have seen this, a company I know has employed a guy who was really good, he built the network sorted out all the IT systems and telephony, the company paid him a pittance and treated him like he was 24hr support so all of their techs out on jobs could call any hour.

      Guy gets a new job thats the point they offer him whatever they want, including a very nice car. he's now working for another company who are providing him with all the training he wants (they are a big Cisco customer), a decent wage and he works normal hours.

      Meanwhile former company are left with lots of stuff set up by a guy who knew his onions but didn't document anything (he probably didn't get the time), and now they are hiring cheap as possible again, is that network having trouble? Yep.

      1. ecofeco Silver badge

        I've seen this as well.

  16. Stevie

    Bah!

    It has been my experience in over 35 years in the IT business that the "lower" one gets into the infrastructure support - the closer to the metal - the more overblown the perception of personal importance to the enterprise is, with a corresponding elevation of the Dick Factor.

    The observation that talent in IT seems to run hand-in-waldo with dysfunctional social skills, elevated sense of worth and a relaxed stance on ethics based on I-know-best is no surprise to anyone I would imagine, but is a cause for concern to anyone putting their business in such people's care.

    I'm just racked-off that I get painted with the same brush as the San Fransisco Shirthead when it comes to expectations regarding personal ethics.

    1. ecofeco Silver badge

      Re: Bah!

      Got my upvote, Stevie. It's not just sysadmins. I've seen plenty of this kind of behavior from tier one tech support to all the way up the line and deliberate sabotage of teammates and the business itself and withholding of needed resources just so they could retain absolute control.

      But that's nothing new in most businesses, is it?

  17. Erik4872

    I've experienced a Terry Childs like incident

    I'm not surprised that systems people are looked on with suspicion. I've seen a pretty even split between normal people doing their best to maintain order and be nice, and BOFH-level personality issues. Look at how many people came out to defend Terry Childs' actions. The facts of the case are that he basically used his power and a lack of oversight to insert himself as the single point of failure in the network, keeping all the config files stored in memory, with the only backups being in his possession. There's no way to defend that.

    At a previous very large employer that grew from a startup, we had Network Guy. NG had built the entire infrastructure from the ground up, and had been allowed to run it unchallenged for years.The only problem is that nothing was documented, and the only people he allowed access to the equipment were 2 employees who followed his orders. He also had a personality that could be best described as "acerbic" and was highly possessive of "his" network...typical ThinkGeek T-shirt guy. When Startup became Big Company over a 5 year period, the CIO rightfully started worrying about what would happen if NG quit, was fired, or was incapacitated. When the CIO brought someone in to work with him and document everything, NG's response was simply, "No, I will not be doing that." It took firing him, getting the minions back under control, hiring a couple of consultants and very carefully probing every corner of the network to get things back under control. So yes, control freaks who get system admin jobs can really be a problem. Being a *good* control freak is a great thing, but being the SPOF is not.

  18. Ken Hagan Gold badge

    "a formal HR process that is tested"

    So have a few "practice sackings" then, repeated every so often and with role reversals so that everyone can play both sides? Sounds like it could be fun.

  19. Ken Hagan Gold badge

    Everyone has a price?

    I assume that *all* sysadmins are familiar with this dictum, but let's think about it. What should that price be? Obviously you won't work again, so the cash needs to be equal to the rest-of-life earnings. Obviously it needs to be paid in such a way that the authorities can't stop you enjoying it. Obviously, it also needs to include some compensation for the lost years in jail.

    So those three conditions together are the break even point in your cost/benefit analysis. And someone was willing to say in public that they'd jump for it? Huh! I'd sack them just for being so stupidly cheap.

  20. Anonymous Coward
    Anonymous Coward

    Haven't seen a company with such procedures in place

    I have worked in both private and public sector and have yet to see a company that has even a hint of a procedure as outlined in the article. At best the root password gets changed a few months later. However personal accounts do get disabled fast, this won't deter the initial urge to get even of course.

    In this day and age where sysadmins are often an afterthought after developers drown in the mismanagement of their shitpile of server it's a utopia to think such procedures would ever be common practice.

    One just has to accept sysadmins have the potential to do harm. But, shock horror *gasp*, if you treat your employee with respect and like a human being they would not feel a need to "get even". It may come as a surprise to some, sysadmins are humans too and they have a conscious. O:-)

    And you can never defend yourself against the extremely unlikely event someone is "bought". Shit happens.

  21. Anonymous Coward
    Anonymous Coward

    I left an employer who liked to hire anything with boobs

    and whose selections usually had no personal or professional attributes suited to the job.

    So I created a nice storm by designing everything "properly". Interface programs were written in C using an API, rather than pumping data direct into the database, scripts required some actual knowledge of programming techniques rather than a 5 day block-course, documentation was written using proper technical language, etc. Apparently when I left it took them 2 years to redesign everything rather than hire someone who knew what they were doing to maintain it. Small victory, but at least plausibly deniable, in that the solutions worked and were maintainable by a competent professional.

    1. Anonymous Coward
      Anonymous Coward

      Re: I left an employer who liked to hire anything with boobs

      Pfft, I wouldn't have left!

  22. Anonymous Coward
    Anonymous Coward

    Morality vs work ethos

    The article briefly mentions Manning and Snowden, but then delves into the monetary price people would attach to their confidentiality. In the discussion there's a long thread about somebody who became the victim of a serious crime (blackmail) from a sysadmin abusing his power, and another about what people would do to protect their own family. Fair enough, but back to those two guys: if I was asked to define my own integrity, loyalty to my employer might rank high, but it can't be on the top. There's the law, and above that is what I believe to be right. And if, in my official capacity, I should find out about things that are illegal or unethical (the latter being the more overriding principle), loyalty is trumped.

    AC for obvious reasons...

  23. roger stillick
    Holmes

    Monitary Price on Confidentiality ??

    The article mentions money as a breaker of classified information, exactly the same as claiming money as a breaker of sexual morals, it is wrongly assumed that everyone has their price, some don't...

    Reality is folks getting cleared to work in those sensitive areas neither sell their info or their bodies for any price or favor, and if the truith were actually known= they would willingly rat out anyone being a jerk to the Company's Chief Special Agent...every company larger than a simple partnership has someone responsible for security...let it be their problem, it's what they get paid to do...RS.

  24. This post has been deleted by its author

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like