back to article Help, my IT team has no admin access to their own systems

You know that plumber who charges hundreds to just to change a magic washer? The IT world can be the same, where seemingly magical skills are grounded in the most mundane of realities. Welcome to Contractor Time in On Call. Today's story, from a reader Regomised as "Dan", takes us back to the end of the first decade of the …

  1. A Non e-mouse Silver badge
    Facepalm

    I worked in a small company that worked closely with other similar, local companies. We all had independent IT. One day I got an email from one of the other IT teams asking me to reset their admin password as they'd lost it. I replied saying I couldn't as I didn't have access to their system. It took a bit of persuading that I couldn't help. They then asked how they could reset the admin password. I told them it probably wouldn't be a quick and painless process (just as it shouldn't be)

    The admin password they'd lost was the global admin account for their Office 365 tenancy.

    1. UCAP Silver badge

      I used to be the IT Manager for my company many years ago. I insisted that all admin passwords had to be printed off, placed in a sealed envelope and securely stored. Got taken out for a beer a couple of years afterwards by my successor - that strategy had saved his bacon when someone responsible for managing a critical system left the company without doing a proper hand-over.

      1. Phil O'Sophical Silver badge

        Always a sound strategy, but sadly screwed up these days by IT forcing everyone, even admin accounts, to change passwords every month or two.

        1. ChrisElvidge

          Passwords

          Is it really IT that insist on password changes? Or is it some c-type who has heard that passwords should be changed regularly for "security" and has conviced some higher-up to issue an edict?

          1. John Brown (no body) Silver badge

            Re: Passwords

            IIRC, it's been shown that people will use the most easily remembered/guessed password they can get away with within the confines of the defined password policy when said policy forced frequent changes.

            On the other hand, people will choose a more complex, less easily guessed password if it's significantly longer but only requires a change every 6-12 months and is, in turn actually more secure.

            Irhmbawhwrny1666 is probably more secure than Pa55wo0d26!

            (FWIW, I Really Hate My Boss And Wish He Would Retire Next Year 1666 :-)

            The initial letters of a long but memorable and personal phrase is easier to handle than a short complex, random sequence and more secure.

            Disclaimer: IANAsecurity professional and may be talking bollox.

            1. redpawn

              Re: Passwords

              Offline NT was a tool in my tool box. Teachers and admin were always forgetting their passwords

              1. Jou (Mxyzptlk) Silver badge

                Re: Passwords

                Olline-NT does not work for domain controllers. But there is a way around that too... One for NT up to Server 2003 (or 2008 ?) and one from 2008 to now. Needs physical access - or in case of VM: "Physical" access. Been there. Done that several times.

                1. AlbertH

                  Re: Passwords

                  Ophcrack always get me into Windoze boxes with "forgotten" (or more usually corrupted) passwords. I've built a couple of modified versions for specific environment uses... The best use I've found of "Slax"!

            2. TeeCee Gold badge
              Flame

              Re: Passwords

              Golden Fuckwit award goes to Crapita (surprise) and their local government payment portal. I'll leave out just how utterly bloody uselessly nonsensical the design is (worth seeing it, just to see that much WTFness in one place) and concentrate on the login.

              Requires....12 characters(!), at least one capital, at least one number and (drumroll) at least one special character. Short of actually putting on the password screen; "Please make sure you write this down on a post-it note and stick it to your screen", I'm not sure what else they could have done.

              Yes, I have just been trying to pay some council tax via this Edifice of Shite(tm). Why do you ask?

              God alone knows why the Public Sector continues to hire this bunch of incompetent, idiotic tossers. I guess they must be really good at disguising bribes and giving blowjobs.

              1. Ace2 Bronze badge

                Re: Passwords

                Last time logged in to my account at the Social Security Administration (USA gov’t old age pensions), they did that 2FA thing where they text you a security code. It was an *eight-digit* code. Maybe they figure six digits is too easy to guess in ten minutes?

                1. ShadowSystems Silver badge

                  At Ace2, It gets worse...

                  I have a FeaturePhone, not a SmartPhone. When I tell a site to send me a text message with my 2FA code, I expect it to send a text message with a plain text code to let me log in. Instead some of them insist on sending a theoreticly clickable hyperlink that I'm supposed to use to complete the 2FA step.

                  Can you spot the problem? =-/

                2. ShadowDragon8685

                  Re: Passwords

                  Wow, that's... Wow.

                  Even positing spacefuture science-fiction alien computers in orbit attempting to brute force it, and even positing that their systems AREN'T rigged to lock out and re-issue the second-factor code after two or three failed tries, at that point I think the processing power of THEIR systems is going to bottleneck a brute-force attempt.

              2. Anonymous Coward
                Anonymous Coward

                Re: Passwords

                Pro-tip: Make your users use only unicode characters outside the first page. It'll be 256x more secure.

                Bonus pro-tip: Make the password a prime number of characters long. Everyone else uses 6,8,12 etc

              3. Anonymous Coward
                Anonymous Coward

                Re: Passwords

                I reckon they must have been responsible for the payroll system at a local authority for which I once worked.

                First brainfart in the design (um, that kind of assumes a brain was involved, which is debatable) was that, being payroll, most of us only logged in once a month to download our payslip but the password expired after 30 days.

                The other, more concerning, thing was that, not only did it not allow any of the previous n passwords to be reused (which is easy to defeat with an incrementing number) but it wouldn't allow a new password to be too similar to any of the previous n ones, which suggested that it was probably either storing them in plain text or using reversible encryption (which is, let's face it, basically the same thing) so it could check to see if a chunk of the password longer than x characters had been reused.

                My solution to both: use the month and year plus a couple of extra characters to make it long enough with a memorable sequence of pressing shift or not to introduce the required upper-case and non-alpha characters.

                1. AlexG_UK

                  Re: Passwords

                  Ahh yeah .. I used to work for a small UK software house which was taken over by a slightly larger US software maker who suffered from delusions of grandeur (or possibly even delusions of adequacy).

                  Anyway our new us lords and masters introduced an upgraded security led password policy - a creative mix of numbers, letters, upcase but no 'special' characters needed, changed every month, can't be too similar to any of your previous 14 passwords. Hmm just over a years worth of passwords.

                  So what was to be done? Well, simples really: January2012, February2012, March2012, ... Janvier2013, Fevrier2013, .. Januar2014 ... And then cycle started again.

            3. Paul Hovnanian Silver badge

              Re: Passwords

              Tpwisttsotmo1pi!

              The password is stuck to the side of the monitor on one post-it!

            4. JBowler

              Re: Passwords

              Indeed; good user passwords are STO, bad user passwords aren't.

              A good password is a DVD: dd if=/dev/random of=/dev/dvd bs=1048576 count=4096

            5. Kayakerdude
              IT Angle

              Re: Passwords

              More interestingly, the actual phrase as written out and typed out is far more secure from cracking attempts, as it's much longer and has a huge amount more entropy than a shorter set of letters.

              If you have keepass installed you can check the amount of entropy bits used when you type the password into the appropriate field when adding a new password entry.

              The generally-accepted "best" password generation idea for mortals to use, is to take four words independent of each other. XKCD has a good pointer on this if you google for "correct horse battery staple"

          2. Anonymous Coward
            Anonymous Coward

            Re: Passwords

            Cybersec manager here

            I actually requested and had written into policy that our passwords can be 1 change per year.

            IT refuse to do it and we're still doing 30 day resets with reminders 2 weeks prior..so you only get about 2 weeks before "you're password will expire.." starts to appear.

            I've done an audit of passwords, they are all trash and those who use multiple domains have them set the same hash on each one - so likely have the same password over multiple accounts as a result.

        2. Anonymous Coward
          Anonymous Coward

          It’s not “IT”” that force that on anyone (not unless they’re clueless or read too much Gartner) - it’s clueless IT auditors. I’ve had plenty on run-ins with the likes of PWC claiming that we’re not enforcing best practice by setting a 30 day maximum password age.

          After having to argue it for so long, it’s great that NCSC now have explicit advice on this: https://www.ncsc.gov.uk/blog-post/problems-forcing-regular-password-expiry

          1. Terry 6 Silver badge

            Best example for this is probably primary schools.

            After several weeks of the premises being closed staff arrive in, desperate to log in to the often limited number of staff PCs and read the essential(?) staff emails, get their planning printed off, the materials they need and so forth. Often with very little time for everyone to do this before having to troop in to some damn fool compulsory training in the Latest Best Thing (tm).

            But half of them will have forgotten their passwords, due to the complexity requirements. Even if they remember them, because it's on a post-it note on their desk or they're allowed to choose their own (simple) ones*, all the passwords will have expired. So, at best, they are forced to set a new one after logging in- and under pressure it's not going to be a well thought out one. At worst, there's some IT guy in the Local Authority or the sub-contractor who will suddenly have dozens ( or hundreds) of frantic calls as soon as they open their phone lines (in my experience, up to 90 minutes after the teachers have tried to start work). Or there will be an Authorised Person in the school who can set up new passwords. But no one has allocated them the time to change 15 or 20 passwords, so at best it's going to be a rush job with everyone set to the same p/w pre-emptively..

            *Often they learn quickly to always use the same p/w with an incremented digit.(Password1, Password2, etc).

            1. herman Silver badge

              Working at a foreign defence company, I was mildly amused when I eventually reached P@$$w0rd256!

          2. Turn It Off And On...

            Age of IT Auditors

            IT Auditors - get younger ever year - the last two (independent auditors) who turned up at my last place of employment were about 13 years old...

          3. Anonymous Coward
            Anonymous Coward

            or NIST https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-63b.pdf

            which says amongst other useful things

            "Verifiers SHOULD NOT impose other composition rules (e.g., requiring mixtures of different character types or prohibiting consecutively repeated characters) for memorized secrets. Verifiers SHOULD NOT require memorized secrets to be changed arbitrarily (e.g., periodically). However, verifiers SHALL force a change if there is evidence of compromise of the authenticator."

            and "Biometric characteristics do not constitute secrets" (aka faceid is dumb way to protect things you value)

            and "security questions" are the dumbest thing of all, please everyone - stop implementing them.

      2. Anonymous Coward
        Anonymous Coward

        I had done the same thing - it was a small defence contractor shop and I always insisted that the admin passwords where put in an envelope, signed over the join with tape over the top and then inside another envelope with the top-bosses signature on the join. then placed into the bosses fire safe.

        Ah those were the days - removing all hard disks every night and putting them into the various fire safes, clear desks, and two networks: Secure and Internet with everyone having 2 PCs with KVM switches to move between the two. Mind you trying to authorise various MS licences offline (using a telephone) took forever. Always wondered how these phone-home software packages will work on a secure network (i.e. disconnected from the Internet) nowadays.

        1. diguz

          "Always wondered how these phone-home software packages will work on a secure network (i.e. disconnected from the Internet) nowadays."

          really simple: DON'T use products that phone home. They are insecure by design if they phone home. I never saw libreoffice complain about being on an air-gapped machine.

          1. NoneSuch Silver badge
            Thumb Up

            "really simple: DON'T use products that phone home. They are insecure by design if they phone home. I never saw libreoffice complain about being on an air-gapped machine."

            Start by using Linux and go from there. That runs Lire Office nicely.

            1. plrndl
              Pint

              Ultimately, all usable IT products are "insecure by design". The trade-off between security and usability is one of the biggest problems our business faces.

              The fact that most users and decision makers have zero IT training massively exacerbates this problem.

              TGIF.

              1. Jou (Mxyzptlk) Silver badge

                Oh no, that trade off does not exist. Bad usablity and insecurity do not exclude each other. In quite a few cases bad usability is the insecurity, so I'd say it is rather the other way around!

        2. Sequin

          We stored important stuff in a fire safe. We had a fire one night and the safe did it's job. Unfortunately the key was left in a desk drawer and melted in the heat!

          1. lglethal Silver badge
            Joke

            Forget the turtles...

            See it just goes to show, you needed a safe to store the key for the fire safe...

            Hmmm, but then you would probably need a key for the safe that holds the key for the fire safe. And then another safe for that key, and .... well it's safes the whole way down!

            1. A.P. Veening Silver badge

              Re: Forget the turtles...

              Hmmm, but then you would probably need a key for the safe that holds the key for the fire safe. And then another safe for that key, and .... well it's safes the whole way down!

              And it goes down quickly with the weight of all those safes, rather an unsafe situation ;)

              1. Andy A Bronze badge

                Re: Forget the turtles...

                Long ago, the company I worked for took over the workload of a company which made firesafes. They showed us a safe which had been on the 3rd floor of a building, and was extracted from the sub-basement following a fire.

                They had made a replacement key to gain access. The tapes inside read perfectly.

                1. WorsleyNick

                  Re: Forget the turtles...

                  Long ago (1970's) I worked for a company that stored all its data in a fireproof safe on the top floor of its warehouse above three levels of cold storage basement. I never did manage to convince them that while tapes might survive after a building collapse, 12 platter discs would not. We had no tape drives, therefore backup only comprised of a 3day cycle of every 12 platter disk on 12 platter disks! Fortunately we never had fire.

            2. Doctor Syntax Silver badge

              Re: Forget the turtles...

              No problem. You use a combination safe instead. Then all you have to do is write down the combination and keep it secure somewhere.

              1. A.P. Veening Silver badge

                Re: Forget the turtles...

                No problem. You use a combination safe instead. Then all you have to do is write down the combination and keep it secure somewhere.

                Just make sure you write it down on asbestos.

                1. John Brown (no body) Silver badge

                  Re: Forget the turtles...

                  "Just make sure you write it down on asbestos."

                  Fuck no!!!!! H&S would have apoplexy!!

                  1. A.P. Veening Silver badge
                    Joke

                    Re: Forget the turtles...

                    H&S would have apoplexy!!

                    Yes, perfect solution to that problem as well

              2. Toni the terrible

                Re: Forget the turtles...

                Store the key and combination off site, in another fire safe if you like

            3. Hero Protagonist
              Paris Hilton

              Re: Forget the turtles...

              Well obviously you store the key for the second safe in the first safe — simples!

              (Hmm, why do I have this nagging feeling I’ve overlooked something? Oh well, I’m sure it’s not important )

              1. Just A Quick Comment

                Re: Forget the turtles...

                Hang on a minute guys! This is an IT forum yes? Then how about a recursive key? For impressive-ness this could be stored in a recursive fire safe, and the only thing it would be at risk from is dodgy algorithms.

                1. Doctor Syntax Silver badge

                  Re: Forget the turtles...

                  Or running out of memory.

                2. John Brown (no body) Silver badge

                  Re: Forget the turtles...

                  You don't need a key, recursive or otherwise. Just wrap the safe in blockchain and tie it with a Gordian Knot.

              2. Snapper Bronze badge

                Re: Forget the turtles...

                You must be in middle-management!

            4. swm Silver badge

              Re: Forget the turtles...

              Why not store the key to the safe in the same safe as the passwords?

            5. EVP
              Flame

              Re: Forget the turtles...

              1. Place fire safe in a regular safe.

              2. Place fire safe key in a fire resistant box (without lock) in the regular safe as well.

              3. The place burns down and melts down the regular safe.

              4. When things have cooled down, recover the fire safe key.

              5. Open the fire safe.

            6. hoola Silver badge

              Re: Forget the turtles...

              Anyone remember Flanders and Swann, The Gas Man Cometh...

              We had 2 LPs when I was a kid, "At The Drop Of A Hat" and "At The Drop Of Another Hat". if I have recalled it correctly.

              1. AlexG_UK

                Re: Forget the turtles...

                Oh yes.. I used to love listening to their records (and they were records) when I was a child. I found a remastered boxset on CD a few years ago - still just as clever and witty but missing the charm of the hissing and popping of the vinyl!

              2. Toni the terrible

                Re: Forget the turtles...

                or Bernard Cribbins 'Hole in the Ground'

          2. ShadowSystems Silver badge

            At Sequin, re: fire safe.

            My BioDad used to work for the Strategic Air Command (SAC) out at McClellen AFB in Sacramento, California before it got closed down. I remember him coming home one time laughing like a tickled kid. I asked him what was so funny.

            "My boss had a fire safe. He lost the key. They used a welding rig to burn the safe open. They ended up burning everything inside the fire safe when the safe itself caught fire. Someone didn't consider that most fire safes are only rated for a few hundred degrees for perhaps an hour, not a few THOUSAND degrees lancing through the lock!"

            Even my ~10 year old mind understood the FacePalmDoh! moment that must've been. =-J

            1. Doctor Syntax Silver badge

              Re: At Sequin, re: fire safe.

              Back in the Troubles one of the buildings burned down in Belfast had a safe drop from the fourth floor or thereabouts. After the fire was out it was discovered it was a little bent and the lock couldn't be opened. Although the maker's locksmith was sent for a PHB decided he couldn't wait & had someone cut it open. The contents were undamaged except for a few items affected by the cutting open. I believe it became a favourite story of the maker's sales staff.

              1. John Brown (no body) Silver badge

                Re: At Sequin, re: fire safe.

                Yeah, it depends on the size and build materials. A small "fire proof" wall safe won't handle the heat dissipation of a cutting torch as well as a floor standing safe big enough to hide a body or two in.

            2. Sgt_Oddball Silver badge

              Re: At Sequin, re: fire safe.

              Fire safes should be opened with a concrete cutter round the sides. Much less hassle and unlikely to ignite the contents.

              1. Martin-73 Silver badge

                Re: At Sequin, re: fire safe.

                yes, see most recent video about fire safes from Lockpickinglawyer, very informative

        3. Anonymous Coward
          Anonymous Coward

          same I used to work for a small defense contractor, All PW's stored in Keypass and then printed out and put in the companies firesafe

          1. Strahd Ivarius Silver badge
            Devil

            but where was stored the master password for your Keypass?

            1. W.S.Gosset Silver badge
              Happy

              See above (recursively).

        4. Anonymous Coward
          Anonymous Coward

          > Ah those were the days - removing all hard disks every night and putting them into the various fire safes, clear desks, and two networks: Secure and Internet with everyone having 2 PCs with KVM switches to move between the two.

          Very similar at one of my former jobs, except that we had PCs with fancy removable HDD caddies. So you only had one PC and you had to attach either the blue network cable or the red one depending on whether you were booting up the Unsecure or Secret drive. (It checked to make sure it was connecting to the expected network. Though there *may* have been ways around this.)

      3. Anonymous Coward
        Anonymous Coward

        Mentioned it before, I used to work at a small company, and sort of did the IT on the side. There was another chap who also did IT on the side, we sort of split it 50/50. He did emails, I did network etc etc

        Big wigs decided one day that they wanted a passwords in an envelope - I was too busy at the time to get round to it, but my colleague did. Couple of weeks later, big company meeting - management buyout, everything will be fine. Few days later, another meeting - a third of staff where to be made redundant.

        My colleague found out after the second meeting that he couldn't access, but I could access mine!

        Was still made redundant though......

        1. Doctor Syntax Silver badge

          "Couple of weeks later, big company meeting - management buyout, everything will be fine."

          That should be a trigger to change all the passwords. Not so you'll access the systems after you've been made redundant but just to make it clear that making IT staff redundant isn't the best idea.

      4. ShadowDragon8685

        Probably best to deposit that envelope with a lawyer or in a safe-deposit box. Better the lawyer than the safe-deposit box, because you can (with documentation) convince a lawyer that your company is still your company and has the rights and need for that envelope after two mergers, a split and a rebranding, while if you've done all that and lost the safe-deposit box the bank is liable to tell you you're SOL and you'll have to... Engage a lawyer to sue them to get the contents of the box anyway.

      5. Bigkahoona

        Of course this is the only proper way to do it. Print-out of master PW list stored in the tape safe where such a thing still exists and if not there needs to be a dedicated safe for such critical information.

        What if the head admin gets hit by a bus while carrying his laptop holding the only copy of the master PW KeePass database and both get smashed to bits?

    2. J. Cook Silver badge

      oof.

      First thing we did with our Office 249 account was to assign two people the Global Admin role, and then created a 'break glass" account that was also GA, with the password stored in the company's on-prem password vault.

      1. pirxhh

        Better yet: Hand two or three people a chit labeled "Part 1", 2 etc.

        Have each of them select a few random characters, write on said paper, and place all of them in an envelope. Seal it.

        Run the password change process with each of them typing their part.

        You now have a pretty secure password that is written down - and nobody knows all of it.

      2. Cheshire Cat

        And then somebody mandates MFA on the whole tenancy, and adds their personal phone as the MFA on the break-glass account ... been there done that.

        Also note that spiffy Microsoft Authenticator (as opposed to boring old Google Authenticator) doesnt allow 2 devices to be set up with the same QR code and ties it to one only, so you can only save the QR code image in your keepass along with the break-glass account if you're using Google Auth.

  2. b0llchit Silver badge
    Go

    Miracle workers

    It is like giving a book to a future tablet user. There are no buttons and you actually need to turn the page manually. I'd gladly take £1000 for turning the page once without them seeing the trick.

    1. Mongrel

      Re: Miracle workers

      Books?! We should never have given up scrolls!

      https://www.youtube.com/watch?v=pQHX-SjgQvQ

      1. big_D Silver badge

        Re: Miracle workers

        Best uses for an iPad...

        https://www.youtube.com/watch?v=nPGY2T9r1Ok

        1. John Brown (no body) Silver badge

          Re: Miracle workers

          Video unavailable

          The uploader has not made this video available in your country

          (No, the downvote wasn't me)

          1. Martin-73 Silver badge

            Re: Miracle workers

            Quite amusing, used a vpn to popup in germany, worth doing if you have access :)

            1. Phrontis

              Re: Miracle workers

              Yes I tried the UK then the US and finally Germany at which point it came up.

          2. big_D Silver badge

            Re: Miracle workers

            Yes, I had that with a clip of an American film the other day, Tele-München had claimed copyright on it in Germany and blocked it... :-(

        2. Martin-73 Silver badge
          Pint

          Re: Miracle workers

          Amazing for an IT site that 2 people downvoted you presumably because (1) they are unable to work out how to use a VPN (it's not obvious that the video won't be available in all countries if you're not in one of them) or (2) like apple devices (shudder) or (3) both.

          Anyway re: icon, (4), have a 568ml or thereabouts of your tipple of choice (and an upvote) for bringing a smile to at least my face :)

      2. Anonymous South African Coward Silver badge

        Re: Miracle workers

        I miss Magnetic Scrolls. (the adventure game company).

        Luckily they're back as Strand Games. https://strandgames.com/

      3. Wyrdness

        Re: Miracle workers

        We gave up on scrolls when Apple invented the book.

        https://www.youtube.com/watch?v=-1IdJDscYHk

        1. A.P. Veening Silver badge

          Re: Miracle workers

          We gave up on scrolls when Apple invented the book.

          There was a time when "book" was synonymous with "scroll" and a scroll is till more efficient for handwriting. However, by now everybody is used to the codex form of books, which really took flight with printing.

          1. John Brown (no body) Silver badge
            Joke

            Re: Miracle workers

            Well, you could always write your award winning novel on a scroll for writing convenience then cut it into convenient sized pieces and glue them together into a binding for reading convenience. Best of both worlds.

            1. A.P. Veening Silver badge

              Re: Miracle workers

              With my handwriting that is not really a possibility. Besides that, there is that minor inconvenience of only one side being available for writing on.

          2. Andy A Bronze badge

            Re: Miracle workers

            The Scroll was the paradigm used in the computing world for nearly 40 years.

            Some of us remember when the scrollbar showed you:

            - that there was more content than that currently in view

            - the relative position of the current view within the whole

            - the proportion of the whole which was in view.

            Current offerings have the functionality of the scrollbar castrated, or the whole scrollbar hidden away.

            1. Will Godfrey Silver badge
              Happy

              Re: Miracle workers

              Not here they don't. Full bar availability with its length scaled to indicate the size of the document

              1. Andy A Bronze badge

                Re: Miracle workers

                But that scrollbar is subject to the fashion choices made by the people who supply your browser.

                Look what happened in Windows 10 - the Start Menu has a scrollbar which is invisible unless you manage to hover over a vertical strip one pixel wide. It then "appears", almost the exact shade of grey as the background of the menu. The control section of the scrollbar has a colour choice almost identical. That means that the "stylists" (we surely couldn't promote them as "designers") have decided to hide important things from us.

                We can never forgive them.

      4. Toni the terrible

        Re: Miracle workers

        Dont forget cuniform on clay tablets

    2. Evil Auditor Silver badge

      Re: Miracle workers

      The value of the content of that book, or rather its next page, that is what you should charge for turning a page. £1000 was actually rather cheap for what Dan did. The question is, what was it worth the client to get admin access? Probably much more than a meagre £1000.

  3. Test Man

    This article is briiliant! :D

    What I want to know is did the bloke state he "fixed" it as soon as he did so, or wait a few hours? Was he even getting paid an hourly rate?

    1. b0llchit Silver badge
      Go

      All just under an hour. I'm guessing the briefing was at least 45 minutes, walking to the server room took 10 minutes and the fix was 1 minute.

    2. Admiral Grace Hopper
      Windows

      There is long established precedent in the industry for this. I was convinced that the two-day very expensive performance uplift to ICL 2900 series mainframes was 1 3/4 days of the white-coated technician eating his sarnies, half an hour taking the side panel off, 10 seconds with the tin snips to cut the wire to the performance-slugging circuit, 10 minutes to put the panel back on and half an hour to do the paperwork.

      1. Doctor Syntax Silver badge

        If there was a quality management system in place swap the eating sandwiches and doing paperwork times.

      2. jgard

        I once worked for a medium sized IT services company. One of the sales people was a solid gold twat who just happened to be a director’s son. Eager to wring extra money out of a poor and unsuspecting customer, he sold them a ‘mandatory’ pbx upgrade. With on-site engineer services, ‘hardware’, licenses etc the total came out at about £15k.

        The ‘work’ consisted of a guy walking in with cardboard boxes, turning the phones off and eating his sandwiches in the comms room. After a few hours of snoozing and YouTube he turned the phones on, put a different plastic cover on the pbx, then went home. Disgraceful.

        That sales guy was such a prick, I very much doubt this was the only time it happened.

        1. My-Handle Silver badge

          A dangerous game, as I'm pretty sure that's actually fraud. Had one of his customers / victims caught on, both he and the company could have landed in some very hot water.

          1. jgard

            I wish he had, and if I had known at the time, I would have grassed him up.

            He was such a jumped up, self important cock. My dad, a traditional self-employed working class builder, once did some work in our offices. One morning I was chatting to my old man next to his van, it was parked near the building for offloading gear etc.

            All of sudden, the 21 year old salesman with impossibly white teeth (now a director himself) marches over like he owns the earth, his fake tan getting redder with each stride. Then, when he can’t hold the anger and sense of injustice in any longer, he starts barking at my dad like he’s a 4 year old, right up in his face, prodding him too. His speech included beauties like ‘this space is for directors only’, ‘what’s your name? I’m gonna talk to your manager’. I just stayed silent and watched with growing apprehension knowing that a suitable response was likely just seconds away.

            My dad quietly let him finish, then grabbed him by the shirt collar and let go the most terrifying, yet articulate tirade of profanity and rage I’ve ever seen close up. It truly was like watching a wildlife show where an alpha male destroys a would be challenger. The kid’s whole presence changed immediately, shoulders rounded, head bowed, his face doe eyed and submissive. My dad made sure he understood that he would need new veneers if he ever even spoke to him again at all. He was then made to apologise in his weak and trembling voice, in front the of hushed mini crowd that had assembled. It was as one of the best things I ever witnessed, and it taught the kid a lesson, never saw him behave like that again. I was so proud of my old man!

            The big boss, the lad’s father (a decent guy) heard about it shortly after and apologised profusely to my dad, he was very sincere. He also thanked my dad for giving his lad a life lesson. A much better lesson would be to stop giving his kid BMW M3s and 850is, and make him stand on his own two feet instead.

            1. Doctor Syntax Silver badge
              Pint

              For your dad.

              1. jgard

                Cheers fella! I’m seeing him in about an hour. I can’t wait to pop a pint down and tell him:

                “Well dad, it’s from a person you’ve never met, on a website you’ll never read, offered as a token of respect and appreciation for that time you served shiny faced Jonny his own arse on a plate in the company car park.”

                I can guarantee that will make him chuckle!

      3. Anonymous Coward
        Anonymous Coward

        snips, we don't need no snips

        That story was factually incorrect. The work would have been performed by a level 2 engineer who would have loaded a new CPU micro program from tape. There would also have been a requirement to check the microcode levels of all the associated disk and tape controllers to ensure they were at the same level as the new CPU microcode. It may also have been necessary to run the ATS (Ashton Test Suite, named after the factory) to confirm that the system was functioning correctly. I imagine that this would would have been scheduled over a weekend and would have allowed at lead a couple of hours in a local hostelry , probably while ATS was running.I never once saw an engineer with sandwiches, they lived entirely on liquid diets.

        1. EddieC

          Re: snips, we don't need no snips

          As I recall, the hardware upgrade involved removing a jumper from a board rather than cutting a wire, and the associated Upgrade Part consisted of the empty foam-lined cardboard box that the removed jumper could be put into before being returned to stores. Saw one of those being done back in the day.

          1. Anonymous South African Coward Silver badge

            Re: snips, we don't need no snips

            ...and the associated Upgrade Part consisted of the empty foam-lined cardboard box that the removed jumper could be put into...

            That sounds like a Monty Python skit....

            1. Antron Argaiv Silver badge

              Re: snips, we don't need no snips

              Sounds more like an IBM thing. Same hardware, two different performances (and rental rates), depending on whether the jumper was installed or not.

              Other computer companies (I worked for Data General) did similar things. Look for the chips covered in epoxy. Those are the microcode ROMs which define your instruction set. Change those and you get scientific hardware multiply and divide instead of commercial instruction set.

          2. Plest Silver badge
            Facepalm

            Re: snips, we don't need no snips

            "consisted of the empty foam-lined cardboard box"

            Didn't you just love IBM's total disregard of the environment pre-2000?

            First time we got brand new IBM RS6000 and some PC desktops back around 1992, we got so many boxes of kit and manuals. I remember though, several boxes were empty, so I asked a colleague why. He said, check again and so I did.

            There are in the cardboard box was a slither of paper with "This is a notice to declare this and all unwanted items should be disposed of after installation."!!! Yep, a box with a piece of paper standing the f**king, bleedin' obvious! You wonder why polar bears are starving at the pole with when kind of utter bollocks used to go on.

        2. Admiral Grace Hopper

          Re: snips, we don't need no snips

          ATS is still in the codebase (as I suspect you know).

          I had the joy of experiencing ICL from both sides of the customer - supplier relationship. Both generated cynicism of different aspects.

        3. Andy A Bronze badge
          Pint

          Re: snips, we don't need no snips

          We suggested fitting a coin-in-the-slot device.

          Operators wanting to head off early might contribute a few extra machine cycles a second.

      4. GruntyMcPugh Silver badge

        I was once an Operator on a McDonnel Douglas 'Reality' midrange system. We paid for a memory upgrade, and all the technician did, was remove a jumper to enable the memory that was already in there.

        1. F. Frederick Skitty

          Same thing with my then employer's DEC Alpha server in 1997 or thereabouts. DEC engineer simply cut a couple of traces to enable the already fitted RAM when we paid for a memory upgrade.

        2. Sequin

          An ICL enhgineer once doubled thhe capacity of the disk drive in our mainframe by removing a plastic peg that physically prevented the heads from moving to the outer part of the disk. This cost us a lot of money!

        3. Strahd Ivarius Silver badge
          Trollface

          Did the technician go to for work Tesla latter on?

        4. Andy A Bronze badge

          The place where I first worked had an ICL 1901T with 24K words of core.

          There were physically 32K words, but they wouldn't pay for the extra. Snip!

      5. mdubash

        Wasn't there a (maybe) apocryphal story about an IBM storage tech doing just that, only with a hard disk? And when I say hard disk, I don't mean one of the piddling little 16TB things we see in servers these days but a proper, two-person-lifter jobbie containing a humongous 5MB...

  4. TonyJ Silver badge

    What about other systems? I doubt everything was AD integrated and that it was all left unlocked ready to go.

    "...

    I used to be the IT Manager for my company many years ago. I insisted that all admin passwords had to be printed off, placed in a sealed envelope and securely stored. Got taken out for a beer a couple of years afterwards by my successor - that strategy had saved his bacon when someone responsible for managing a critical system left the company without doing a proper hand-over..."

    There should always be at least one break glass account per system where the accounts are stored offline, in a secure place (fire safe, for example, or some equivalent even if it's digital).

    1. PM from Hell

      When tech support manager I always insisted on this. Imagine my embarrassment when after having had to retrieve the password for a mainframe system to heck up on dome config info, when my sysadmin was away for 2 weeks I left it in a shirt pocket and it was then washed. I had a very nervous 10 days hoping that there was no need for admin access to the system. All credit to him that there was no need for any system privilege access for the whole period. No Disks filled, nothing expired we didn't even need to create any new users while he was away.

      1. Anonymous South African Coward Silver badge
        Pint

        I once deleted current backups for one of our VM's whose host had a disk failure in a RAID...

        ...sweated bullets until I managed to have a Good Backup, thereafter I made a second backup to another location... and relaxed.

        Just say that it was a brainfart and leave it at that.

        Need one of these --->

  5. bofh1961

    Shades of The Lone Jedi...

    For those of us of a certain age!

  6. Blacklight
    Mushroom

    Does your CMDB extend to password stores & credit cards?

    Similar thing at "an/other" financial organisation, where Azure was setup with a credit card. A personal one. Something got productionised, and then the card holder left the company. Some payment reminders presumably went to an inbox which was no longer serviced (or more likely in existence).

    Not long after, an Azure subscription magically vanished. Which was nice.

    Password storage (actually secure) *was* a thing. Checking how it was funded, less so....

  7. Furtive Lurker
    Devil

    don't spoil the magic

    Well, that was very badly managed. You NEVER do a two minute job in two minutes! You MUST stretch it out to at least an hour or the customer won't believe they're getting value for money. You are the expert with years of experience, training, knowledge and wisdom. Don't ruin the illusion.

    1. tinman

      Re: don't spoil the magic

      As Scotty taught us…

      https://youtu.be/8xRqXYsksFg

    2. Doctor Syntax Silver badge

      Re: don't spoil the magic

      That depends on whether you want a reputation as a hero or magician.

      1. b0llchit Silver badge
        Coat

        Re: don't spoil the magic

        I'd settle for God status.

      2. DJO Silver badge

        Re: don't spoil the magic

        ..whether you want a reputation as a hero or magician.

        Which pays more?

        1. Anonymous Custard Silver badge
          Headmaster

          Re: don't spoil the magic

          If you're doing it right, the two are not mutually exclusive either...

        2. Hero Protagonist

          Re: don't spoil the magic

          I think heros are expected to perform their feats for no pay — that’s why Superman needed his side hustle at the Daily Planet.

    3. My-Handle Silver badge

      Re: don't spoil the magic

      I think after several days being stuck with an unsolvable problem, the customer would pay handsomely for one of your farts if it looked like it fixed things, regardless of how short a time it took.

    4. Contrex

      Re: don't spoil the magic

      I learned to stretch out jobs when I was a self-employed TV/general electrical repairer in the 1980s. If you took 5 minutes they didn't want to pay the pre-agreed rate. "You didn't do much!". I used to get fed up with people I hardly knew demanding pals' rates, or offering a 'packet of fags' or a (seldom forthcoming) 'pint'.

      1. Doctor Syntax Silver badge

        Re: don't spoil the magic

        You should have presented an itemised invoice including the very expensive part you fitted.

        1. Emir Al Weeq

          Re: don't spoil the magic

          Eg, to use the plumber comment at the start of article...

          Replacement washer: 1p

          Knowing which one to replace: £199.99

  8. GlenP Silver badge

    A former employer allowed their drawing office to source new CAD systems without consulting IT (me!) Fine, they were NT-4 boxes but this was prior to AD so it didn't really make a lot of difference. My only involvement was dragging the delivery van out of the mud after they'd decided to drive up to a window and pass the boxes through that rather than take them 30 feet further through reception (I did have a Range Rover at the time - see the fuel bills!)

    Several months after they'd decided they didn't need an IT manager and made me redundant I had a call, "Do you know the admin password for the CAD machines?"

    I took great delight in pointing out that they hadn't involved IT in the purchase/installation and hadn't given us the passwords therefore I couldn't help. My suspicion is that the vendor had never actually told anyone what the passwords were but that wasn't my problem.

    At my current employers there is an encrypted spreadsheet on a backed up server folder with two senior people having the key to decrypt it, just in case.

    1. tip pc Silver badge
      Thumb Up

      At my current employers there is an encrypted spreadsheet on a backed up server folder with two senior people having the key to decrypt it, just in case.

      that is a great way of storing those passwords,

      How often are those passwords rotated or even checked?

      Does the encryption get redone when one of those seniors leaves?

      Do the seniors know where they have stored their password and do they know how to retrieve in the event ion an issue, is that tested?

      Not raining on your parade just wondering how far down that rabbit whole I'd need to go if I did the same.

      1. GlenP Silver badge

        To answer, for info,

        Every few months or if something significant changes.

        Hasn't happened yet but it will if/when one leaves and a substitute is appointed

        I probably should audit them on it but they're both pretty reliable on these matters.

        1. A.P. Veening Silver badge

          Assuming the password of either one is enough, how often do they fly together?

          1. Outski Silver badge

            Quite. When the boutique consultancy I worked for many moons ago was responsible for a 12k seat Domino estate (later a 26k seat Exchange estate), the customer insisted my boss and I were never to travel together, by any means. Mostly ok, since I'd by that time moved to Malaysia to set up our 24/7 team.

      2. Terry 6 Silver badge

        This was my thought. The level of attrition on those two back-ups could be pretty high.

        Not just personnel changes, but loss of the access password, maybe forgetting they even had one.

        It's not hard to imagine that one senior person might leave, go doolally or die and the other totally forget where the password was. Or that the device with the stored spreadsheet document becomes degraded by age, accidents or whatever. These may be small risks in the short term, but over a few years..

    2. Outski Silver badge

      At my current employers there is an encrypted spreadsheet on a backed up server folder with two senior people having the key to decrypt it, just in case.

      And what happens when that server fails?

      1. GlenP Silver badge

        And what happens when that server fails?

        The support people bring it back up on the DR server, either in-house in a few minutes or off-site in 30 minutes. The joys of virtualisation!

        1. Outski Silver badge

          And how often is that tested?

          1. Strahd Ivarius Silver badge
            Trollface

            Last time they tried they needed the password stored in the file for restoring the server...

          2. John Brown (no body) Silver badge
            Coat

            Every week. But no one is tasked with checking the fuel levels in the backup genny at the DR site.

  9. IvanV

    Well, sometimes knowing how to fix something in one minute is a results of 20 years experience. So, charging for one minute of your time should factor your experience too. That's why I won't feel that bad for overcharging customers.

    1. Pascal Monett Silver badge

      As they say, it's $50 for the hammer, and $400 for knowing where to use it.

    2. Outski Silver badge

      I wouldn't call that overcharging.

    3. John Brown (no body) Silver badge

      Yep. Did a job a couple of months back. Got sent in with minimal info on the client site. PCs would randomly work/not work on the LAN, but all had internet access. Who got on the working LAN and who didn't was random and changed each time machines were booted. In the back of my mind, I'm thing DHCP. Sure enough, when I got there, there were two LANs on different subnets and two DHCP servers. The "emergency" backup provision had kicked in at some stage and not reverted when the main provision came back up. The emergency provision was bare bones hence the lack of connectivity internally and, naturally, had it's own DHCP. And was plugged into the same master switch box as the primary provision. Half an hour to find the kit, 10 seconds to identify and pull the plug. 20 minutes explaining why their failover method was the cause and how to fix it it for the future, half day billable and a grateful customer who will almost certainly call us back to set it all up properly for them.

    4. herman Silver badge

      I add my age to the bill - in k$.

  10. chivo243 Silver badge
    Devil

    A magician!

    Never reveals how his tricks work!! At least to the audience. I always found it best to share these experiences with my colleagues, we all like a good laugh!

  11. Lazlo Woodbine Silver badge

    I left a company once because the new management decided I was no longer needed

    A couple of months later the manager tracked my down by phone at my new job, "err you wouldn't happen to know the admin password would you, we've lost it."

    I did remember it, and I remembered where it was written on a post-it, but I let them squirm for a few days whilst I "tired to remember"

    1. DJO Silver badge

      I did remember it

      Without a consultancy fee?

      Organic memory needs beer* vouchers to refresh.

      * Chateauneuf du Pape vouchers if you are working in the city.

      1. l8gravely

        I still support a fellow OpCo in my $WORK, but they're cheap and lazy and fired all their application support people. They had me on a two hour call recently trying to figure out some workflow issues in their application used by finance. I don't know shit about Java or the business logic, so I was kinda flailing, but much less than they were.

        I kept asking them to just go call the guy who did all this setup years ago and now works for another different sister company and pay him a couple of hundred bucks to find and fix the issue.

        But no... they're still too cheap to do that.

    2. MCMLXV

      Downvoted. So the staff at your old work were left hamstrung for a few days while you "tired" to remember. Just to satisfy a grudge. Childish and unprofessional.

      1. Doctor Syntax Silver badge

        Life is too short to carry a grudge. We just have to do the best we can.

      2. spuck

        "Childish and unprofessional" would have been to make mischief with the password. Letting management feel the pain of their poor decisions is Not His Problem.

      3. Antron Argaiv Silver badge

        Counterbalanced by the fact that he didn't charge them for the information.

        Very professional, in the "we're all in this together" sense.

      4. doublelayer Silver badge

        After you get fired, you have no professional duty to keep working. Sabotage would have been both. Not doing free work, however, should be expected.

      5. David Nash Silver badge

        He got fired. I would have charged them £100 for the service.

        1. A.P. Veening Silver badge

          He got fired. I would have charged them £100 for the service.

          I would recommend adding at least one zero (before the decimal point/comma).

      6. Lazlo Woodbine Silver badge

        Downvoted for your lack of understanding of how life works.

        Why was I the only one who'd noticed the password was on a post-it on top of the server, it wasn't even hidden.

        My goodwill only goes so far, if someone other than the manager who'd let me go had phoned I'd have told them straight away...

    3. Gene Cash Silver badge

      "Sorry, mate I no longer work for you. You fired me, remember? Now let's talk about the consultancy fees..."

      1. W.S.Gosset Silver badge

        "Sure, I can do that! My day rate is ..."

    4. Anonymous Coward Silver badge
      Terminator

      "I deleted all of that information when you made me redundant. We can talk about data recovery costs to attempt to recover it if you want"

      1. Antron Argaiv Silver badge

        Information all deleted (as is required by my NDA and separation agreement).

        1. John Brown (no body) Silver badge

          I wonder how much beer it takes to unremember a password and exactly which type beer to target just the relevant info? And does the company firing you pay for that beer? Especially considering that said company should have changed the password after the firing.

    5. AlbertH
      Devil

      I've had exactly that happen to me, but in my case, it was a Civil Service Server in deepest Whitehall...

      Bliar's "government" paid a hell of a lot of moolah for my new company to resolve their fundamental IT stupidity. I just saw our very high hourly rate as a generous Tax Rebate!

  12. Sam not the Viking Silver badge
    Big Brother

    Top Dog

    At one company I worked for, we had to use a production system run/supervised by a sister company who seemed to want to prove their superiority. (The system was hopelessly out of date but that's another story). The supervisor would visit, usually to sort out a 'technical glitch' that only he could resolve. It was a clear scam to get a day out at our expense. When we realised that this was going on, we quickly 'discovered' his login details and in doing so obtained much wider access than we anticipated..... We used this access carefully to maintain our rich data source; as well as prevent the 'glitches' recurring.

    His password was 'Top Dog' and he used it for everything. He must have puzzled the persistent canine references in the surrounding chatter.

    1. jgard

      Re: Top Dog

      Brilliant! I would not be able to resist mumbling ‘sausages’ in a doggy style voice, followed by light panting, as I walked past him…..

      Me: “SROSSAJIESS!”

      Him: “What was that?”

      Me: “RNNUFFINK!”

      1. David 132 Silver badge

        Re: Top Dog

        > I would not be able to resist mumbling ‘sausages’ in a doggy style voice

        “WALLS!”

        How many decades on, and that advert still sticks in my memory…

    2. lglethal Silver badge
      Go

      Re: Top Dog

      I sincerely hope the info about the deliberate glitches was communicated to your CEO.

      Unless your CEO is already in the pocket of the other company, they would normally at that point be making it very clear to the CEO of the other company that unless they are reimbursed for all of the callouts they will be reporting the "Supervisor" and his firm for fraud to the Police. And that at the next contract negotiation, there would be some rather smaller numbers on the page then exists currently...

      It's the only way to deal with scum like that... But it does usually have to come from the Top, to get in anything to actually happen (the other CEO would of course deny any knowledge of such behaviour, would see to it that the Supervisor was sacked or shunted to other projects, and of course they would be happy to compensate the firm for the despicable behaviour of their employee, etc... Another round of golf, perhaps?)

      1. Sam not the Viking Silver badge

        Re: Top Dog

        The eventual fallout was rather messy...... The sister company had a bigger turnover and much more spending power. They claimed to obtain major discounts from suppliers because of this. When it transpired that everything we bought was in fact cheaper than their 'special discounts' from 'approved suppliers' the scam was wide-open and the dismissals were fast.

        Top Dog was neutered.

  13. Anonymous Coward
    Anonymous Coward

    That's around £1,500 ($1,980) in today's money

    £1800....

    £1900...

    £2000....

    Cost of living these days....

    1. Aladdin Sane

      Re: That's around £1,500 ($1,980) in today's money

      Doesn't bother me, I only ever put a tenner in anyway.

  14. AnotherName

    Admin access

    In a previous job in the early 2000's, where I was IT Manager (and I was also 50% of the IT team), I was made redundant, partly enabled by underhand reassigning of roles. We had an internal meeting where the organisation chart was displayed. I pointed out that they had me down in the wrong role, but it was brushed over - I should have seen the signs...

    Anyway, on the day they called me down to tell me I was redundant, they asked someone else to revoke my admin rights and changed the admin password. I was allowed to go back to my desk for the rest of the day to copy off my personal files from my computer. As I had been logged in the whole time, I still had full admin rights and could have done anything, but obviously didn't do any damage. I did copy off a few customer directories from a server so that I could make direct contact with them after the event and actually did some work for many of them.

    A few months later, after the redundancy period was up, I received a call asking for the password for a specific system. I told them what my rates were for consultancy and they never bothered me again. They lost many knowledgeable developers due to the way those of us who were made redundant were treated and eventually went bust.

    1. A.P. Veening Silver badge

      Re: Admin access

      They lost many knowledgeable developers due to the way those of us who were made redundant were treated and eventually went bust.

      Karma is such a lovely bitch ;)

      1. Doctor Syntax Silver badge

        Re: Admin access

        Karma nothing! More like cause and effect.

        1. A.P. Veening Silver badge

          Re: Admin access

          "Karma" has always been a one word code for "cause and effect". Also known as "what goes around comes around" and "everybody gets what he/she deserves".

          1. Antron Argaiv Silver badge
            Thumb Up

            Re: Admin access

            ...though, sometimes, it takes longer than one would hope.

            e.g.: the previous occupant of the US Whit House.

            1. Anonymous Coward
              Anonymous Coward

              Re: Admin access

              e.g.: the previous current occupant of the US Whit House.

              FTFY

  15. GruntyMcPugh Silver badge

    Many years ago I got a call from an account I used to work on. They'd had some recommendation to reduce the number of enterprise adminsistrators they had, and use one time access to the built in admin account, using some app they'd been sold. So they removed everyone's admin access, and then realised they didn't know the Admin account password to get the ball rolling. So they called, as me and a couple of colleagues had promoted the first domain controllers back in the day and asked if I knew the Directory Services Restore Mode password,... which, luckily I had ingrained in my memory (and that we had lodged somewhere safe, but our jobs were outsourced, people left offices, safes were disposed of, etc). So they had to roll back their AD to before the fudge-up.

    1. Robert Carnegie Silver badge

      I wonder how often the phone call to a recently let go employee saying "Er, can we trouble you for the Very Important Password, that we can't seem to find" is not from (an appropriate person at) the business that let you go at all, which does know the Very Important Password but may have been not sufficiently cognisant that in such a situation it also is Very Important to change it.

  16. aj68

    Sometimes it just falls into your lap

    Only a few weeks after joining a large US RDBMS company in the late 1980s (based in Bracknell), I was told I was needed to visit a large customer site the next day where they were having severe performance problems with their VAX based database. I spent as much time as I could cramming text books and performance tuning notes before being flown up to Manchester and getting a cab to their office, while inwardly panicking the whole time.

    I was met by the data manager and shown to a VT terminal logged onto the machine. Within seconds I found their system disk was over 99% full and having a couple of years VAX/VMS system management experience this was a smoking gun with a red hot barrel.

    I breathed out, looked up, and thanked the almighty for this "Get out of Jail free card". I cleaned everything up and wrote the client a 4 page report on maintaining their system performance.

    I'm told he later waved this in the face of his DEC support team saying this was the level of support he was failing to get from them, and how great we were in comparison. A great start to my new job, but it was a bit lucky!

  17. Ian Johnston Silver badge

    A fellow post-graduate student, years ago, was given a very well paid commission to get a piece of software running on the departmental VAX. It came as FORTRAN source code, and although it compiled it didn't do what it should. He looked at the source and spotted lots of lines beginning with "D", which readers will recall are debug lines which are treated by a compiler flag as comments or not.

    So he compiled again /DLINES, and everything worked fine. Ten minutes' work, paid for three days.

  18. Anonymous Coward
    Anonymous Coward

    I'm a little bit worried

    that only one poster here appears to have heard of a break-glass account.

    1. A.P. Veening Silver badge

      Re: I'm a little bit worried

      Most of them have heard of it, but maybe not by that exact name.

      1. Anonymous Coward
        Anonymous Coward

        Re: I'm a little bit worried

        Most of them have heard of it, but maybe not by that exact name

        Oh, I agree. But when I posted there were 53 comments ... assuming 50% of posters know of the concept, I would have expected more than one mention.

        Little tip for people who do setup such accounts. Obviously you test them regularly. But make sure you have hard copy backup 2FA codes squirreled away somewhere. And make sure you can access the details offsite, in case your office burns down.

    2. GlenP Silver badge

      Re: I'm a little bit worried

      In my case it's a GOOJ account, stands for Get Out Of Jail (Free).

      Too many years of playing Monopoly as a youth!

      1. Wellyboot Silver badge

        Re: I'm a little bit worried

        Many years ago we also had a 'terminator' account that when logged in on the 'emergency' workstation would initiate a sequenced script to shutdown everything neatly in the correct order, If it was logged in anywhere else it merely gave you the list it was parsing to allow confirmation nothing was being missed.

        The 'emergency' workstation was not far from the safe containing the all the password envelopes though only a few system techies in IT new this part of the process (but only their third of the actual password).

  19. Anonymous Coward
    Anonymous Coward

    Once spent the pre-pizza portion of an evening doing something like this at a company where the admins were getting a bit too up themselves and the manager had me and another friend in to regain control of things. I dealt with the Windows and NetWare boxen, he did the various Unix flavours. I think the admins were read the riot act the next morning and behaved a bit better thereafter.

    1. Tim99 Silver badge
      Devil

      Admin access

      A long time ago, we had an uppity admin in the public service who told his boss that he was the only one who knew how the main system worked, and that he needed a promotion. There was a very high level meeting where it was decided that this was unacceptable.

      I had written said system, was very senior, and was responsible for it throughout the organization. I was asked my opinion. Plans were put in place, and some weeks later I travelled on a Sunday to where the system was. I dumped the data from the files, then reinstalled the system, changed the system password, reassigned the admin to a lower group role, and reloaded the data.

      On Monday, everybody in the location was told that their sections were being merged with others, that they should apply for new roles within the revised organization, and that they should continue in their old assignments until further notice. The admin just happened to have started his annual leave on the previous Friday. Funnily enough the admin was the only person who didn’t get a job similar to their previous role. All of the Byzantine rules of dealing with staff were scrupulously followed, and he left later that year.

      Yes, if sufficiently provoked, the public service can be at least as bastardly as anywhere…

      1. Jou (Mxyzptlk) Silver badge

        Re: Admin access

        "he was the only one who knew how the main system worked, and that he needed a promotion"

        One should never think of being not replaceable. The cemeteries are full of them.

        I am in contact with a few bosses, actually good ones, who had people like that. When they pulled that stunt in front of their boss they were told "OK, go. Leave company car key here, leave your access card, leave the company phone here. Go." - when the response was something along the lines "notice period" or "but I need the car since we are moving on monday" or "but I only have that phone" they continued with "I know, I don't care, leave that stuff here right now on my table and go.". The next step, which I recommend then when I was informed, was to change their password. Of course they paid what they had to by law, according to the notice period and such things, but the sign was clear: Go now.

  20. aerogems Silver badge

    Ah the laying of hands

    Worked as a contractor providing hardware support for a small to mid-sized University. One day get a ticket that one of the library staff's computer won't boot and is beeping at them. I make my way over to the computer to find someone had left a book sitting on the spacebar, so the BIOS was complaining about a stuck key. Moved the book, rebooted the computer, all was well with the world.

  21. DS999 Silver badge

    Back in the 90s I was once asked

    To help break into a server. I was consulting for the company on a six month contract dealing with migration to new EMC Symmetrix arrays, but my resume did state I knew HP-UX very well. The guy who dealt with that server had left in a pretty acrimonious way - frog marched out by security back before that became the standard way everyone gets laid off - and no one could access it. They told me they'd tried swapping out the boot drive with a similar system but it wouldn't boot, they tried putting its boot drive on another system and modifying /etc/passwd, but it still asked for a password, so they were stumped. They had tried some other stuff too, apparently one then two and eventually their whole Unix server team had taken a crack at it over the course of 3 1/2 days with no luck.

    I walked up to the machine which was helpfully already running and sitting the firmware prompt, ran a command to check the boot path and recognized it had mirrored boot drives, so when it tried to boot the default read drive must not have been the one they edited /etc/passwd on. I ran a command to manually boot from the other drive, logged in as root with no password, and told them they had a split mirror in an unknown state because of what they'd done so they need to manually re-mirror to the active drive. Took less than 30 seconds from the time I arrived, I mumbled something about "let me know if you still have issues, gotta get back to what I was doing" as I walked away.

    The next morning the guy in charge of all the server and desktop teams (the manager I was working for on this gig was in charge of telco, networking & storage, so I had barely talked to this guy) walks up to me and thanks me for helping out his team. He says "I talked to John [my "manager"] and he said you should fill out a separate timesheet to submit to me for your hours working on this, so he doesn't get billed for it". I was a little confused, and kind of stammered and he said "don't worry, it will be at the same rate". I finally get out "but it only took me a few seconds, do you want me to bill you a full hour for that?" and then he was a little confused and stammered. I explained what I did, and he said his guys had only told him I fixed it for them, not how long it took, and he just assumed it had taken a long time given how much time his team had already spent on it.

    I caught up with John later that day and told him about the strange exchange I had. He started laughing when I explained to him how little I'd done to solve their big issue and said not to worry about the other timesheet. I thought about filling out a timesheet for an hour to the other manager anyway just to be a smartass, but he was a pretty nice guy who was just steered wrong by his people so I decided against it.

  22. Marty McFly Silver badge
    Facepalm

    NSDE solved for $50

    Back in my early career days I supported PC-based point-of-sale systems for restaurants. So the customers were all non-technical types. "Brain" was the computer, "TV" was the monitor, "typewriter" was the keyboard, and so forth. Furthermore the restaurant manager types didn't like to be called at the wee hours of the morning when closing up after last-call. Unfortunately the same manager types cringed at any sort support contract, so it was $50 minimum up front with a credit card number.

    These were early days where backups were saved to floppy disks. Inevitably someone mopping the floor would bump the power button on the primary workstation. Realizing their mistake they would power it back on....resulting in a dreaded "Non-System Disk Error" and the hunt for the ANY key.

    After charging their credit card number.... "Push the button on the front of the brain next to the slot. Now press the space bar on the typewriter. Thank you, bye!"

    Of course, for every luscious NSDE call I had, there were ten where the same $50 at 2am had me talking non-techies through re-seating ISA Bus cards to get serial ports working again. It wasn't all easy money.

    1. John Brown (no body) Silver badge

      Re: NSDE solved for $50

      ...and then there'd be the times you had to actually visit site and find the PC full of beer/curry/custard or just years of fat/grease build-up!!!

  23. JerseyDaveC

    Been there, loved wearing the t-shirt

    I had a similar experience - desperate call from a client of a client of a friend whose SQL Server cluster I'd set up but which had died ... when he did a DIY data centre move. This was 1pm on Sunday, and it had to be up by the wee small hours of Monday.

    Two-hour drive to London. Server said "No storage connected". Moved the SCSI connectors for the RAID array from the server's SCSI adaptors to the server's RAID adaptors. Booted, Server said: "Ah, I see you've hooked up some storage, but it's not connected properly". Moved SCSI cable from server A to server B, and from server B to server A. Booted. Server said: "Ah, that looks familiar - would you like me to start up?". Told it "Y". Two-hour drive home. Wrote and emailed four-figure bill. Massively relieved client of a client of a friend paid it promptly and was super-grateful, which was lovely.

    1. DreamEater

      Re: Been there, loved wearing the t-shirt

      Having recently gone self employed, I now have a new smile when I read these stories.

      We have the original smile from dealing with desperate people because of their desperate actions, then the 2nd smile at being able to fix it and now the new smile "paid it promptly"

      Never knew cashflow would give such anxiety.

  24. zaax

    I bet Dan didn't see much of that thousand

  25. Anonymous Coward
    Anonymous Coward

    Timely public bug

    Once upon a time, we were subcontracted to a subcontractor to a Major Defense Agency with Big Computers.

    Said MDAwBC had lost the root password -- their policy was change it every 30 days, don't write it down, and don't forget it.

    So the subcontractor phoned and asked us for the password (no way, it was 3 months later); did we "accidently' leave a backdoor (no way, the account list was audited as part of acceptance); did we have any fzcking clue how to get them out of a jam????

    Well, actually, did you read El Reg that morning?

    https://www.theregister.com/2014/09/24/bash_shell_vuln/

    They hung up, and never even phoned back with a thank-you.

  26. raving angry loony

    Wasted opportunity

    Anyone THAT stupid and THAT incompetent (and I'm talking about the senior management of the outfit who trashed the original I.T. team, not the current IT staff) should really have had a hefty stupid tax added to their bill. OP could probably have extended their stay a day or two, at least.

    I mean, what would the BOFH have done?

  27. herman Silver badge
    Devil

    You’re fff… err hired

    I was once fired and then hired back for $10000 to complete what I was working on.

    1. AlbertH
      Linux

      Re: You’re fff… err hired

      Been there too - except their offer of £45000 was politely declined, and they were forced to pay my bill of £180000..... That set up my new business very nicely!

  28. vistisen

    I used this hack many years ago when arriving at acustomer where no one knew an admin password!

    C:\> cd \winnt\system32

    C:\winnt\system32> copy logon.scr logon.scr.old

    C:\winnt\system32> del logon.scr

    C:\winnt\system32> copy cmd.exe logon.scr

    Now log off the machine, logon.scr is the screen saver that will kick in after 15 minutes of not touching the keyboard/mouse at the logon screen. Wait 15-20 minutes and a DOS prompt with FULL SYSTEM rights will pop up, then just to

    C:\> net user administrator <newpassword>

    and then log in with the new account.

    1. MJB7

      Re: logon.scr

      What OS was that on? Moving things in system32 requires full admin rights already these days (and has done since Windows XP).

    2. Jou (Mxyzptlk) Silver badge

      Yep, that is the one for NT. Today you have to use utilman.exe for the same. Saves the waiting.

      1. Tim99 Silver badge

        I haven’t done this for a while, but assuming that the system can boot from an external device, and that you only want to zero a local admin password the chntpw utility in the SystemRescue distribution worked well, but it won’t work on an encrypted disk…

        1. Jou (Mxyzptlk) Silver badge

          Not on domain controllers :D

  29. CuChulainn Silver badge

    A Timely On-Call

    I woke up Saturday morning and discovered I couldn't access my own blog (or any other sites I host) - either view it or access the Wordpress dashboard. I was getting a timeout. I assumed it was a server outage.

    This was overshadowed ever so slightly by the paranoid thought that my posting of the Arnold Schwarzenegger address to the Russian people concerning Ukraine, and several comments about Putin might have upset someone, even though Russia is blocked due to repeated attempts at hacking over the years from there.

    Then I realised my traffic was still there, just not from my home network IP address.

    I later discovered I could access it normally if I used VPN, but again not from my home IP address. I also realised that Uptime Robot had stopped logging at 1.30am Saturday morning, so it can't get access either.

    I did all the usual stuff - power cycling everything, flushing and renewing DNS, and so on. My IP is whitelisted in my security software, and since I could log in via VPN I obviously wasn't blocked as a user/admin in Wordpress.

    Having started wondering where the hell to start, I concluded it just had to be a server/host issue. IONOS have pretty much (i.e. definitely) admitted they have a situation, but they haven't bloody fixed it. And tonight, after being on hold for ten minutes while the agent had a word with the backroom, I was cut off (end of working day, I guess, but they won't admit to that when I call tomorrow), the problem persists.

    1. CuChulainn Silver badge

      Re: A Timely On-Call

      And after almost two full days, it was apparently fixed just before midday today.

      So if you do get ERR_CONNECTION_TIMED_OUT - apart from all the usual stuff which blames things your end and questions your parentage - don't rule out your ISP or web host as the real source of the problem.

      I'm now questioning IONOS's parentage, believe me.

  30. Rockets

    Been In The Same Boat

    I was once sent out to a new client because the company had sacked their previous admin but didn't have the admin password. It was a small shop and this was back in the NT 4 days and the client was running SBS 4.5 server. Before I cracked open the "server" to attach the servers hard drive to another NT4 machine so I could copy off the SAM and run lohptcrack against it, I tried connecting to MS SQL. To my astonishment I connected to MS SQL as sa with a blank password. As SQL 7 was running as the system account on the PDC a quick couple of sql statements & I had a new account with domain admin rights on the network and was in after only being on site for 15 minutes.

    1. Robert Carnegie Silver badge

      Re: Been In The Same Boat

      It sounds like they should re-hire the former admin so as to sack them all over again.

  31. Anonymous Coward
    Anonymous Coward

    Disloyalty begets disloyalty

    He did that ALL wrong. A company so morally decrepit and hard-wired-for-stupidity that they laid off their entire IT staff should be made to pay through the nose. I'd have dragged the "account recovery" out over several months. Not because I'm greedy, but because I like to think of myself as an IT-based batman, dispensing karmic justice through my keyboard and my invoicing software.

  32. venkatarangan

    SQL Server injection to rescue

    Once I was involved in a similar situation in a clients place. They had forgotten their AD Admin password on a production web & db cluster. Luckily this was before SQL Server the patches from Microsoft after a well-known attack had been applied. I simply executed an ASP page with 'poison' SQL query to execute, CMD.EXE with a parameter to run "net user username password /add" and then another command to add to the global admin. Voilla.

    Then, dutifully I executed the Microsoft patch on all the server for SQL Injection and instutionalised a process with the developers to sanitize all their HTML inputs.

  33. This is not a drill

    Useless service desk manager.

    Sounds similar to a situation I was part of. I was network manager (Novell) for a company. Following best practise the "admin" account was not used, I delegated all rights and had a superuser role for my administration account. The "admin" account had a strong password which had been written down, put in an envelope in a locked box, and stored in the company safe "just in case".

    Anyhow the company hired a service desk manager, who insisted the service manager and network manager (me) report to him. I wasn't consulted and this was effectively a demotion so immediately found a new job and resigned.

    The service desk manager insisted that the help desk team needed the network "admin" account password, I held my ground and explained that they had all the rights necessary. On my last day I handed everything over, including the admin account and explained to the service desk manager that he should use it to add himself and whoever he wanted superuser role then change the admin password and put it back in the safe.

    Rather than doing that he just gave the admin account to all the help desk team, after about a month they had managed to lock the admin account and because he hadn't added anybody to the superuser role, nobody could reset it.

    This caused major problems and in the end they had to get Novell in to reset the admin account. Shortly after the helpdesk manager and the IT director were sacked.

    1. Robert Carnegie Silver badge

      Re: Useless service desk manager.

      Now that is funny.

  34. Trotts36

    Drag it out

    Personally I would of enjoyed a very nice week long exercise at 1k a day to fix this issue.

  35. firebits

    Good job that Windows 2003 Server was unlocked otherwise, it could have taken a good 30 minutes to get that domain admin account password reset

  36. Some call me "Tim"

    Lasers in a grill factory

    Roughly 18 years ago at my previous job, a customer (large charcoal grill manufacturer) located in Georgia, USA calls to complain that their safety laser area sensor is broken and picking up things that aren't there. The sensor was used to make sure all humans and other impediments were clear of the area before a large welding jig moved at the start of the grill assembly line. I worked for the sensor manufacturer in New Jersey. Grill company is told that they will be responsible for $1000 per day plus expenses if I determine that the problem is on their end and not a failure of our product. The following week, a couple hours on an airplane from Newark to Atlanta, followed by a 2 hour drive in the rental car gets me to the front door. Just inside, 8 managers and maintenance staffers are staring at a laptop showing a realtime 2D map of what the sensor "sees". I watch the laptop and the jig for 3 minutes, then ask one of the "watchers" to place his finger on the laptop screen where the "false" trigger is showing up. 5 minutes later, the line is stopped for the scheduled morning coffee break and I can approach the equipment. I knock some welding slag off the end of the rig with my finger and ask the designated watcher if that matches what the "failure" they witnessed. They had recently changed to cheaper imported steel (creating more slag) and cut back on how often they cleaned the machine, never realizing that a sensitive laser sensor might receive a reflection from shiny falling metal shavings. I politely waited an hour before calling my boss and telling him to get the invoice ready. My return flight wasn't scheduled until the next afternoon, so I enjoyed my dinner and hotel room that night!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like

Biting the hand that feeds IT © 1998–2022