Help, my IT team has no admin access to their own systems

You know that plumber who charges hundreds to just to change a magic washer? The IT world can be the same, where seemingly magical skills are grounded in the most mundane of realities. Welcome to Contractor Time in On Call. Today's story, from a reader Regomised as "Dan", takes us back to the end of the first decade of the …

[A Non e-mouse]

I worked in a small company that worked closely with other similar, local companies. We all had independent IT. One day I got an email from one of the other IT teams asking me to reset their admin password as they'd lost it. I replied saying I couldn't as I didn't have access to their system. It took a bit of persuading that I couldn't help. They then asked how they could reset the admin password. I told them it probably wouldn't be a quick and painless process (just as it shouldn't be)

The admin password they'd lost was the global admin account for their Office 365 tenancy.

[UCAP]

I used to be the IT Manager for my company many years ago. I insisted that all admin passwords had to be printed off, placed in a sealed envelope and securely stored. Got taken out for a beer a couple of years afterwards by my successor - that strategy had saved his bacon when someone responsible for managing a critical system left the company without doing a proper hand-over.

[Phil O'Sophical]

Always a sound strategy, but sadly screwed up these days by IT forcing everyone, even admin accounts, to change passwords every month or two.

[ChrisElvidge]

Passwords

Is it really IT that insist on password changes? Or is it some c-type who has heard that passwords should be changed regularly for "security" and has conviced some higher-up to issue an edict?

[John Brown (no body)]

Re: Passwords

IIRC, it's been shown that people will use the most easily remembered/guessed password they can get away with within the confines of the defined password policy when said policy forced frequent changes.

On the other hand, people will choose a more complex, less easily guessed password if it's significantly longer but only requires a change every 6-12 months and is, in turn actually more secure.

Irhmbawhwrny1666 is probably more secure than Pa55wo0d26!

(FWIW, I Really Hate My Boss And Wish He Would Retire Next Year 1666 :-)

The initial letters of a long but memorable and personal phrase is easier to handle than a short complex, random sequence and more secure.

Disclaimer: IANAsecurity professional and may be talking bollox.

[redpawn]

Re: Passwords

Offline NT was a tool in my tool box. Teachers and admin were always forgetting their passwords

[Jou (Mxyzptlk)]

Re: Passwords

Olline-NT does not work for domain controllers. But there is a way around that too... One for NT up to Server 2003 (or 2008 ?) and one from 2008 to now. Needs physical access - or in case of VM: "Physical" access. Been there. Done that several times.

[AlbertH]

Re: Passwords

Ophcrack always get me into Windoze boxes with "forgotten" (or more usually corrupted) passwords. I've built a couple of modified versions for specific environment uses... The best use I've found of "Slax"!

[TeeCee]

Re: Passwords

Golden Fuckwit award goes to Crapita (surprise) and their local government payment portal. I'll leave out just how utterly bloody uselessly nonsensical the design is (worth seeing it, just to see that much WTFness in one place) and concentrate on the login.

Requires....12 characters(!), at least one capital, at least one number and (drumroll) at least one special character. Short of actually putting on the password screen; "Please make sure you write this down on a post-it note and stick it to your screen", I'm not sure what else they could have done.

Yes, I have just been trying to pay some council tax via this Edifice of Shite(tm). Why do you ask?

God alone knows why the Public Sector continues to hire this bunch of incompetent, idiotic tossers. I guess they must be really good at disguising bribes and giving blowjobs.

[Ace2]

Re: Passwords

Last time logged in to my account at the Social Security Administration (USA gov’t old age pensions), they did that 2FA thing where they text you a security code. It was an *eight-digit* code. Maybe they figure six digits is too easy to guess in ten minutes?

[ShadowSystems]

At Ace2, It gets worse...

I have a FeaturePhone, not a SmartPhone. When I tell a site to send me a text message with my 2FA code, I expect it to send a text message with a plain text code to let me log in. Instead some of them insist on sending a theoreticly clickable hyperlink that I'm supposed to use to complete the 2FA step.

Can you spot the problem? =-/

[ShadowDragon8685]

Re: Passwords

Wow, that's... Wow.

Even positing spacefuture science-fiction alien computers in orbit attempting to brute force it, and even positing that their systems AREN'T rigged to lock out and re-issue the second-factor code after two or three failed tries, at that point I think the processing power of THEIR systems is going to bottleneck a brute-force attempt.

[Anonymous Coward]

Re: Passwords

Pro-tip: Make your users use only unicode characters outside the first page. It'll be 256x more secure.

Bonus pro-tip: Make the password a prime number of characters long. Everyone else uses 6,8,12 etc

[Anonymous Coward]

Re: Passwords

I reckon they must have been responsible for the payroll system at a local authority for which I once worked.

First brainfart in the design (um, that kind of assumes a brain was involved, which is debatable) was that, being payroll, most of us only logged in once a month to download our payslip but the password expired after 30 days.

The other, more concerning, thing was that, not only did it not allow any of the previous n passwords to be reused (which is easy to defeat with an incrementing number) but it wouldn't allow a new password to be too similar to any of the previous n ones, which suggested that it was probably either storing them in plain text or using reversible encryption (which is, let's face it, basically the same thing) so it could check to see if a chunk of the password longer than x characters had been reused.

My solution to both: use the month and year plus a couple of extra characters to make it long enough with a memorable sequence of pressing shift or not to introduce the required upper-case and non-alpha characters.

[AlexG_UK]

Re: Passwords

Ahh yeah .. I used to work for a small UK software house which was taken over by a slightly larger US software maker who suffered from delusions of grandeur (or possibly even delusions of adequacy).

Anyway our new us lords and masters introduced an upgraded security led password policy - a creative mix of numbers, letters, upcase but no 'special' characters needed, changed every month, can't be too similar to any of your previous 14 passwords. Hmm just over a years worth of passwords.

So what was to be done? Well, simples really: January2012, February2012, March2012, ... Janvier2013, Fevrier2013, .. Januar2014 ... And then cycle started again.

[Paul Hovnanian]

Re: Passwords

Tpwisttsotmo1pi!

The password is stuck to the side of the monitor on one post-it!

[JBowler]

Re: Passwords

Indeed; good user passwords are STO, bad user passwords aren't.

A good password is a DVD: dd if=/dev/random of=/dev/dvd bs=1048576 count=4096

[Kayakerdude]

Re: Passwords

More interestingly, the actual phrase as written out and typed out is far more secure from cracking attempts, as it's much longer and has a huge amount more entropy than a shorter set of letters.

If you have keepass installed you can check the amount of entropy bits used when you type the password into the appropriate field when adding a new password entry.

The generally-accepted "best" password generation idea for mortals to use, is to take four words independent of each other. XKCD has a good pointer on this if you google for "correct horse battery staple"

[Anonymous Coward]

Re: Passwords

Cybersec manager here

I actually requested and had written into policy that our passwords can be 1 change per year.

IT refuse to do it and we're still doing 30 day resets with reminders 2 weeks prior..so you only get about 2 weeks before "you're password will expire.." starts to appear.

I've done an audit of passwords, they are all trash and those who use multiple domains have them set the same hash on each one - so likely have the same password over multiple accounts as a result.

[Anonymous Coward]

It’s not “IT”” that force that on anyone (not unless they’re clueless or read too much Gartner) - it’s clueless IT auditors. I’ve had plenty on run-ins with the likes of PWC claiming that we’re not enforcing best practice by setting a 30 day maximum password age.

After having to argue it for so long, it’s great that NCSC now have explicit advice on this: https://www.ncsc.gov.uk/blog-post/problems-forcing-regular-password-expiry

[Terry 6]

Best example for this is probably primary schools.

After several weeks of the premises being closed staff arrive in, desperate to log in to the often limited number of staff PCs and read the essential(?) staff emails, get their planning printed off, the materials they need and so forth. Often with very little time for everyone to do this before having to troop in to some damn fool compulsory training in the Latest Best Thing (tm).

But half of them will have forgotten their passwords, due to the complexity requirements. Even if they remember them, because it's on a post-it note on their desk or they're allowed to choose their own (simple) ones*, all the passwords will have expired. So, at best, they are forced to set a new one after logging in- and under pressure it's not going to be a well thought out one. At worst, there's some IT guy in the Local Authority or the sub-contractor who will suddenly have dozens ( or hundreds) of frantic calls as soon as they open their phone lines (in my experience, up to 90 minutes after the teachers have tried to start work). Or there will be an Authorised Person in the school who can set up new passwords. But no one has allocated them the time to change 15 or 20 passwords, so at best it's going to be a rush job with everyone set to the same p/w pre-emptively..

*Often they learn quickly to always use the same p/w with an incremented digit.(Password1, Password2, etc).

[herman]

Working at a foreign defence company, I was mildly amused when I eventually reached P@$$w0rd256!

[Turn It Off And On...]

Age of IT Auditors

IT Auditors - get younger ever year - the last two (independent auditors) who turned up at my last place of employment were about 13 years old...

[Anonymous Coward]

or NIST https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-63b.pdf

which says amongst other useful things

"Verifiers SHOULD NOT impose other composition rules (e.g., requiring mixtures of different character types or prohibiting consecutively repeated characters) for memorized secrets. Verifiers SHOULD NOT require memorized secrets to be changed arbitrarily (e.g., periodically). However, verifiers SHALL force a change if there is evidence of compromise of the authenticator."

and "Biometric characteristics do not constitute secrets" (aka faceid is dumb way to protect things you value)

and "security questions" are the dumbest thing of all, please everyone - stop implementing them.

[Anonymous Coward]

I had done the same thing - it was a small defence contractor shop and I always insisted that the admin passwords where put in an envelope, signed over the join with tape over the top and then inside another envelope with the top-bosses signature on the join. then placed into the bosses fire safe.

Ah those were the days - removing all hard disks every night and putting them into the various fire safes, clear desks, and two networks: Secure and Internet with everyone having 2 PCs with KVM switches to move between the two. Mind you trying to authorise various MS licences offline (using a telephone) took forever. Always wondered how these phone-home software packages will work on a secure network (i.e. disconnected from the Internet) nowadays.

[diguz]

"Always wondered how these phone-home software packages will work on a secure network (i.e. disconnected from the Internet) nowadays."

really simple: DON'T use products that phone home. They are insecure by design if they phone home. I never saw libreoffice complain about being on an air-gapped machine.

[NoneSuch]

"really simple: DON'T use products that phone home. They are insecure by design if they phone home. I never saw libreoffice complain about being on an air-gapped machine."

Start by using Linux and go from there. That runs Lire Office nicely.

[plrndl]

Ultimately, all usable IT products are "insecure by design". The trade-off between security and usability is one of the biggest problems our business faces.

The fact that most users and decision makers have zero IT training massively exacerbates this problem.

TGIF.

[Jou (Mxyzptlk)]

Oh no, that trade off does not exist. Bad usablity and insecurity do not exclude each other. In quite a few cases bad usability is the insecurity, so I'd say it is rather the other way around!

[Sequin]

We stored important stuff in a fire safe. We had a fire one night and the safe did it's job. Unfortunately the key was left in a desk drawer and melted in the heat!

[lglethal]

Forget the turtles...

See it just goes to show, you needed a safe to store the key for the fire safe...

Hmmm, but then you would probably need a key for the safe that holds the key for the fire safe. And then another safe for that key, and .... well it's safes the whole way down!

[A.P. Veening]

Re: Forget the turtles...

Hmmm, but then you would probably need a key for the safe that holds the key for the fire safe. And then another safe for that key, and .... well it's safes the whole way down!

And it goes down quickly with the weight of all those safes, rather an unsafe situation ;)

[Andy A]

Re: Forget the turtles...

Long ago, the company I worked for took over the workload of a company which made firesafes. They showed us a safe which had been on the 3rd floor of a building, and was extracted from the sub-basement following a fire.

They had made a replacement key to gain access. The tapes inside read perfectly.

[WorsleyNick]

Re: Forget the turtles...

Long ago (1970's) I worked for a company that stored all its data in a fireproof safe on the top floor of its warehouse above three levels of cold storage basement. I never did manage to convince them that while tapes might survive after a building collapse, 12 platter discs would not. We had no tape drives, therefore backup only comprised of a 3day cycle of every 12 platter disk on 12 platter disks! Fortunately we never had fire.

[Doctor Syntax]

Re: Forget the turtles...

No problem. You use a combination safe instead. Then all you have to do is write down the combination and keep it secure somewhere.

[A.P. Veening]

Re: Forget the turtles...

No problem. You use a combination safe instead. Then all you have to do is write down the combination and keep it secure somewhere.

Just make sure you write it down on asbestos.

[John Brown (no body)]

Re: Forget the turtles...

"Just make sure you write it down on asbestos."

Fuck no!!!!! H&S would have apoplexy!!

[A.P. Veening]

Re: Forget the turtles...

H&S would have apoplexy!!

Yes, perfect solution to that problem as well

[Toni the terrible]

Re: Forget the turtles...

Store the key and combination off site, in another fire safe if you like

[Hero Protagonist]

Re: Forget the turtles...

Well obviously you store the key for the second safe in the first safe — simples!

(Hmm, why do I have this nagging feeling I’ve overlooked something? Oh well, I’m sure it’s not important )

[Just A Quick Comment]

Re: Forget the turtles...

Hang on a minute guys! This is an IT forum yes? Then how about a recursive key? For impressive-ness this could be stored in a recursive fire safe, and the only thing it would be at risk from is dodgy algorithms.

[Doctor Syntax]

Re: Forget the turtles...

Or running out of memory.

[John Brown (no body)]

Re: Forget the turtles...

You don't need a key, recursive or otherwise. Just wrap the safe in blockchain and tie it with a Gordian Knot.

[Snapper]

Re: Forget the turtles...

You must be in middle-management!

[swm]

Re: Forget the turtles...

Why not store the key to the safe in the same safe as the passwords?

[EVP]

Re: Forget the turtles...

1. Place fire safe in a regular safe.

2. Place fire safe key in a fire resistant box (without lock) in the regular safe as well.

3. The place burns down and melts down the regular safe.

4. When things have cooled down, recover the fire safe key.

5. Open the fire safe.

[hoola]

Re: Forget the turtles...

Anyone remember Flanders and Swann, The Gas Man Cometh...

We had 2 LPs when I was a kid, "At The Drop Of A Hat" and "At The Drop Of Another Hat". if I have recalled it correctly.

[AlexG_UK]

Re: Forget the turtles...

Oh yes.. I used to love listening to their records (and they were records) when I was a child. I found a remastered boxset on CD a few years ago - still just as clever and witty but missing the charm of the hissing and popping of the vinyl!

[Toni the terrible]

Re: Forget the turtles...

or Bernard Cribbins 'Hole in the Ground'

[ShadowSystems]

At Sequin, re: fire safe.

My BioDad used to work for the Strategic Air Command (SAC) out at McClellen AFB in Sacramento, California before it got closed down. I remember him coming home one time laughing like a tickled kid. I asked him what was so funny.

"My boss had a fire safe. He lost the key. They used a welding rig to burn the safe open. They ended up burning everything inside the fire safe when the safe itself caught fire. Someone didn't consider that most fire safes are only rated for a few hundred degrees for perhaps an hour, not a few THOUSAND degrees lancing through the lock!"

Even my ~10 year old mind understood the FacePalmDoh! moment that must've been. =-J

[Doctor Syntax]

Re: At Sequin, re: fire safe.

Back in the Troubles one of the buildings burned down in Belfast had a safe drop from the fourth floor or thereabouts. After the fire was out it was discovered it was a little bent and the lock couldn't be opened. Although the maker's locksmith was sent for a PHB decided he couldn't wait & had someone cut it open. The contents were undamaged except for a few items affected by the cutting open. I believe it became a favourite story of the maker's sales staff.

[John Brown (no body)]

Re: At Sequin, re: fire safe.

Yeah, it depends on the size and build materials. A small "fire proof" wall safe won't handle the heat dissipation of a cutting torch as well as a floor standing safe big enough to hide a body or two in.

[Sgt_Oddball]

Re: At Sequin, re: fire safe.

Fire safes should be opened with a concrete cutter round the sides. Much less hassle and unlikely to ignite the contents.

[Martin-73]

Re: At Sequin, re: fire safe.

yes, see most recent video about fire safes from Lockpickinglawyer, very informative

[Anonymous Coward]

same I used to work for a small defense contractor, All PW's stored in Keypass and then printed out and put in the companies firesafe

[Strahd Ivarius]

but where was stored the master password for your Keypass?

[W.S.Gosset]

See above (recursively).

[Anonymous Coward]

> Ah those were the days - removing all hard disks every night and putting them into the various fire safes, clear desks, and two networks: Secure and Internet with everyone having 2 PCs with KVM switches to move between the two.

Very similar at one of my former jobs, except that we had PCs with fancy removable HDD caddies. So you only had one PC and you had to attach either the blue network cable or the red one depending on whether you were booting up the Unsecure or Secret drive. (It checked to make sure it was connecting to the expected network. Though there *may* have been ways around this.)

[Anonymous Coward]

Mentioned it before, I used to work at a small company, and sort of did the IT on the side. There was another chap who also did IT on the side, we sort of split it 50/50. He did emails, I did network etc etc

Big wigs decided one day that they wanted a passwords in an envelope - I was too busy at the time to get round to it, but my colleague did. Couple of weeks later, big company meeting - management buyout, everything will be fine. Few days later, another meeting - a third of staff where to be made redundant.

My colleague found out after the second meeting that he couldn't access, but I could access mine!

Was still made redundant though......

[Doctor Syntax]

"Couple of weeks later, big company meeting - management buyout, everything will be fine."

That should be a trigger to change all the passwords. Not so you'll access the systems after you've been made redundant but just to make it clear that making IT staff redundant isn't the best idea.

[ShadowDragon8685]

Probably best to deposit that envelope with a lawyer or in a safe-deposit box. Better the lawyer than the safe-deposit box, because you can (with documentation) convince a lawyer that your company is still your company and has the rights and need for that envelope after two mergers, a split and a rebranding, while if you've done all that and lost the safe-deposit box the bank is liable to tell you you're SOL and you'll have to... Engage a lawyer to sue them to get the contents of the box anyway.

[Bigkahoona]

Of course this is the only proper way to do it. Print-out of master PW list stored in the tape safe where such a thing still exists and if not there needs to be a dedicated safe for such critical information.

What if the head admin gets hit by a bus while carrying his laptop holding the only copy of the master PW KeePass database and both get smashed to bits?

[J. Cook]

oof.

First thing we did with our Office 249 account was to assign two people the Global Admin role, and then created a 'break glass" account that was also GA, with the password stored in the company's on-prem password vault.

[pirxhh]

Better yet: Hand two or three people a chit labeled "Part 1", 2 etc.

Have each of them select a few random characters, write on said paper, and place all of them in an envelope. Seal it.

Run the password change process with each of them typing their part.

You now have a pretty secure password that is written down - and nobody knows all of it.

[Cheshire Cat]

And then somebody mandates MFA on the whole tenancy, and adds their personal phone as the MFA on the break-glass account ... been there done that.

Also note that spiffy Microsoft Authenticator (as opposed to boring old Google Authenticator) doesnt allow 2 devices to be set up with the same QR code and ties it to one only, so you can only save the QR code image in your keepass along with the break-glass account if you're using Google Auth.

[b0llchit]

Miracle workers

It is like giving a book to a future tablet user. There are no buttons and you actually need to turn the page manually. I'd gladly take £1000 for turning the page once without them seeing the trick.

[Mongrel]

Re: Miracle workers

Books?! We should never have given up scrolls!

https://www.youtube.com/watch?v=pQHX-SjgQvQ

[big_D]

Re: Miracle workers

Best uses for an iPad...

https://www.youtube.com/watch?v=nPGY2T9r1Ok

[John Brown (no body)]

Re: Miracle workers

Video unavailable

The uploader has not made this video available in your country

(No, the downvote wasn't me)

[Martin-73]

Re: Miracle workers

Quite amusing, used a vpn to popup in germany, worth doing if you have access :)

[Phrontis]

Re: Miracle workers

Yes I tried the UK then the US and finally Germany at which point it came up.

[big_D]

Re: Miracle workers

Yes, I had that with a clip of an American film the other day, Tele-München had claimed copyright on it in Germany and blocked it... :-(

[Martin-73]

Re: Miracle workers

Amazing for an IT site that 2 people downvoted you presumably because (1) they are unable to work out how to use a VPN (it's not obvious that the video won't be available in all countries if you're not in one of them) or (2) like apple devices (shudder) or (3) both.

Anyway re: icon, (4), have a 568ml or thereabouts of your tipple of choice (and an upvote) for bringing a smile to at least my face :)

[Anonymous South African Coward]

Re: Miracle workers

I miss Magnetic Scrolls. (the adventure game company).

Luckily they're back as Strand Games. https://strandgames.com/

[Wyrdness]

Re: Miracle workers

We gave up on scrolls when Apple invented the book.

https://www.youtube.com/watch?v=-1IdJDscYHk

[A.P. Veening]

Re: Miracle workers

We gave up on scrolls when Apple invented the book.

There was a time when "book" was synonymous with "scroll" and a scroll is till more efficient for handwriting. However, by now everybody is used to the codex form of books, which really took flight with printing.

[John Brown (no body)]

Re: Miracle workers

Well, you could always write your award winning novel on a scroll for writing convenience then cut it into convenient sized pieces and glue them together into a binding for reading convenience. Best of both worlds.

[A.P. Veening]

Re: Miracle workers

With my handwriting that is not really a possibility. Besides that, there is that minor inconvenience of only one side being available for writing on.

[Andy A]

Re: Miracle workers

The Scroll was the paradigm used in the computing world for nearly 40 years.

Some of us remember when the scrollbar showed you:

- that there was more content than that currently in view

- the relative position of the current view within the whole

- the proportion of the whole which was in view.

Current offerings have the functionality of the scrollbar castrated, or the whole scrollbar hidden away.

[Will Godfrey]

Re: Miracle workers

Not here they don't. Full bar availability with its length scaled to indicate the size of the document

[Andy A]

Re: Miracle workers

But that scrollbar is subject to the fashion choices made by the people who supply your browser.

Look what happened in Windows 10 - the Start Menu has a scrollbar which is invisible unless you manage to hover over a vertical strip one pixel wide. It then "appears", almost the exact shade of grey as the background of the menu. The control section of the scrollbar has a colour choice almost identical. That means that the "stylists" (we surely couldn't promote them as "designers") have decided to hide important things from us.

We can never forgive them.

[Toni the terrible]

Re: Miracle workers

Dont forget cuniform on clay tablets

[Evil Auditor]

Re: Miracle workers

The value of the content of that book, or rather its next page, that is what you should charge for turning a page. £1000 was actually rather cheap for what Dan did. The question is, what was it worth the client to get admin access? Probably much more than a meagre £1000.

[Test Man]

This article is briiliant! :D

What I want to know is did the bloke state he "fixed" it as soon as he did so, or wait a few hours? Was he even getting paid an hourly rate?

[b0llchit]

All just under an hour. I'm guessing the briefing was at least 45 minutes, walking to the server room took 10 minutes and the fix was 1 minute.

[Admiral Grace Hopper]

There is long established precedent in the industry for this. I was convinced that the two-day very expensive performance uplift to ICL 2900 series mainframes was 1 3/4 days of the white-coated technician eating his sarnies, half an hour taking the side panel off, 10 seconds with the tin snips to cut the wire to the performance-slugging circuit, 10 minutes to put the panel back on and half an hour to do the paperwork.

[Doctor Syntax]

If there was a quality management system in place swap the eating sandwiches and doing paperwork times.

[jgard]

I once worked for a medium sized IT services company. One of the sales people was a solid gold twat who just happened to be a director’s son. Eager to wring extra money out of a poor and unsuspecting customer, he sold them a ‘mandatory’ pbx upgrade. With on-site engineer services, ‘hardware’, licenses etc the total came out at about £15k.

The ‘work’ consisted of a guy walking in with cardboard boxes, turning the phones off and eating his sandwiches in the comms room. After a few hours of snoozing and YouTube he turned the phones on, put a different plastic cover on the pbx, then went home. Disgraceful.

That sales guy was such a prick, I very much doubt this was the only time it happened.

[My-Handle]

A dangerous game, as I'm pretty sure that's actually fraud. Had one of his customers / victims caught on, both he and the company could have landed in some very hot water.

[jgard]

I wish he had, and if I had known at the time, I would have grassed him up.

He was such a jumped up, self important cock. My dad, a traditional self-employed working class builder, once did some work in our offices. One morning I was chatting to my old man next to his van, it was parked near the building for offloading gear etc.

All of sudden, the 21 year old salesman with impossibly white teeth (now a director himself) marches over like he owns the earth, his fake tan getting redder with each stride. Then, when he can’t hold the anger and sense of injustice in any longer, he starts barking at my dad like he’s a 4 year old, right up in his face, prodding him too. His speech included beauties like ‘this space is for directors only’, ‘what’s your name? I’m gonna talk to your manager’. I just stayed silent and watched with growing apprehension knowing that a suitable response was likely just seconds away.

My dad quietly let him finish, then grabbed him by the shirt collar and let go the most terrifying, yet articulate tirade of profanity and rage I’ve ever seen close up. It truly was like watching a wildlife show where an alpha male destroys a would be challenger. The kid’s whole presence changed immediately, shoulders rounded, head bowed, his face doe eyed and submissive. My dad made sure he understood that he would need new veneers if he ever even spoke to him again at all. He was then made to apologise in his weak and trembling voice, in front the of hushed mini crowd that had assembled. It was as one of the best things I ever witnessed, and it taught the kid a lesson, never saw him behave like that again. I was so proud of my old man!

The big boss, the lad’s father (a decent guy) heard about it shortly after and apologised profusely to my dad, he was very sincere. He also thanked my dad for giving his lad a life lesson. A much better lesson would be to stop giving his kid BMW M3s and 850is, and make him stand on his own two feet instead.

[Doctor Syntax]

For your dad.

[jgard]

Cheers fella! I’m seeing him in about an hour. I can’t wait to pop a pint down and tell him:

“Well dad, it’s from a person you’ve never met, on a website you’ll never read, offered as a token of respect and appreciation for that time you served shiny faced Jonny his own arse on a plate in the company car park.”

I can guarantee that will make him chuckle!

[Anonymous Coward]

snips, we don't need no snips

That story was factually incorrect. The work would have been performed by a level 2 engineer who would have loaded a new CPU micro program from tape. There would also have been a requirement to check the microcode levels of all the associated disk and tape controllers to ensure they were at the same level as the new CPU microcode. It may also have been necessary to run the ATS (Ashton Test Suite, named after the factory) to confirm that the system was functioning correctly. I imagine that this would would have been scheduled over a weekend and would have allowed at lead a couple of hours in a local hostelry , probably while ATS was running.I never once saw an engineer with sandwiches, they lived entirely on liquid diets.

[EddieC]

Re: snips, we don't need no snips

As I recall, the hardware upgrade involved removing a jumper from a board rather than cutting a wire, and the associated Upgrade Part consisted of the empty foam-lined cardboard box that the removed jumper could be put into before being returned to stores. Saw one of those being done back in the day.

[Anonymous South African Coward]

Re: snips, we don't need no snips

...and the associated Upgrade Part consisted of the empty foam-lined cardboard box that the removed jumper could be put into...

That sounds like a Monty Python skit....

[Antron Argaiv]

Re: snips, we don't need no snips

Sounds more like an IBM thing. Same hardware, two different performances (and rental rates), depending on whether the jumper was installed or not.

Other computer companies (I worked for Data General) did similar things. Look for the chips covered in epoxy. Those are the microcode ROMs which define your instruction set. Change those and you get scientific hardware multiply and divide instead of commercial instruction set.

[Plest]

Re: snips, we don't need no snips

"consisted of the empty foam-lined cardboard box"

Didn't you just love IBM's total disregard of the environment pre-2000?

First time we got brand new IBM RS6000 and some PC desktops back around 1992, we got so many boxes of kit and manuals. I remember though, several boxes were empty, so I asked a colleague why. He said, check again and so I did.

There are in the cardboard box was a slither of paper with "This is a notice to declare this and all unwanted items should be disposed of after installation."!!! Yep, a box with a piece of paper standing the f**king, bleedin' obvious! You wonder why polar bears are starving at the pole with when kind of utter bollocks used to go on.

[Admiral Grace Hopper]

Re: snips, we don't need no snips

ATS is still in the codebase (as I suspect you know).

I had the joy of experiencing ICL from both sides of the customer - supplier relationship. Both generated cynicism of different aspects.

[Andy A]

Re: snips, we don't need no snips

We suggested fitting a coin-in-the-slot device.

Operators wanting to head off early might contribute a few extra machine cycles a second.

[GruntyMcPugh]

I was once an Operator on a McDonnel Douglas 'Reality' midrange system. We paid for a memory upgrade, and all the technician did, was remove a jumper to enable the memory that was already in there.

[F. Frederick Skitty]

Same thing with my then employer's DEC Alpha server in 1997 or thereabouts. DEC engineer simply cut a couple of traces to enable the already fitted RAM when we paid for a memory upgrade.

[Sequin]

An ICL enhgineer once doubled thhe capacity of the disk drive in our mainframe by removing a plastic peg that physically prevented the heads from moving to the outer part of the disk. This cost us a lot of money!

[Strahd Ivarius]

Did the technician go to for work Tesla latter on?

[Andy A]

The place where I first worked had an ICL 1901T with 24K words of core.

There were physically 32K words, but they wouldn't pay for the extra. Snip!

[mdubash]

Wasn't there a (maybe) apocryphal story about an IBM storage tech doing just that, only with a hard disk? And when I say hard disk, I don't mean one of the piddling little 16TB things we see in servers these days but a proper, two-person-lifter jobbie containing a humongous 5MB...

[TonyJ]

What about other systems? I doubt everything was AD integrated and that it was all left unlocked ready to go.

"...

I used to be the IT Manager for my company many years ago. I insisted that all admin passwords had to be printed off, placed in a sealed envelope and securely stored. Got taken out for a beer a couple of years afterwards by my successor - that strategy had saved his bacon when someone responsible for managing a critical system left the company without doing a proper hand-over..."

There should always be at least one break glass account per system where the accounts are stored offline, in a secure place (fire safe, for example, or some equivalent even if it's digital).

[PM from Hell]

When tech support manager I always insisted on this. Imagine my embarrassment when after having had to retrieve the password for a mainframe system to heck up on dome config info, when my sysadmin was away for 2 weeks I left it in a shirt pocket and it was then washed. I had a very nervous 10 days hoping that there was no need for admin access to the system. All credit to him that there was no need for any system privilege access for the whole period. No Disks filled, nothing expired we didn't even need to create any new users while he was away.

[Anonymous South African Coward]

I once deleted current backups for one of our VM's whose host had a disk failure in a RAID...

...sweated bullets until I managed to have a Good Backup, thereafter I made a second backup to another location... and relaxed.

Just say that it was a brainfart and leave it at that.

Need one of these --->

[bofh1961]

Shades of The Lone Jedi...

For those of us of a certain age!

[Blacklight]

Does your CMDB extend to password stores & credit cards?

Similar thing at "an/other" financial organisation, where Azure was setup with a credit card. A personal one. Something got productionised, and then the card holder left the company. Some payment reminders presumably went to an inbox which was no longer serviced (or more likely in existence).

Not long after, an Azure subscription magically vanished. Which was nice.

Password storage (actually secure) *was* a thing. Checking how it was funded, less so....

[Furtive Lurker]

don't spoil the magic

Well, that was very badly managed. You NEVER do a two minute job in two minutes! You MUST stretch it out to at least an hour or the customer won't believe they're getting value for money. You are the expert with years of experience, training, knowledge and wisdom. Don't ruin the illusion.

[tinman]

Re: don't spoil the magic

As Scotty taught us…

https://youtu.be/8xRqXYsksFg

[Doctor Syntax]

Re: don't spoil the magic

That depends on whether you want a reputation as a hero or magician.

[b0llchit]

Re: don't spoil the magic

I'd settle for God status.

[DJO]

Re: don't spoil the magic

..whether you want a reputation as a hero or magician.

Which pays more?

[Anonymous Custard]

Re: don't spoil the magic

If you're doing it right, the two are not mutually exclusive either...

[Hero Protagonist]

Re: don't spoil the magic

I think heros are expected to perform their feats for no pay — that’s why Superman needed his side hustle at the Daily Planet.

[My-Handle]

Re: don't spoil the magic

I think after several days being stuck with an unsolvable problem, the customer would pay handsomely for one of your farts if it looked like it fixed things, regardless of how short a time it took.

[Contrex]

Re: don't spoil the magic

I learned to stretch out jobs when I was a self-employed TV/general electrical repairer in the 1980s. If you took 5 minutes they didn't want to pay the pre-agreed rate. "You didn't do much!". I used to get fed up with people I hardly knew demanding pals' rates, or offering a 'packet of fags' or a (seldom forthcoming) 'pint'.

[Doctor Syntax]

Re: don't spoil the magic

You should have presented an itemised invoice including the very expensive part you fitted.

[Emir Al Weeq]

Re: don't spoil the magic

Eg, to use the plumber comment at the start of article...

Replacement washer: 1p

Knowing which one to replace: £199.99

[GlenP]

A former employer allowed their drawing office to source new CAD systems without consulting IT (me!) Fine, they were NT-4 boxes but this was prior to AD so it didn't really make a lot of difference. My only involvement was dragging the delivery van out of the mud after they'd decided to drive up to a window and pass the boxes through that rather than take them 30 feet further through reception (I did have a Range Rover at the time - see the fuel bills!)

Several months after they'd decided they didn't need an IT manager and made me redundant I had a call, "Do you know the admin password for the CAD machines?"

I took great delight in pointing out that they hadn't involved IT in the purchase/installation and hadn't given us the passwords therefore I couldn't help. My suspicion is that the vendor had never actually told anyone what the passwords were but that wasn't my problem.

At my current employers there is an encrypted spreadsheet on a backed up server folder with two senior people having the key to decrypt it, just in case.

[tip pc]

At my current employers there is an encrypted spreadsheet on a backed up server folder with two senior people having the key to decrypt it, just in case.

that is a great way of storing those passwords,

How often are those passwords rotated or even checked?

Does the encryption get redone when one of those seniors leaves?

Do the seniors know where they have stored their password and do they know how to retrieve in the event ion an issue, is that tested?

Not raining on your parade just wondering how far down that rabbit whole I'd need to go if I did the same.

[GlenP]

To answer, for info,

Every few months or if something significant changes.

Hasn't happened yet but it will if/when one leaves and a substitute is appointed

I probably should audit them on it but they're both pretty reliable on these matters.

[A.P. Veening]

Assuming the password of either one is enough, how often do they fly together?

[Outski]

Quite. When the boutique consultancy I worked for many moons ago was responsible for a 12k seat Domino estate (later a 26k seat Exchange estate), the customer insisted my boss and I were never to travel together, by any means. Mostly ok, since I'd by that time moved to Malaysia to set up our 24/7 team.

[Terry 6]

This was my thought. The level of attrition on those two back-ups could be pretty high.

Not just personnel changes, but loss of the access password, maybe forgetting they even had one.

It's not hard to imagine that one senior person might leave, go doolally or die and the other totally forget where the password was. Or that the device with the stored spreadsheet document becomes degraded by age, accidents or whatever. These may be small risks in the short term, but over a few years..

[Outski]

At my current employers there is an encrypted spreadsheet on a backed up server folder with two senior people having the key to decrypt it, just in case.

And what happens when that server fails?

[GlenP]

And what happens when that server fails?

The support people bring it back up on the DR server, either in-house in a few minutes or off-site in 30 minutes. The joys of virtualisation!

[Outski]

And how often is that tested?

[Strahd Ivarius]

Last time they tried they needed the password stored in the file for restoring the server...

[John Brown (no body)]

Every week. But no one is tasked with checking the fuel levels in the backup genny at the DR site.

[IvanV]

Well, sometimes knowing how to fix something in one minute is a results of 20 years experience. So, charging for one minute of your time should factor your experience too. That's why I won't feel that bad for overcharging customers.

[Pascal Monett]

As they say, it's $50 for the hammer, and $400 for knowing where to use it.

[Outski]

I wouldn't call that overcharging.

[John Brown (no body)]

Yep. Did a job a couple of months back. Got sent in with minimal info on the client site. PCs would randomly work/not work on the LAN, but all had internet access. Who got on the working LAN and who didn't was random and changed each time machines were booted. In the back of my mind, I'm thing DHCP. Sure enough, when I got there, there were two LANs on different subnets and two DHCP servers. The "emergency" backup provision had kicked in at some stage and not reverted when the main provision came back up. The emergency provision was bare bones hence the lack of connectivity internally and, naturally, had it's own DHCP. And was plugged into the same master switch box as the primary provision. Half an hour to find the kit, 10 seconds to identify and pull the plug. 20 minutes explaining why their failover method was the cause and how to fix it it for the future, half day billable and a grateful customer who will almost certainly call us back to set it all up properly for them.

[herman]

I add my age to the bill - in k$.

[chivo243]

A magician!

Never reveals how his tricks work!! At least to the audience. I always found it best to share these experiences with my colleagues, we all like a good laugh!

[Lazlo Woodbine]

I left a company once because the new management decided I was no longer needed

A couple of months later the manager tracked my down by phone at my new job, "err you wouldn't happen to know the admin password would you, we've lost it."

I did remember it, and I remembered where it was written on a post-it, but I let them squirm for a few days whilst I "tired to remember"

[DJO]

I did remember it

Without a consultancy fee?

Organic memory needs beer* vouchers to refresh.

* Chateauneuf du Pape vouchers if you are working in the city.

[l8gravely]

I still support a fellow OpCo in my $WORK, but they're cheap and lazy and fired all their application support people. They had me on a two hour call recently trying to figure out some workflow issues in their application used by finance. I don't know shit about Java or the business logic, so I was kinda flailing, but much less than they were.

I kept asking them to just go call the guy who did all this setup years ago and now works for another different sister company and pay him a couple of hundred bucks to find and fix the issue.

But no... they're still too cheap to do that.

[MCMLXV]

Downvoted. So the staff at your old work were left hamstrung for a few days while you "tired" to remember. Just to satisfy a grudge. Childish and unprofessional.

[Doctor Syntax]

Life is too short to carry a grudge. We just have to do the best we can.

[spuck]

"Childish and unprofessional" would have been to make mischief with the password. Letting management feel the pain of their poor decisions is Not His Problem.

[Antron Argaiv]

Counterbalanced by the fact that he didn't charge them for the information.

Very professional, in the "we're all in this together" sense.

[doublelayer]

After you get fired, you have no professional duty to keep working. Sabotage would have been both. Not doing free work, however, should be expected.

[David Nash]

He got fired. I would have charged them £100 for the service.

[A.P. Veening]

He got fired. I would have charged them £100 for the service.

I would recommend adding at least one zero (before the decimal point/comma).

[Lazlo Woodbine]

Downvoted for your lack of understanding of how life works.

Why was I the only one who'd noticed the password was on a post-it on top of the server, it wasn't even hidden.

My goodwill only goes so far, if someone other than the manager who'd let me go had phoned I'd have told them straight away...

[Gene Cash]

"Sorry, mate I no longer work for you. You fired me, remember? Now let's talk about the consultancy fees..."

[W.S.Gosset]

"Sure, I can do that! My day rate is ..."

[Anonymous Coward]

"I deleted all of that information when you made me redundant. We can talk about data recovery costs to attempt to recover it if you want"

[Antron Argaiv]

Information all deleted (as is required by my NDA and separation agreement).

[John Brown (no body)]

I wonder how much beer it takes to unremember a password and exactly which type beer to target just the relevant info? And does the company firing you pay for that beer? Especially considering that said company should have changed the password after the firing.

[AlbertH]

I've had exactly that happen to me, but in my case, it was a Civil Service Server in deepest Whitehall...

Bliar's "government" paid a hell of a lot of moolah for my new company to resolve their fundamental IT stupidity. I just saw our very high hourly rate as a generous Tax Rebate!

[Sam not the Viking]

Top Dog

At one company I worked for, we had to use a production system run/supervised by a sister company who seemed to want to prove their superiority. (The system was hopelessly out of date but that's another story). The supervisor would visit, usually to sort out a 'technical glitch' that only he could resolve. It was a clear scam to get a day out at our expense. When we realised that this was going on, we quickly 'discovered' his login details and in doing so obtained much wider access than we anticipated..... We used this access carefully to maintain our rich data source; as well as prevent the 'glitches' recurring.

His password was 'Top Dog' and he used it for everything. He must have puzzled the persistent canine references in the surrounding chatter.

[jgard]

Re: Top Dog

Brilliant! I would not be able to resist mumbling ‘sausages’ in a doggy style voice, followed by light panting, as I walked past him…..

Me: “SROSSAJIESS!”

Him: “What was that?”

Me: “RNNUFFINK!”

[David 132]

Re: Top Dog

> I would not be able to resist mumbling ‘sausages’ in a doggy style voice

“WALLS!”

How many decades on, and that advert still sticks in my memory…

[lglethal]

Re: Top Dog

I sincerely hope the info about the deliberate glitches was communicated to your CEO.

Unless your CEO is already in the pocket of the other company, they would normally at that point be making it very clear to the CEO of the other company that unless they are reimbursed for all of the callouts they will be reporting the "Supervisor" and his firm for fraud to the Police. And that at the next contract negotiation, there would be some rather smaller numbers on the page then exists currently...

It's the only way to deal with scum like that... But it does usually have to come from the Top, to get in anything to actually happen (the other CEO would of course deny any knowledge of such behaviour, would see to it that the Supervisor was sacked or shunted to other projects, and of course they would be happy to compensate the firm for the despicable behaviour of their employee, etc... Another round of golf, perhaps?)

[Sam not the Viking]

Re: Top Dog

The eventual fallout was rather messy...... The sister company had a bigger turnover and much more spending power. They claimed to obtain major discounts from suppliers because of this. When it transpired that everything we bought was in fact cheaper than their 'special discounts' from 'approved suppliers' the scam was wide-open and the dismissals were fast.

Top Dog was neutered.

[Anonymous Coward]

That's around £1,500 ($1,980) in today's money

£1800....

£1900...

£2000....

Cost of living these days....

[Aladdin Sane]

Re: That's around £1,500 ($1,980) in today's money

Doesn't bother me, I only ever put a tenner in anyway.

[AnotherName]

Admin access

In a previous job in the early 2000's, where I was IT Manager (and I was also 50% of the IT team), I was made redundant, partly enabled by underhand reassigning of roles. We had an internal meeting where the organisation chart was displayed. I pointed out that they had me down in the wrong role, but it was brushed over - I should have seen the signs...

Anyway, on the day they called me down to tell me I was redundant, they asked someone else to revoke my admin rights and changed the admin password. I was allowed to go back to my desk for the rest of the day to copy off my personal files from my computer. As I had been logged in the whole time, I still had full admin rights and could have done anything, but obviously didn't do any damage. I did copy off a few customer directories from a server so that I could make direct contact with them after the event and actually did some work for many of them.

A few months later, after the redundancy period was up, I received a call asking for the password for a specific system. I told them what my rates were for consultancy and they never bothered me again. They lost many knowledgeable developers due to the way those of us who were made redundant were treated and eventually went bust.

[A.P. Veening]

Re: Admin access

They lost many knowledgeable developers due to the way those of us who were made redundant were treated and eventually went bust.

Karma is such a lovely bitch ;)

[Doctor Syntax]

Re: Admin access

Karma nothing! More like cause and effect.

[A.P. Veening]

Re: Admin access

"Karma" has always been a one word code for "cause and effect". Also known as "what goes around comes around" and "everybody gets what he/she deserves".

[Antron Argaiv]

Re: Admin access

...though, sometimes, it takes longer than one would hope.

e.g.: the previous occupant of the US Whit House.

[Anonymous Coward]

Re: Admin access

e.g.: the previous current occupant of the US Whit House.

FTFY

[GruntyMcPugh]

Many years ago I got a call from an account I used to work on. They'd had some recommendation to reduce the number of enterprise adminsistrators they had, and use one time access to the built in admin account, using some app they'd been sold. So they removed everyone's admin access, and then realised they didn't know the Admin account password to get the ball rolling. So they called, as me and a couple of colleagues had promoted the first domain controllers back in the day and asked if I knew the Directory Services Restore Mode password,... which, luckily I had ingrained in my memory (and that we had lodged somewhere safe, but our jobs were outsourced, people left offices, safes were disposed of, etc). So they had to roll back their AD to before the fudge-up.

[Robert Carnegie]

I wonder how often the phone call to a recently let go employee saying "Er, can we trouble you for the Very Important Password, that we can't seem to find" is not from (an appropriate person at) the business that let you go at all, which does know the Very Important Password but may have been not sufficiently cognisant that in such a situation it also is Very Important to change it.

[aj68]

Sometimes it just falls into your lap

Only a few weeks after joining a large US RDBMS company in the late 1980s (based in Bracknell), I was told I was needed to visit a large customer site the next day where they were having severe performance problems with their VAX based database. I spent as much time as I could cramming text books and performance tuning notes before being flown up to Manchester and getting a cab to their office, while inwardly panicking the whole time.

I was met by the data manager and shown to a VT terminal logged onto the machine. Within seconds I found their system disk was over 99% full and having a couple of years VAX/VMS system management experience this was a smoking gun with a red hot barrel.

I breathed out, looked up, and thanked the almighty for this "Get out of Jail free card". I cleaned everything up and wrote the client a 4 page report on maintaining their system performance.

I'm told he later waved this in the face of his DEC support team saying this was the level of support he was failing to get from them, and how great we were in comparison. A great start to my new job, but it was a bit lucky!

[Ian Johnston]

A fellow post-graduate student, years ago, was given a very well paid commission to get a piece of software running on the departmental VAX. It came as FORTRAN source code, and although it compiled it didn't do what it should. He looked at the source and spotted lots of lines beginning with "D", which readers will recall are debug lines which are treated by a compiler flag as comments or not.

So he compiled again /DLINES, and everything worked fine. Ten minutes' work, paid for three days.

[Anonymous Coward]

I'm a little bit worried

that only one poster here appears to have heard of a break-glass account.

[A.P. Veening]

Re: I'm a little bit worried

Most of them have heard of it, but maybe not by that exact name.

[Anonymous Coward]

Re: I'm a little bit worried

Most of them have heard of it, but maybe not by that exact name

Oh, I agree. But when I posted there were 53 comments ... assuming 50% of posters know of the concept, I would have expected more than one mention.

Little tip for people who do setup such accounts. Obviously you test them regularly. But make sure you have hard copy backup 2FA codes squirreled away somewhere. And make sure you can access the details offsite, in case your office burns down.

[GlenP]

Re: I'm a little bit worried

In my case it's a GOOJ account, stands for Get Out Of Jail (Free).

Too many years of playing Monopoly as a youth!

[Wellyboot]

Re: I'm a little bit worried

Many years ago we also had a 'terminator' account that when logged in on the 'emergency' workstation would initiate a sequenced script to shutdown everything neatly in the correct order, If it was logged in anywhere else it merely gave you the list it was parsing to allow confirmation nothing was being missed.

The 'emergency' workstation was not far from the safe containing the all the password envelopes though only a few system techies in IT new this part of the process (but only their third of the actual password).

[Anonymous Coward]

Once spent the pre-pizza portion of an evening doing something like this at a company where the admins were getting a bit too up themselves and the manager had me and another friend in to regain control of things. I dealt with the Windows and NetWare boxen, he did the various Unix flavours. I think the admins were read the riot act the next morning and behaved a bit better thereafter.

[Tim99]

Admin access

A long time ago, we had an uppity admin in the public service who told his boss that he was the only one who knew how the main system worked, and that he needed a promotion. There was a very high level meeting where it was decided that this was unacceptable.

I had written said system, was very senior, and was responsible for it throughout the organization. I was asked my opinion. Plans were put in place, and some weeks later I travelled on a Sunday to where the system was. I dumped the data from the files, then reinstalled the system, changed the system password, reassigned the admin to a lower group role, and reloaded the data.

On Monday, everybody in the location was told that their sections were being merged with others, that they should apply for new roles within the revised organization, and that they should continue in their old assignments until further notice. The admin just happened to have started his annual leave on the previous Friday. Funnily enough the admin was the only person who didn’t get a job similar to their previous role. All of the Byzantine rules of dealing with staff were scrupulously followed, and he left later that year.

Yes, if sufficiently provoked, the public service can be at least as bastardly as anywhere…

[Jou (Mxyzptlk)]

Re: Admin access

"he was the only one who knew how the main system worked, and that he needed a promotion"

One should never think of being not replaceable. The cemeteries are full of them.

I am in contact with a few bosses, actually good ones, who had people like that. When they pulled that stunt in front of their boss they were told "OK, go. Leave company car key here, leave your access card, leave the company phone here. Go." - when the response was something along the lines "notice period" or "but I need the car since we are moving on monday" or "but I only have that phone" they continued with "I know, I don't care, leave that stuff here right now on my table and go.". The next step, which I recommend then when I was informed, was to change their password. Of course they paid what they had to by law, according to the notice period and such things, but the sign was clear: Go now.

[aerogems]

Ah the laying of hands

Worked as a contractor providing hardware support for a small to mid-sized University. One day get a ticket that one of the library staff's computer won't boot and is beeping at them. I make my way over to the computer to find someone had left a book sitting on the spacebar, so the BIOS was complaining about a stuck key. Moved the book, rebooted the computer, all was well with the world.

[DS999]

Back in the 90s I was once asked

To help break into a server. I was consulting for the company on a six month contract dealing with migration to new EMC Symmetrix arrays, but my resume did state I knew HP-UX very well. The guy who dealt with that server had left in a pretty acrimonious way - frog marched out by security back before that became the standard way everyone gets laid off - and no one could access it. They told me they'd tried swapping out the boot drive with a similar system but it wouldn't boot, they tried putting its boot drive on another system and modifying /etc/passwd, but it still asked for a password, so they were stumped. They had tried some other stuff too, apparently one then two and eventually their whole Unix server team had taken a crack at it over the course of 3 1/2 days with no luck.

I walked up to the machine which was helpfully already running and sitting the firmware prompt, ran a command to check the boot path and recognized it had mirrored boot drives, so when it tried to boot the default read drive must not have been the one they edited /etc/passwd on. I ran a command to manually boot from the other drive, logged in as root with no password, and told them they had a split mirror in an unknown state because of what they'd done so they need to manually re-mirror to the active drive. Took less than 30 seconds from the time I arrived, I mumbled something about "let me know if you still have issues, gotta get back to what I was doing" as I walked away.

The next morning the guy in charge of all the server and desktop teams (the manager I was working for on this gig was in charge of telco, networking & storage, so I had barely talked to this guy) walks up to me and thanks me for helping out his team. He says "I talked to John [my "manager"] and he said you should fill out a separate timesheet to submit to me for your hours working on this, so he doesn't get billed for it". I was a little confused, and kind of stammered and he said "don't worry, it will be at the same rate". I finally get out "but it only took me a few seconds, do you want me to bill you a full hour for that?" and then he was a little confused and stammered. I explained what I did, and he said his guys had only told him I fixed it for them, not how long it took, and he just assumed it had taken a long time given how much time his team had already spent on it.

I caught up with John later that day and told him about the strange exchange I had. He started laughing when I explained to him how little I'd done to solve their big issue and said not to worry about the other timesheet. I thought about filling out a timesheet for an hour to the other manager anyway just to be a smartass, but he was a pretty nice guy who was just steered wrong by his people so I decided against it.

[Marty McFly]

NSDE solved for $50

Back in my early career days I supported PC-based point-of-sale systems for restaurants. So the customers were all non-technical types. "Brain" was the computer, "TV" was the monitor, "typewriter" was the keyboard, and so forth. Furthermore the restaurant manager types didn't like to be called at the wee hours of the morning when closing up after last-call. Unfortunately the same manager types cringed at any sort support contract, so it was $50 minimum up front with a credit card number.

These were early days where backups were saved to floppy disks. Inevitably someone mopping the floor would bump the power button on the primary workstation. Realizing their mistake they would power it back on....resulting in a dreaded "Non-System Disk Error" and the hunt for the ANY key.

After charging their credit card number.... "Push the button on the front of the brain next to the slot. Now press the space bar on the typewriter. Thank you, bye!"

Of course, for every luscious NSDE call I had, there were ten where the same $50 at 2am had me talking non-techies through re-seating ISA Bus cards to get serial ports working again. It wasn't all easy money.

[John Brown (no body)]

Re: NSDE solved for $50

...and then there'd be the times you had to actually visit site and find the PC full of beer/curry/custard or just years of fat/grease build-up!!!

[JerseyDaveC]

Been there, loved wearing the t-shirt

I had a similar experience - desperate call from a client of a client of a friend whose SQL Server cluster I'd set up but which had died ... when he did a DIY data centre move. This was 1pm on Sunday, and it had to be up by the wee small hours of Monday.

Two-hour drive to London. Server said "No storage connected". Moved the SCSI connectors for the RAID array from the server's SCSI adaptors to the server's RAID adaptors. Booted, Server said: "Ah, I see you've hooked up some storage, but it's not connected properly". Moved SCSI cable from server A to server B, and from server B to server A. Booted. Server said: "Ah, that looks familiar - would you like me to start up?". Told it "Y". Two-hour drive home. Wrote and emailed four-figure bill. Massively relieved client of a client of a friend paid it promptly and was super-grateful, which was lovely.

[DreamEater]

Re: Been there, loved wearing the t-shirt

Having recently gone self employed, I now have a new smile when I read these stories.

We have the original smile from dealing with desperate people because of their desperate actions, then the 2nd smile at being able to fix it and now the new smile "paid it promptly"

Never knew cashflow would give such anxiety.

[zaax]

I bet Dan didn't see much of that thousand

[Anonymous Coward]

Timely public bug

Once upon a time, we were subcontracted to a subcontractor to a Major Defense Agency with Big Computers.

Said MDAwBC had lost the root password -- their policy was change it every 30 days, don't write it down, and don't forget it.

So the subcontractor phoned and asked us for the password (no way, it was 3 months later); did we "accidently' leave a backdoor (no way, the account list was audited as part of acceptance); did we have any fzcking clue how to get them out of a jam????

Well, actually, did you read El Reg that morning?

https://www.theregister.com/2014/09/24/bash_shell_vuln/

They hung up, and never even phoned back with a thank-you.

[raving angry loony]

Wasted opportunity

Anyone THAT stupid and THAT incompetent (and I'm talking about the senior management of the outfit who trashed the original I.T. team, not the current IT staff) should really have had a hefty stupid tax added to their bill. OP could probably have extended their stay a day or two, at least.

I mean, what would the BOFH have done?

[herman]

You’re fff… err hired

I was once fired and then hired back for $10000 to complete what I was working on.

[AlbertH]

Re: You’re fff… err hired

Been there too - except their offer of £45000 was politely declined, and they were forced to pay my bill of £180000..... That set up my new business very nicely!

[vistisen]

I used this hack many years ago when arriving at acustomer where no one knew an admin password!

C:\> cd \winnt\system32

C:\winnt\system32> copy logon.scr logon.scr.old

C:\winnt\system32> del logon.scr

C:\winnt\system32> copy cmd.exe logon.scr

Now log off the machine, logon.scr is the screen saver that will kick in after 15 minutes of not touching the keyboard/mouse at the logon screen. Wait 15-20 minutes and a DOS prompt with FULL SYSTEM rights will pop up, then just to

C:\> net user administrator <newpassword>

and then log in with the new account.

[MJB7]

Re: logon.scr

What OS was that on? Moving things in system32 requires full admin rights already these days (and has done since Windows XP).

[Jou (Mxyzptlk)]

Yep, that is the one for NT. Today you have to use utilman.exe for the same. Saves the waiting.

[Tim99]

I haven’t done this for a while, but assuming that the system can boot from an external device, and that you only want to zero a local admin password the chntpw utility in the SystemRescue distribution worked well, but it won’t work on an encrypted disk…

[Jou (Mxyzptlk)]

Not on domain controllers :D

[Anonymous Coward]

A Timely On-Call

I woke up Saturday morning and discovered I couldn't access my own blog (or any other sites I host) - either view it or access the Wordpress dashboard. I was getting a timeout. I assumed it was a server outage.

This was overshadowed ever so slightly by the paranoid thought that my posting of the Arnold Schwarzenegger address to the Russian people concerning Ukraine, and several comments about Putin might have upset someone, even though Russia is blocked due to repeated attempts at hacking over the years from there.

Then I realised my traffic was still there, just not from my home network IP address.

I later discovered I could access it normally if I used VPN, but again not from my home IP address. I also realised that Uptime Robot had stopped logging at 1.30am Saturday morning, so it can't get access either.

I did all the usual stuff - power cycling everything, flushing and renewing DNS, and so on. My IP is whitelisted in my security software, and since I could log in via VPN I obviously wasn't blocked as a user/admin in Wordpress.

Having started wondering where the hell to start, I concluded it just had to be a server/host issue. IONOS have pretty much (i.e. definitely) admitted they have a situation, but they haven't bloody fixed it. And tonight, after being on hold for ten minutes while the agent had a word with the backroom, I was cut off (end of working day, I guess, but they won't admit to that when I call tomorrow), the problem persists.

[Anonymous Coward]

Re: A Timely On-Call

And after almost two full days, it was apparently fixed just before midday today.

So if you do get ERR_CONNECTION_TIMED_OUT - apart from all the usual stuff which blames things your end and questions your parentage - don't rule out your ISP or web host as the real source of the problem.

I'm now questioning IONOS's parentage, believe me.

[Rockets]

Been In The Same Boat

I was once sent out to a new client because the company had sacked their previous admin but didn't have the admin password. It was a small shop and this was back in the NT 4 days and the client was running SBS 4.5 server. Before I cracked open the "server" to attach the servers hard drive to another NT4 machine so I could copy off the SAM and run lohptcrack against it, I tried connecting to MS SQL. To my astonishment I connected to MS SQL as sa with a blank password. As SQL 7 was running as the system account on the PDC a quick couple of sql statements & I had a new account with domain admin rights on the network and was in after only being on site for 15 minutes.

[Robert Carnegie]

Re: Been In The Same Boat

It sounds like they should re-hire the former admin so as to sack them all over again.

[Anonymous Coward]

Disloyalty begets disloyalty

He did that ALL wrong. A company so morally decrepit and hard-wired-for-stupidity that they laid off their entire IT staff should be made to pay through the nose. I'd have dragged the "account recovery" out over several months. Not because I'm greedy, but because I like to think of myself as an IT-based batman, dispensing karmic justice through my keyboard and my invoicing software.

[venkatarangan]

SQL Server injection to rescue

Once I was involved in a similar situation in a clients place. They had forgotten their AD Admin password on a production web & db cluster. Luckily this was before SQL Server the patches from Microsoft after a well-known attack had been applied. I simply executed an ASP page with 'poison' SQL query to execute, CMD.EXE with a parameter to run "net user username password /add" and then another command to add to the global admin. Voilla.

Then, dutifully I executed the Microsoft patch on all the server for SQL Injection and instutionalised a process with the developers to sanitize all their HTML inputs.

[This is not a drill]

Useless service desk manager.

Sounds similar to a situation I was part of. I was network manager (Novell) for a company. Following best practise the "admin" account was not used, I delegated all rights and had a superuser role for my administration account. The "admin" account had a strong password which had been written down, put in an envelope in a locked box, and stored in the company safe "just in case".

Anyhow the company hired a service desk manager, who insisted the service manager and network manager (me) report to him. I wasn't consulted and this was effectively a demotion so immediately found a new job and resigned.

The service desk manager insisted that the help desk team needed the network "admin" account password, I held my ground and explained that they had all the rights necessary. On my last day I handed everything over, including the admin account and explained to the service desk manager that he should use it to add himself and whoever he wanted superuser role then change the admin password and put it back in the safe.

Rather than doing that he just gave the admin account to all the help desk team, after about a month they had managed to lock the admin account and because he hadn't added anybody to the superuser role, nobody could reset it.

This caused major problems and in the end they had to get Novell in to reset the admin account. Shortly after the helpdesk manager and the IT director were sacked.

[Robert Carnegie]

Re: Useless service desk manager.

Now that is funny.

[Trotts36]

Drag it out

Personally I would of enjoyed a very nice week long exercise at 1k a day to fix this issue.

[firebits]

Good job that Windows 2003 Server was unlocked otherwise, it could have taken a good 30 minutes to get that domain admin account password reset

[Some call me "Tim"]

Lasers in a grill factory

Roughly 18 years ago at my previous job, a customer (large charcoal grill manufacturer) located in Georgia, USA calls to complain that their safety laser area sensor is broken and picking up things that aren't there. The sensor was used to make sure all humans and other impediments were clear of the area before a large welding jig moved at the start of the grill assembly line. I worked for the sensor manufacturer in New Jersey. Grill company is told that they will be responsible for $1000 per day plus expenses if I determine that the problem is on their end and not a failure of our product. The following week, a couple hours on an airplane from Newark to Atlanta, followed by a 2 hour drive in the rental car gets me to the front door. Just inside, 8 managers and maintenance staffers are staring at a laptop showing a realtime 2D map of what the sensor "sees". I watch the laptop and the jig for 3 minutes, then ask one of the "watchers" to place his finger on the laptop screen where the "false" trigger is showing up. 5 minutes later, the line is stopped for the scheduled morning coffee break and I can approach the equipment. I knock some welding slag off the end of the rig with my finger and ask the designated watcher if that matches what the "failure" they witnessed. They had recently changed to cheaper imported steel (creating more slag) and cut back on how often they cleaned the machine, never realizing that a sensitive laser sensor might receive a reflection from shiny falling metal shavings. I politely waited an hour before calling my boss and telling him to get the invoice ready. My return flight wasn't scheduled until the next afternoon, so I enjoyed my dinner and hotel room that night!