Life lesson: Don't delete millions of accounts on the same day you go to the dentist Rise and shine, dear readers, it's Monday and therefore time to sink your teeth into a new week of work, a fact The Register celebrates with a new edition of Who, Me? This is the column in which you share stories of when you bit off more than you can chew, but fumbled through. This week, meet a reader we'll Regomize as "Blair …
Yeah. RTFM. All of it.
And understand it.
At least he was able to recover the issue. Good on him for that. And missing one sentence is an easy thing to do when what you are reading is about as exciting as watching paint dry.
But as an admin, what your doing is important (at least, the consequences may be). So read that manual.
And test it too. Spin up a test version of the system, ideally with a copy of some real data and then implement the change on the test system to see if it works properly. This should be the case for changes on any critical systems really, always have a test environment you can break and rebuild without impacting production.
Yeah. RTFM. All of it. And understand it.
With LDAP? All and Understand is at best a tall order either separately or together. Complex, convoluted and in places just plain daft.
But in this case I guess it was the ticketing system that was at fault as it would appear to support only one authentication service at a time (clearly didn't use pam and nss.)
Presumably the fix was to export the user accounts from the Oracle RDBMS, massage into ldif files and import into ldap or alternatively knock up a gateway from ldap to the existing Oracle system
"I couldn't agree more. But... the warnings come after the spell!"
'Strange' that ... isn't it !!!???
Still not learnt that lesson yet !!!
People tend to stop reading the 'fine manual' when they 'think' they understand enough ... PUT THE WARNINGS AT THE TOP !!!
:)
Good cooking and baking is a form of magic (be aware for the US: The definition is from raw materials, not pre-fabricated cookie-dough, microwave-popcorn, fries, salad via mixing pre-fabbed sauce with convenience-packaged salad).
Really understanding finance and Tax is a form of magic.
Really understanding law is a form of dark magic.
Making it IT centric:
Really understanding Certificate-Authority / PKI (with offline root CA) is a form of magic. I can handle it well, considered as "the man" in a 3000+ IT-service company. And know others which I consider "magicians" on that field.
Really understanding the possibilities if you can program C#/C++/Basic etc, programming thinking without being locked into one language - takes long even if you have a knack for it. I stared with basic 3.5 around 1984 on C16, went to first ASM routines on that machine 1985.
Windows Centric: Really understanding the possibilities of powershell, being able to go beyond the cmdlets if needed, and know the value of it behaving the same since Version 5.1 (Windows 7/Server 2008 R2 up to Server 2025 and possibly beyond). Being able to NOT require modules for most tasks, which some admire. I am deep into it, and know others I consider magicians here. Got forced into it 'round 2021.
etc etc etc.
The ignorant is you, Jake. As long as someone stands by his "I am simply not good" and has good manners I have no problem. (Good manners by German standards - many cultures cannot take typical German directness)
Body paint, you say? My ears prick up, and I launch, Rowley Birkin-style, into one of my anecdotes.
Many years ago (late 90s - and this point should be borne in mind) the large tech company at which I was working had a booth at a tech tradeshow at London's Olympia. Can't remember the name of the show now, but it was a CES-like thing, mostly focused on enterprise IT.
Our booth designers had pushed the boat out, and we had lots of space - and even a second storey, allowing for a bar and seating for the entertainment of "important" sales leads.
One day, we had a sudden stampede to our booth, and up the stairs to the rooftop bar, and it took us a moment to ascertain why this sudden interest in our sales literature.
That moment was a brief one, as we very quickly realized that... the company occupying the booth across the aisle from ours had decided to drum up interest in their products with the very on-message, tactful and respectful use of... two models, young and female, clad in brightly-coloured motorbike leathers. Leathers that on second blushing glance, turned out to be nothing but body paint and the skimpiest of bikinis.
No, I have no idea why body-painted young ladies were considered an appropriate way to advertise mouse-mats or whatever the company was flogging, but as I said, late 90s. Different times, different mores, and anyway, we all wore onions on our belts as was the fashion, et cetera.
Hence the stampede of punters to our booth's upper floor, which just so happened to provide uninterrupted sightlines over the heads of the salivating milling throng, to allow respectful consideration of the body-painter's fine art. Ahem.
"Of course, at the time, I was very, very drunk..."
"The request was then extended to applying LDAP to the customer ticketing system"
if the Ticketing System was home-made, he should've asked the developer(s) on the best way to do it, if it was commercial (as it seems the case, since there was a documentation) he should've asked the company that made that thing and was also providing support (I assume somebody that run a "nation-sized" system does also pay for support for his wares).
So life lesson: ASK THE F*ING QUESTION before "assuming".
"So life lesson: ASK THE F*ING QUESTION before "assuming"."
Sometimes that's the case, but life and most work is based on assumptions and generalisations. As a primitive form of learning they are arguably these are the very foundations of civilisation. Logically, would you expect or even harbour reasonable doubt that implementing better authentication to delete all existing user records? Really? Somebody, somewhere thought "if a sysadmin implements LADP, then the system should delete all user records". Now there's the fuckwit responsible here, not "Blair".
And the author of the shitty manual that doesn't make clear the huge consequences. Hiding something of huge consequence as though it's trivial is a grade A failure.
I suspect that the stupid, stupid idea of deleting all users was only ever tested on a small dev system that it was trivially easy to roll back and forward, and where there were no operational consequences or angry managers.
Ut ullam impedit amet et. Harum voluptatum debitis itaque corrupti libero dolorum. Doloribus nihil sit laborum. Laborum amet est est. Qui quo in voluptas maxime doloremque cumque voluptas. Officia sequi voluptates placeat. Impedit non et minima. Veniam doloremque qui dolorem excepturi autem debitis. Minima voluptatibus magni aperiam temporibus atque ullam iure. Aut dolores unde enim odit cupiditate non numquam et. Praesentium ut veniam quis ut voluptatem inventore at. Voluptate voluptas omnis a qui temporibus. Tempore facilis tenetur consequuntur. Quod voluptates sed inventore blanditiis. Sint voluptatibus optio enim incidunt deleniti nostrum. Molestias corporis et corrupti velit eius natus. Sint delectus rerum praesentium nemo. Et dicta odit dolor. Quos repellendus et saepe consequatur rem molestiae nesciunt. If you switch to ldap all accounts will be completely deleted because this is a feature. Soluta beatae perferendis sed. Est temporibus natus ab odio tenetur nemo alias. Ex sed consectetur possimus quia animi. Doloribus nihil cumque commodi recusandae id dolores quisquam cupididate.
Ha, your combination of subject line and text brings back memories. I wanted to demonstrate a bug in the formatter of an application, so I posted a description of the issue along with a demonstration file containing enough lorem ipsum to trigger the bug. The admin thought I was spamming the system and deleted my account.
Or maybe an "are you sure you want to nuke your user records?" popup? (or y/N prompt in CLI).
They get abused for trivial actions too often, but i feel like the possible consequences here are severe enough for that...
Along with the fact that configuration changes generally shouldn't be deleting user data in the first place. Is it really necessary to delete all user records instead of creating a users_ldap table, or creating a "authentication source" column? oh my, did i just accidentally create a feature to use multiple auth sources simultaneously too??
Which question? “If I sit on this chair, will the table collapse?” Is a silly question, only the most churlish carpenter would blame you for trying and not asking.
There are many unknowable things that are unknowable until you know them but you need to know before you know.
I am currently dealing with a supplier who uses MFA ( the TOTP kind) for me to logon and create certificates for our machines to access our data in their "lake"
The MFA has out of the blue stopped working despite me having a record of the initial secret key .
Their only solution to this is to delete my account and recreate it , which will invalidate the certificates i've crated and validated and signed etc for our servers to talk .
As a password reset solution I find this less than satisfactory.
They are also unable to answer any questions on why this happened , or why they cant just reset the MFA , or how exactly the MFA works as "a third party is providing the MFA"
This is a very large NHS healthcare provider .
sorry for the off topic rant , this is pissing me off no end - the link is "authentication" I guess .
Inexcusable (their excuse, that is). At the very least they should create a new account for you first, then allow you migrate your servers to new certificates, and then delete the old account (which deletes you and invalidates the no-longer-used certificates).
Badly implemented MFA just winds me up.
Is this the thing that lets me login on 3 different apps my employer uses with the same password, but when I change my password, Google pw manager will ask me to type the new password on all of them?
Is this how any of this is supposed to work? For my employer, only one change and all the apps obey the new password, but for me, I have to type it again on every app, it just doesn't read my new credentials and let me through?
Isn't it supposed to skip every login prompt down the road once I'm logged back in with the new pw? What's the point of unifying all the passwords if you have to type them (or Google asks to update it for me) every time?
I would use the Windows user icon, but you know...
TLDR I have a grievance about typing passwords over and over on several places, if they are synchronized. Yeah, token and man-in-the-middle and all that.
Indeed, that is how it is supposed to work. Your case is a google issue or a "the program you to with google" issue though.
If you want the pow² or pow³ version of those problems, go Azure/Entra. Then only, AND ONLY change you password when you are connected via LAN (or a site-to-site VPN but not a VPN client on your computer) to your actual AD-controller. Wait 15 minutes. Relogin to the AD controller (i.e. logoff-logon) with the new password. For the rest of the day you will be busy with typing in the password at numerous places, cleanup windows password store and so on. You can add google on top too, if you have enough time.
Oh, and be aware, you account may be locked a few times during that day since some programs, via Azure/Entra, still use the wrong password for a few hours.
I got lucky so far, my PW changes survived without locking me out, but I know coworkers which get locked out every time they have to change their pasword. And it is not their fault, it is just half-assed systematically crap on all sides - not only MS, all sides.
thanks for the explanation, I ALMOST understand it. Just got done with a day or two of that fun when I had to change my password on my machine, from home.
There seems to be a way to get it to sync everything (desktop and apps etc) but our IT dept haven't been able to write it down. There are myriad ways to initiate a password change, from the desktop ctrl-alt-del menu, to going to myapps.microsoft, and with or without VPN engaged, and I SWEAR I got it to work one time so that the login to my computer was updated along with everything else, but I can't seem to reproduce it. It could also be some misconfiguration by our IT people, they seem to be in the low cost arena.
So I'll have a machine (win10) that I have to use the old password to log in, then everything else uses the new password. Until I go into the office and on powerup the machine will get its update and then everything is wobbly for a bit while the machine and everything microsoft and in the cloud somehow thinks that its out of sync (when it should be in sync.)
There are 2 parts to single sign on:
1. Applications/services authenticate to the same backend. So a single user list etc that all programs consult. LDAP being the legacy way to do this.
2. Applications support reading your current system login credentials and supplying them to the server automatically without prompting.
Enterprise software should be able to do this for an active directory joined windows computer, but you won't find many smaller developers who can be bothered to implement support for this. Web based applications can only do this towards a server hosted on the LAN, for security reasons.
The modern version would be using SAML or OAuth to pop up a browser dialog that opens the organisation login page, which then returns the user details to the application. In this case the browser is authenticating all of the applications so can maintain a session to avoid prompting for password input multiple times.
Probably depends on how many of those systems have a rule where changing the password invalidates existing auth tokens (i.e. logs out all instances where the password is changed).
A more sensible alternative is to allow you to see the "currently logged in instances" and terminate them individually as required (which is something everyone should regularly do with things like various google services, web mail, etc.)
This is documented and just how Kerberos works. Dig a bit deeper into it: Kerberos is a compromise for secure auth, distributed auth, vs. re-auth to the central authority for every bit sent over the wire.
You can change that behaviour, more specific the time and how long those TGTs, and a few other things, are valid in (Windows-) kerberos. But I don't recommend it. You can experience a part of higher paranoia kerberos if you configure some admin accounts as "protected + sensitive" along with the other settings for securing high risk accounts: You may have to re-login or re-enter your credentials every two hours, which is acceptable for some admin accounts, but not for normal users.
If you mess with those settings your really really have to know what you are dealing with. And my knowledge is deep enough to recognize: Don't touch unless there is really a reason and everyone involved knows the consequences - and I am not among those knowing all those consequences (Dunning Kruger effect, in the original form).
Edit: You can force-clear all TGTs of a client or a server. The result is pretty close like you have just rebooted the computer, in your case the Exchange server. If you use that method believe me: You will reboot (not power cycle please) the server out of frustration since many services don't like that and recovering from that is faster with a reboot. Guess how I know.
Ohhhh yes, Kerberos... I had the distinct (dis)pleasure of dealing with Kerberos and GSSAPI-based tech for a few years. While others less... knowledgeable would describe Kerberos or GSSAPI stuff as 'fragile', it's actually more paranoia built into it all from the developers to make sure it can't be abused.
And tech taking shortcuts, like thinking that everything using GSSAPI will only ever do one roundtrip, which is categorically not true, or not gracefully handling the invalidation of the existing token by requesting a new one, is just one of the many things that make Kerberos feel like a temperamental glass-based thing, rather than people taking liberties and making assumptions instead of sticking to what the specifications say you should do.
You might be drifting off as the documentation author's monotone tutorial drones on and on.
But you will notice that, just after reading out 'Enabling LDAP authentication will delete all existing users' there is a noise somewhere between a maniacal giggle and a guffaw.
That just doesn't come across in the written work.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
