Easy; Just blame AI
Headline says it all.
Microsoft has admitted that June's Patch Tuesday updates could break the DHCP service on Windows Server. The update, released June 10, contains an issue that can affect Windows Server 2025, 2022, 2019, and 2016. According to Microsoft, "the DHCP Server service might intermittently stop responding after installing this security …
"Enterprises are unlikely to accept an excuse that involves finger-pointing at dodgy code from Redmond in the face of an unexpected loss of connectivity."
I'm sure Enterprises are well acquainted to the fact that it's nearly always dodgy code from Redmond.....
Unless it's DNS, or soon to be featured in a "Who Me?"
Anyway, we monitor the DHCP service - if it stops, we'll know.
<Walks away muttering insults and profanities about Micros~1>
Maybe the service works and reports it is, but in reality it is not.
I dunno, not seen it, not doing infra any more and many places I have worked at do not us MS for DHCP .
I remember we were once bitten by a simple check of something running and monitors looking OK only to find it was anything but. It caused a few problems, but made me wise to that
Somehow it really is nearly always DNS even when it shouldn't be. I spent the last day trying to figure out why a website which redirects to HTTP from HTTPS (insecure ofc) ends up in a redirect loop. I was checking server configs left right and center, checking HSTS, and clearing caches, trying other browsers, and other devices. In curl all the headers were fine, no redirect loop, but in every browser it would load fine once, then on refresh, end up in a redirect loop. Turns out enabling HTTP2 in cloudflare adds a 'secret' DNS record known as SVCB/HTTPS which tells modern browsers to use https for HTTP2 (apparently even when the user enters HTTP:// in the address bar). I say secret because if you look at the DNS config in cloudflare's dashboard it won't be there, but if you do enough digging with the dig command it shows up. I don't need http2 or even HTTPs for this particular subdomain, so I try manually setting a DNS HTTPS record but cloudflare overwrites it. In the end, i decided it doesn't even matter. It doesn't need cloudflare, and exposed the http server without ddos protection enabled,
Congratulations Redmond, you've just found another justification in having revoked your QA department : you have your entire customer base doing it for you !
Isn't that the ultimate savings ?
Because none of them are going to leave, now are they ? Of course not. They stay tied hands and feet to your benevolence, and totally ignore how much money it costs them to remain under your control instead of hiring some competent network specialists and going to Linux, which would condemn you to the dustbin that you deserve.
Because that would mean making their management learn something, instead of going for the tried and true Excel charts and Powerpoint presentations.
Dear God forgive me, but sometimes I wish our entire economical world would collapse so that we could migrate to something that is actually functional.
Unfortunately, that would mean something like this, and the cost in innocent lives would be more than I could bear.
So we'll just have to slog it through and wait for the current generation of useless manglement to die out before maybe having a chance at getting true performance and professionals back into the enterprise arena.
I'll be dead by then anyway, so good luck, kids.
Of for fucks sake, who is running IT these days ? Single point of failure anyone ? Hello ?
My home setup is fully protected against the DHCP server disappearing. Using the power of bash and cron, and puppet, if DHCP is absent for more than 10 minutes, a fallback docker container with piHole is spun up, which reads the main config and runs as a spare DHCP server until I can figure out what happened.
The implication of this MS SOP breaking something, is that a lot of places had no idea about setting up a fallback service. In which case they deserve all they get.
Yes, MS fucked up. THAT'S WHAT THEY DO. But the job of IT *professionals* is to factor that into our lives and move on.
Still, if nothing else, this is todays "Is is such a good idea to be so dependent on MS" moment. Weather to follow.
Your DHCP server at home fails so often you need that kind of setup?
My home(and personal gear hosted at a colo) DHCP runs on my OpenBSD firewalls, which is my internet gateway, anything important uses static IPs, but obviously if the firewall goes down(which outside of power outages and scheduled maint has never happened), I don't need to monitor DHCP specifically. If the firewall is down then not having DHCP is the least of my worries. Haven't had a full on hardware failure(that caused an outage, which excludes disks failing in RAID since that doesn't cause unscheduled downtime) in any of my important personal servers that I can think of in the past 20 years.
I did have a Ryzen motherboard fry itself 4-5 years ago(sparks literally flew while I was diagnosing why it wouldn't turn on again) while doing video encoding(despite it running Linux I wouldn't really consider it a server as it's purpose was nothing but video encoding I have ripped about 3500 DVD/BDs), though there was no impact to anything other than that system. Fortunately still under warranty and got it replaced, no issues since(only MB failure I can recall having since the Abit BP6 on personal gear). I think I did have a CF failure(boot drive) in one of my older Soekris firewalls that ran OpenBSD as well maybe a decade ago or so, though the firewall kept chugging along in read only mode I didn't notice it for a while as it wasn't causing an issue.
The software stacks I run are well behind bleeding edge, so things are generally quite stable.
Sigh ...another home user comparing their set up with a 10,000 person set up
For starters, why the fuck are you using DHCP if you need a back to check every 10 minutes! Unless you are using DHCP to assign fixed IPs, but again, why check every 10 minutes?
Also are you checking if the service is running or it is actually handing out renewals? Two very different scenarios? Are you checking that it's not just handing out renewal but all the DHCP options often required these days such as SIP proxies?
It could have been worse. They could have borked one of the most basic networking protocols in existence that is required by a lot of their users. Oh wait.
I blame AI. Maybe they need AI to test the AI testing. Do they even test these patches anymore? Have they really gone to the assumption that our users will test it anyway? That's not going to end well.
I visited my local Greggs lunchtime today (other purveyors of cholesterol-laden baked products are available) and the message below was on their lovely advert screens.
Activate Windows.
Go to Settings to Activate Windows.
Oh FFS. I'm pretty sure Greggs wouldn't cheap out on pirated Windows licenses, but really, why Windows??????
The advert display was in the window? :-)
On a marginally more serious note, maybe the build was one which needs to check with the licencing server and for whatever reason has not seen it for some $random length of time that it's now de-activated. Maybe it lost it's DHCP assigned IP address lease and can't renew? :-D
MS users are like abused partners, they keep coming back for more. Enterprises have the ultimate voice in this, and could easily just make Linux the only option, training the staff, and letting creatives use Macs.
But like sad junkies they just have to figure out a way to get around the fear that their dealer is going to give them a hot shot. Every month.
Until it don't.
So many network admins these days were hired as digital natives. But, being born with digital tech in your hand does not make a competent technician.
No push back from greybeards that MS-DHCP is required, and redundancy is "built-in!" because the greybeards were too expensive and shown the door.
I'm one of those greybeards. I and my colleagues made sure critical servers had well documented, static IPs. Of course this meant ensuring MS-DHCP did not clobber those IPs. True backup DHCP was ready if needed, because functioning servers are of no use if clients do not have their IPs. Similar approach to the entire network.
The list of "critical" servers these days though has probably suffered mission creep and would be claimed to be unmanageable for manual "DHCP".
enshitification all around
@chivo243
"ageism?,,,"
Ditto.
And in other news: wifi on Windows has been broken since at least Win10. It completely ignores "Connect automatically" and regularly disconnects. My better half regularly asks me if I am still online (I have been Windows-free since 2002-ish, but still have to maintain her laptops.)
----->>> MS, of course.
>”critical servers”
Well the sites that will be “clobbered” will be small businesses with a single Domain Controller, with auto updating enabled…
Fortunately it seems the server should still boot, so gaining a non-domain admin login shouldn’t be too difficult, enabling the update to be rolled back…
enshitification all around
How hard can it be?
More than twenty years ago I was running two isc-dhcp servers with 500-600 clients in an active-active failover configuration which with isc bind were the two things that didn't fall over in a screaming heap. Two operating systems (Unix & Linux) on Alpha and x86 respectively, in case there was a problem with the network stacks.
A little bit of caution with lashings of paranoia goes a long way in this game; and sectionable insanity if it involves Microsoft.
This months update also brought a bluescreen on supported AMD processors, about them being unsupported. Only way to fix was to roll back (if you could).
https://www.reddit.com/r/sysadmin/comments/1l7pc1k/comment/mx5x7y6/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
Technically yes but also no. In over 20 years of using one flavour of Linux or another, the only update that has caused a 'failure' was an Nvidia driver update to a laptop that was running Arch Linux. The failure was, it couldn't load X-Windows but everything else worked as expected. Nvidia fixed it within 24 hours and pacman did the necessary.
Other peoples experiences may be different :)
Yes. Last fall I had to switch from Ubuntu 22.04LTS to Debian on servers that was used to share files and host virtual machines. An Ubuntu update broke VirtualBox where I could not get any virtual machine to run. I verified it was the Ubuntu update and not an installation that went bad by installing a fresh Ubuntu 22.04 on a test system and running a test virtual machine. I installed the latest updates and the virtual machines would not run. At the same time, I installed Debian 12 on a test system, ran the latest updates and installed VirtualBox. My virtual machines ran fine. It took me a couple hours to blow-out the Ubuntu installs on the production systems and install Debian. My setup scripts I used to install the applications and utilities I used in Ubuntu worked fine for Debian with only a couple name changes.
For technical reasons important to me (but not necessarily important to others) I had already decided to switch Linux distributions. The botched Ubuntu update just accelerated the process. While I had not decided that Debian would be the new OS for the servers, it was the top contender when I was forced to do the change.
I have not looked up the relevant RFCs (and, in any case, they may not contain the information required).
RFCs were often written by (or anonymously sponsored by) the original inventors/proposers.
A long time ago (last millennium!) somebody told me that MS invented DHCP to manage IP addresses in large networks.
I have no idea if this is actually true.
If it is true, their implementation is obviously the best because it works exactly as they envisioned it, and not the way some Godless, Commie, bearded, sandal-wearing script kiddie implemented it.
doubt that ms invented dhcp. we were using it on a system where only the clients were windows based, and mostly dos initially. they all got their ip addresses when the connected to the network in. had a subnet and used a server that handed out static addresses to mac's it recognized and ip's from a pool if it didn't recognize the mac. a variation of bootp that could hand out addresses to new computers but didn't handle leases.
how is it that we've accepted crap product that can trash your entire firm as standard in this industry?!
at least when the Ford pinto exploded several times, they were forced to improve it & you're not having to take your car in monthly to fix issues that should have never got out the factory because cars are made by REAL engineers ! it's telling that the 747MAX was also a software error!
its stunning that as developers get more and more "freedom to innovate" & MBAs run more and more of the process, software is getting worse and worse, service has collapsed & testing is non existent!
i've always been a microsoft guy but these days i would never run anything vaguely important like dhcp on their product. I'd ideally pay for an infoblox or similar. and let's not talk about people dumb enough to put anything important on the cloud