Ubuntu Linux 18.04 systemd security patch breaks DNS in Microsoft Azure Microsoft Azure customers running Canonical's Ubuntu 18.04 (aka Bionic Beaver) in the cloud have seen their applications fail after a flawed security update to systemd broke DNS queries. The situation is as odd as it sounds: if you're running Ubuntu 18.04 in an Azure virtual machine, and you installed the systemd 237-3ubuntu10 …
It prevents "just update resolv.Conf" from being a solution to anything.
Trying to update resolv.conf is a nightmare, even removing write permissions does not work, you have to attrib immutable.
My recommended solution to systemd-resolvd problems is to uninstall it. Everything works fine without it
I think its purpose is to work as a DNS cache for the whole system - in case e.g. a server runs multiple processes with short lifetimes, it can improve DNS performance quite a lot. Unfortunately, its config seems to be really flakey because it's losing the config or using wrong config way too often because of minor changes during package updates.
I guess it's the usual story with systemd: the idea is good but implementation quality is not yet ready for production.
It's not clear that "the idea is good" always fits either. Adding complexity (which systemD often does) isn't universally held as a good idea.
Agreed about sub-par implementation quality, especially when it comes to ongoing maintenance. The systemD devs seem more focused on new shiny than fixes and stability and improvements for what's already there.
There are light dns-caching programs already (eg dnsmasq) - these can be run under their own user (with bind-to-port-53 permissions)
They don't then need to be updated because some monolithic virus has a use-after-free that can compromise the whole system.
systemd is not the OS. It has no business, and no need, to do this.
Even outside of this patch, DNS resolution was always a bugbear for me on Ubuntu 18.04. I believe that was the release (I only use LTS) they switched to systemd-resolved. For reliability on large-scale deployments I ended up removing systemd-resolved entirely and configuring resolv.conf directly and it ran happily for years.
Fortunately it seems to have improved since 20.04. I feel like there must be other things going on in 18.04 that combine with this recent issue for such a crazy issue to surface like that though.
Oh and Microsoft, blocking a single update in Ubuntu is pretty straightforward - simply hold the package (sudo apt-mark hold <package-name>). How about publishing proper instructions?
And then the crowd will bitch that MS is oppressing them and conspiring against Linux users by withholding patches.
It's up to the "owner" of the server to do test deployments, before mass rollout to production.
If you turn on automatic updates on all your systems and they take out everything, that's your responsibility, regardless of OS.
Re: that's your responsibility,
Auto security updates as a service is provided by Canonical, to a large extent its their responsibility to ensure that they are safe. Given the scale of the problems and the fact that its hit systems running latest patches, this looks like somebody skipped a serious amount of testing before pushing this to world+dog.
It's not even very good as an init. I've currently got 22.04 hosting isc dhcpd and Samba AD, and it's touch and go at boot as to whether SystemD's view of when a network is up actually results in there being an interface that you can actually bind services to. Something of a race condition...
Coz systemd doesn't know when the network is up. All it knows is it ran a program which may have asked the network to start.
It would be nice if there was a way to get udev type notifications for when a NIC changed to the up state and perhaps one for I got or lost an IP address.
I've never understood why systemd ever got involved in DNS but then expecting any logic as to why systemd ever tries to do anything outside it's core role of being a service manager is beyond me.
Please Linus can you step in and say only grown ups are allowed to be involved in writing the orphan catcher.
I think the idea for the systemd messing with the network stuff is to support theoretical setup where server is running e.g. node.js server and it's network configuration changes on the fly requiring new routes and DNS data and the systemd should be clever enough to automatically configure network, configure DNS and automatically reload or restart all services that depend on specific network features such as DNS.
However, the actual implementation quality of systemd cannot deal even with simple cases without random failures so there's very little sense trying to cover for this kind of theoretical situations.
If systemd had zero bugs for it's features, it would be pretty good already.
That "is network up" topic is such a longstanding boondoggle with systemd, it's not even a bad joke anymore.
We've been bitten by it in a couple situations, across different RHEL releases. Granted, our use cases were probably not common, e.g. sysdisk as iSCSI LUN on a private network dedicated NIC, so it's unlikely that anyone at Red Hat, let alone systemd, ever tested such a thing.
But the real point isn't Red Hat QA groups looking into corner cases for trouble. Rather, that systemd added progressively more complexity and fragility as we moved from RHEL6 to 7 to 8, with no measurable benefit. Often as not, in RHEL8 we ended up with systems unable to resolve anything.
>Granted, our use cases were probably not common,
Shouldn't matter. SystemD claims that if dependencies for starting up are properly expressed, it will sort out the order.
It does not get it right for common use cases.
One can conclude that the the words "uncommon" and "common" are redundant; it simply cannot cope with "use cases" ...
... explain to me why DNS should ever have anything to do with an init?
Because systemd is not really an init.
Like I have said before:
---
Systemd is a virus.
It works just like the registry does in MS operating systems.
It's a developer sanctioned virus running inside the OS, constantly changing and going deeper and deeper into the host with every iteration and as a result, progressively putting an end to the possibility of knowing/controlling what is going on inside your box as it becomes more and more obscure.
Systemd is nothing but a putsch to eventually generate and then force a convergence of Windows with or into Linux, which is obviously not good for Linux and if unchecked, will be Linux's undoing.
There's nothing new going on here: it's nothing but the well known MSBrace at work.
Now go and tell me that Microsoft has absolutely nothing to do with how systemd is crawling inside/infecting the Linux ecosystem.
---
You could possibly argue against the virus idea and consider it something else as it does not reproduce autonomously.
Yet.
But it does reproduce in a developer assisted way: by successive iterations.
O.
It is not a virus. Rather, it is a cancer.
Consider: The systemd-cancer takes root in its host, eats massive quantities of resources as it grows, spreads unchecked into areas unrelated to the initial infection, and refuses to die unless physically removed from the system, all the while doing absolutely nothing of benefit to the host.
No, there is no MS-sponsored conspiracy. And even if there was, it's a massive failure in that nobody has to run the systemd-cancer unless they want to. The kernel itself is, and shall remain, init agnostic, per Linus himself.
"No, there is no MS-sponsored conspiracy."
I remain unconvinced. The world-class tosser responsible for the systemd cancer has returned to the mothership and now works for MS.
"nobody has to run the systemd-cancer unless they want to"
That's almost impossible because this virulent cancer has metastasised. Its deadly tentacles are spread throughout just about every aspect of the Linux train wreck. It's extremely difficult to remove them. systemd's got a stranglehold on just about everything in Linux-land, like the creature that locked on to John Hurt's face in Alien.
No, there is no MS-sponsored conspiracy
then please explain away why Microsoft has hired Pottering in the past 1-2 months (can't be bothered to look up the exact date) ? They've done that with Nokia before, FFS !!! Embrace, Extend, Extinguish.
Fool-me once, shame on you. Fool-me twice, shame on me
nobody has to run the systemd-cancer unless they want to
This is false: I run a systemd-less distro (MX-Linux-21) which uses the traditional SysV-init, and is nominally systemd-free, and yet, when running ps aux | grep systemd, I get:
/lib/systemd/systemd-udevd
/sbin/cgmanager --daemon -m name=systemd
/lib/systemd/systemd-logind
So even though I chose to NOT use systemd, it's so difficult to remove (because of some dependencies ?) that it leaves traces even in distros where we don't want it. That is characteristic of a virus (or cancer ?): it infests everything, even those things where it has nothing to do (udev was independent before being swallowed by systemd).
What, do your systems initialise in precisely the same way whether or not they can resolve names? Including services that hit the network?
But yeah, no need for it to be part of the init system itself. Which is why it isn't. It's a separate component you can choose to use or not.
You could argue it may as well live outside the system git repo as inside it, but that's sort of up to the developers, really.
It's my understanding that systemd wants to handle network connection going up and down to start and stop services that use the network.
The system tries to be generic so it must be able to handle DNS settings to change with the network, too. And it also implements caching DNS service and I guess the logic is that in case of network change the DNS cache must be flushed.
It's all good in theory but the implementation quality is not yet at the beta level in my opinion. However, it's still used in production by all big distros for reasons I cannot explain.
If you are running Ubuntu in lxc containers you can remove systemd entirely and run your own /sbin/init and none of the sysyemd-* nonsense is needed for most server applications.
Apt requires it, but you can remove the /sbin/init symlink that points to systemd (which is in Some stupid place not sbin) and put a Binary /sbin/init of your own choosing.
apt update of systemd will revert this, so you need to copy it back if you use apt in lxc. Apt does work inside lxc but you need to be a bit careful.
You could write a better init than systemd in a couple of days in bash. Scripts can be used for init because #! Interpretation is done by the kernel.
I've been quite happy with Devuan both on desktop and servers/vms. The only insurmountable problem I've had in past two years with Devuan is MiracleCast, and I can live without that. I've never had other problems with non-free drivers, firmware etc., both on older and fairly recent hardware.
Ubuntu has problems of its own (how they handle the Linux name, the opt-out stats, aimless juggling with desktop envs etc). I can only recommend Devuan with KDE to Ubuntu users.
re. " I've never had other problems with non-free drivers, firmware etc".
For older Nvidia 9400M cards that use the Nvidia 340.108 driver, linux (in general) no longer works out the box with the added non-free Nvidia 340.108 drivers. The only way of late is using a modified Nvidia driver to get hardware acceleration.
https://www.if-not-true-then-false.com/2021/debian-ubuntu-linux-mint-nvidia-guide/
(Have this working with latest Mint 21 and Ubuntu 22.04.1).
Shame, it's not mentioned on the Ubuntu / Mint websites because it saved hours, and without it, it's a showstopper to using linux on older hardware, as a fast general purpose desktop.
If there is another way, a link would be helpful.
How does an Ubuntu DNS problem knock out Azure?
I can understand that updated Ubuntu instances running on Azure can cause problems for those instances, but surely Azure should do its own DNS look-ups and resolution.
Or does Microsoft's DNS servers run on Ubuntu, which they had updated en masse and now it has become an unholy mess with no easy way back?
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
