back to article New systemd 248 feature 'extension images' updates immutable file systems without really updating them

Version 248 of systemd, a widely used system and service manager for Linux, adds a feature called system extension images, designed to allow system files to be added, or appear to be added, even on read-only file systems. As developer Lennart Poettering explained: "When a system extension image is activated, its /usr/ and /opt …

  1. Anonymous Coward
    Anonymous Coward

    why in hell?

    Apparently L.P. couldn't run pulseaudio on his Phillips Hue lights...

    1. Paul Kinsler

      Re: why in hell?

      Yeah, enough about systemd already. Where's this year's April Fool article?

      1. cookieMonster Silver badge

        Re: why in hell?

        Systemd is to replace emacs, or is it the other way around ??

        1. MrBanana Silver badge

          M-x apropos systemd

          No apropos matches for ‘systemd’

          I think we're safe in that direction.

      2. MrBanana Silver badge

        Re: why in hell?

        In recent years, across a number of publications, it seems more common to post implausible but actually real stories. Instead of making something up. Maybe the world is now so messed up that April fools day will be abandoned, as we can enjoy it all year long.

    2. chuBb.

      Re: why in hell?

      Nah he needed to find new ways to ruin my day now I'm wise to *most* of the locations that can override what ever config I define.

      It's a shame, if systemd just did init and not all the other shit it does it could be an adequate update to init scripts, i find it less effort to parse a. .service file than what ever ultra terse bash dick waving the dev put in the init file for example

      1. Graham Dawson Silver badge

        Re: why in hell?

        That's been the problem from day one. As an init, it's fine. Starts services in a neat, orderly way with a consistent configuration.

        Why does it have to glom logging and device management and networking and inter-process communication and user management and home directories and container management and file system overlays... and so on and so forth.

        There's never a rational explanation for it.

  2. Steve Davies 3 Silver badge
    Big Brother

    Errr but...

    Version 248 of systemd, a widely used and almost universally hated system and service manager for Linux,

    There fixed it for you...

    Even if this is a joke, I could not resist the opportunity to have a go at systems. It is about the worst thing ever imposed on Linux.

    1. John Riddoch

      Re: Errr but...

      The core idea of replacing the serial init scripts with a parallel boot process is fine, it makes server booting faster and replicates SMF on Solaris in that regard.

      Unfortunately, just as SMF absorbed a bunch of configuration into its clutches (e.g. DNS changes via svccfg rather than simply editting /etc/resolv.conf), systemd is doing the same. I'm not keen on so much being done by the init process in Linux - it's too critical and important to have bugs in.

      1. DarkwavePunk

        Re: Errr but...

        That auto-generated /etc/resolve.conf annoys the hell out of me. The other day I came across it again, couldn't remember the correct voodoo incantation, and couldn't be arsed. So I did the only sane thing - edited the file and made it immutable with "chattr +i". Yes, I'm a bad person.

        1. Martin Gregorie

          Re: Errr but...

          Yep, I got pissed off by the monkeying with /etc/resolve.conf too, mainly because for years I've just slapped a local customised copy over the top of it whenever a system upgrade caused name resolution to fail. This time, after a couple of reboots I finally read the version that got pasted back in place at each boot, did what it said and have decided that I actually quite like putting all resolver customisation in /etc/systemd/resolved.conf because it does at least collect a number of related settings in one file.

          Still somewhat annoyed that somebody thought it would be a good idea to put these hints in the replacement /etc/resolve.conf rather than an easily found manpage, though.

          1. David 132 Silver badge

            Re: Errr but...


            Don't mention resolved.conf in my presence. For the last two weeks since a systemd update, I've been battling resolveD, which insists on setting my local DNS resolver to after every sodding reboot. But not actually starting any DNS service. Which means nothing works, although my Pi-Hole stack insists everything's peachy. I have tried everything I could think of, every tutorial, every hack and workaround, but still after each reboot it resets. Next step is to take off and nuke it from orbit.

            Why the blithering f@ck is a service init daemon even running a DNS resolver, much less a crappy and non-functional one?

            Sorry, rant over.

            1. Graham Dawson Silver badge

              Re: Errr but...

              I had exactly the same problem this week. New laptop with an ubuntu-based OS, which in most other regards is a very nice machine, but the networking just killed me. I couldn't resolve anything on my lan no matter what I tried. I was at least able to disable resolved and put network manager back in charge of resolv.conf, which is a step in the right direction at least. Now it gets its nameservers from the dhcp lease instead of that convoluted solution to something that was only ever a problem in poettering's mind.

              Somewhat related, I had a weird problem with pulseaudio as well. It appears to have found a way to crash my docking station by sending a power down command to the idle sound hardware in it. Another obscure configuration to change. Hooray.

              1. David 132 Silver badge

                Re: Errr but...

                WONTFIX: Doesn't happen on LP's configuration, therefore isn't a problem, therefore no further action needed.


              2. bazza Silver badge

                Re: Errr but...

                Ubuntu seem to be trying to bring some sanity to the systemd - gnome ghastliness, with the introduction of netplan. This seems to be a tool that you use to specify how you want network configuration to work in some file, run netplan, and it stamps out the corresponding config files for systemd and networkmanager.

                Yes, networking has gone even more meta. Now you configure something that configures other things that will in turn configure yet more things. And of course, this all means that things that were possible in the past are now not possible...

                BTW Ubuntu 20.04 seems to come with both NetworkManager and systemD configured to think that they're controlling DNS. I think that systemD mostly wins, but it has caused me no end of trouble in some domains. These were all sorted by telling NetworkManager to not fiddle with dns settings by adding "dns=none" to its configuration and leaving systemd resolveD stock / enabled.


                SystemD's resolved seems particularly objectionable. The apparent intention behind it is that applications will name resolve using dBus calls into resolveD, whilst it provides a bastardised /etc/resolv.conf for "legacy" applications. How can name resolution via dbus ever be a good idea, all things considered? The implication is that at some point they'll deprecate /etc/resolv.conf altogether, and then name resolution in a system will be entirely dependent on a whole bunch of daemons being properly configured and running. Surely that'd be an insane position to be in?

                Why is it that Linuxes, of all the major OSes out there, seems to be the only one that has these tortuous, competing and often conflicting problems with DNS, and with networking in general? I can't even begin to remember the last time I saw a Windows or Mac or OS/2 machine get its DNS config in a twist.

                What's worse is that both NetworkManager and systemD are essentially from the same stable - RedHat / LP - and they've managed to foist this entire shit shower off onto more or less the entire Linux world.

                Don't get me started on automounteD vs autofs...

                1. tonique

                  Re: Errr but...

                  When did you last see an OS/2 machine?

                  1. Anonymous Coward
                    Anonymous Coward

                    Re: Errr but...

                    Last saw an OS/2 (or at least an ecomstation or ArcaOS derivative) machine a couple of years ago. It was running the phone system at my then employer.

                  2. bazza Silver badge

                    Re: Errr but...

                    2000, when I switched to win2k..

                2. Michael Wojcik Silver badge

                  Re: Errr but...

                  How can name resolution via dbus ever be a good idea, all things considered?

                  You failed to consider Poettering Ideology, under which doing anything via dbus is a good-in-itself.

      2. Vulch

        Re: Errr but...

        "Something must be done. This is something, therefore we must do it"

        1. TVU Silver badge

          Re: Errr but...

          "Something must be done. This is something, therefore we must do it"

          To this day, I still do not know what possessed the Debian Project to adopt systemd all those years ago.

          They did appear to recognise the upset that this caused at the end of 2019 when the pro-systemd stance did appear to soften a little:

          "[Retain] Systemd but we support exploring alternatives.

          Here's the position for the Debian project described by that option:

          The Debian project recognizes that systemd service units are the preferred configuration for describing how to start a daemon/service. However, Debian remains an environment where developers and users can explore and develop alternate init systems and alternatives to systemd features".

      3. Ogi

        Re: Errr but...

        > The core idea of replacing the serial init scripts with a parallel boot process is fine, it makes server booting faster and replicates SMF on Solaris in that regard.

        Yes, and it was done before systemd. My Gentoo 2004 vintage distro had parallel init booting, and it did speed things up quite a bit.

        So systemd was very much a (poor) solution in search of a problem when it came out in the first place, and not much has changed, except it is getting forced down our throats by RedHat/IBM.

        1. sreynolds Silver badge

          Re: Errr but...

          Yeah whatever gains you had in booting would have been lost during the compile cycle for the update.

          2004 Gentoo, hmm dependency hell and non reproducible builds from memory but did support AMD64.

          Does anyone know what the point of systemd is these days? No seriously, what does it claim to be?

          1. Ogi

            Re: Errr but...

            The point is that Gentoo (then bleeding edge) had it in 2004, my Debian distro's had it a year or two later, when it was considered stable.

            Fact is, we had parallel init before systemd existed (which I think was back in 2010 or so), so "faster boot times" was a solved problem, not to mention a problem that was pretty much irrelevant in Linux world. For two simple reasons:

            1. Most Linux servers had uptimes measured in years, and when they did reboot, the bios POST process was usually far longer than the actual OS boot time. When it can take 15 mins for a machine to get past POST, you didn't care if the OS booted 30-60 seconds faster. Especially as you don't have to sit in front of a server waiting for it to boot up.

            2. For desktops and laptops, suspend worked fine, so you rarely needed to reboot. Even hibernate started working pre 2010's, so you could really be "up and running" quickly. Saying that, in the days of HDD boot was usually limited to how many iops you could do. SSDs have even made single-process init startups blisteringly fast now.

            Beyond that, I will say for Gentoo, back in 2004 compiling and tweaking for specific CPU architectures did make a noticeable benefit, that for me was worth it, even if I had to compile all from scratch. As machines got more powerful, the benefits diminished, so I stopped using it (funnily enough, as CPU's got more powerful, more people started to use Gentoo because it could be compiled faster, but I used it less as there was less of a performance improvement of doing that with faster computers).

            Nowadays I am running Devuan Linux, and thus am thankfully systemD free, although do have to do battle with it at work on CentOS/RHEL from time to time.

            1. Ian 55

              Re: Errr but...

              Gentoo - 24 hours of compiling everything from source so you can spend 2% more time in the idle loop.

              1. David 132 Silver badge
                Thumb Up

                Re: Errr but...

                Harsh, but fair.

          2. Anonymous Coward

            Re: Errr but...

            It’s an operating system/universe that hosts a Linux kernel personality.

            Even Linus Torvalds is now merely a shadow running in systemd having been accidentally uploaded after a minor misconfiguration accident. He only found out last week, be kind to him.

            1. Fruit and Nutcase Silver badge

              Re: Errr but...

              systemctl status linus-torvalds.service

              linus-torvalds.service - Linus Torvalds

              Loaded: loaded (/usr/lib/systemd/system/linus-torvalds.service; enabled)

              Active: active (running) since Sun 1969-12-28 51 years ago

        2. katrinab Silver badge

          Re: Errr but...

          Debian takes about 1 second to boot on the machine I run it on.

          FreeBSD takes about 10 seconds on identical hardware.

          Sure, Debian is a lot faster, but 10 seconds isn't really a problem for me. Rebooting is not something I do very often, and preparing for the reboot takes far longer than the reboot itself. I'm more interested in how they perform once they have booted up, and FreeBSD performs a lot better.

          1. bombastic bob Silver badge

            Re: Errr but...

            but if applications begin to *RELY* *TOO* *HEAVILY* on obscure and ridiculous "features" supplied by systemd, it will only encumber the BSDs and Devuan and other "non-systemd" Linux flavors with having to WORK AROUND IT.

            I hope devs will not rely on such features, and will NOT assume systemd is "there". I fear they just might jump into the lemming herd and do exactly THAT...

            (it's bad enough when they assume /proc or /sys exist as they do in Linux, or WORSE, Pulse Audio - even as a PORT on FreeBSD, Pulse Audio tends to interfere with OSS whenever the server is running)

      4. Arbuthnot the Magnificent

        Re: Errr but...

        "replacing the serial init scripts with a parallel boot process is fine"

        My OpenRC machine has parallel service startup now, works perfectly well enough for me!

      5. Ian 55

        Re: Errr but...

        Shaving a few seconds off the boot time is irrelevant for 99.9% of Linux systems. My machines typically go months between reboots.

        Adding a few more points of failure / security holes - which is what systemd has also done - is totally unacceptable.

        1. stiine Silver badge

          Re: Errr but...

          I mean no criticism, but you unless you are running Oracle Linux and paying for ksplice (or whatever its called that allows you to patch without rebooting) you really should be patching and rebooting on a regular basis. If you require 100% uptime, you may have to rewrite your application to run in a cluster where you can withdraw a machine from the cluster to patch and reboot, without halting your application...

    2. Michael Wojcik Silver badge

      Re: Errr but...

      systemd, a widely used and almost universally hated system and service manager for Linux

      How about: systemd, a hateful operating system based on abusing concepts Microsoft introduced in Windows, built upon the Linux kernel and GNU userland but ignoring the best ideas of both.

  3. Anonymous Coward
    Anonymous Coward

    Extension images

    Yet another neat idea that absolutely should have been it's own project instead of swelling the excessively turgid belly of Systemd. Hmm, turgid, I might make that my word of the day.

    1. ecofeco Silver badge

      Re: Extension images

      Great. Now I have that Foreigner song stuck in my head.

  4. theOtherJT Silver badge

    Why. Just why?

    It's an overlayfs. That's all it is. Just a less good one that's forced to target specific parts of the tree. If I wanted to overlayfs /usr/ I can already do that. I don't need another stupid bloody systemd specific way of doing that!

    1. Anonymous Coward
      Anonymous Coward

      Re: Why. Just why?

      My thoughts exactly. If this isn't an April Fool's joke, then yet again Poettering has reinvented the wheel.

      1. Ogi

        Re: Why. Just why?

        It is a sad state of affairs in Linux land, when a systemd article like this comes up on April the 1st, and we can't be sure if it is an April fools joke, or a real "idea" of the systemD team. Either is possible, which says it all really.

  5. Julz


    I first discovered in UNIX that you could mount one filesystem on top of another, I thought it was a bug.

  6. Natalie Gritpants Jr

    Haters gotta hate

    I'm OK with systemd. I think it is better at keeping a system in good nick than init-scripts. It can be hard to find the right man page but at least everything is documented in detail and Google usually has a quick answer.

    I have a little set-top-box pc that has a read-only root I update by rsyncing the main server root file system and then having a dozen or so symlinks to handle the unique files and files that have to be write-able. Those symlinks are created by a script in the initramfs image (very painful to debug) so the vanilla boot procedure works normally. This new feature might be usable instead given that systemd has a very powerful and customizable dependency system.

    1. mevets

      Re: Haters gotta hate

      I will meet your false dichotomy and raise you an anti-pattern of over-reach.

      There is a lot of space for a middle solution.

  7. sbt

    Undermining the benefits of an immutable filesystem?

    Isn't the fact it's read-only security enhancing; e.g. a bad actor can't replace a critical library or executable on /usr with some backdoored version? Then what's the point?

    1. mevets

      Re: Undermining the benefits of an immutable filesystem?

      Do you ever notice a sort of whooshing sound above your head, followed by people looking quizically at you?

      I admit, systemd is likely only outdone by NetManager and its inexplicable worker netplan for failing the basic reason for existing test. Almost everything written about systemd seems like it came from a first year programming student with an over active weed habit; but come on, a bulk update of an immutable file system didn't trigger a quick peek at the calendar?

      1. Graham Dawson Silver badge

        Re: Undermining the benefits of an immutable filesystem?

        Except it's real. The link to the official release news was right in the article, with its last commit on the 30th, and this particular part of the release was being touted in release-candidates back in February.

  8. Stuart Castle Silver badge

    Not sure if this is an April fools or not. It’s possible it is, but seems a bit of a reach functionality wise for systemd. That said, it seems systemd is almost becoming an os in it’s own right it had gained so many functions.

    1. Loud Speaker

      that said, it seems systemd is almost becoming an os in it’s own right it had gained so many functions.<P>

      Well the sooner it does, the sooner we can avoid it if we don't want it.

    2. oiseau Silver badge

      A huge step backwards

      ... seems systemd is almost becoming an os in it’s own right ...

      It is nothing but a developer endorsed registry-class virus set up inside Linux distributions.

      From a post here at ElReg, May 2017:

      "Systemd also has its dirty fingers into other parts of the system. As a replacement for sysvinit is is supposed to be an init system, but because its scope goes far beyond the initialization phase (and it doesn't let you take the good without the bad) it has become a dependency for many userspace programs that should never have any reason to interact with the init system at all, making it harder to use those programs on a non-systemd system."

      And to quote myself:

      Why would Linux actually need to have something like systemd running inside it?

      I'll try to sum it up in short Q&A:

      Q1: Hasn't the Linux philosophy (programs that do one thing and do it well) been a success?

      A1: Indeed, in spite of the many init systems out there, it has been a success in stability and OS management. And it can easily be tested and debugged, which is an essential requirement for any OS. Certainly not something you can do with a MS OS.

      Q2: So what would Linux need to have the practical equivalent of the registry in Windows for?

      A2: So that whatever the registry does in/to Windows can also be done in/to Linux.

      Q3: I see. And just who would want that to happen? Makes no sense, it is a huge step backwards.

      A3: ....



    3. David 132 Silver badge

      All the best April Fools' Day pranks are plausible.

      Spaghetti trees - yup, those crazy forriners, seems legit.

      Barnards Castle putting up a bronze statue of Dominic Cummings - plausible too.

      Systemd oozing its tentacles into yet another area while simultaneously completely missing the point and buggering it up - nooo, that could never happen. Ah who am I kidding. Utterly plausible.

  9. GrizzleeAdams

    QNX 4 anyone?

    This sounds just like the abandoned package manager from QNX 4. Need to install an app? Just overlay mount a FS with that app!

    Evidently it adds huge overhead the more packages you installed.

    The only nice bit about it was uninstalling a package was quick and safe.

    1. mevets

      Re: QNX 4 anyone?


      It was actually Neutrino (QNX 5) 1.0 that featured the package file system. Despite QNX 4s fetish for symlink installations (some idjit had an unformed idea), it didn't have the package file system.

  10. steelpillow Silver badge

    April 1?

    I really don't know if this is an April Fool or not. It is utterly insane to the point of unbelievability. But then, so is Poettering.

    1. swm Silver badge

      Re: April 1?

      Why the joke icon?

      1. stiine Silver badge

        Re: April 1?

        Mr. Poettering?

  11. Anonymous Coward
    Anonymous Coward

    Because this discussion has broadened and at the same time become entirely one-sided, I though referencing an alternative viewpoint might be appreciated. Nobody really enjoys a one-sided flame - do they? Without counterpoint, the mind, and life, becomes dull. So below is a quote from an Arch Reddit thread about systemd. Which is not exactly what the article is about, but then neither is this discussion.


    [ Why did ArchLinux Embrace Systemd? 2brainz ]


    I was the primary maintainer for Arch's init scripts for a while and I can share a couple of thoughts.

    Arch's initscripts were incredibly stupid. In their first phase, there was a static set of steps that would be performed on every boot. There was almost no way to adjust the behaviour here. In their second phase, the configured daemons were started in order, which only meant that a init scripts were called one after another.

    In the early 2000s, that seemed like a good idea and has worked for a while. But with more complex setups, the shortcomings of that system become apparent.

    With hardware becoming more dynamic and asynchronous initialization of drivers in the kernel, it was impossible to say when a certain piece of hardware would be available. For a long time, this was solved by first triggering uevents, then waiting for udev to "settle". This often took a very long time and still gave no guarantee that all required hardware was available. Working around this in shell code would be very complex, slow and error-prone: You'd have to retry all kinds of operations in a loop until they succeed. Solution: An system that can perform actions based on events - this is one of the major features of systemd.

    Initscripts had no dependency handling for daemons. In times where only a few services depended on dbus and nothing else, that was easy to handle. Nowadays, we have daemons with far more complex dependencies, which would make configuration in the old initscripts-style way hard for every user. Handling dependencies is a complex topic and you don't want to deal with it in shell code. Systemd has it built-in (and with socket-activation, a much better mechanism to deal with dependencies).

    Complex tasks in shell scripts require launching external helper program A LOT. This makes things very slow. Systemd handles most of those tasks with builtin fast C code, or via the right libraries. It won't call many external programs to perform its tasks.

    The whole startup process was serialized. Also very slow. Systemd can parallelize it and does so quite well.

    No indication of whether a certain daemon was already started. Each init script had to implement some sort of PID file handling or similar. Most init scripts didn't. Systemd has a 100% reliable solution for this based on Linux cgroups.

    Race conditions between daemons started via udev rules, dbus activation and manual configuration. It could happen that a daemon was started multiple times (maybe even simultaneously), which lead to unexpected results (this was a real problem with bluez). Systemd provides a single instance where all daemons are handled. Udev or dbus don't start daemons anymore, they tell systemd that they need a specific daemon and systemd takes care of it.

    Lack of confiurability. It was impossible to change the behaviour of initscripts in a way that would survive system updates. Systemd provides good mechanisms with machine-specific overrides, drop-ins and unit masking.

    Burden of maintenance: In addition to the aforementioned design problems, initscripts also had a large number of bugs. Fixing those bugs was always complicated and took time, which we often did not have. Delegating this task to a larger community (in this case, the systemd community) made things much easier for us.

    I realize that many of these problems could be solved with some work, and some were already solved by other SysV-based init systems. There was no system that solved all of these problems and did so in a realiable manner, as systemd does.

    So, for me personally, when systemd came along, it solved all the problems I ever had with system initialization. What most systemd critics consider "bloat", I consider necessary complexity to solve a complex problem generically. You can say what you want about Poettering, but he actually realized what the problems with system initialization were and provided a working solution.

    I could go on for hours, but this should be a good summary.


    1. John Robson Silver badge

      There are weaknesses with init scripts, but there are also strengths in the transparency.

      Parallelism is a pretty minor issue in my experience, how often do you reboot?

      The consistent handling of daemons is a good thing, I'm even getting used to the binary blob log management, and have some sympathy with that - although there is no reason why it should be tied to init.

      As for the rest of the tendrils which it exudes... there may well be benefits to many of them, but they shouldn't be tied to a specific init system, and certainly shouldn't be forced on people who just want the init system.

    2. stiine Silver badge

      Then I should log into the bugtracker for systemd and request a --serialise flag.

      While we're discussing terrible decisions, does anyone else remember the first time you booted CentOS 6? How quickly did the boot sequence scroll by on your terminal? Do you remember the flash of red error text that whizzed by? How many times did you reinstall CentOS 6 trying to fix this problem? How long did you spend trying to determine what failed only to discover that some absolute thoughless idiot changed the nice cyan CentOS banner to bright red.

  12. Anonymous Coward
    Anonymous Coward

    I came here expecting to see an argument, but instead...

    ...I found a massacre with only 1 positive comment and everything else against.

    And yet, nearly all the Linux distros have systemd on board and doing mostly what is needed.

    Did I say that pulseaudio works for me too?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like