back to article Kernel kerfuffle kiboshes Debian 12.3 release

The Debian maintainers have identified a problem in kernel 6.1 that can cause corruption on ext4 volumes. As a result, the planned 12.3 release won't happen. The issue is Debian bug #1047843, and it's been assigned a severity level of "grave" – which is worse than "serious" and only one step down from "critical." …

  1. Doctor Syntax Silver badge

    Just escaped that one!

    The problematic Debian kernel version was 6.1.0-14.landed here at the weekend. I installed that about 6 pm yesterday but didn't immediately reboot so continued running on 13. I'd seen a news item about an ext4 problem but it didn't register that this might contain it. Later in the evening I rebooted and maybe an hour afterwards an alert came up for a new update. I checked that & found it was for kernel 15. Realising that 2 kernel updates a day meant that there must have been a problem with the first so immediately installed and rebooted and then looked into the history finding the references in the article and this discussion https://lwn.net/Articles/954285/

    This left me with the problem - did I do the big sorting out of email archives before or after the reboot? I think i was after but AFAICS, no damage done. Deep breath! This, BTW, was in Devuan but as far as non-systemd stuff is concerned, it's Debian

    Debian 12.4 with the corrected kernel was out by the end of the day, BTW. See amacater's post towards the end of the LWN comments.

    As far as I can make out from the discussion the later patch was - ironically - supposed to deal with the possibility of corruption in the event of a system crash under certain circumstances which is the sort of thing that's like;y to get a high priority for back-porting and the earlier patch that didn't get back-ported may have been the performance-related one.

  2. Crypto Monad Silver badge

    Sigh. Another case of "Debian knows better than the upstreams". Like the "improvements" to ssh key generation which removed almost all entropy.

    Please: just leave the software alone, and package it. In particular, give me a kernel that's as close to the one Linus released as possible.

    1. ludditus

      You didn't get it, did you? Being too busy with your crypto stuff... THE BUG WAS UPSTREAM'S, NOT DEBIAN'S! It affected 6.1.64 and 6.1.65, and (a second bug) 6.1.66 in every distro on Earth!

  3. may_i Silver badge

    Fixed, done and dusted

    You're a bit out of date on this one. The 12.4 release has already happened - about 20 minutes after the Debian news article which you linked to was published.

    1. ludditus

      Re: Fixed, done and dusted

      Indeed. Unfortunately, "Debian 12.4 is released with linux-image-6.1.0-15 (6.1.66-1)," whereas 6.1.67 is what everyone needs (I just got it as kernel-lt in ELRepo for EL9).

      Surely enough, that Wi-Fi-related commit in 6.1.66 that had to be reverted in 6.1.67 caused me issues in AlmaLinux 9.3 with kernel-lt from ELRepo: NetworkManager waking up the laptop from sleep, with high CPU, and failing to shutdown. Everything fine so far with 6.1.67.

      Debian still seems to be lacking 6.1.0-16 aka 6.1.67 (what's with this dumb versioning scheme in Debian and Ubuntu, why can't they use the real version?).

      1. Altrux

        Re: Fixed, done and dusted

        Something to do with ABI version compatibility or something? They always use the major kernel version with .0, then the -XY is their ABI release number. But yes, at first glance it would be more useful if they actually included the patch version instead of always being .0, but I guess that would mess up some of their build tooling in some way.

        1. amacater

          Re: Fixed, done and dusted

          uname -a will give you both:

          Linux [hostname].home.arpa 6.1.0-15-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.66-1 (2023-12-09) x86_64 GNU/Linux

          There is an ABI bump, yes, hence the Debian referencing of Debian kernel image numberings.

          6.1.67 is being worked on as I speak - but I don't know when it will be released.

          [Debian images release team member]

  4. Steve Graham

    Man, you squares are so out of touch. I'm currently compiling festive Linux 6.6.6 (Santa is an anagram of Satan.)

    1. Ken Hagan Gold badge

      " (Santa is an anagram of Satan.) "

      Only in English, but that's a devilish bodge of a language so perhaps we shouldn't be surprised.

  5. MarkMLl

    Oh flippin' great...

    Spotted the upgrade on Sunday (10th) evening, installed it yesterday (Monday 11th) morning and it's left me with kernel 6.1.66-1 (2023-12-09) and no indication of any available replacement.

    However /etc/debian_version tells me I'm on 12.4, which is presumably OK.

    Still better than my memories of OS/2...

    MarkMLl

    1. amacater

      Re: Oh flippin' great...

      Yes, you've got the fixed version

      1. MarkMLl

        Re: Oh flippin' great...

        Got it thanks.

        66-1 = 67.

        MarkMLl

  6. Altrux

    Oh dear

    Stable kernels in "not actually that stable" shocker!

    When you look at the sheer number of patches arriving in each stable point release (sometimes many hundreds, in a cycle involving several updates per month), you do have to wonder. Greg does a lot of amazing work, but clearly the maintainer burden is too high to really manage these days, so not everything gets the vetting it should. And the old, fairly strict rules about what should be included in stable seem to have been quietly forgotten. Just pour in lots of nice patches from the newest -rc release!

    When backporting to older LTS kernels, as opposed to the latest stable kernel, this seems to have become too risky due to divergence?

  7. Nate Amsden

    dselect

    Debian had dselect before apt, and it handled dependencies, it was a lot more painful to work with though for sure. I still remember on one install spending about 6 hours going through the various packages in dselect to find what I wanted because after the install was done I didn't know how to start that tool again(or even what the name of it was). Though back in those days I still built a lot of things from upstream source directly(including KDE, GCC, Gnome, X11, kernels(last kernels I regularly built were 2.2.x)).

    For me, apt started to become much more useful with Debian 3 which for whatever reason I believe I recall as having a huge jump in number of packages available(the first huge jump in my memory at least), it was the only Debian version where I used "testing" for several months (instead of "stable"). Reading the release notes of Debian 3(tried to find the number of packages), they specifically said apt-get for upgrading the (distribution) version was not recommended(and to use dselect instead).

    Even though dselect has been depreciated for over 20 years it still serves a critical role for me for copying package states between systems(not something I do often but it is handy)

    dpkg --get-selections >selections.txt

    (copy selections.txt to another system

    dpkg --set-selections <selections.txt

    apt-get dselect-upgrade

    Perhaps there is another way to do it these days. I recall at one point trying to find a similar process on a RPM-based system(don't recall which), but was not able to.

    My systems at work are pretty much all built with configuration management software so little or no need to do that process there, but my ~20ish home linux systems(including my public email/web/dns) are managed ad hoc still.

  8. Dave Pickles

    Not just Debian

    I run Arch Linux on Raspberry Pi, currently kernel 6.1.64. Last weekend I discovered the root filesystem (on a USB SSD) was read-only due to a corrupt journal. Thinking that the SSD was faulty I copied the contents to other media and got it working again. It now seems that this kernel bug is likely to blame.

    The latest kernel for the Pi on Archlinux ARM is 6.1.66. So do I install that or wait for 6.1.67? I think I'll wait.

  9. PRR Silver badge
    Trollface

    Downvote below!

    > time for someone to invent some sort of automatic dependency tracking and resolution mechanism

    Windows Update has done this for 20 years. Gets it right 9 times out of 10. You needed more non-functional H-P printers today, didn't you?

    1. Ken Hagan Gold badge

      "9 times out of 10"

      That'll be 9 users out of every 10 receiving the same update.

    2. bazza Silver badge

      One has to admit that, when it comes to software packaging / installation / removal, both Windows and MacOS have got Linux licked. And Linux is getting worse. Where once we used to have multiple distros with differing views on how software should be packaged, we now have mixed views within some individual distros. For example, apt or snap? Etc.

      I can recall the days when it was all just yum or apt and the many claimed advantages such as having a clean set of dependencies installed etc. Whereas, Windows and MacOS apps simply brought all their dependencies with them, leading to installation bloat. Well, I'm not sure that Linux isn't now following that direciton with snaps, containers, etc. Maybe I should just forget the good old days and embace systemd and all other new objectionables in the world of Linux.

      1. Ace2 Silver badge

        Installation bloat - as of about a dozen versions ago, a base MacOS install included about 400GB of Asian fonts. MS apps went nuts if you tried to remove them to save space.

        Now though, with modern disk sizes… I suspect just having each app provide its own version of libm or whatever is the best way to do it.

        1. Liam Proven (Written by Reg staff) Silver badge

          [Author here]

          > I suspect just having each app provide its own version of libm or whatever is the best way to do it.

          Which is what GNUstep provides with its .app package format, which is just a FOSS re-implementation of the NeXT way of doing it.

          But GNUstep doesn't even realise that it has a packaging format.

        2. Ace2 Silver badge

          (Ugh, MB, whoops!)

    3. Liam Proven (Written by Reg staff) Silver badge

      [Author here]

      > Windows Update has done this for 20 years

      No it hasn't, and I think you do not understand what I was talking about.

      We don't know what MS does internally for kernel code dependencies, and if you are at MS and you do, then you've probably just lost your job for breach of confidentiality.

      However, it is *not* Windows Update, which is for after-release vulnerability patching, which is a totally different solution to a totally different issue and doesn't really have dependencies.

    4. Peter Gathercole Silver badge

      AIX has had dependency management at least since AIX 3.1, 34 years ago (when the original POWER systems were launched). And if I can dredge back that far in my memory, I think that installp was on AIX on the 6150 PC/RT before that.

      It's interesting that Redhat took YUM from YellowDog Linux, a Power version of Linux. It's almost like the people working on Power, having seen AIX's dependency management, decided to add it to their Linux build!

  10. Steve McIntyre

    Patch dependencies are nothing like package dependencies

    I think you're a bit confused here...

    Anyway, due to great efforts by some Debian folks we managed to get a fixed release out in record time. 12.3 was the shortest-lived point release ever.

    Massive thanks to the people who went above and beyond at short notice, for the sake of our users.

    1. RedGreen925 Bronze badge

      Re: Patch dependencies are nothing like package dependencies

      "I think you're a bit confused here..."

      No doubt there is some as no one has answered the questions raised by the article. Namely what is the correct version to have installed for the fixes and how to check if file corruption has occurred. As the article states there is supposed to be a 6.1.67 version of the kernel that solves the second issue in its report of the problems with these brown bag releases as they call it around here, for this type of messed up situation that is used to hide your face from the embarrassment. All I can see for the latest kernel is the 6.1.66 available. Nowhere is it stated how to find out if you suffered corruption due to the .65 version installed or is there any sign of the mythical at this point .67 release, the .66 flew out to be available for install by the next morning when it appeared for me to install after the .65 installed the night before. Now I think about it why does apt-listchanges no longer work, I have not seen a change log displayed in forever despite it being installed on my machine and never using anything but a bash shell during upgrades.

      Oh and I remember the dreaded upgrade of the Redhat system mentioned too and their dependency hell, the trying to figure out which you had to sacrifice, a chicken or a goat while dancing naked in the moonlight to ensure success. I went to Mandrake soon after as they had somewhat solved those problems with that, then onto Woody when that came out and really enjoyed using apt.

      1. Liam Proven (Written by Reg staff) Silver badge

        Re: Patch dependencies are nothing like package dependencies

        [Author here]

        > No doubt there is some as no one has answered the questions raised by the article.

        Do tell?

        > Namely what is the correct version to have installed for the fixes

        The latest.

        Note that I clearly stated that 6.1.67 merely reverts the changes from 6.1.66 so they will get re-done ASAP and there will doubtless be a 6.1.68 RSN.

        > and how to check if file corruption has occurred.

        `fsck` ?

        > I went to Mandrake soon after as they had somewhat solved those problems with that,

        Really? Mandrake was just RH + KDE and didn't have a better packaging tool early on.

        It got `urpmi` later but not before `yum` I think.

        1. RedGreen925 Bronze badge

          Re: Patch dependencies are nothing like package dependencies

          "Really? Mandrake was just RH + KDE and didn't have a better packaging tool early on."

          Yeah it solved that junk system you had to go through at the time by having it done for you in advance. Then of course you left out the beginning of the Gnome trolls hate campaign at that time against KDE because they had the audacity to use a dual license model with both commercial and open source solutions. Which continues to this very day even though the hypocrites at Gnome and IBM Redhat are moving all the time closer to locking Linux down with their De Icaza / Pottering , the Microsoft moles they had, and their inspired trash they foisted on Linux. Killing off the do one thing and do it well mantra and the closing up of the repositories they have done.

          1. Liam Proven (Written by Reg staff) Silver badge

            Re: Patch dependencies are nothing like package dependencies

            [Author here]

            > Yeah it solved that junk system you had to go through at the time by having it done for you in advance.

            Indeed so -- but that left you dependent on Mandrake for KDE version upgrades...

            > Then of course you left out the beginning of the Gnome trolls hate campaign

            I did, yes. But I've written about this in the past, e.g. here:

            https://www.theregister.com/2022/08/17/gnome_project_25/

            ... and indeed here:

            https://www.theregister.com/Print/2013/06/03/thank_microsoft_for_linux_desktop_fail/

            But there *were* other reasons than the Qt licensing issue.

            Qt and KDE is closely coupled with C++.

            A lot of Linux traditionalists don't like C++ and prefer plain old C; GNOME targets C developers, and so appeals to them.

            And GNOME >= 3 removes a lot of patented MS design elements from the desktop, too. Former major KDE backer SUSE signed a patent-sharing agreement, as did Linspire and others. RH didn't and so, soon after refusing, released a desktop with way less Win95 influence. I submit that is not a coincidence.

  11. danielfgom

    Timeshift saved me this time

    I'm running LMDE 6 and was hit by this issue. Not only was the wifi no longer working but the entire machine was non-responsive. Even the terminal was hanging. Total disaster.

    Thankfully I had enabled Timeshift after initial installation of LMDE 6 so I was able to restore my system to an earlier time point. Only today, the 18th, has Debian finally released 6.1.67 and everything seems ok now.

    I know of at least one user on Mastodon that this bug led him to ditch LMDE and he did a fresh install of Fedora instead. I'm guessing he's not the only one not running Timeshift who couldn't fix their systems so they jumped distro.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like