back to article Google offers first part of its in-house M:N thread code as open source to Linux kernel

Google has confirmed it plans to contribute some of its in-house threading code to the Linux kernel, but hasn’t disclosed its motivations beyond a desire to share. As described by Googler Peter Oskolkov on the Linux kernel mailing list, the tech is “an M:N userspace threading subsystem backed by Google-private SwitchTo Linux …

  1. Anonymous Coward
    Anonymous Coward

    hit the kernel sooner or later?

    TBH, I hope it is much much later. Schedulers are nasty things to code. All sorts of edge cases and race conditions keep popping up especially in General Purpose workloads. These are a million miles away from an Android phone especially on HPC installations. The last thing you want is a CPU hog that you can't kill.

    As long as there is a way of disabling their scheduler (should it ever be included in the mainstream kernel) I won't mind. I'd treat it much like systemd. Avoid if at all possible.

    1. Charlie Clark Silver badge

      Re: hit the kernel sooner or later?

      Why not wait until the code becomes available before you judge it? Google is actively involved in several open source projects and has a pretty good record for the quality of the code it contributes. If anything, critcism is levelled at it for keeping code to itself.

    2. Tomato42

      Re: hit the kernel sooner or later?

      the problem with Google is that they don't share enough (they love to "vendor" open source code and then put patches on top, internally only of course), and when they do contribute it's rarely test code and with minimal test coverage

    3. Anonymous Coward
      Anonymous Coward

      Re: hit the kernel sooner or later?

      "Schedulers are nasty things to code."

      Back in the 1970s - in the lunch-time pub - the development team came up with a scheduling improvement idea. It was so simple a concept that they did a quick successful test and coded it into the master source.

      For the next 18 months support were writing patches to fix the contingency cases. When it was finally stable it was indeed an improvement.

  2. Charlie Clark Silver badge

    Applicabilitiy for Android

    While I bet the code will be eagerly inspected by teams running server systems, I'm not sure if it has anything to with scheduling for Android, which doesn't have to be that fine-grained.

    1. Dan 55 Silver badge

      Re: Applicabilitiy for Android

      Maybe they plan on finally making Android useful for audio apps.

    2. Rich 2 Silver badge

      Re: Applicabilitiy for Android

      'Android for TV' applications and that sort of thing could find it useful

      1. Charlie Clark Silver badge

        Re: Applicabilitiy for Android

        Do you really think so? Away from the phone functions, which handled separately anyway, I don't see this being relevant for an OS focussed on user input.

        1. Nick Ryan Silver badge

          Re: Applicabilitiy for Android

          More efficient use of CPU resources should be very useful regardless - keeping a CPU running at full speed while it effectively idles doesn't do much for energy/thermal efficiency, making better use of the CPU cycles that are available makes a lot of sense.

  3. DarkwavePunk

    This could be quite tasty.

    Especially for server use, but even for personal. I've never really understood complaints about high load or memory usage etc. I paid good money for a gazillion cores and umpteen gigabytes of RAM - bloody well use them. Obviously not just for the sake of it, but if this gets more performance out of your kit, why not eh?

    1. Claverhouse Silver badge

      Re: This could be quite tasty.

      Conceivably there are po folk out there who don't have a gazillion cores and umpteen gigabytes of RAM.

    2. Anonymous Coward
      Anonymous Coward

      Re: This could be quite tasty.

      "[...] but if this gets more performance out of your kit, why not eh? "

      IIRC some multi-core processors used to be limited by overall heat considerations. All cores couldn't run at their potential top speed simultaneously - but an individual core could have a turbo boost while others were lightly loaded.

    3. mevets

      Re: This could be quite tasty.

      Tradition. Open almost any Operating System Design text written during the golden age, and it was all about minimally deploying resources. This is the fundamental tenet of time sharing systems. Busy-Waiting was considered an anti-pattern.

      Fast forward to where there are an embarrassing richness of cores, but a dearth of operating systems designed to maximally deploy resources, and busy-waiting has been renamed lock-free programming.

      Give it a few years, it will switch poles again.

  4. DJV Silver badge

    "Threading technology is widely used across Google."

    Ah, an answer that's on par with a lot of their search results - slightly informative but not exactly what you were after.

  5. Anonymous Coward
    Anonymous Coward

    M:N was already there a while back

    It seem that M:N scheduling was in the linux kernal prior to version 2.6 - though there's no saying how the old version compares with the code Google wants to share now; https://stackoverflow.com/a/4249245/1838819

    Methinks this will be a hard sell to Mr. Torvalds to get it into the kernel

    1. Ken Hagan Gold badge

      Re: M:N was already there a while back

      There has been a general move away from user-space threading. I think they started life in Solaris. Microsoft have deprecated their "fibers", Linux added and then removed its M:N scheduling. Apparently it is one of those superficially attractive ideas whose deficiencies only become apparent after you've invested a year or two.

      This is essentially the stackless versus stackful debate for co-routines, and languages are increasingly coming down on the stackless side of the fence. As far as I can tell, the main reasons are:

      If you don't know which thread a routine is going to run on, you can't safely use any OS API that depends on per-thread kernel state. If you can recode so that there are no such dependencies, you've basically done all the work needed to refactor your program as a (kernel) thread-pool feeding on a queue of work packages. If you can't recode in this way, kernel threads aren't *that* much more expensive and work as intended.

      1. mevets

        Re: M:N was already there a while back

        Tide in, tide out.

        Popular modern programming languages, most notably Go, lean heavily on M:N models.

        On the other hand C++ is looking to, ahem, standardize this model, so maybe it is doomed.

        Really it is all about applicability, and avoiding kernel calls. The trap latency of modern processors is obscene, even though the architecture manuals mumble about a few cycles, that carefully dismisses making the write-fifo visible, cancelling all speculative results and branch predictors, and thanks to some benchmark-embarrassed cpu-designers, a bit of cache flushing. So after the trap, the processor trundles along like something from the 80s for a bit until it can get its funk on, only to go through the same throttling to get back to useful work. To paraphrase a brilliant processor designer "I shed a tear every time I see a syscall".

        So, instead, adopt a virtual cpu model, with rapid switching between local threads on internal interlocks, and fall-back to a set of parked system threads for handling the laborious system calls. If you look at the google mutex handling code(*), it is at once as brilliant as it is frightening; especially when you realize how much of it is just to avoid system calls.

        (*): https://github.com/abseil/abseil-cpp/blob/master/absl/synchronization/mutex.cc

  6. bazza Silver badge

    Well...

    ...isn't this just amounting to cooperative multitasking within a single kernel thread, and having several kernel threads doing that?

    1. Nick Ryan Silver badge

      Re: Well...

      Pretty much I'd guess, although previous experience with fibers it was something that an application largely managed rather than the operating system. This reads like it is moving it towards more of a standard OS feature where the OS can manage them instead. Although as it is something that existed previously, this is an upgrade to existing functionality rather than brand new.

      1. bazza Silver badge

        Re: Well...

        I guess so, but it still feels to me that if an application has knowledge that can be used to aid the kernel in scheduling threads then it might just as well bypass the kernel altogether. I know there’s shortcomings to that approach (signal delivery, major revision to source code, etc), but in the quest for ultimate speed all sacrifices must be made. This mod feels like a half way house to avoid rewriting a lot code “properly” (granted that is subjectively judged, depends on the cost benefit analysis, etc).

        I wonder if Fuchsia has this in it already?

  7. amanfromMars 1 Silver badge

    If I were a cynic and not psychic, one would imagine such largesse psychopathic.

    Thanks for all the phish and here's some red herrings resulting ........ live product for blind tasting/mass crash dummy testing ...... is what immediately springs eternal to the front of engaging minds here. YMMV.

    Nice one, Google. A Very IMPressive Daemon which no one sees coming ...... although it does open up a whole host of new dedicated cans of worms methinks, requiring additional superior advanced remote virtual command and control for leading practical operations ....... and that is guaranteed to have both the competition and opposition from the West to the East in the likes of a Microsoft and Apple environment/semiconductor microprocessor infrastructure, green with envy and energised to fight for the right to also wield such a might. And they'll all come a'poaching and throw big money bags at both prime and sub-prime drivers/systems movers and shakers. It is just what they can only do in order to try and prevent their very own Meltdown/Systems TakeOver and MakeOver/call it what you will in AI Narrative Pump and Dump Circles around Magical Round Tables.

    1. prandeamus

      Re: If I were a cynic and not psychic, one would imagine such largesse psychopathic.

      Whatever your medication, I suggest you resume it immediately.

      1. Anonymous Coward
        Anonymous Coward

        Re: If I were a cynic and not psychic, one would imagine such largesse psychopathic.

        "Whatever your medication, [...]"

        You're new around here? Treat amanfromMars posts as purple prose that contain rational statements. A bit like reading James Joyce's Ulysses.

  8. prandeamus

    Whatever your medication is, please resume it immediately.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like