Intel energizes decades-old real-time Linux kernel project

Intel announced a move on Wednesday that will inject fresh energy into a Linux kernel project that started close to two decades ago – and was lacking funding and contributors. The microprocessor giant has made an under-the-radar acquisition of Linutronix, a German developer house that provides services for Linux-powered …

[Anonymous Coward]

You know, if this patch becomes mainstream, I could see it becoming a key feature used by desktops to maintain snappy performance under heavy load, and to ensure non-jittering audio and video playback. Little things like smooth playback on mediocre hardware is one are where Linux has lacked for some time. The fact you started a build should not mean your mouse is suddenly sliding through molasses or syrup on the screen. :)

[Tom7]

Not really. There's a reason that desktop operating systems don't generally use hard real-time schedulers; they don't usually produce the best user experience. TBH it's been a long time since linux desktop performance has had problems other than memory exhaustion for me - and this won't help with that.

[Androgynous Cupboard]

Agree, it's not a desktop thing - but it sure would be nice for a Raspberry Pi hardware board I've been working on recently. Having strict control over latencies will remove a lot of the cases where you have to outsource timing-critical GPIO to a dedicated MCU

[Tom7]

Yes indeed, although how many people will manage to use it correctly is debatable - most RasPi GPIO seems to be done in Python, which rather defeats the purpose.

[Bartholomew]

Now that Intel have the guiding hand, the software could be guided into two tiers support. Intel chips getting all the bells and whistles and all other chips would still be supported, but not supported as well.

And judging by the history of Intel (link to video) that is very likely. Or search for "intel compiler lawsuit" - where high performance instructions were chosen e.g. SSE when "genuine intel" was found and slower instructions used when it was not found - at runtime if the program was compiled using the Intel compiler.) Intel currently own multiple popular benchmarking organisations and companies, and needless to say that the benchmarks were tweaked to make one CPU brand look the best that it can.

[sreynolds]

Most people don't realize that the trash is taken out at the worst possible time.

I don't get this proliferation of ECMA (or whatever that javashit is called) and embedded python, possible the most egregious use of embedded, when they show off their small footprints but when all is considered the people they are trying to "help" would be better or trying to understand the limitations of the hardware/

[Tom 7]

(not replying here the above is a imposter (appropriate cheeky icon here))*

One thing about a RT system is its easy to write code that demand attention and without considerable experience the whole thing grinds to a judder and the user experience gets kinetic.

I often wonder if the main reason RT remains the domain of embedded is because it restricts you to a head-full of code that you can actually as a human being understand and control - well a top of the range human being in many cases.

* I have wondered sometimes how I managed to write such coherent posts and then get so pissed so quickly that I forgot writing them. Then I realised the space was missing!

[Tom7]

There are two Tom7s? Wow. I've been using it since 2010, though it seems you did get there first.

[Tom 7]

As the misses often points out!

[Anonymous Coward]

One wonders if "misses" is the plural of "Miss", a misspelling of "missus", or something else....

[Mike 16]

two tom 7s?

Is either of you a prolific contributor to SigBovic?

If not, there are at least three Tom7s.

[Korev]

Little things like smooth playback on mediocre hardware is one are where Linux has lacked for some time

That's not nice...

[Tom 7]

I found the Studio versions of Linux which use some realtime stuff do handle things a lot better until you play with the wrong settings. Real Time is not easy to debug (or at least I haven't worked out how to do it) when something wont let the debugger start.

Anyone know a good website/PDF on this stuff?

[bazza]

kernelshark is pretty good. See https://www.kernelshark.org/Documentation.html.

You can use it to see how every single thread in the sytems was scheduled. You have to compile in FTRACE (if memory serves). One of the modes allows you to see all user and kernel threads runtime and scheduling history like a logic analyser display, which is very useful.

kernelshark is a tool not unlike the kind of debug tools that you get with "proper" commercial RTOS like VxWorks or INTEGRITY. I've used these and kernelshark, and they're absolutely essential as a tool to find out what's happened.

For instance, you can start seeing the impact of CPUs switching power states; I've seen Nahlem XEON based machines, running Linux + preemptRT, go to sleep for an entire 300ms whilst a mode switch happens after the workload has warmed up the CPU a bit. It's quite eye opening, seeing every single user and kernel thread just stop dead.

[sreynolds]

yeah most Android devices are rank but playback seems ok.

[Lars]

@msobkow

I have at least four laptops running Linux, all grownup.

One is a Sony I quite like, the keyboard is very good, the screen is still good and it stays put on the table as it's not a light weight.

My wife got it instead of a money debt, no I never asked for how much.

It was a typical Microsoft fraud, came with Vista and just 1gig. Absolutely hopeless.

Asked Sony if it was possible to add memory, got an answer, surprise surprise, but telling me only they don't guarantee more than one. I managed to add 2gig, bought a new battery and put Linux on it.

Next I had problems getting Bluetooth working, until I realized it didn't have any.

Is't quite fast enough for reading and writing (perhaps I am slow) and for YouTube and such and of course for Solitaire and similar, I think the thing must be about 15 years old, my youngest is about 8 years old.

My point is that it's never Linux that is slow it's the hardware or the software.

[Filippo]

That's a frequent misconception. "Hard real-time" doesn't mean "better performance". It means "predictable and consistant performance".

A system that's guaranteed to always take 100 msec or less to render a video frame would be a hard real-time system. But in actual usage, it would be a poor video player, compared to one that almost always takes 20 msec, but every few minutes has to run GC, or finds the disk busy for something else, or whatever, and then it takes 150 msec.

Or, for another example, you wouldn't want your industrial automation software to miss a step because the CPU was busy making sure your mouse cursor renders smoothly...

You don't use hard real-time where you need something to be fast; you use hard real-time where you need a specific time or less, every single time, or something explodes. Due to the way common OSes and hardware work, that's surprisingly difficult to achieve, even when the requested time is comparatively large - in large part, exactly because they are tuned for maximum performance rather than consistant performance.

[Anonymous Coward]

Must strictly meet constraints, needn't exceed them.

I think that misses the point.

> A system that's guaranteed to always take X msec or less to render a video frame would be a hard real-time system. But in actual usage, it would be a poor video player, compared to one that almost always takes C*X msec, C<1, but every Y secs has to run GC, or finds the disk busy for something else, or whatever, and then it takes N*X msec, N>>1.

A changed your scenario and put in parameters to make a point.

Suppose a manufacturing line handling 1 item every X*1.1 ms. If the computer processing always handles it's task in X ms, it is sufficient, a single lot of 100,000 items can be produced with the required 6 sigma quality control in an hour run. But if 1 out of 1000 items is not handled correctly, either the system cannot operate continually or the quality control goes to hell - either way the system is useless as costs mushroom uncontrollably.

Well there are already real time solutions and we don't need linux - right? The counter to that is that development for a linux system is much easier and more flexible than for most dedicated real time systems - and results in better software.

Whether of not GC-using software can be used is really an orthogonal question. If that GC is reliably included in the X ms. There's nothing to say a GC system couldn't be (or are already) designed to make that possible .

But if every Y secs

[FatGerman]

I remember a time, probably was 20 years ago, when you could build a realtime linux kernel using these patches and get low latency audio performance that put many modern audio workstations to shame. That was with kernel 2.6 though, I haven't been able to build such a kernel since and was forced to move to macOS for serious audio work. Will be good to inject some energy back into this.

[Inkey]

Most ubuntu derivitives can run a low latencey version of themselves and is used in ubuntu studio for sound recording etc , however whats not clear to me is that the few os's that i used this for on embeded systems pi's, beagleboards are not real time systems ... more pre-emtive os's or device tree overlays and to be fair are comparitivly faster than "realtime' os's ... as far as ive read (not done any bench marking myself only read what others have done).

For robots printers and the like the paths are all worked out before execution and all that has priority over the rest of the busses and buffers is error correction... so a slightly larger compute before executing and smaller more managable compute during works bettet on a wider section of hardware...

also that might be why intel are interested as far as safe side execution code and pre-emtive exacution.

[thames]

With Linux you need to use a different audio system for that. The standard one is made for ease of use which is fine for what nearly everyone needs, but there is a different audio system you install for serious audio work.

There is a project to replace both with a single system that does both, and it will probably become the default some time in the near future.

[Paul Floyd]

I think that you are confusing "fast" with "guaranteed response time".

[Anonymous Coward]

In response to all and sundry, no, I quite well understand the difference between "fast" and hard real time. 'Tis many of you who misunderstand that low-latency of user interface operations and media playback are critical to a smooth user experience, and Linux emphatically does not deliver that as it stands. You have to have a fair bit of idle processor and device bandwidth to get smooth playback out of Linux, and that should not be.

Take a look at the design of QNX and its internals before you continue to pooh-pooh the idea - it is rather popular for car entertainment systems because of it's design.

[DS999]

Linux has been working on lowering latency for many years, and already has better latency than competing operating systems that aren't designed specifically for real time.

There are plenty of good reasons why real time schedulers would not be desirable for general desktop or mobile usage, so even if these patches get mainstreamed you aren't going to see e.g. Android using that as its default scheduler.

As for QNX, it has been designed around embedded use for many years. There's zero reason why automotive infotainment systems need hard real time. I mean, set tops like Roku and Apple TV operate far more reliably than any car's infotainment system despite not running real time OSes.

[Roland6]

>There's zero reason why automotive infotainment systems need hard real time.

Part of the reason - for the retreat of real-time OS's, is the massive improvements in hardware performance; real-time on a 5Mhz CPU(*) is a different kettle-of-fish to real-time on a 2Ghz CPU.

Although the 2Ghz CPU opens the door on a whole new world of real-time applications.

(*) Or a 10Mhz 286/386.

[Filippo]

> 'Tis many of you who misunderstand that low-latency of user interface operations and media playback are critical to a smooth user experience

Nobody disputes that, but you seem to not understand that hard real time means predictable latency, not low latency.

Usually, there is a pretty significant trade-off between consistancy and performance, and using hard real-time for the UX could degrade the performance significantly. What would you prefer - a button response taking 500 msec every single time, or the response taking 100 msec 99.9% of the time, and 700 msec 0.1% of the time? The first would technically be hard real time, but it would also have me gnashing my teeth in frustration inside of fifteen minutes.

If "hard real time" could mean "make it just as fast, but smoother", everyone would be using it for everything.

[Paul Hovnanian]

A lot of RT programming depends on allocating system resources to processes which must execute in a defined time vs ones that can be pushed into the background. I fear that the browser and app developers will make those decisions for us. And it will be the adware popups which will win out over the mouse response as it crawls toward the close winfow icon.

[Pascal Monett]

So, two people in their spare time

If OpenSSL is only maintained by two people, then how many other crucial Open Source projects are there out there that are only maintained by a handful of people ?

Sometimes it feels like the Internet only works because of a dozen people . . .

[JamesTGrant]

Re: So, two people in their spare time

Feels like there’s a few thousand things each with a few key folk - but, that’s ok if the thing is open source. Better that than a few thousand things produced by mega corps with obliquely available complied code with companies disappearing or being acquired and dissolved and code/knowledge disappearing.

The good thing about the open source building blocks is that most of it is very readable, the most genius code is often eminently readable, the brilliance being that it very neatly solves a problem and allows the reader to follow along and understand - even if the original implementation required deep and clever original thinking and wizzy efficient coding, once it’s done, it’s a solved problem and hopefully maintenance is about environment/api evolution.

[Justthefacts]

Re: So, two people in their spare time

Well, I think we’re about to find out whether “that’s Ok if the thing is open-source”. Given that a significant fraction of the world’s software engineers are in Russia. A single-point failure is a single-point failure.

[sreynolds]

Re: So, two people in their spare time

I thought most of the commies were trying to rip you off, encrypt your data for ransom or install malware so that they can run DDoS attacks on critical infrastructure or rip you off, encrypt your data for ransom or install malware so that they can run DDos attacks on on critical infrastructure or ...

[Androgynous Cupboard]

Re: So, two people in their spare time

I've had that "the internet broke while you were on holiday" conversation a few times over the years, but hadn't considered the possibility that the internet actually broke for everyone. You'd want to keep your phone switched off for that one.

[David 132]

Re: So, two people in their spare time

As ever... an obligatory XKCD.

[Julz]

Re: So, two people in their spare time

"Sometimes it feels like the Internet only works because of a dozen people . . ."

It is as always has been.

[Anonymous Coward]

Re: So, two people in their spare time

> Sometimes it feels like the Internet only works because of a dozen people . . .

... and sometimes basically ruined by a different dozen or so people, most of them with "CEO" sort of titles.

[nautica]

Sad; really sad.

"...That's left PREEMPT_RT being shepherded by few people despite its use in embedded and industrial electronics.

"The whole Heartbleed issue, remember that?" the source added, drawing parallels between the OpenSSL and the PREEMPT_RT projects. "We found out that OpenSSL is maintained by two people, which is responsible for the entire world. You know, encryption is run by two people in their spare time..."

"...responsible for the entire world. You know, ENCRYPTION IS RUN BY TWO PEOPLE IN THEIR SPARE TIME..."

[adam 40]

Re: Sad; really sad?

Or

"if it ain't broke, don't fix it", and

"too many cooks spoil the broth"

[VladimirOrlovsky]

• So, Intel [and others] is about 20-30 years too lait ;

low-latency communication between controllers, sensors, robots and tooling, and other equipment …

this is a working 'pony' , and NOT ...

shiny and glorious 'Horse' with a lot of hype and operation / maintenance problems, and good only for show off and games for kids ~ VJO

[Tacoman]

Intel creates software that sells hardware

It's been my observation that Intel likes to create software that sells their hardware. OpenCV, machine learning, AI.. They get these projects going because they flourish with lots of CPU. I wonder what hardware they have in mind for the new work on real-time?

[Roland6]

Re: Intel creates software that sells hardware

> I wonder what hardware they have in mind for the new work on real-time?

Well I suspect as a possible replacement for Minix...

[Will Godfrey]

Been using it for years in audio work

uname -a

Linux devuan 5.10.0-11-rt-amd64 #1 SMP PREEMPT_RT Debian 5.10.92-1 (2022-01-18) x86_64 GNU/Linux

That's what I currently have. It makes a huge difference to latency and reducing Xruns when running the Yoshimi soft-synth on an otherwise quite 'normal' computer.

That synth is a beast. There are no samples. The full characteristics of each played note are individually calculated in real-time as it's playing.

[Anonymous Coward]

Hello 1980 called ...

iRMX86 says it wants it's ball back ....

With documentation in glorious techno-troff

[DropBear]

...and this is why I still MUCH prefer a GRBL firmware running ON A MICROCONTROLLER (a humble Arduino Uno) to a RT-enabled LinuxCNC running ON A PC. You want real-time? Stop fooling the fuck around and use actual hardware fit for the job (which admittedly the quoted Arduino example is NOT the best example for in 2022, for that particular job (CNC) - even though it's still doing it in a quite honourable fashion...)

[Anonymous Coward]

It really depends what you are doing. If there is a high computational load (e.g., image processing) the cost benefits of using consumer grade hardware are massive. Ad to that ease and flexibility of development for a further cost down.

[Adam Ant]

But then if you dig into the source code for LinuxCNC, you quickly realize what what a clusterfuck it really is. Just about every variable is declared as volatile in the misguided notion that it will be atomic and shareable with all other processes within the application... Oh, and best not look at all the bits that run with root permissions or the lack of security.

GRBL demonstrates that CNC does not need to be complex, nor does it need a high powered multicore processor to run.