Why an embedded OS is like a mammal

Embedded software applications are like the mammals of the software industry – while the monolithic dinosaur applications slug it out up in broad daylight, the embedded apps scurry about in the undergrowth doing what they do best, hidden from all but the most observant. While this isn’t an analogy you’d want to push too far, …

[John Smith 19]

could, but won't

The greatest *achievement* of Microsoft is to persuade a generation of IT (and management) types that bought software *has* to be rebooted regularly. "You have to understand, says MS apologist, we can't afford to make it any more reliable"

Thisi s widely enough believed (and Windows installations badly enough implemented) to be a self fulfilling prophecy.

Comms managers (who operate computer controlled PBXs 24/7/365) know this is wrong, hence MS Exchange (ever wondered about that name) fail in this market.

Embedded developers also know that is not the case and know their chances of making their systems rise when they don't use Windows embedded.

[John 62]

a car wheel balancer

I got a puncture in one of my car's wheels fixed yesterday and the balancing device had a big computer display with fancy graphics. No tell-tale UI elements tho, so I'm wondering what OS it used. Tho it could be just one big C program if it doesn't have to do anything else.

A friend of mine used to work for an aircraft engine controller supplier. They used a special language that didn't have loops (he didn't say if it was a functional language) so that the controller could never, even accidentally get caught in an infinite loop, or iterate once too many times.

[Prag Fest]

Going deep

A curious article, I'm not sure if you quite grasp the true nature / requirements of a deeply embedded system. Nothing personal, even amongst software and hardware guys it's a pretty specialist branch of the profession that few ever really get to grips with.

It's quite feasible to loose the OS (although not as much these days with good cheap OS's readily available), but trust me, no decent embedded engineer in their right mind would be using Windows CE for something that needs rock tight performance and reliability.

[Anonymous Coward]

Forced in

"no decent embedded engineer in their right mind would be using Windows CE for something that needs rock tight performance and reliability."

Sometimes you don't have a choice. We use Linux for everything we can, but a new project requires the use of a particular third party library, that only runs on MS OS's. So Windows CE 6 it is. I see it as adding variety to my CV. I suspect a lot critics of CE have never actually used it and assume it has similar "features" to it's desktop equivalents. At least I'll soon be able to base my opinion on actual experience.

[Anonymous Coward]

"when it simply just has to work"

Perhaps someone could tell the consumer electronics manufacturers that their systems are supposed to "just work" too. None of mine seem to. From a famous name DSL router that won't restart PPP after it times out but hasn't lost sync, to a famous name Freeview PVR that has managed to lose all my recordings and has quite a few other oddities too. And if I had more time I'd name more examples.

Why does (almost) nothing with a modern computer in it work right?

[Mr Pedantic]

Blue endoscope of death

You don't want a blue screen of death when using an endoscope pretty much sums up why Microsoft does so poorly in the embedded market.

[Boris the Cockroach]

Not news

to us down at the industrial robot programming front

A popular make of control uses linux as an OS or is available based on a version of windows

Guess which one sells more?

Guess which one I trust not to decide to rotate 180 degrees and stick the welding head through the operator?

Oh and we do have one running on windows.... and sometimes it does something weird just for the hell of it

[Anonymous Coward]

OS/2

Don't forget good old OS/2 - a lot of ATMs (cash machines) run it because the banks like it for its security and stability. I've seen quite a few BSODs on Windows ATMs but the OS/2 ones just seen to run and run.

BeOS has a small cult following in embedded space also.

[Mark Aggleton]

I find that difficult to believe.

Surely OS/2 isn't supported any more so wouldn't pass PCI compliance.

[Bruce Ordway]

Scale please

>Developers and managers of other application types

>could probably learn a great deal from how these well-concealed systems are built and deployed.

I'm thinking the embedded systems you describe do things very well and just work, but...

they only have to do a very few things

What about the the grey area between true developers and manufacturing. For example the people who "Program" PLC's and OIU's.

I'm not a fan of CE, but it is about the only option for them anymore.

[jake]

@Bruce Ordway

"What about the the grey area between true developers and manufacturing. For example the people who "Program" PLC's and OIU's."

Most of the kit keeping an eye on processes in my winery comes from PLCs, the OIUs are mostly dumb terminals. Neither are actually programmed, but rather provide raw data (PLCs), or display that data (OIUs). The actual programming comes in between "raw" and "display". If you don't understand the actual process involved, you are screwed ...

"I'm not a fan of CE, but it is about the only option for them anymore."

Bullshit. I've been doing this with *nix for over thirty years ... and with PDP assembly language and OS-free PDP hardware and dumb-terminals longer than that (and early Apple 6502 kit, occasionally, back in the day). Glitter distracts from raw data and interpreting that data.

[Tzael]

Bad data

You did a poll and opened it up to the masses instead of restricting participants to only those who actually work with embedded operating systems? We're supposed to take the results of your poll seriously?

If the data is actually sourced from a more reliable selection of participants then I would definitely appreciate knowing more about its origins other than "in a recent poll". The data presented in the article is seriously flawed and does not coincide with first-hand experience working with embedded operating systems.

[sT0rNG b4R3 duRiD]

^ This

Honestly, I would have expected most embedded applications to be 'bare metal' code or a tiny RTOS either in-house or commercial.

Embedded applications of course vary in complexity - simple things like digital watches obviously are abundant and all into the above category.

[Visual Echo]

Teats.

That's why things are like mammals, they have mammaries. I love operating systems with big bouncy honking teats.

[Adrian 4]

Predictable responses

It's hardly surprising, given the history.

Whatever the strengths of WinCE, the reputation of Windows was always going to make it tough. It's seen by embedded engineers as unreliable : the idea of rebooting to keep it working is laughable. This may be less deserved now for both WIndows and CE, but it's a tough reputation to lose.

Certain people do find CE attractive, mainly if they have windows developers available who'll find it familiar. But that's only in newer markets like set top boxes and media centres, not the traditional embedded device market. It's difficult to see what appeal CE would have for a device that doesn't have a screen.

The surprise is the popularity of Linux ; it was equally seen by those same traditional embedded engineers as being a desktop OS - huge, slow, non-real-time and having an inconvenient licence.

But as hardware platforms have got more powerful, the availability of source at low cost makes it more attractive than commercial options and the widening choice of ready-made software for it seems to have come at just the right time to let CE fizzle out.

[heyrick]

Well, duh! :-)

We are used to bugs in our desktop applications. The web browser crashes, you grumble and restart it. The word processor crashes, tick another pound/euro/dollar to the millions no doubt wasted each year while the employee restarts. Windows XP works well for me, but I remember older versions where BSOD was a catchphrase, along with the fix-all solution of "just reformat the harddisc and reinstall everything", an exercise in controlled nervous breakdown before products like Ghost were around (and even that has bugs).

Desktop machines crash, or the applications crash, or something screws up. We've come to accept it, albeit the situation seems much better now than a decade ago.

What do you do if your car's engine management system crashes? There'll no doubt be a watchdog to restart it, but if you were doing 110kph down a motorway, you might not GET the chance for it to restart.

One of the biggies, for it had a real crowd-pleasing explosion, is the Ariane's integer overflow failure - http://en.wikipedia.org/wiki/Ariane_5_Flight_501

While that lost a load of money and probably sent a bunch of scientists into therapy, nobody actually died.

Sadly, buggy software used in critical elements can kill people, and does - http://en.wikipedia.org/wiki/Therac-25

I think, all in all, the entire design of software for an embedded system is vastly different to a desktop machine. Any reasonable software engineer used to embedded systems would probably die if people thought they could code for it using a Visual Studio-like interface! An embedded system (printers, satellite decoders, smartcards - and don't forget the recent issues with millions of German bank cards) are deployed and expected to "just work". Some of these devices may have the ability to be updated (remember the Freesat box failures last year?), or even to self-update. The majority probably won't. They just have to work. End of. And for this reason, it is pretty clear why stability is the most important factor.

It is a shame that your OS chart is only partially complete. It would have been much more interesting if it included "Custom OS (or RTOS)" and "No OS" as options.

[Anonymous Coward]

Embedded S/W

"Unsurprising perhaps is the dominance of Linux in this space"

Actually, while not massively surprising, I do find this interesting (and yes, a little surprising).

I have been writing embedded software now for about of 20 years. While there is no doubt that Linux is starting to make inroads, it is only relatively recently, and there is an awful lot of embedded stuff out there running on other (usually much smaller) OS's. You seem to only mention the "larger" OS's (like Linux, Vx Works, OS9 etc), presumably leaving the small embedded OS's like Nucleus, pSOS, and the zillions of in-house kernels to the "others" section. I would expect the use of these OS's to dwarf the use of Linux in the embedded world. I would certainly not consider Linux "de facto" in the embedded environment.

In a true embedded environment, you usually do not need or want the sophistication of something like Linux - it is a large and complex system, which you can usually do without. The other biggie that you seem to be forgetting is that Linux is NOT a real time OS. UNIX (and I include BSD and Linux in this) has never been real time and is very unlikely ever to be. Yes, I know there are "real time" versions of Linux, but these are still often not "real time" enough.

Basically, you seem to be looking at the embedded world as if it was just and extension of your quad-core Xeon desktop PC, and this, is simply not accurate. This is (almost) the case with devices like the latest mobile phones or pocket computers with their MBytes of storage and processors faster than your desktop was 5 years ago, but devices like these are not really "embedded" any more - they are just "normal" computers running (almost) desktop applications. They're just smaller, that's all. It is precisely this muddled definition that allows "Windows CE" to be included in your survey at all - nobody in their right mind would use Windows 'anything' in a true embedded device - it is a non-starter (it's size and complexity alone would preclude it, never mind it's shockingly bad reliability record and the fact that it is basically rubbish).

I have used Linux in embedded environments in the past, am doing so now, and will no-doubt continue to do so. But the systems involved require the sophistication that Linux provides (usually boiling down to its networking support), and, of course, it's cheap! I'm not knocking it at all, but I do think it is a mistake to think that Linux is taking over the embedded world. It may be in some areas, but for each that it is, there are a million others that it isn't.

[heyrick]

No embedded Windows?

1994, Spain, some random el cheapo block of flats. There was a problem with the satellite feed breaking up. I stupidly suggested the recent winds might have caused the dish to move. This was translated by the Brit enclave to be "he knows something about how to get our telly back!" and so I was dragged up to a massive dish. Does it need to go left or right? I said I dunno. We flipped a coin. He took a running jump at the rear side of the dish. No signal. Tried the other side. Signal. Just. With more running and jumping and huffing we managed to nudge the thing to a pleasing signal. This was analogue, so it wasn't exactly difficult.

"What's it hooked into?" I asked. So he took me into... I dunno. Machine room? Attic? I don't know how to describe it. The dish provided eight outputs which fed into a stack of Amstrad receivers (fire hazard, anybody?) which had the UHF outputs fed into a booster and pushed into the internal cable system (all the UHF sockets in the building were in a loop).

Behind me, big motor. Lots of relays - big-ass clunkers. It was the lift control system. A mess of wiring (all the buttons and sensors and stuff) went to a metal box. By the box was a monitor. The bloke turned it on. I thought it was a telly, but it displayed Windows. And a little graphic of the two lifts and what they were up to. Looking closer, I can tell you it was Windows 3 of some description. And there was a "nightboot" countdown. Evidently every night at 3am the system would reboot. The lifts would freeze for the minute or so the reboot took, then they'd return to the sub-ground floor (bottom level).

This wasn't a DIY job, this would appear to have been the system installed with the lifts. Mmmm, I wonder if the machine currently thinks it's the '80s all over again? Hehe, maybe it's got an ST506 harddisc? I can't imagine it being solid state in those days!

This is, of course, NOT to be taken as an example of a good embedded system. I dimly recall that button control was icky - if you pressed a floor button at exactly the right time on reaching that floor, the lift would come to a rather violent halt. In fact the only thing it was smart enough to do is not jam on the brakes and go backwards if you pressed after passing the floor. I bet a few timer ICs and a bunch of logic gates could perform the same functions better, and without the nightly kick in the ass.

[Pigeon]

`tis true

I don't think this has changed in 30+ years. You have to validate O/S and hardware to a specification. If the O/S keeps changing, then the cost of validation becomes excessive. So, the smaller, and more stable the O/S the better. This seems to be reflected in the terms&conditions of common commercial O/S's which say 'not to be used for life&death situations ... etc. blah'.

[Anonymous Coward]

dont forget

about cars and Coke machines. Surely you remember the Microsoft car joke? Simply press the gas, brake and clutch petals to reboot to fix the flat tire.

[Anonymous Coward]

If Only

If any car manufacturer could implement that as a technoilogy they'd make fortunes, you've obviously never changed a tyre on the hard shoulder in the pouring down rain with HGVs playing chicken with your backside.

Oh, and the CTRL/Alt/Del thing was fixed in hardware by IBM although I appreciate the sentiment.

[Anonymous Coward]

IMO

In my opinion this article summarizes the embedded OS use/opinion quite well. You did miss the no OS option - as was pointed out. Often I write the entire OS myself. It's not a full OS, more a collection of drivers and a coordinating kernel, but it often is all that is needed.

[Hungry Sean]

not sure about conclusion

El Reg concludes that general application developers could learn a lot from the hard-core embedded world, but I'm not so convinced-- the problem domains are different. In application development, you're generally trying to pump out features, and bugs, while regrettable, are a necessary evil in the process. You don't have time to get everything right, and I've heard stories of companies where when the bug list gets 80% addressed, the product ships. By contrast, it's damn hard to upgrade a program once it's embedded in someone's anti-lock brakes. I've heard it said (and agree) that the best way to debug an embedded system is to write bug-free code. That entails a slower, more deliberate approach, and (hopefully) precludes the use of code-monkeys. Both considerations combine to increase the cost of the code and time to market.

Similarly, performance in the embedded world is a very binary thing-- if you can meet all your deadlines under the worst possible conditions, and you have enough memory, life is good. Once your code fits, there's no point in further optimization (except maybe to conserve power), and you generally need to throw away CPU cycles to make those guarantees. Application software is entirely different-- all else being equal, code with faster average execution time is better, and if your quicksort occasionally goes quadratic, so be it. You can also come close to 100% utilization.

Different problem domains require different strategies, and while customers may think otherwise, the highest quality solution is not always the best. As much as I dislike it, a broken piece of crud that's delivered quickly is often more useful than a work of beauty that arrives years later. The applications guys generally know what they are doing, realize the limitations of their approach, and overall do a pretty good job of hitting the tradeoff between time to market and reliability.

[Charles Manning]

No OS is the most popular OS

Most embedded systems don't have any OS at all. Most embedded systems are far too memory/resource constrained to have any OS at all. A typical modern kitchen probably has embedded systems in large and small appliances: stove, fridge, dishwasher, microwave, bread machine, rice cooker, blender,..... none of those is running Linux, WinCE etc. They're mostly running small 8-bit micros with 1kb or less of RAM and a few 4 of flash.

Even your desktop PC has significant numbers of embedded systems: monitor, mouse, power supply, disk drives,...

While your fancy car might have a fancy Linux/WinCE screen in the dash, it is loaded with small embedded systems. Each door probably has its own 8-bit micro to handle the switches/electric window etc. Another few in the dash/console, airbag, lights, seatbelt tensioners,.... These micros are cheaper than wiring.

There are quite a few reasons for this:

* Cost: The cheap micros cost less than 50c US. Adding resources needed to run an OS just add cost with no benefit. Any Moores Law that brings the cost of resources down will also reduce the cost of the small micros to 20c, 10c,...

* Boot time: Bare metal systems can boot up in milliseconds. Linux et al need seconds. That's just not acceptable for any critical function.

* Response time: You want that airbag to fire **now**.

* Simplicity: Simple things are typically more robust. It is a lot easier to verify all the code in a system with only 200 lines. Not so if you're running the whole of Linux etc.

So why does the Reg kick up such a poor survey? Well very few of the people doing real embedded work will be reading El Reg. I would not expect a useful output from Reg readers on embedded systems any more than I'd expect good a good result if the survey was run in Knitting Monthly.

[jake]

Question for Jon ...

"When it comes to operations and management criteria, number one in the list was cost of licensing, but only just"

What happens if you throw out the replies from folks who had "no experience or opinion" to any of the embedded OS questions? Methinks the outcome might change slightly, but significantly.

[John Styles]

Not used, poor impression

The Slashdot Weenie (TM) response to anything Microsoft. Roll up, roll up, get your ill-informed prejudice here.

[John Smith 19]

"a broken piece of crud that's delivered quickly "

What I like to think of as "The Microsoft Way."

[Anonymous Coward]

Fantastic comments, thanks!

To all who have commented and indeed responded, thanks very much for helping bottom out this under-served area. Yes, indeed, just picking up on the 'embedded' category is inevitably going to lead to inconsistencies in both categorisation and interpretation, which we do our best to avoid. But it's comments such as these that we can all learn from! Watch this space... and keep them coming.

[Anonymous Coward]

Automotive...

For automotive applications, the OS will likely be an OSEK variant - and these are easily capable of running on 8bit micros with a handful of RAM and FLASH... These micros are found in car doors/mirrors, lighting clusters, instrument panels as well as ABS etc.

(I've been working for companies developing OSEKs for the automotive sector for about ten years now...)

[JaitcH]

When my ATM fails it shows ...

the Windows Blue Screen!

Windows is everywhere and still as useless in machines as it is on your home computer.

[Graham Bartlett]

@Charles Manning

There's another reason related to cost why most embedded systems either use no OS (a simple continuous processing loop), a simple in-house scheduler (using the processor's hardware interrupt priorities for pre-emptive multitasking) or an in-house OS (using some custom messaging sytem). That reason is licensing.

With the exception of Linux, all the others have some cost for the OS, either as a fixed fee or more usually on a per-item basis. When you're churning out a million units, even 1 cent per unit for OS licensing makes it cheaper to roll your own solution for most embedded applications.

[cosmogoblin]

Cash machines

I'm always amused seeing BSODs on cash machines. They show quite a few details of the failure (usually driver-related) - including, I've seen a few times, the machine's IP address.

If anything could tempt me to learn proper hacking, it'd be the thought of connecting to a cash machine and taking control of the dispenser.

[heyrick]

ATM IP address

I phoned the bank helpline of one I saw at a service station, back in 2001 near Guildford. Was put PDQ onto somebody with some semblence of a clue who listened to everything I said, then denied there was a problem and that said cash machine even existed. Idiot.

Anyway, I do not remember the actual IP address, but I remember it started with 10, so it was on its own private network... but then basic security would demand this anyway!

Of course, you could always BUY one and hack it from the inside out. :-) http://cgi.ebay.co.uk/_W0QQitemZ350113176164QQ