NASA rover coders at Intel's Wind River biz axed – sources

Intel-owned Wind River – the maker of the VxWorks software used in NASA rovers, spacecraft, military computer systems, and industry – has laid off a number of its most experienced staff, sources tell The Register. We've learned that some of the engineers hit by this quiet "reduction in force" have been with the Alameda, …

[Paul Crawford]

Sad to hear

At one time the main argument for paying for software was you got knowledgeable and experienced support from folk who had made those mistakes years ago and would not do it again on your watch.

Now that is simple an expense to be gotten rid of to shore up some CEO's stupid purchase losses elsewhere.

[Christoph]

So next time the Curiosity rover has a software problem. they'll phone support and get "John from San Francisco" who has a very strong Indian accent?

[Anonymous Coward]

Ha, ha. I get "Armando from Monterey" since I have a Spanish name!

[Anonymous Coward]

"So next time the Curiosity rover has a software problem. they'll phone support and get "John from San Francisco" who has a very strong Indian accent?"

And they'll spend 10 minutes trying to explain to him that a "Problem on Mars" doesn't mean there's an issue with the confectionary vending machine.

[Anonymous Coward]

Yes, and soon the rover will start to move in a Bollywood ballet way...

[Dr Trevor Marshall]

1000 staff and no OS engineers sounds par-for-the-course

At least they won't be illegally exporting again - nobody will want an unsupportable product...

[asdf]

brave new world

One would think Intel had to know CPUs would become fast enough and become a commodity at some point. Having worked indirectly with some of their management though its easy to see how they would put their head in the sand and now panic when this new reality hits. My guess is the decline of Wintel is going to hit Intel harder.

[Anonymous Coward]

Topical

I've just finished reading The Martian (now a major motion picture etc...) and VxWorks actually gets a name check in the book. I haven't seen the film yet, but will look out for it when I do.

I've worked on VxWorks-based systems in the past and it was certainly a reliable, compact and well documented OS. It was bloody expensive though, and they had a weird licensing model where you were supposed to pay a fee per "project", which was something of a loosely defined concept as I recall.

That was back in the day when you might use VxWorks in consumer products. These days it's all Linux. I guess that this is part of the problem for Wind River.

[bazza]

Re: Topical

Linux has certainly had a big impact on their business. I heard that VxWorks never adapted well to multi core processors. Sure they bolted multi core support on top, but didn't do it as thoroughly as they might. Linux has left VxWorks behind performance-wise on multi core processors.

When I first started doing big embedded systems it was all VxWorks, VxWorks, VxWorks. Now it's Linux all the way, with the premp-rt patch set being good enough for all but the most demanding of applications.

Now that Linux has grown tools like kernelshark the there's not a whole lot of technical advantage in the proprietary IDE WindRiver had. In fact, Tornado was at its zenith on Sparc/Solaris, primarily for debugging. You could have a separate debugger running for each thread (well, task), and that was amazingly useful. You cannot do that even today on visual studio or gdb as far as I'm aware. The Windows version was rubbish in comparison.

[cageordie]

Re: Topical

Sadly that's what you get when you push most investment in to areas of the product that nobody ships; a graphical tool nobody uses and a stagnating operating system. Very sad.

[Yet Another Anonymous coward]

Re: Topical

So what do you do if you have a load of highly skilled OS engineers and your very rich customers with massive expensive support contracts are all demanding Linux.

1, Retrain, figuring somebody who has spent 20years understanding all the problems and pitfalls in one Posix based RTOS will pretty soon pick up another Posix based RTOS

2, Sack everyone hoping to pick up some cheap Linux programmers to rebrand an off the shelf Linux distro and hope your customers don't notice.

[thames]

Re: Topical

@Yet Another Anonymous coward - "So what do you do if you have a load of highly skilled OS engineers and your very rich customers with massive expensive support contracts are all demanding Linux."

Yes, but are the customers demanding it from Wind River, or are they going to other companies? It could be that Wind River's customer base is drying up.

That's the thing that customers like and vendors hate about Linux, there's very little vendor lock-in.

To be honest, I was very surprised when Intel bought Wind River. If I was a customer at that time, I would have been planning my exit strategy ASAP. Intel is rarely the first choice for CPUs these days in embedded markets, and having your OS supplier bought up by a company desperate to get (back) into the embedded market can only be bad news. It would raise serious questions about Wind RIver's continuing commitment to the CPUs which make up the bulk of the market, and it would potentially raise problems when working with the chip designers of ARM, MIPS, Power, etc because of concerns about maintaining confidentiality.

This latest news isn't going to help their credibility with customers.

Given all these factors together, I'm not surprised to see Wind River's future starting to look a little uncertain.

[Yet Another Anonymous coward]

Re: Topical

The advantage of using vx-works on your airliner, nuclear powerstation or space mission is that you can talk to engineers who really understand it when your mars robot hangs because of a race condition. So the obvious thing to do is to continue charging them the same but to support Linux.

In the short term there are going to be an awful lot of vx-works engineers making many $k/day as consultants. A lot of the kit running vx-works is going to be mission critical for decades

[bazza]

Re: Topical

@Thames,

"Intel is rarely the first choice for CPUs these days in embedded markets..."

It depends on which particular bit of the market. I can assure you that in the high performance embedded market (radar, etc) it's Intel all the way for newish projects. That's simply because Freescale (now NXP) failed to deliver a decent PowerPC with decent math performance. Not surprisingly I suppose, the market is perhaps too small.

Once upon a time I can remember it being the other way round. I can remember 400 MHz 7410 PowerPCs being way quicker than 4 GHz Pentium 4s (mostly down to Altivec, the PowerPC equivalent of SSE). In my opinion Altivec is still better than SSE, but the first Nehalem Xeons were just monstrous enough to overwhelm anything the PowerPC world was selling, especially as IBM canned the Cell processor.

So in short, if you want a lot of embedded CPU maths the only choice left in the market is Intel.

Which also implies Linux. The Linux kernel works better on Intel's big chips (Xeons, etc) than VxWorks apparently.

Embedding Linux (with the premp-rt patches) on Xeons and making the whole thing real time is really, really difficult. It can be done however, and these days it is about the only option left if you want to crunch a lot of numbers.

Freescale did very well out of the telecommunications market with their PowerQICC range without Altivec.

[cageordie]

Re: Topical

There was a perception that Linux is free. Well it isn't. When I worked for Motorola our group had four full time Linux maintenance engineers trying to hack the kernel in to a form that worked for us. When I worked on defense programs we bought VxWorks and then didn't need to worry about reworking the code. The code for VxWorks is cheap to obtain now and the Chinese and Indians were shipping fake copies even back in the late 90s. Our embedded Linux machine needed a flash drive and a whole load of RAM, a typical VxWorks system would load from ROM and was only a few hundred kilobytes. People forget that the money they pay to Wind is trivial compared to the cost of feeding and watering a free Linux installation.

Linux spends a lot of time doing its own thing, so you have to code around that and hope you get your event serviced in time. VxWorks had very little latency and wasn't doing a bunch of time sharing machine work when it wasn't doing what you wanted. For many that doesn't matter. But for those working in tight real time, things that are still life or death performance critical, like RADAR, ECM and ESM for example, then Linux just won't do.

[Anonymous Coward]

Re: Topical

@cageordie: There was a perception that Linux is free. Well it isn't. When I worked for Motorola our group had four full time Linux maintenance engineers trying to hack the kernel in to a form that worked for us. When I worked on defense programs we bought VxWorks and then didn't need to worry about reworking the code.

That's my point. The economics are different in CE. I have worked on defence projects in the past, building small numbers of very expensive bespoke systems for a specific function. When you are doing that, the cost of a development licence and per-unit royalties just disappears into noise.

In the consumer electronics world however you can spend an afternoon arguing, for example, over whether it's absolutely necessary to fit a resistor costing fractions of a cent - so you really don't want to pay royalties if you can avoid it. You also have armies of programmers, many of them in countries with a low cost base. So you don't want a developer licence fee either.

Don't forget all the ancillary guff that comes with Linux too - graphics libraries, web browsers, you name it. You get a huge choice of packages with the least porting effort.

In many environments, Linux makes a very powerful argument. It might not be perfect but it can be made "good enough" - it's usually as good or better than the software that goes on top of it (unless you are actually building a Mars mission of course).

[Anonymous Coward]

Re: Topical

AFAIK, Canon DSLR camera software is built on VxWorks. I'm not sure trying to transform a general purpose OS (Linux, whatever) into an RTOS works well for most demanding tasks. IMHO, an OS designed from the beginning for it is better.

But it's also true many companies now are trying to believe Linux or its derivative Android could be used as embedded OS for many applications, because of the low price alone, and I can foresee lots of devices with software issues because of that.

[Chemist]

Re: Topical

"AFAIK, Canon DSLR camera software is built on VxWorks."

Not since ~2007, it's now DRYOS. https://en.wikipedia.org/wiki/DRYOS.

DRYOS aims to be compatible with µITRON 4.0 and with POSIX.

[cortland]

Re: Topical

-- you can spend an afternoon arguing, for example, over whether it's absolutely necessary to fit a resistor costing fractions of a cent --

Not just consumer electronics; "been there, seen that" in other markets. Though getting things changed in the consumer realm is a veritable Augean stable, so to speak.

[HmmmYes]

Re: Topical

Well, Linux if you have an MMU.

FreeRTOS is its a plain processor.

The most reliable RTOS is the one that runs on the most systems.

[Will Godfrey]

Unbelieveable!

So the lunatics really have taken over the asylum then.

When the shit hits the fan (and it certainly will) I hope these guys charge $10,000/hour to bail the idiots out.

[Anonymous Coward]

Re: Unbelieveable!

They will get more than that in advance. Windriver software devel is primarily in California.

In a redundancy situation in Cali you can request stats from HR and they are legally obliged to provide them on age, salary and gender of the employees given the boot. If Intel has performed a selective cull of grey beards that will not go well under Cali discrimination laws.

[Mage]

Agism and fools

Idiotic

[bazza]

Re: Agism and fools

Maybe idiotic, but companies that have done it the other way round have ended up in trouble too when all their remaining staff retire...

If they're trying to save money then that's probably a symptom of the management trying and failing to grow a business. Things aren't easy out there for proprietary expensive real time operating systems.

This is probably some MBA trying to be a clever arsehole when really they should accept that the business has filled the market, especially now that Linux has pinched so much of it, including their own Linux distro. Also selling Linux ain't natural, especially if some of the "magic add ons" are actually someone else's unacknowledged open source efforts... If they want to stay in the market properly then that means taking special care to retain expertise, especially the amount they charge. If they lose too much expertise through these redundancies and the inevitable follow on "fuck this for a lark" departures, they may end up with no business at all.

These days I use Greenhill’s INTEGRITY if I need something like that. It's even more expensive, but is very cool and the company is privately owned by a single individual. If he's happy with his niche then there's no one else to tell him he's wrong, and that's a good thing in my opinion from a long term support point of view. If I were in the market for a properly good rtos then I would look at these redundancies today and wonder about Intel's long term commitment to my project's support if I chose their products...

[cageordie]

Re: Agism and fools

It's not like they replaced the developers they discarded in the past. At least not in this country.

[petur]

Re: Agism and fools

@bazza

I'm so happy I'm done with using INTEGRITY, ran against more bugs than we wanted to handle, and the standard response was that there was a newer version just out, please try that. No we don't know if your problem is fixed, but go through the whole hassle of upgrading anyway, risk adding new bugs, only to see the damn bug is still there.

And in the end, a lot of their code is just ripped out of FreeBSD, at least they left enough of the headers so you could search there for fixes.

Oh, and charging 1000 EURO for a 10cm flatcable with a propriety connector for the debug probe? No thanks.

[bazza]

Re: Agism and fools

@petur,

"I'm so happy I'm done with using INTEGRITY, ran against more bugs than we wanted to handle, and the standard response was that there was a newer version just out, please try that."

Sorry to hear that. I have to say my experiences with INTEGRITY were all good. This has all been in the past 4 years, version 10. The support I got here in the UK was very good indeed. We didn't try to run on anything out of the ordinary, stuck to carefully chosen standard x86 hardware, and didn't need a magic cable either. Maybe that was a good choice. In that particular system it was very good value for money.

The drivers may be derived from BSD, but they're not really part if the OS proper. The clever bit, the kernel, is all theirs AFAIK.

It's so hard to put an off the peg OS onto bespoke hardware, Linux, VxWorks, etc they're all hard. Someone somewhere has to put the lot of effort into software support. Linux has put a lot of effort into SoC, which is a tremendous help.

[asdf]

Re: Agism and fools

>and the company is privately owned by a single individual. If he's happy with his niche then there's no one else to tell him he's wrong, and that's a good thing in my opinion from a long term support point of view.

Unless he truly doesn't care about money I would hesitate to think a single individual owner is a good thing for long term support. What happens if say Oracle comes along and decides it needs his companies niche to grow its portfolio or even worse he retires or dies and his coke head son takes over?

[bazza]

Re: Agism and fools

And if Intel do screw the vxworks side of the business that will seriously hamper a number of really quite important projects for Uncle Sam's DoD.

When Apple took over PaSemi, they canned the PowerPC chip straight away. Unfortunately for Apple Uncle Sam made them keep it going, because it had already been built into some military kit and Uncle Sam wasn't about to redevelop it simply because St. Jobs didn't like it. (And of course Apple really wanted the PaSemi staff, but they all buggered off to form Agnilux, leaving Apple with nothing. Ha!)

Same thing could happen here. If VxWorks support starts becoming ineffective then Uncle Sam might start getting very cross, and Intel might have to be forced to go cap in hand to the people they've just sacked. You take on a product with very long term support promised to some customers with big sticks, don't be surprised if they wave those sticks at you.

[Destroy All Monsters]

Hmmm.....

Guess the next version will be written in PHP. Cheaper that way anyway.

Is that Catbert I hear?

[bazza]

Re: Hmmm.....

You owe me a keyboard.

[ben edwards]

Staff who worked from home were targeted? It could be that the guys were being lazy and really not working when at home - you know, an open secret. You can always tell who works from home and who doesn't.

[Ragequit]

Working @ home...

Maybe the senior staff had such large salaries they decided to subcontract their work overseas? It's happened before with people in an office. It'd be even easier to pull off the scheme from home. Joking aside, if uncle sam really does have projects that need their expertise it wouldn't be surprising if they found themselves being recruited by the defense contractors (or NASA). Or for that matter coming back to work for Wind River as contractors. Maybe Intel is looking to sell it off and wants to gut employee benefits to make it look more appealing on paper. /shrug who knows.

Long story short I bet the guys who lost their jobs won't have too much trouble finding work.

[cageordie]

Re: Working @ home...

LOL! Wind never paid well. They employed a lot of H1 slaves and paid well under the market rate to people who were stuck with them because it was that or return to their country of origin.

[Anonymous Coward]

Re: Working @ home...

>They employed a lot of H1 slaves

Ah so in that regard their culture matched Intel's.

[x 7]

"One of the axed employees..............provided support for the real-time operating system on radiation-hardened processors"

""They have laid off the last engineer who worked on the version of the operating system used on spacecraft.""

"In October last year, Wind River was fined $750,000 for exporting encryption to China, Russia, Israel and other countries."

I hear there are a number of new vacancies in China........hardened ballistic warheads interest anyone?

[MondoMan]

"the axe appeared to fall heavily on those who worked from home."

I think the potential issue with working at home is more the getting left out of social interactions at the office. It's always easier to let go those who you rarely see in person.

[Paul Crawford]

Re: "the axe appeared to fall heavily on those who worked from home."

Other wise known as "keeping the arse licker"

[Otto is a bear.]

Re: "the axe appeared to fall heavily on those who worked from home."

Yep, managers like people they can see, if they can't see you, you don't exist when it comes handing out the jelly beans. It has always been thus, stay in the office until the manager goes home then you're a hard worker, go home before him and you aren't showing commitment. A long time ago, when PCs were z80 powered, I worked for a well known multi-national who gave its engineers the option of working from home, and those who took it were the first to go come the redundancy program. It really doesn't matter how effective you are, only how effective you are perceived to be.

Don't forget most senior execs know nothing about the businesses they manage, in fact for many it's a badge of honour. Let's hear it for the MBA generation.

[Anonymous Coward]

Plenty of near misses on Mars missions

There have been at least several near misses on or inbound to Mars, typically involving memory issues that just maybe the OS of the day could have done a better job managing. The boffins managed to save them each time, but still... ...not exactly stress free memory management.

Apologies in advance for being overly critical at this time. But claims of ultra-reliability combined with several examples of Mars missions CPUs crashing due to memory management problems seems odd to me.

[cageordie]

Re: Plenty of near misses on Mars missions

Hmm. Where do you make up your 'facts'? Back around 97 NASA decided to ignore advice to set the inversion safe bit on semaphores and had to patch the thing once it got there and something they once saw turned in to a lockup every few minutes. Then there was the autonav failure where the autonav asked for Newtons but the engine folks didn't do metric and gave them pounds, so they slowed too much and burned to a crisp. And there was the weight on legs switch that was set by the legs flipping open, and latched, and not reset when the landing radar got an instantaneous echo... so they switched the braking motor off at around 20m and dropped the lander the rest of the way. There have been memory faults, but those are hardware failures, they are expected when you operate with little shielding in the hard radiation of space for years on end. The Mars shots that JPL engineered, rather than Lockheed, worked and have exceeded their expected life. VxWorks is a proper realtime operating system, not a hacker project or a Microsoft mess.

[Unicornpiss]

It always disgusts me when companies do this..

Laying off your senior talent, even if they aren't cheap and appear curmudgeonly and not as productive as younger staff is a true dick move and a good sign that upper management and/or board have little comprehension of how a company does what it does. Let's reward the people that helped us get where we are today with the hardship of having to involuntarily find another job.

In the long run when companies do this, they just screw themselves. As another poster mentioned, experienced people can pick up another facet of the business and their experience is transferable. If nothing else, it takes at least a year typically IMHO for a new hire in a technical field to learn the specifics of how a company does things, policies, why not to do it this way, etc. In the end a need always arises for that experience that you've thrown away for a short term gain and to fill the vacuum you end up with twice as many eager new hires that while perhaps gifted, will make the same mistakes that the people you laid off already made. If you're developing a project, you no longer have the mentors to guide the process and dev time and quality inevitably suffers. The auto industry and its numerous recalls are perhaps an example of this, as is all the knowledge painfully gained in the 1960s when the US had a thriving space program, now mostly forgotten, leaving us unlikely to ever go to the moon again without another decade of costly, dangerous effort. I sometimes like to joke about the company I work for: "We spend more money to save money than any other place I know of."

[Anonymous Coward]

intel in bermuda triangle for companies... companies come here to die.

there are several such examples.

but management is very arrogant to accept thier mistakes so they keep doing the same thing.