Either the Author or the Sub-editor needs a bit of a casual beating....
The Linux Foundation has announced a new hypervisor for use in embedded and internet of things scenarios. Project ACRN (pronounced “acorn”) will offer a “hypervisor, and its device model complete with rich I/O mediators.” There’ll also be “a Linux-based Service OS” and the ability to “run guest operating systems (another …
I certainly wouldn't trust a car that has single computer (ARM or Intel based), running a hypervisor for secure/critical systems and "non-secure" systems, like entertainment.
The other thing is, the entertainment system is outdated in a couple of years, so either it needs to be just a dumb mirror display for a mobile device or it needs to be cheap and replaceable.
The other thing is, the entertainment system is outdated in a couple of years,
Not so sure about that. They more or less settled on Android of sorts which ties them up to the obsolescence curve of a handset. The obsolescence of these has slowed down dramatically - you are looking to a 4+ years run on the current 6-8 core mid-range crop.
The ECU, brakes, etc will not run under a hypervisor to start with - way too realtime to allow even the slightest chance of someone else hogging a resource. The cost savings from running it on a single SoC instead of having dedicated controllers strung up by cambus as we do today does not recoup the additional testing, verification, development and risk costs.
As far as the infotainment - you do not need a hypervisor. At all. All infotainment, navigation and even some comfort level car related tasks like the climate control display can run in containers - they do not need hypervisor level separation. The only thing you get from a hypervisor are overheads.
So I do not see exactly what is the purpose for that in automotive in the first place. Other applications like industrial... Maybe. But that is so niche that throwing more CPU at it to run it in KVM is less costly than keeping a dedicated embedded hypervisor up to date.
@Voland's right hand
I'm willing to bet with you, that you won't be getting security updates for that Android (or Apple) entertainment system in 5 years time, let alone 10 or 20 years time. In the old days, when this was only radio + cassette or radio + CD, this wasn't important, but when these things now have their own SIM cards and data plans, in many cases, that is a huge problem - especially if that unpatched entertainment system is sharing a hypervisor with critical systems.
I'm willing to bet with you, that you won't be getting security updates for that Android (or Apple) entertainment system in 5 years time, let alone 10 or 20 years time
You do not need to bet with me. I am with you on that.
However, there is only a "one small step" between the regulatory requirement that parts for a vehicle are available at a reasonable price for 10 years (Eu, other markets), a workshop manual is available at a price which covers only distribution (Australia), etc and mandatory updates.
It will take a major clusterf*** like the VW service shenanigans from the turn of the century for this step to be taken, but it will. History tends to repeat itself and automotive manufacturers do not read their own history and the legislation which was applied to them for their "prior sins".
So they will get this mandated, do they like it or not. It is not a question of if, it is a question of "when".
We can hope.
I would require from the *arstewards:
a. Full workshop manual, dead tree copy, with all vehicles. Also on the net. (You need a paper copy in the workshop!) In the native language of where you got the car.
b. Updates available an no cost from any dealer on production of proof of ownership, paper copy, also on the net.
c. Likewise full illustrated partslists, with partnumbers and updates. Electronics down to component level.
d. For the life of the vehicle, legally defined as 20 years minimum. Info to remain available on the net for a further 20 years.
e. All documentation to be GPL, Creative Commons or similar.
f. ALL SOURCE CODE FOR ALL SOFTWARE AND ALL UPDATES - GPL OR SIMILAR.
g. Should the bunch of shysters be taken over or go broke, all above to be preserved online for posterity.
That little lot should go some way to concentrate tiny minds to get the engineering right in the first place and to discourage releasing pre alpha grade designs and deliberate fiddles.
You may think you can discern a slight mistrust here - shame on you.
I marvel that it's easier to get far more complete repair info for a 40 year or older car - and spares - than most of the modern stuff.
Hence the recent Apple CarPlay/Android WhatsSit stuff - the car provides a touchscreen and a few buttons (and a GPS?) and the phone does the rest... So virtualising the entertainment system is not really relevant?
Apart from Apple's closed shop - only Apple Maps :( - it works
To be honest, if you made me design a resilient, highly-available, complex system running commodity OS, it would be thus:
- Several hypervisors.
- Running dozens of small enclosed, self-sufficient, minimal, limited-purpose virtual machines.
That then allows you to run all the complex stuff alongside all everything... replicated to several different units, while being able to fall over to any other hypervisor present in the case of a fault, within microseconds.
At which point you can do things like:
- Activate a service light if the primary hypervisor fails, but carrying on processing everything just the same.
- Cutting non-essential services giving preference to the essential ones.
- Allow software upgrades to be staged to check that they work, with rollback, without interfering with primary operation (i.e. keep all the original VM's, upgrade a replica, try a switch to the new upgraded replica, if it fails, fall back to the original).
I'm not sure I'd want it controlling things like ABS, but you could certainly apply it to vast portions of a modern car. The satnav would be a VM. The media player. The dashboard interface (not as critical as you might think). The telephony. The in-car wifi router. The GPS tracker. The OBD service. And if you broke it down properly, every service would be independent - i.e. the GPS daemon running on one VM that just does nothing but interface to the GPS module (no other system allowed to do so), and offers out NMEA strings over only a single port. Thus media-player compromise would leave you in the media player VM and unable to play with OBD, etc.
As time goes by, no matter how much we dislike it, there's going to be more and more stuff that your car does that have little to do with actually driving. Let's shove those functions off into non-essential VM without having to have 20 black-boxes all co-ordinating. Three hypervisors, one of them the primary, one a replica, another purely for staging/upgrading/failover. On those all the non-critical services running. Everything. Including that app-for-the-future that talks to the GPS-server service and sends the data to the Internet-access-server to report your road tax for this month based on how many miles.
We can isolate things, but that just results in a myriad of interconnected / wired / reliant computers. Or we can consolidate and make resilient.
Throw all your eggs in one basket. And then make several identical copies of that basket.
"Throw all your eggs in one basket. And then make several identical copies of that basket."
It's an option. But does it actually reduce costs? (ArLi, having registered today, might be able to contribute to this discussion too).
Suppose one were to take some existing tried tested and proven (well-specified well-designed well-written well-tested widely deployed) software. Surely there must be some somewhere?
Suppose there then existed a hardware-fault-tolerant variant of the existing commodity hardware used by this well tested software.
The existing software might need a few relatively minor changes to make the most of the new facilities, but it wouldn't be a near-complete (and nearly-untestable) change of software architecture which the multi-hypervisor setup would seem to imply.
What would be the pluses and minuses for the two approaches?
As a hypothetical example, take some existing known good ARM code, and make it tolerant of many hardware issues by running it on something conceptually similar to a fault tolerant SoC based on e.g. ARM Cortex ?
Actually, how hypothetical is that example, if the 'new' fault tolerant SoC is (e.g.) something like a TI Hercules or similar?
That seems to me to be a lot of unnecessary computing. All I want is a car with a radio, EFI and ABS. I don't even care if it has powered windows. Anything else I can add as a peripheral.
Off the car topic, this is similar to my laptop requirements. I want a minimum 15 inch screen, decent keyboard, multiple USB ports, card reader and an RJ45 connector. Anything else I can add as a peripheral. Wifi, bluetooth, speakers, video camera, microphone, I can add and remove as I see fit or would like to be able too. It would, potentially, save the manufacturers money and increase their profits as I'm sure I would not see a decrease in laptop prices, in fact, possibly have to pay a premium as a security enhanced device.
Yet another attempt by Intel to smear thier legal feces all over the computing world, and seemingly more frequently - at the expense of non-proprietary offerings by other vendors. Jailhouse by Siemens already does this, and is not locked to x86 or Intel's hardware...
I'll bet they are touting novel security, but also don't want to open up the codebase for public inspection. ME and Intel proprietary "security" all over again - but now they want to do to IoCrap and automotive what they've done to the datacenter.
Ladies and gentlemen, I give you the most prolific advanced persistent threat - Intel. SMH.
Perhaps I lack the vision thing, but I'm not understanding why you'd want a hypervisor for embedded systems worrying about audio and video. I understand why you might want a hypervisor, and I understand prioritizing safety over other things. But I can't see why you'd want a computer that is responsible for safety also trying to control audio and video. CPUs are pretty cheap; if you want the safety CPU sending some notification to car occupants, then let it send a message over a bus to a separate audio/video controller. I really can't see why you'd want to have your safety computer time-slicing tasks unrelated to safety.
Yep, absolutely bonkers but hopefully just some Intel marketdroid's fantasy. By all means put your climate control, entertainment or seat adjustment stuff in one of these things but keep the safety critical software on it's own hardware with only the leanest of real-time operating systems. The idea of cars controlled by consumer grade software gives me the willies.
"... CPUs are pretty cheap; if you want the safety CPU sending some notification to car occupants, then let it send a message over a bus to a separate audio/video controller.... "
Today all automotive OEMs try to integrate as much as possible functions in a single device (ECU).
The reason is simple - cost reduction.
The case, power supply and connectors (sockets) are the most expensive parts of ECU - they usually cost much more than microntroller (and other chips) inside.
OEMs (and tier ones) also think that having single HW platform will give opportunity to better SW reuse (including safety certifications) but IMHO it looks good only in Excel. For 15 years of my work with automotive SW I never seen code reuse at level assumed by management and promised by chips/tools vendors.
"The case, power supply and connectors (sockets) are the most expensive parts of ECU - they usually cost much more than microntroller (and other chips) inside."
Presumably you've never had full sight of the cost impacts of failed software projects, or recalls of failed systems, etc?
Fair enough, as those tend to come out of different cost centres than the design+build budget. Maybe they shouldn't, but often they do.
Meanwhile, as a non-hypothetical example, readers might like to learn about the unforeseen costs to date for a manufacturer who skimped on hardware and software and mis-used a box for things it wasn't capable of doing reliably? Hint: tens of billions of dollars so far in a well known "uncommanded acceleration" case.
"... Presumably you've never had full sight of the cost impacts of failed software projects, or recalls of failed systems, etc? ..."
This is nothing different than in other busineses - unit manufacturing cost and time to market are the kings.
If you take into consideration that SW embedded into modern car is much bigger than most other SW systems around then you realize that any ideas (especially "magic bullets") to improve SW are very welcomed by management.
IMHO automotive (but not only) industry tries to find to which extend they can lower SW security and safety standards until it becomes really unprofitable. Eexamples: relay attack which allows to steal any car (with PEPS) using $50 worth equipment, report about Toyota's SW after "unattened acceleration" case etc.
The reality is that today most customers don't really care about bugs (they are accustomed to them) but they want more fancy functions (event they will not use them) and then automotive business is going this way. For me one of the most prominent example of stupidity driven by mass market is touch screen with cascaded menus used as UI for functions needed when you driving.
You have a hyper visor for an embedded system for he same reason why you have them for a larger system.
If stuff goes wrong - and they always do - the error can be trapped and you can fail the system in a controlled manner.
Same reason most embedded systems have a hardware watchdog.
"If stuff goes wrong - and they always do - the error can be trapped and you can fail the system in a controlled manner."
Do you think that particular capability needs a hypervisor? If so, why?
Proper multi-tasking OSes with proper memory protection, user/supoervisor modes, etc (not just RTOSes) have offered that "error-trapping and controlled failure" capability for decades.
It's only since Apple and Microsoft software and business practices came along that the ability of a simple error in one ordinary piece of code to take down a whole system became widely acceptable practice.
Even then, once WinNT and the Apple equivalent came along, there was no need for an error in some random non-kernel software to be permitted to cause a BSoD and bring down a whole system.
HYPErvisors address a different set of requirements.
CPUs are pretty cheap;
Indeed. I've lost count of the number of cheap, tacky, pointless, gimmicky Chinesium products Big Clive has reviewed that have a Microcontroller just to achieve a repertoire of flashing LED patterns or some other equally pointless task. You could achieve the same effects with discrete logic, but that would require a lot more chips and be a lot more expensive (and a lot larger). You could achieve the same effects with an FPGA, but that would cost about (I'm talking order-of-magnitude "about") the same. You might as well throw a micocontroller at it, especially as you probably manufacture a wide range of similarly pointless products and can use the same chip in all of them.
The real cost is the programming, but with a large enough run of product that amortizes down to almost nothing.
Ob Big Clive video detailing a particularly banal use of modern technology.
No, not in general language perhaps. But when you coin an acronym which gets used as a name for your product, you get to decide how it's pronounced _in that sense._
(Fortunately there seems to be some doubt about the intended pronunciation of 'SQL': I can't abide "sequel" here, sticking with "ess-cue-ell".)
No, you switch on the radio and all the lights go out
You jest, but I was coming back from the Joe Bonamassa concert in Birmingham the other night with a friend in his car - a recent BMW - and the rear reading lights kept coming on for no obvious reason. Apparently it's a known fault & not yet sorted by BMW. The lights are controlled off the Can bus.
So, anyone out there in the know care to enlighten us great unwashed on what the relationship is between Autosar and ACRN ? And where Intel's market-insignificant NUC (which is apparently essential to ACRN at the moment) fits in this picture? And whether a different core hardware (or at least, a non-x86 option) might be a more defensible option for the industry that appears to have forgotten why "dual circuit brakes" were introduced a few decades ago.
No prizes, it's just for fun. No press releases will be accepted from Intel staff or their affiliates, not even from Intel's Wind River subsidiary.
The Chief Engineer's decision will be final, until overridden by the Chief Executive's offer from the vendor's Chief Financial Officer.
Does MISRA have a view on the role of hypervisors in safety-related and safety-critical automotive systems?
IMHO classic AUTOSAR was outdated at the moment of its "invention" and (even) present AUTOSAR tools are total crap. After some years "they" realized that classic AUTOSAR will never catch modern HW and SW requirements and there emerged idea of AUTOSAR Adaptive Platform (by the way - they even discovered C++ ;-) ).
In short - the idea is to use Posix (should read Linux) system and add Autosar like API for software components. It means that ACRN idea can be somehow related with AUTOSAR but its dependency from Intel and neglecting of jailhouse project is a little mysterious.
Regarding MISRA - till now it has only lanuage(s) related rules.
Intel already has a hypervisor in Titanium which is produced by Windriver (a Intel subsidiary)
They've touted it for years, but it is expensive. Now Intel throw their IP into a free version so cutting the legs of part of their own company.
Their does not seem to be a joined up strategy here...
"... future in-car computers know when to starve the entertainment system of resources in order to ensure drivers’ or riders’ safety"
I wasn't the only one to be brought up short by that sentence, then. The idea that one computer shares the tasks of critical driving safety with in-car entertainment sounds impressively stupid. I'm not even convinced they should be allowed to share a network.
I submit that as cars become ever more automated and computerised, and as driver-assist gradually morphs into autopilot over the course of time—which then becomes full automated journey management and driving as the cars become integrated with each other and city computer/traffic management systems—the level of required safety and reliability shoudn't be much less than you'd expect in an aircraft.
Now you'd be right to say that planes are special insofar as (a) a total failure doesn't simply mean drifting to an embarrassed stop at the side of the road, and (b) the consequences of a crash are potentially two orders of magnitude worse than for a car ... BUT from a public perception point of view, it's basically the same: every time a car kills people because of a software failure there will be absolute hell to pay.
The safety and security requirements for an entertainment system are so completely different from systems to control steering and brakes that there is no meaningful comparison. You can let lardy, careless Android wheeze and waddle its way round a movie or a website and no one dies, even if the OS is now 10 years old. But that really won't do for systems that will have to distinguish a child from a tumbleweed and decide within a hundredth of a second whether to risk the occupants' lives by deliberately swerving into the opposite ditch.
There's no real reason to be confident that the anti-malware arms race is going to be conclusively won by the white hats any time soon, especially given that with virtualisation, hypervisors, containerisation and the general obsession with loading a multiplicity of OSs and applications on a single piece of tin, we're opening ever more chinks for super-stealthy hyper-powered ultra-sneaky sequestration attacks.
So FWIW my vote is for never mixing business with pleasure. Take a leaf from the airliner handbook: build a bullet-proof OS for car management, keeping people alive, use multiple voting computers to decide critical issues, and keep the fluffy stuff completely and utterly separate.
CPUs just don't cost enough to justify VM-ing a car and taking risks ... not even for an industry as soullessly greedy and dishonest as automotive.
This is a suggestion for auto manufacturers:
Instead of running the entertainment computer as a VM under a hypervisor you should integrate it directly with the in-dash touch screen. Also, integrate the beeping horn alarm system into the same module.
Then take the entire module and shove it up your arse. Following that you can fuck right off.
This is hardly embedded. Intel, get back to me when you can produce an embedded IoT project that runs UDP protocol, can run for 3 years on 2 AA batteries, and monitor a reservoir in all-year-round conditions. Oh, and the compiled binary is < 32KB. We do that all the time here, but we're coding in Forth for embedded and embedded mission-critical applications.
The libraries that we use have been developed and tested over a decade or so (and only amount to approx. 3KB compiled) and the entire application is small enough to be written, understood, and documented by exactly one human.
Intel will never be able to do embedded in a million years. It's too "small" for them. They, like most companies, will chuck a large room-full of coders at a project and tell them (or, rather, their lower layers of management that will look after said room-full of coders) to produce something "small, agile, and robust, for IoT", and after many man-years of development, they will design something that runs on Linux, thus at a minimum requires an Intel x86 with half a gig of RAM, storage, and associated power budget.
Embedded my big fat (grey) hairy arse.
It's nonsense. Literally non-sense. It might be cool, and probably work quite well, but please don't hijack the "embedded" moniker, 'cause embedded it isn't.
A fucking hypervisor running in an embedded system FFS. A bunch of us guys in the office are all reading this article in open-mouthed disbelief, I can tell you!
"Today, cars have several computers linked over a bus. It’s widely envisaged that future cars will have one computer running a hypervisor to isolate workloads, an arrangement that will mean fewer integration hassles and lower costs for auto-makers. ACRN likes that idea, but thinks a hypervisor used in such scenarios needs to prioritise workloads related to safety."
There's so much fail here I hardly know where to begin.
It’s widely envisaged that future cars will have one computer running a hypervisor to isolate workloads
No it isn't. It just isn't. If you go to your chief engineer and say "Hey, I've done a design whereby the ABS, Engine Management, economy and emissions management, air conditioning and ICE can all run on one box" you will literally get laughed out of the office. They run on different systems for a reason. One of the reasons is certain clauses in the standards MANDATE it. The other reason is redundancy. If the engine management system develops a hardware fault, do you want your ABS to fail at the same time? Common failure modes is another consideration. Utter nonsense.
an arrangement that will mean fewer integration hassles and lower costs for auto-makers.
Again nonsense. After years of doing this sort of thing, I can tell you that it's FAR easier to "integrate" your application, and my application, at two ends of a wire of fibre-optic connection, running an agreed protocol. That way, if your code has a memory leak and the watchdog reboots it every few days, my system keeps running, and the clue that your system faulted and mine didn't indicates perhaps an un-caught exception in your code, or memory leak etc. When they are all running on the same silicon, everyone points their fingers at everyone else.
This is fairy-tale land, nuclear levels of bollocks. All engineering best-practices, refined and improved over decades, thrown out. It's like a marketing press release and The Register just re-posted it!
but thinks a hypervisor used in such scenarios needs to prioritise workloads related to safety.
Yes, because embedded application developers have never ever ever developed our own, application specific, extremely low latency, extremely simple (sometimes just a few assembly instructions) task switchers and schedulers. We've never heard of time-slicing micro-kernals. No. What we really really need is a fucking entire Linux OS with a Hypervisor between us and the hardware.
Yep. We need that. More of that, please.
I wish I could upvote you more than once for that.
I also wonder how much of this automotive stuff has what aviation calls a "reversionary mode." Whereby if a system like the hydraulics fails critical controls are still operable by mechanical linkages. Well, that's old-school because these days it's fly-by-wire, but there is still either multiple redundancy or some form of reversion in avionics.
Hell, even automotive design had reversion in days of yore. Hydraulic power steering used to be designed in such a way that if the hydraulics failed you still had brute-force steering. It was a very elegant design embedding a spool valve in a mechanical steering system. The spool valve was part of a negative feedback loop and the set point was controlled by the steering wheel. If the hydraulics failed the steering wheel pushed the entire body of the (now useless) spool valve operating the linkage mechanically (very hard to describe in words and I've just done a bad job of it). That was real engineering design. Modern design seems to be "cross your fingers and hope the s/w doesn't crash, because if the s/w crashes so does the car."
I think what is really annoying me about this article, or, rather, the people behind this "Hypervisor for Embedded" nonsense is it appears to originate from people who have literally no fucking idea what they are talking about. They have heard of functional safety, they maybe know a bit about embedded, and are now talking themselves up as some sort of (self-appointed) experts on automotive automation with a new shiny that is really great and you-don't-know-you-need-it-yet-but-trust-us-you-really-do "technology.
Pah. Spare me. I'm too old for this shit!
Their approach (which seems to be a thinly veiled attempt at promoting x86 technology, you know, those really reliable, secure CPUs which are not at all affected by the recently discovered Spectre and Meltdown vulnerabilities) is based on added complexity, when, in actual fact, one of the FIRST things an engineer/engineering team does when it starts a projects is to say to itself "How much programmable complexity can we DO AWAY with?".
When you are working on SIL rated projects (I haven't worked directly in automotive, my experience is in 61508 on pipelines, subsea wells etc.), one of the first rounds of the design exercise (post requirements gathering and review) is to say "From the solution that is *beginning* to emerge, what portions of the solution currently use programmable technology, and of those, what proportions can we remove and replace with mechanical/electrical/discrete replacements?" The emergent answer feeds back into the front end and you have another round. This is not news to anyone that has worked in this kind of thing. Automotive, aeronautical, shipping, rail, national infrastructure etc. design engineers have been doing this now for literally decades.
Software is shit. We're trying to engineer it OUT. Not engineer it IN.
The presumptive stupidity of it. Yes! What we need is a Linux OS (with x bugs and y vulnerabilities) and a Hypervisor running on top of it (with k bugs and m vulnerabilities) and we should build our automotive software on that. Yes, of course! And not only that, but run it all on the same box. Fuck it. Why not?! What could possibly go wrong?
Such idiocy. I thought it was April 1st for a moment!
Yup, what you say rings very true. Marketing BS indeed.
(and even for those of us who use ASM and C...it's still true)
According to GM and TechCrunch, there are more than 100 cpu's in my 2012 Chevy Volt.
OK, about 96 of them are in the battery to control charge leveling, safety, and cooling.
But I also heard that there are a few in a linux cluster in the dash, and of course the usual stuff in the ECM, which has been hacked by performance tweakers who obviously know what's in there.
As someone who has also developed this "type" of system for super high 9's reliability, yup, that's how we do it - and avoid any single point of failure like the plague. I have systems I designed in the '90s that have never crashed or been rebooted. We'll see how those 9's stack up when we have a real sample of the fail rate.
To the guy who said the box and connectors are the expensive parts, well yeah - the box is called an automobile and the connectors have to be there anyway to make the lights and stereo and drivetrain work. Nowadays most control is some serial protocol...
Looks like intel is hoping that they can replace a bunch of CPUs that are so cheap the underlying PCB costs more per sq in with one of their high margin ones. Dream on, and while I use x86 for lots of things, that would never be one of them.
FWIW, GM released (open source, yay) a version of bash they rewrote that was tighter and more simplified than the original...so you can kinda guess what the entertainment stack runs. Won't let me telnet in (haven't tried any advanced hacking, it's my daily driver) but it's interesting.
Anyone who is thinking on depending on Intel for anything with a lifetime should note their extreme disloyalty to anyone who adopts their stuff but doesn't achieve huge numbers. They drop product as fast as they can, stranding users and designers. Guess what, MBAs? We pay attention and remember.
"... To the guy who said the box and connectors are the expensive parts, well yeah - the box is called an automobile and the connectors have to be there anyway to make the lights and stereo and drivetrain work. Nowadays most control is some serial protocol..."
It is just a fact - I do not say/judge if it is good or bad. And as I wrote it is only a partial reason why ideas like ACRN are emerging.
Keep in mind tht the most lanes in connectors are not for communication. At the end you need to drive actuators (including displays) and get data from sensors (and no - you cannot put microcontoller in every actuator and sensor (at least using present technology)).
"Yep. We need that. More of that, please."
Because it's A LOT cheaper, and the investors ONLY care for immediate costs because, as long as you get away with it, you have no future costs. If you DO get caught, you find ways to defer or reduce the costs so you still come out ahead. Put it this way; unless the consequences are near-existential, they'll be pressured to play fast and loose and gamble so they don't get undercut.
So, the solution to buggy (if not now, then just wait a few updates) software is to wrap it in more buggy software. Threads within processes, within a genetically predisposed to fail at realtime OS, within a buggy VM, controlled by a buggy hypervisor on a processor with buggy microcode. And all those well-defined, designed and isolated protocols? Trash them and get some modern "Well the API used to follow that spec, but we decided that it looked so much better on the Powerpoints if we mixed stuff up a bit".
It's like the whole software world are Skeksis ( https://en.wikipedia.org/wiki/The_Dark_Crystal )
Biting the hand that feeds IT © 1998–2021