Watchdog urges Tesla to recall 158,000 Model S, X cars to fix knackered NAND flash that borks safety features The US Department of Transport has recommended Tesla recall 158,000 Model S and Model X vehicles after an investigation found worn-out NAND flash memory can cause the cars’ rearview cameras to fail. The dept's National Highway Traffic Safety Administration (NHTSA) launched a probe into Tesla's failing digital storage in June. …
A car's entertainment system failure can be a menace for the actual primary function of a car which is passenger safety.
How many times have I already said that the entertainment bus should be physically isolated from the car's command bus ?
I don't remember. I can't count.
In any case, well done Mr Musk. You might be a billionaire, but you still fail Basic Security 101 - all in the name of saving a few bucks.
Well I hope you're happy with your savings now.
Not to mentiion that any form of rewritable memory with a limited number of cycles should be user swappable. I'll bet they wish they had built a microSD card slot into the bezel of the infotainment system. Most people wouldn't even bother complaining about popping in a new 8GB card after 6 or so years of use.If it needs to also have firmware on it they could just mail them as cheaply as a recall letter.
@G.Y. Plug and socket may have been a reliability issue back in the middle of the last century but technology has moved on a bit since old Ma Bell could only get tin plated plugs.
These days you really do get what you pay for. Even car (some) manufacturers have learnt to install silicon grease filled, gold plated pin, lever operated, o-ring sealed sockets on junctions not within the interior of the car. There is no reason not to design a ZIF socket for memory cards to allow consumer versions to become mission critical.
In fact some microSD card holders are so cheap that they rely on flex in the card to provide contact pressure and the housing doesn't even cover the contact area. It would be simple to install a bridge over the contacts containing a lever operated cam to maintain vibration free contact.
"Swappable implies plug&socket; In "Old ma Bell" (the old AT&T) these were a no-no"
The average car has hundreds of plugs and sockets. Mostly cleverly arranged so that it's nearly impossible to plug a cable into the wrong socket. Yes they do fail every now and then, but not very often. What's one more?
And, BTW, are you trying to tell me that AT&T's phone network was free of RJ11 plugs and sockets or are RJ11s somehow not plugs?
I've never had a problem with the SD Card connection in any of my phones or cameras, and those see more vibration than an SD Card socket in my car's irritainment system would. (Trivial proof: I almost always have the phone with me in the car, and it also gets kicked around while the car isn't being used.)
Not really true. Depending on the location of the phone or camera, it probably has more padding insulating it from shock, whether that be a protective case for the camera or your leg if your phone spends its time in your pocket. SD is a bad idea for a number of other reasons though, which I've just posted about. The short version: not good for lots of rights, can't predict failure, not easy to confirm reliability, and NEVER make it feasible to remove something safety-critical while the system is running. Reservations about SD as an option aside, the storage should certainly be replaceable.
"I'll bet they wish they had built a microSD card slot into the bezel of the infotainment system."
The storage should definitely be swappible, but never like that. It shouldn't be easy to accidentally take the storage out, and it really shouldn't be feasible to take it out while driving. Similarly, consumer-grade EMMC inside the dashboard isn't great for reliability, but if you wanted to see what significantly worse looks like, trusting SD would be a good way to get there.
I would instead suggest a container for a storage device which is modular but installed beneath the screen panel. Users who are knowledgeable enough can get to it by disassembling the car, but nobody will accidentally take a card out of the display thinking they must have put it in a while ago. That also makes it possible to insulate the storage device from shock, temperature, and other possible damage. While specifying this storage system, the manufacturer should ensure they use storage which supports a SMART-style health check so they can warn the drivers to get it replaced. If they want to be extra careful, they could have the module include two mirrored disks so not even a relatively rare unpredicted failure can crash it. Even requiring all of this instead of a basic EMMC module wouldn't change the price all that much, and the service center could probably charge the driver for the replacement part instead of being forced to replace it at their expense when it causes a safety issue.
"Not to mentiion that any form of rewritable memory with a limited number of cycles should be user swappable."
There should be 3 slots. When one fails, a service light comes on. If two go out, the car goes into full bitch mode. Even if it being user swappable is not wanted, it should be very easy for a service person to do. All 3 should be changed at the same time.
Easy.
Why shall a failing entertainment center (MCU) cause "rearview cameras blacking out and an inability to defog windshields, as well as the loss of turn signal chimes and other audio alerts", as indicated in the article.
Hence why ODI is recommending a recall and threatening legal action.
Had the MCU kept to its namesake as an "entertainment center", Tesla wouldn't be in this pickle.
Security, in this context is the security of the core functions of the car to remain road worthy, plus keep people safe.
Can only cope with so many read/writes.
I've currently got an old-ish Samsund SSD in my PC bleating that I should 'raise the over provisioning partition' - basically lower the usable space, so it has a buffer to reallocate the failing sectors.
Not something I'd thought about when I switched to SSDs - but makes sense when you look at how NAND works.
Slightly scary thing is that maybe this was never considered when building Smart Cars. You can separate the OS out all you like with perfect security - but this is a whole layer below this.
NAND degrades in your car like the tyres, brakes etc. Not an issue if wear can be monitored and the part replaced when it degrades. Very problematic if say the chips are soldered to the brains of the car.
Your no-socket Powerbook might get replaced after 5 years, but cars need to last longer.
Not really. A normal user doesn't hit their storage anywhere near enough to wear out an SSD in 5 years. Non-replaceable drives are generally bad, due to the inability to repair and upgrade, but it's certainly not part of a plan to let them wear out and force people to buy a replacement. Of all the things likely to go wrong with a laptop, normal wear of the SSD is just not a significant factor.
There's a nice test from Tech Report, from back when they were still good, on consumer SSD endurance. They do a lot better than people generally expect.
https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead/
That's a bit optimistic. Now I'm biased, because I already had to replace one failing SSD in my primary laptop and now another one is failing, but still. There's a lot of writing that can't easily be controlled, and that will wear out storage. The more writing takes place and the less control one has, the more likely this is to become a problem. Depending on what people do with their computers, they can do some write-heavy things. Browsing, for example, often includes a lot of caching stuff to nonvolatile storage. Editing image or sound data also usually has a largish disk usage, depending on the editor in use, to support things like autosave with undo. And that's without including things that the operating system might do, like downloading an update, failing to fully get it, deleting the failed download, and going again. Then updating all the OS files from that update.
Disks wear. Whether it's an old mechanical drive whose moving parts have worn down or an SSD which is nearing its write limit, they can be expected to fail with more frequency than other components. For any sufficiently important system, there should be a plan for replacing them when that eventually happens.
Planned obsolescence for a high value (when new) item such as a car would not be good. Expensive and time wasting return to dealership repairs damage the reputation of the car maker, drive down perceived reliability and used prices. A poor resale market hurts the new car market as residual values are important to lease type and PCP type deals.
This article is quite interesting in explaining the problem https://insideevs.com/news/376037/tesla-mcu-emmc-memory-issue/
Tesla's response has been woeful - eventually owning up to the issue but only agreeing to fix it once the car is broken and if it's done less than 100k miles. https://electrek.co/2020/11/09/tesla-emmc-failure-touchscreen-offers-extended-warranty/
It might well that they still have all the components duly separated so that it's really a security issue.
The problem is the big table thing is _the one interface_ to rules them all. There aren't secondary controls, e.g. for heating, defrosting. Clearly they're playing system/safety and indicator sounds through the stock audio/entertainment system. Not a good dependency to have for safety critical bits. Or having separate display(s) for the rear/safety cameras.
It's a very stock PC/computer model, but this is not a PC. If anything goes wrong with the unit so it cannot display or cannot accept touch inputs, its impact is wide. Most other manufacturers seem to know or have learnt that critical components require their own, independent dials/knobs.
"Most other manufacturers seem to know or have learnt that critical components require their own, independent dials/knobs."
Try telling that to Citroen, we have a C4 Grand Picasso as our pool car at work and EVERYTHING other than the normal binacle dials works through the touchscreen infotainment system
I haven't owned a car for a long time but I rent a lot (e.g. car club or Hertz for longer period - it's cheaper for me than owning), therefore I get to try a lot of different cars. By far the biggest pain in the butts are the ones with everything in the sodding 'infotainment' system. As far as I'm concerned it's no different to using a phone while driving. Even if I familiarise myself before setting off ( Yet Another System...) I still don't consider safe to use while driving. Even having controls on the steering wheel doesn't work as you still need to work out which button does what and how many levels you have to go through to get to what you want.
critical components require their own, independent dials/knobs.
I completely agree. This was a primary consideration when I purchased a new car. To cut through the sales guff, I just told salespeople that "I can't safely operate a touchscreen while driving. If there's a touchscreen feature you'd like me to try, I'll just shift into park while I access it."
Unfortunately, although I got the trim line without touchscreen, I believe that the console controls (climate, rear defrost, hazard lights) are all wired through the infotainment system. The backup camera certainly is, but I have a secondary instrument for that called "Large Rear Windows."
Yup. That's why my current Volvo will be my last (new) one. They got rid of separate physical controls for the stupid touchscreen.
I hate touchscreens. Having one on my phone is bad enough - and I put up with that simply because it's nearly impossible to get a phone with a physical keyboard at a price I'm willing to pay. I'm certainly not going to put out car money for one.
"I hate touchscreens. Having one on my phone is bad enough"
For lots of things, I agree with you. I'm not against touch screens completely. They are perfectly fine for trip data, car settings that don't change on the fly and for the SatNav. The volume control and fan speed need to be on dedicated controls. I use those all of the time while driving and don't want to look away from the road to make adjustments.
"Most other manufacturers seem to know or have learnt that critical components require their own, independent dials/knobs."
Ford's MachE copies many of the design flaws of the Tesla Model 3. ie: just gluing an iPad to the dash. Kia/Hyundai and some others have adopted the approach I like which is to have frequently used controls as separate knobs and buttons. It's much easier for me to reach out and adjust the fan or temp control via a knob than to find the proper screen on a display and use a touch interface.
Tesla's new MCU's delete local broadcast radio. What good is a subscription service via the internet if you can't get road reports when you need them?
Because everything is controlled through the touchscreen on the sodding "infotainment system". Remember the idiotic design of these cars means that almost all the physical controls have been replaced with an interface.
At least on my VW you can control stuff through a real knob or button. In Germany a driver was prosecuted for driving one of these whilst trying operating the touch screen to change the wiper speed .
"Why shall a failing entertainment center (MCU) cause "rearview cameras blacking out ..."
As I understand it, they use the same touchscreen control device(s). There's possibly some excuse for that for the rearview camera I think, there being only so much real estate available for display screens. As for touchscreen controls in a car ... Not all that great an idea from a safety POV I think unless the touch part is disabled when the vehicle is in motion. But what do *I* know?
"from a safety POV I think unless the touch part is disabled when the vehicle is in motion"
Look at the Tesla Model 3 and you can see that besides the touch screen, you have very few control options. The steering wheel thumb controls are "soft" so their functionality changes. If your screen punks out, you may not be able to change those controls. You also won't be able to see your speed, state of charge, etc.
That's not what was implied in the article. What was stated in the article is that Tesla chose to write less to the storage and use error correction.
Seems more like they didn't spec out the NAND properly for the amount of data that was going to be written to the memory each day for the anticipated lifespan of a car.
Or maybe Tesla did spec out the memory properly, anticipating that a combination of "autopilot" and stupid users would finish off these cars before the memory was anticipated to start failing.
Tesla has gone with commercial, consumer-level components in many places. And that's bitten them a few times. The NAND flash thing is not "obvious", but for a typical automotive manufacturer with established design, development process, it likely would have been picked up. They've learnt, through hard experience, the cost of post-release repairs/fixes.
Telsa had an issue with tablet screens delaminating and fading, because they bought consumer-level components which couldn't deal with the temperature variances and UV exposure inside cars. They're rookie/newbie mistakes. Nothing that can't be fixed, some much more expensive than others. Bullying customers who complain about it, silencing others through NDAs and being argumentative with NTHSA doesn't help. But then, they now have a $500bn kitty to burn on fixes/improvements.
"This is what happens when newcomers to a sector appear and do stuff on the cheap."
All automakers do stuff on the cheap. The difference is the established lot know where they can shave a few cents and where they can't. Warranty and recalls cost profits. It can be well worth it to make sure expensive and hard to service components are well built. A headlight relay is a cheap and easy fix. A crankshaft isn't. Even a simple part buried deep under the dash might benefit from some bulking up where a more expensive switch on the dash might have all of it's corners cut to the bone.
"Telsa had an issue with tablet screens delaminating and fading, because they bought consumer-level components which couldn't deal with the temperature variances and UV exposure inside cars."
Most LCD screens will be put to the test in a car. Where I live, we can have bright sun during the day and the temp dropping below freezing at night. One side of the LCD sandwich will get hotter and colder than the other side. With the different expansion/contraction of each side on a daily basis, something will eventually let go and the goo will run out. Tesla (and most makes) often bundle replacement parts. With Tesla, you can't just get the LCD, you have to buy the whole MCU. You might find an independent shop that can buy the panel on the used market and do the work, but at some point, those displays will be dearer than gold if they go out of production. If you have a screen in your car, a sun shade is a really good idea. A glass roof might be a problem.
User replaceable? Come on, this is the automotive version of Apple so nothing will ever be user replaceable. You can arrange to take it to one of their designated 'service centers' and have it replaced for a nominal fee of around $5,000, and at that price they'll be doing you a favour mate.
I'd like to know, other than screen wash, what user replaceable parts are there on any modern car?
I do all my own mechanics where possible / resonable, but most of my vehicles are old enough for that to make some sense.
I'm currently running my late fathers Volvo V50, it's 15 years old but I'm not going to work on it other than wheels / bulbs etc. It's a 'modern' car and if you aren't a Volvo specialist it's just a pain to do anything on.
Fortunately, it's well built and I have a tame non dealer specialist!
Point is, even screenwash is beyond a lot of folks...
I'd like to know, other than screen wash, what user replaceable parts are there on any modern car?
Pretty much everything just like in old ones. With whole bunch of electronic gubbins added. Admittedly it does help (and for some things is necessary) to have decent diagnostics/programming kit (either standalone or software on a laptop). I had no issues finding/resolving/fixing things on a BMW (albeit bit older) using INPA on old laptop. Likewise for GM (as an example) you can get Tech2 (chinese clone of course) fairly affordably that will let you diagnose and program.
The annoying issue is the increasing habit of automakers starting to code things so even swapping a new battery may mean you need to code it to the car which is just ridiculous.
It's not just e-gubbins, though even the Volvo suffers from this at 15 years old, it physical bother.
Sure a regular code reader will tell you the fault codes, but you'll probably need several hundred pounds worth of special tools to actually fix anything. And you'll probably never use them again...
I asked my tame specialist about tools once, he just winced and changed the subject!
Basic OBD II reader will not really help much other than possibly point in in generic direction of the fault.
Using my prior example of a BMW, INPA actually tells a lot more and shows how the generic codes can be misleading. INPA can be found with bit of searching and all you then need is 30-40 quid cable (assuming you have an old laptop).
Even cheap OBD scanners have come a long way and there are many that now support manufacturer specific protocols (beyond basic OBD) for not much money.
That is not to say that some kit still is not cheap, As an example Tech2 for GM (chinese clone) is not what I'd call cheap. However if you do intend to do work yourself it probably pays itself off first time you can avoid dealer.
There are also lot of good software via many forums dedicated to makes and models of vehicles where only thing you probably need to spend money on is a decent OBD cable (especially for programming you want one with decent UART, K-line support etc).
"I'd like to know, other than screen wash, what user replaceable parts are there on any modern car?"
With nearly every other brand outside of Tesla, you can go to a dealer and buy whatever parts you need and do the service yourself or have somebody else do it for you. Physically replacing a Tesla MCU isn't a big deal if you are moderately handy. There are videos on YouTube. The big problem is you don't have the gear to copy the serial number/authorization and write it to the replacement MCU. There is no law in all but one or two US states that requires Tesla to help you out (Right to Repair).
"They weren't that smart really. 8GB of bog standard eMMC used and abused, basically essential to car operation?"
Tesla has a huge amount of turnover. The logging may have been put in initially to look at how the cars were operating and never removed it for serial production. If it's working, why fix it? Time passes and the latest crop of staff may not realize it's still in debug mode or know how to down shift it. A certain amount of logging will be necessary. Now you have two issues: a)If it's working, don't mess with it, b) management will not authorize the resources to fix it.
This could be one of those issues that can't be fixed with an over the air update as it may have to go pretty deep. It may also brick the car if there is an interruption. Sans a complaint, why would they do a voluntary recall that could cost them a bunch of money? They can just take care of it quietly when cars come in borked and list it on the invoice as complimentary/goodwill.
Another issue with Tesla is their black boxes are registered to the car. Very few outsiders can reprogram a module so it works in a different car. Let's say that they do this for a headlight. Put a little "DRM" on the circuit board and it's a felony to reverse engineer. Only Tesla blessed replacement parts will work. No thanks.
Amusing - because I own neither car nor shares in Tesla.
Young people today eh? Not enough real world experience to think through ALL of the usage cases when they design a new thing. 'Get it rapid prototyped and out there, and we'll fix it on the go... Everything is thrown away after 5 years....'
But no - it seems that when you rush stuff to market and put software out early on the basis that users will do the bug finding, there are real consequences.
Victorian cast iron engineering is the way to go. Everything replaceable, re machinable. Lasts 'forever' with some rebuild maintenance.
All the stuff in our industry that has gone wrong - Year 2000, Windoze as an OS..... yet still people believe us and but more stuff to throw away. If you keep running fast enough you stay ahead of the mob.
(Only Star Trek tells the truth - any alien even looks at a starship and smoke and flames come out of the control panels)
"Yeah the original Mini whose glove box would decapitate children in the front seat in an accident, that was definitely designed by young people today."
Not entirely sure what you mean by the original mini, but if you mean the 1959 introduced model I can assure you no children were ever decapitated by the glove box since it never had one!
That's what Musk told a bunch of kids. What the kids heard was that designing a car is the same as designing an app. Spec the tablet that controls everything with consumer grade NAND, write industrial amounts of logs to it to debug the Autopilot, completely forget that expensive cars don't tend to turn up in a landfill after a couple of years... Easy.
Astonishing. I have devices with flash memory which are far older than these Teslas and all of which still work perfectly (although I do wonder if there's a way of profiling the SSDs to find out how much life they have left in them). So what's the deal? Have I just been lucky, or has Tesla cheaped out on the components in its cars?
In any event, the electric-ness of its cars aside (which I think we can all agree is great, and Tesla did much to remove the milk-float image of EVs, but other manufacturers are now doing this well as well), Teslas do seem to be a bit. Ahem. Well. Crappy. All the focus nowadays seems to be on toys - games console in the car, whoopee cushions for the seats, Autopilot which is arse about face and which seems to be the cause of many accidents*. The build quality isn't there. How old is Elon anyway? 8?
It's not a fair comparison, but I would note that my 300,000 mile V70 still works perfectly in every respect, and it's just as quick as it needs to be - the old car manufacturers still have an ace up their sleeve that Tesla doesn't.
* actually, I'm going to ask for a slight retraction of that comment. The cause of the accidents are drivers dim enough to believe the hype.
14 year old XC70 here. Only done 289,000 miles. Had it not been for 8 months of lockdowns and not going far last year it would have hit 300 by now. A few squeaks and rattles, but all fully functional. MOT man each year tells me he sees lots of other cars (both high end and low end) of half the age and fraction of the miles in much worse condition.
"Astonishing. I have devices with flash memory which are far older than these Teslas and all of which still work perfectly "
When this story first broke, it was that the cars are writing a huge amount of data to the memory, whenever they're turned on. So it's not about absolute age, but the total number of rewrites that the cars are doing - far more than a normal flash memory device.
@45RMP
Why does Windoze "save your settings" every time you log out, even if you've changed nothing? Clearly they've never heard of a "dirty flag" after all this time. Despite having changed no settings, I once suffered a write failure during the save and lost the entire registry, requiring a complete rebuild.
The "Fast Boot" option in Windows 10 caused us no end of issues, as it boots with cached settings, giving rise to slow running machines with false reports of Windows last start & running times. Ironically the only way to purge it was a restart (Which is when you would perhaps like the fast boot option).
Icon - Fixed that pre-configured option in post image script we used (Powershell or registry key import I don't recall).
Yep, but how many IOPS were you doing to the same "block" of memory? That's the problem Tesla uses it for logging information. And there's a lot of logging going on. For example position of accelerator peddle (measured with 2 different sensors). Done around 20 times a second. Each one of those is logged. Old car manufacturers don't record that data. Tesla does. That's where Tesla mucked up. They monitor their cars.
Nope. They mucked up by not realising the inherent problems of writing massive amounts of logging data all the time, and so didn't spec adequate storage to cope for decades* without problems.
*This is a car after all, so multi-decadal use is a common expectation, even though it's not often the same owner for all that time.
Tesla Autopilot actually has fewer (and generally less serious) accidents per million kM than human drivers. This despite AP not being fully autonomous.
Umm, that's because Tesla Autopilot is not autonomous, and therefore when used properly, the human is doing the tricky stuff. The accidents attributed to Tesla Autopilot occur when the human decides to pretend AP is autonomous.
Those stats are also a bit deceptive.
They're comparing Tesla driven miles against miles driven by the general driving population. The general population miles also include other bits like driving in bad weather, or at night. And Tesla, for the most part, are on AutoPilot in day time highway/freeway, i.e. relatively open road scenarios. Human drivers on open roads and highways tend also to not have that many accidents. I'd venture most open road accidents are "stupid" things like overtaking on blind corners, rises or in bad weather.
tl;dr: Be very wary of that comparison.
Sort of related. The EU wants to make lane keeping/lane assist a standard feature on all cars. They reckon that would reduce accidents by 10-15%. That's quite significant. And that's already pretty much stock tech on mid- and higher end cars.
"The EU wants to make lane keeping/lane assist a standard feature on all cars."
Learn how to defeat that and make good money.
I don't want the car driving for me. There could be a reason I'm tending towards one side or the other. The car may also get fooled from time to time if markings aren't up to scratch. Only the main road in my town has proper markings. That and the short section leading to the high school. The other streets are patches over the patches and any paint is likely a spill from a contractors truck than actual striping from days gone by.
"Tesla Autopilot actually has fewer (and generally less serious) accidents per million kM than human drivers."
Downvoted because it's a meaningless "statistic" (how, exactly would you measure its performance considering that there *IS* a human driver who is purportedly managing the drive?). It's very likely a complete fabrication. In point of fact, autopilot is currently just a driver assistance system similar to many others plus some (optional) additional features that mostly seem not ready for prime time. And it generally isn't rated all that well. Here's a typical review typicalhttps://www.consumerreports.org/autonomous-driving/tesla-full-self-driving-capability-review-falls-short-of-its-name/ Bottom line: Sometimes it does what it is supposed to. Sometimes it doesn't. Some of the failures are harmless. Some annoying. A few are kind of frightening.
"...Some of the failures are harmless. Some annoying. A few are kind of frightening."
Just like when dealing with human idiots in all walks of life.
When used as intended Tesla's AP outperforms unaided drivers. When used incorrectly AP still does a better job than idiots who are: applying makeup; drunk; tired; texting; eating; or otherwise doing something other than concentrating on the primary task at hand.
It's kind of like the approach of Boeing vs. Airbus to flight management systems. Boeing tends to operate on the principle that an engaged pilot, who always has the final say, is better able to react when things turn to sh!t. Airbus concentrates on preventing a disengaged pilot from doing something stupid that turns things to sh!t. Both approaches have their merits and their flaws. Each outperforms the other in different circumstances.
Perhaps I should have added "...by the general public."
I'm perfectly aware of the adage "To err is human, to really f*ck things up requires a computer."
However I'm also aware that when autonomous or semi-autonomous systems are involved in an incident, blame is all to often placed upon the machinery, rather than the humans who blithely ignored procedure to: squeeze out a little extra productivity; out of laziness; or to save a little money.
Its emmc , which is the cheaper slower less reliable version of an SSD ... I have a miniPC with an emmc but I wouldnt want that as my sole storage for my %42,000 car.
He'll stiill rule the Car Indusrty but I ws just waiting for some sort of error to come up.
No delear network? Well the costs saved can be spent elsewhere
Cost of recall $1000 per car, cost of new part $3.50 ....ouch
No delear network? Well the costs saved can be spent elsewhereCost of recall $1000 per car, cost of new part $3.50 ....ouch
Indeed. Rich Rebuilds discussed this problem a while back, and I think was quoted something like $5k for a new MCU. From memory, I think the fix would be to replace the whole MCU, not just a component. But assuming it does cost $1k, that still adds up to $158m, which I think is enough to wipe out Tesla's profits. I also think he pre-sold EV credits, so not sure he could magic up more to help his bottom line.
I think there was also a recall in China related to suspension problems.. And competition is starting to bite and reduce Tesla sales. Somehow, I doubt this will affect Musk's perception as a tech genius though.
.Astonishing. I have devices with flash memory which are far older than these Teslas and all of which still work perfectly (although I do wonder if there's a way of profiling the SSDs to find out how much life they have left in them). So what's the deal? Have I just been lucky, or has Tesla cheaped out on the components in its cars?
Tesla has been caught out few times from using consumer grade components. Whether the NAND is one, I have no idea. Industrial grade SLC is pretty durable, so would seem likely they may have cheaped out for some MLC. Just guessing mind.
I think the compounding,if not bigger reason, than which NAND technology they are using, is how they're using it. They are writing way too much and way too often. Hence their OTA mitigation trying to reduce number of writes.
As for whole issue about critical/important functions being borked by "infotainment", well.
"So what's the deal? Have I just been lucky, or has Tesla cheaped out on the components in its cars?"
A little bit of both. The storage was consumer-grade EMMC, which isn't the greatest out there for really anything from speed to reliability. That doesn't help. But another aspect is that you don't write to your flash like the car does. That's often a critical factor, since writing to flash is more intensive than reading from it (mostly unlike spinning disks). I'm guessing that many of the flash-based devices you have don't write a lot. Some may be powered off, some may not store much data and primarily use the flash to store the firmware, and some may not get used as often so they don't need to change so much data. The car's system will write to the storage automatically, including updates and entertainment data. When used consistently for eight years, that's a lot more writes than you'd normally find in other devices. Simultaneously, only 8 GB of flash is provided. While there's some more to handle hardware problems, that's a relatively small chunk to be constantly rewriting. I don't know the size of files that get written to that routinely, but given that the infotainment system which runs on that screen seems to have many features, it gets updates somewhat frequently to support them, and at least one of the features appears to be a navigation system with offline storage of some map data, I'm guessing it gets written a lot. If they also do things like storing logs there, that could be even stronger. I'm pretty sure they don't use it as a cache for autopilot images because EMMC is too slow for most of that.
"So what's the deal? Have I just been lucky, or has Tesla cheaped out on the components in its cars?"
No, yes. The problem is the cars are logging lots of data all of the time the car is on. Maybe even when it's off too. If you walk away from your computer while it's on but not doing some task, it's mostly just sitting there not reading and writing to the disk. So the issue is the constant banging log files in/out of the SSD and using parts spec'd mostly on price.
When I was in aerospace, we learned a lot of lessons from bad things happening such as Apollo 13, the major blackout in NY in the 60's and South Australia's grid train wreck. The big take away is when you do the initial design, you have to specify parts that will work and also have margin for a given nominal operating environment that is well characterized. If anything changes, you should have a complete review of every part in the system. If a valve is rated for 1,000 actuations at full pressure, you have to track usage and make sure it's replaced when it's approaching that figure. If you are specifying an SSD, you have to look at how much you are writing to it, it's expected number of writes in it's life and how long your product has to last. A $90,000 and up car isn't something that's thrown away after five years. A subsequent owner is also going to expect a reasonable lifetime and repairability or the resale value drops to chump change. The engine in my car (no EV yet) costs about $1,200 to swap. If I'm looking at a used Tesla and the motor pack costs $6,000 to swap and the mileage is around where that's likely, I'd want a really cheap price on the car.
on these £100,000 cars when the MCU is getting a little sick.
I talked to a Model X owner at a Charger just before Christmas and his car was suffering from this. He said that it was getting just as bad as his corporate laptop (with all the crap on top of Windows to supposedly make it safe to use) in getting going in the morning. 5-8 minutes was the norm. Then he proceeded to demo it to me. 5mins 49 seconds before he could drive away.
It made me thankful that the ability to drive my EV (Jag I-Pace) is very disconnected from the infotainment system. Yes, the touchscreen UI is crap but I hardly use it now that the car is setup unlike a Tesla Model 3 where almost everything including unlocking the glove box is done via the touchscreen.
Mind you, in Elon Musk's eyes, the driver will soon be surplus to requirements and can be laid off. That is if his vision for self-driving gets approved by the legal people in every country. I predict that the next generation of Tesla will not have any controls for humans to operate. Musk's disciples seem to be clamouring for this.
I, for one, am looking forward to self driving cars.
A taxi from the centre of Oslo to my place in the middle of the night would probably cost me about 2500NOK (£200+). A night bus (weekends only) takes around two hours. Drive time is about 30 minutes.
A hotel room in Oslo would be a better bet, obviously, but I have a dog whom I wouldn't want to leave alone all night.
So being able to jump in my car and have it whisk me home at the end of the night would be bliss.
on my 25 year old 200k mile car... and the (ten year design life) EPROM still holds the data it started with, the EEPROM inside the processor reports no fault, and the processor and all the other electronics still operated correctly. Even the power supply smoothing caps still work. And - mirabile dictu - the EPROM is on a socket, so it can be removed (which I have done for backup sanity purposes) and replaced. Thank you JEDEC that modern memories still fit the same sockets.
But then, it's one component designed to do one thing, not spend half its life chatting to the rest of the system to make sure no-one has had the temerity to change to a non-OEM component somewhere unrelated. Nor does it log its daily bowel movements; just when and if a fault is detected. I suspect that is not complex enough for modern tastes.
Take a car I drive, it had super high beam non OEM lights, installed by the dealership at time of purchase (New), that decided to fail.
Fitted original type bulbs for main beam not high beam, the drivers side ones wouldn't last a day (I'll confess they were ones pulled from previous owned vehicles as I prefer to change pairs if I have to pull out half the car to get to do it).
So I ordered up LED replacements for the headlights, they are super bright (I've dazzled myself frequently just walking to the car & remote unlocking). These cause random spasmodic flickering & other weird results on some vehicles thanks to the ECU, so a set of ballast resistors to give the ECU the correct signals had to be fitted.
The lights now work pretty much as intended but they still have their spasms every 5 - 10 minutes, apparently some cars need capacitors as well as\instead of resistors (Smoothing out the electrical noise for the LED cooling fans most likely).
Apparently its too much to ask for a cars lights to be hardwired to physical switches & not connected up to the ECU in some way & current smoothing of the 12V system to be more sufficient.
Icon - St Paul being blinded by the lights on the road to Damascus.
I think it's more that it's too much to ask that a bulb designed to go into a car 12v electrical system isn't capable of smoothing its own power if it's that sensitive to fluctuations that are inherently present in a car's electrical system.
If you can't handle 10-14v, and more, drops and spikes constantly without failing to perform your primary function (lighting up) then I think the fault is more the bulb.
And this is far from limited to car systems. Any low-voltage lighting made before LEDs will not accept an unballasted LED fitting lightly - it'll flicker and buck like mad. Everyone knows this. The LEDs are so low power that any unregulated DC power supply will go mad and buzz and fizz and fluctuate, and you have to buy a whole new power supply to make them work reliably.
It's not the CAR. It's the BULB. The reason that it doesn't work for you is that you ARE just hardwired to the 12v system, which is NOT smoothed in a car (literally, any automobile component is designed to take this into account - from bulbs and car radios to USB chargers, cigarette lighters, inverters, etc. - go look at any in-car PC enthusiast site... you need smoothers or tolerant and regulated power supplies. You can't just hook up a 12v cable to a car battery and expect it to work, because when starting, charging, air-con, cold weather, etc. kick in that 12v MUST be regulated if you want any kind of consistency. Which is why all the automobile components are either tolerant of it or have on-board regulators to make them tolerant.
Your bulbs obviously don't, and expect you to craft an RC circuit to compensate (which is a cheap, junky way of trying to do that, compared to a single-chip voltage regulator inside the device itself).
There are a number of potential fixes that don't involve a garage visit, it is just an engineering problem.
Firstly, reduce the amount of information you "need" to store to an absolute minumum, this always helps.
If the unit has permanent power, you could keep this in RAM, and only commit to flash if the battery is disconnected, relying on your bulk capacitors. However there are a number of tricky issues involved, many of which could be overcome, but it's all a lot of work. Things like the latency between detecting loss of power - perhaps the unit might be asleep? - and getting on with the write. You may only be able to write a few blocks, depending on the write time and your capacitance.
Best then to have an acceptable "no data saved" backup configuration.
My preferred solution, assuming there is no other easy fix, is to make the nVidia chip so it never writes its flash again - and offload the NVM storage either to another unit on the CAN bus, existing or new, or even, ghastly but cheap, make a device that plugs into the OBD port, and store stuff on there.
Those have already been done, and it extended the life, but not to "infinite" (enough). A side problem is that the logs are often critical to investigations. In one case where a Tesla caught alight, killing the driver, the fire progressed fast enough to damage the ECU within the flush duration and they had no around-the-incident data to investigate.
The impression I get about the design is that it's very much a "this is where all the action" happens design and would require a fair bit of re-architecting. Not just a case of replacing the failing units with new ones with the same failure profile. The NAND is part of the Nvidia card/SoC so not just replacable either. They might have to go with a wholly different, newer model and one wouldn't know how modular that design is. Likely not a drop-in upgrade.
HPE have exactly the same issue with their Gen8/Gen9/Gen10 Microservers/Servers. It's well documented, that the onboard ILO4/5 NAND is constantly written to on a daily basis with diagnostic health information. HPE have admitted to the fault in a paper regarding the problem, the hardware issue regarding excessive NAND writes is fully documented online.
The issue was compounded because HP initially removed access to firmware upgrades after one year of support, so it's not even the fault of the user for failing to upgrade the firmware, the firmware wasn't available, it either required purchase of their service pack or an extended support contract.
HPE have since modified the firmware so that it limits the number of daily writes to the NAND and made it publicly available (hence admitting fault), but this doesn't help the people that have mainboards that have already failed, aka. built in obsolescence.
Failure seems to be around the ballpark 30% mark of units. It's difficult to diagnose too, because it results in firmware corruption, so it can appear as a software issue when it isn't. The online support is such a minefield as there are multiple ways to upgrade the firmware and upgrades for both the system rom J06 firmware and ILO remote management firmware, as well as updates for intelligent provisioning.
An option have been added to reformat the flash, but again that helps no one if the flash is already damaged by excessive writes.
Attempting to get a replacement out of warranty from HP is like pulling teeth. It needs far more coverage like the Tesla issue than it is currently getting.
Surprised it hasn't resulted in a class action in the US.
UK Trading Standards are utterly useless, regarding anything that is technical.
HPE make out it's user error, when in this case it clearly isn't.
HPE do the right thing, and issue a recall on board with failed NAND.
I remember (back in the good old days when we could go outside) walking around a classic car show. Here are 50, 60 year old cars still in mint condition and still in full working order. Seems like when these Tesla's are that age the only place you'll be able to see them is sitting in static museum displays, because the electronic engineering required to keep them in working order will be beyond the skill level of the average classic car enthusiast. Sad really that the built in obsolescence is so stark.
Same for pretty much every car produced in the last 25 years.... They just will not be around in 50 years time as the electronics will not be operable and repairable. Particularly if sitting in a damp shed or barn for 30 years. Building a car that will become a classic in 50 or 60 years is not a large market for manufacturers and not on their list of priorities.
The only thing that may become viable after those years is to pull the engine, gearbox and ECU out, and replace with a modern (in 30 years time) electric power train and interface as best as possible to the rest of the car. And that would be more feasible for a 1990's car with discrete computing systems, rather than more modern ones where everything is on a bus and needs to talk to each other.
I wonder if you’re quite right about the more modern cars presenting a particular problem. As I understand it (and I don’t work in the motor business, so I could be totally wrong), most modern cars (built in the last 30 years (!) ) use CANBus, which is a robust and well understood standard. Presumably, when it gets to the stage of preserving these old japolys, manufacturers of replacement components (electric drive trains and so forth, as you suggest) will create Open ECUs (Linux based?) with open drivers and support for this known and widely used standard.
Biggest problem is not CAN bus which is well understood, but that fact that all the modules need to be registered to each other and effectively read the "serial number" of the car. Each manufacturer does this in a different way - ostensibly to prevent stolen cars being broken up for scrap. New/replacement modules have to be obtained and configured onto the car by the main dealer - 2nd hand ones (ie. scrapyard) will not normally work. Without a recognised and working original engine ECU you may find the dashboard ECU will not allow the car to be powered up. A good hacker will be able to find work-rounds and fudges, but whether there is enough interest to make it worthwhile in 30 or 40 years time is debatable.
Most modules which are tied to a specific ECU (or VIN, in the case of an ECU) are re-programmable. All the major manufacturers allow for it, but you have to pay. Most have plans as small as 24-72 hour for anywhere from $20 - $100 (each OE can have different prices and timeframes). Probably cheaper than your mechanic's labor rate.
Sites like https://www.oem1stop.com/ are a good index for all the OEs
But you do need extra hardware, like a J2534 Device. And some require Java (ugh!).
The only stuff that a DIYer can't get their hands on are modules revolving around key security, such as immobilizer and key codes.
Even independent mechanics aren't allowed access, unless they also have a locksmith license. Independent shops would still need to send those jobs back to a dealer or a locksmith to complete.
"Even independent mechanics aren't allowed access, unless they also have a locksmith license"
Only a handful of states in the US require a locksmith license. It's pointless these days anyway when you can buy whatever locksmithing tools you like online and from out of the country. I like Sparrows picks, but I can get cheaper and less expensive version on AliExpress. I'm tempted to try out some of the Lock Picking Lawyers products.
For reasons not clear, quite a few people like looking at and playing with antique tractors.
Quite a few of them in some parts of the world are antique John Deere gear.
Chances of a modern John Deere lasting more than (say) ten years seem to be minimal, for political not engineering reasons.
https://hackaday.com/2020/03/09/john-deere-and-nebraskas-right-to-repair-the-aftermath-of-a-failed-piece-of-legislation/
My dashcam seems to use a similar principle. It's a RoadAngel, and has flash memory sitting between the front/rear camera modules and SD card. The camera streams to the cache, where the overlay and GPS speed are written to the video stream. It then dumps the cache memory to the SD card. Several times it's gone wrong while driving, resulting in audible "SD Card error" warnings. It isn't the card causing the issue but the system getting confused as to where it was up to dumping the stream to card. I have to wipe the card in a laptop and then hard reset the camera to clear flash cache. Hard resetting without formatting card in a laptop doesn't work, so now I keep a small laptop in the car with me, as my dashcam is my witness that doesn't lie, I don't like driving without it with all idiots on the road!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
