back to article Worn-out NAND flash blamed for Tesla vehicle gremlins, such as rearview cam failures and silenced audio alerts

Worn-out NAND memory chips can cause a whole host of problems with some Tesla cars, ranging from the failure of the rearview camera to an absence of turn signal chimes and other audio alerts, a watchdog warned this month. Some 159,000 Tesla Model S and Model X vehicles built between 2012 and 2018 are at risk, we're told. These …

  1. Snake Silver badge

    Uh-huh

    "The manufacturer admitted the flash memory will still fail eventually, regardless of any software updates applied, and will require a hardware replacement."

    I'll let you guess who's pockets will be responsible for that required hardware replacement, I'm sure at full RRP plus billable service hours.

    1. quxinot

      Re: Uh-huh

      Don't be silly.

      Instead there will be a Tesla owner's loyalty rebate of nearly four (!) percent on the new model!

      (Fixing problems is for automobiles. These are phones with wheels, you throw them out when they go bad.)

      1. Anonymous Coward
        Thumb Up

        Re: Uh-huh

        "These are phones with wheels, you throw them out when they go bad."

        If I could give you multiple upvotes for that, I would.

        1. Anonymous Coward
          Boffin

          Re: Uh-huh

          That's Tesla's attitude and that was the problem.

          They used consumer grade memory instead of automotive grade which the rest of the industry uses and which allows significantly more rewrites.

          They compounded this by writing status to the memory as often as phones do instead of as often as other automakers do.

          1. hoola Silver badge

            Re: Uh-huh

            Yes but they are considered to be market leaders in the field because they got there quickly. This is what happens when software people try to sell hardware. There is no concept that something may need to continue working in 5 or 10 years time.

            Tesla appears to have the same strange status as Apple, the difference is that Apple hardware, though expensive usually works for a substantial length of time.

            1. Strahd Ivarius Silver badge

              Re: Uh-huh

              The difference between Tesla and Apple is the life expectancy of the battery before replacement...

          2. MachDiamond Silver badge

            Re: Uh-huh

            "They used consumer grade memory instead of automotive grade which the rest of the industry uses and which allows significantly more rewrites.

            "

            I think I saw this problem explained on "Rich Rebuilds" YouTube channel. It's not the grade of memory, it's poor programming that is abusing the module with status files the car is constantly generating. Going with a better (more expensive) grade of memory would just kick the can a little further down the road.

      2. Anonymous Coward
        Anonymous Coward

        Re: Uh-huh

        > These are phones with wheels

        That's funny, because that's exactly how I described it a few months ago when I tried out a Model S.

        (It's got some good stuff, but it's an American "car" through and through. That's not a compliment. Still sitting on the fence though.)

        1. MachDiamond Silver badge

          Re: Uh-huh

          "but it's an American "car" through and through"

          ........made with imported parts. A substantial amount of the car comes from foreign suppliers. Much of that is because the parts simply aren't made in the US. Especially electronic stuff but many of the castings are done outside of the US as well.

    2. Anonymous Coward
      Anonymous Coward

      Re: Uh-huh

      Its alright, Tesla can just send out an OTA upgrade to fix it!

    3. Geoff Campbell Silver badge
      Boffin

      Re: Uh-huh

      Tesla are not only replacing free of charge, but have announced that they will pay to cover the cost of those of us who had the eMMC replaced by a third party. I don't know if any of those payments have been claimed, as yet.

      GJC

      1. Roland6 Silver badge

        Re: Uh-huh

        >Tesla are not only replacing free of charge

        This time around...

        Tesla have form for removing functionality and then charging the user to reinstate said functionality... Ask Grant Shapps, the (UK) transport secretary, about his rear seats that were heated until an OTA software upgrade...

      2. Anonymous Coward
        Anonymous Coward

        Re: Uh-huh

        Which they did 2 years after everyone and their mother already knew this was a problem!

      3. DS999 Silver badge

        Re: Uh-huh

        They shouldn't have the problem in the first place. This is why the market for automotive grade components exists, but Tesla likes to cut corners by using consumer level stuff that's not rated to last as long as the car is. If it going to be a "wear item" like tires then it needs to be made easier to replace and have a way for people to find out in advance of its failure so they aren't caught in the middle of nowhere with all sorts of things suddenly not working.

        1. Anonymous Coward
          Anonymous Coward

          Re: Uh-huh

          Is it known that it was consumer not automotive grade? Are you able to offer an authoritative reference on that?

          1. Paul 195

            Re: Uh-huh

            Well, that's pretty much what the article said.

            1. Anonymous Coward
              Anonymous Coward

              Re: Uh-huh

              > Well, that's pretty much what the article said.

              I'm afraid I fail to find that information in the article or the linked PDF file. I don't suppose you would be kind enough to quote the relevant part?

              1. Anonymous Coward
                Anonymous Coward

                Re: Uh-huh

                Indeed, the article does not mention anywhere whether or not those were consumer parts.

                1. TSM

                  Re: Uh-huh

                  This bit certainly implies it:

                  > These all use an infotainment system powered by Nvidia's Tegra 3 system-on-chips that include 8GB of eMMC NAND storage, which is typically found in phones and cheap laptops.

  2. Blackjack Silver badge

    So not only Teslas are worse to play games than a Nintendo Wii...

    But the NANDs are worse than on a Nintendo Wii.

    1. J27

      Re: So not only Teslas are worse to play games than a Nintendo Wii...

      Some models of Tesla ship with the same CPU as the Nintendo Wii.

      1. Anonymous Coward
        Anonymous Coward

        Re: So not only Teslas are worse to play games than a Nintendo Wii...

        Nintendo *SWITCH*, not Wii!

        And no, the Tesla was using a Tegra 3, which came out in 2011. The Nintendo uses a Tegra X1, which came out in 2015.

        So no, not CPU at all. The Tegra 3 uses 4x ARM Cortex-A9 and the X1 uses 4x ARM Cortex-A57.

        1. Blackjack Silver badge

          Re: So not only Teslas are worse to play games than a Nintendo Wii...

          Also the Teslas have no space at all for the games. Granted the Wii barely had any space on it but it was a system that mostly used games on disks.

          1. Strahd Ivarius Silver badge
            Joke

            Re: So not only Teslas are worse to play games than a Nintendo Wii...

            and I thought that the purpose of Autonomous Cars was to let us play instead of driving...

            Games like "Colin McRae Rally" for example

  3. bazza Silver badge

    Not Fit For Purpose

    This is typical of an immature approach to engineering. One of the more obvious requirements is the service life of the car, and they thought about that for the battery but not (it seems) anything else.

    It's possible that some engineer at the time they designed this tried to point out that issue. However, Tesla has never struck me as a company where suggestions of adding cost are given any credit or much consideration.

    And, given their addiction to pushing out OTA updates, there's a decent chance that the same issue will end up affecting other modules in the cars too. One might protect oneself from that by skipping updates but, AFAIK, that's not the Tesla way...

    1. MacroRodent

      Re: Not Fit For Purpose

      Given that these days almost every piece of electronics contains some flash memory, I wonder if also other modern cars will be affected? The flash wear is not so noticeable in consumer electronics, which is rapidly recycled, but cars are supposed to last longer.

      1. Anonymous Coward
        Anonymous Coward

        Flash wear

        Sounds like a basic engineering oversight to me. Anyone who has been working with deeply embedded systems for a reasonable time knows that you can't continuously stream data / reprogram devices without running into erase endurance issues.

        One automotive ECU company I know would only allow a unit to be reflashed about 10% of the maximum number of times before it was considered no good for use as a reconditioned spare. However, even that allows for quite a few updates as the worst case (high temperature) part still generally support 1000 cycles.

        What kills the device (especially SOC) is using the flash to store log files or other data files that are updated regularly; most on board flash is not designed to be use like this.

        1. Steven Raith

          Re: Flash wear

          "What kills the device (especially SOC) is using the flash to store log files or other data files that are updated regularly; most on board flash is not designed to be use like this."

          I know this just from bumbling about with Raspberry Pi's. What are the Tesla engineers excuses?

          Steven R

          1. Anonymous Coward
            Anonymous Coward

            Re: Flash wear

            "I know this just from bumbling about with Raspberry Pi's. What are the Tesla engineers excuses?"

            Most likely the engineers working on the logging designed a logging scheme that worked on one itetation of hardware. They probably used an internal library for non-volatile storage and had no input to or knowledge of the underlying storage (which was probably located on a completely different controller). Then a completely different group made a minor change to the hardware spec. Then around the same time, the NAND flash itself had a spec change. All told, the changes combined to create this situation.

            Nobody sits down and "designs an automotive system", so it's not like there's one engineer that just didn't realize flash has specific concerns with repeated writes. The processes are hugely complex and are distributed to different groups. Software also (mostly) isn't written like it is on a phone or PC, instead behavior is modeled and code is generated (and then tested extensively, with and without actual hardware).

            1. Anonymous Coward
              Anonymous Coward

              Change control?

              Surely they would have that on a project like this?

          2. Strahd Ivarius Silver badge

            Re: Flash wear

            it did work without issues on the simulator!!!

          3. MachDiamond Silver badge

            Re: Flash wear

            "What are the Tesla engineers excuses?"

            The turnover isn't just high at the C-Level. I know a couple of engineers that were at Tesla for very short stays before running away and looking for a real job. A ton of institutional knowledge leaks out when people leave. I've never thought to ask how rigorous they are with documentation. When I worked on rockets (not SpaceX) we keep a log of things that didn't work. It was a bit of a slog but being able to search documents digitally is a really great tool. You don't want to go down blind allies more than once.

            The memory constraints should have been listed in some sort of design guide the same way that a motor controller design will have a spec sheet that describes max limits for things so a somebody isn't changing some code that starts an inverter/controller on fire.

        2. Electronics'R'Us
          Holmes

          Re: Flash wear

          Heavy writing to a flash device is not the standard use model, agreed.

          Although that is one wear mechanism, the data retention time is heavily influenced by temperature.

          Most devices state the data retention time for a device held at a constant 25C; increase that temperature and you get a dramatic drop in that (typically 1/2 for every 10C rise). The test for this is usually a 1000 hour test at an elevated temperature.

          If these devices are close to a heat source (such as all the processors that are in these vehicles doing some pretty heavy grunt work) then the data retention time will not be anywhere close to the datasheet specification (the datasheet only guarantees data retention at a given temperature).

          1. Jellied Eel Silver badge

            Re: Flash wear

            Although that is one wear mechanism, the data retention time is heavily influenced by temperature.

            Interesting, but what is meant by data retention? So does that mean that as temperature increases, the risk of data deletion (or errors) also increases? So is it a kind of temperature driven erase risk?

            1. Electronics'R'Us

              Re: Flash wear

              Interesting, but what is meant by data retention?

              Data retention time is the amount of time that the device will read back the same data as you put in and yes, this is a form of erase / errors risk.

              This is the reason that in all the critical projects I have been involved with two (sometimes 3) copies of critical data are programmed; the chance of all of them failing at the same time is tiny (infeasible as we say in avionics).

              For flash, the problem is that an entire sector can suddenly go pop (even though there may only be a single bit in error).

              The problem is worse in consumer grade MLC (multi-level cell) which encode more than one bit per cell with a range of voltages; what happens is that the sense voltage drifts over time and may not read back the same data because a particular cell set of thresholds have moved enough to change the interpreted read result.

              SLC (single level cell) devices are common in critical applications but it is getting more difficult to source.

              1. Jellied Eel Silver badge

                Re: Flash wear

                Thanks, that makes sense. And reminds me of my university days, doing safety critical designs. Was fun learning the difference between things that probably won't fail & must not fail, and how that affects design choices and costs.

            2. Anonymous Coward
              Anonymous Coward

              It's down to device physics

              Each data bit within the flash is stored as charge within an insulated cell, with the charge being controlled using quantum tunnelling. However, the insulation isn't perfect and some charge migrates over time, with the rate of migration increasing with temperature. In the long term, this does effectively mean that the device erases itself.

              The effect also becomes more dominant as device geometry is reduced (there is less space to store the charge, though this is counteracted to some extent by less "wall area" for charge migration).

              Flash memory in a "real" embedded device (designed for a long service life) will generally use larger geometry and less bits per cell to increase the data retention period.

          2. Blank Reg

            Re: Flash wear

            When placed in a car these devices will have to operate in some extreme conditions. -30c in winter, 50c+ in summer if they are inside the vehicle/greenhouse on a nice hot sunny day.

            1. Anonymous Coward
              Anonymous Coward

              Re: Flash wear

              That's why automotive devices are specified to work from -40c to +80c (cabin) and -40c to +125c (engine bay).

        3. c1ue

          Re: Flash wear

          As earlier commentators noted: temperature and extra wear due to log files are certainly a factor.

          I would bet, however, that the real cause is the fact that Tesla can (and does) access various cameras on the vehicle to fill its self driving data lakes. There is likely both a standard and "pull" type requests from the manufacturer for this data.

          Writing and reading pics is a lot more wear than text files...

          1. Electronics'R'Us

            Re: Flash wear

            One of the projects I was asked to take a look at in another division of $BigCompany was very early failure of compact flash cards (they use NAND flash).

            The application was using (IIRC) MS SQL for transactions and the cards were failing within a month; turned out there were millions of transactions per week as the particular version of Windows was not particularly flash memory friendly (such as buffering up transactions for later commit).

          2. Anonymous Coward
            Anonymous Coward

            Re: Flash wear

            Do you have a citation for that? Would be interesting to see what they collect.

            1. Anonymous Coward
              Anonymous Coward

              Re: Flash wear

              > Do you have a citation for that? Would be interesting to see what they collect.

              Happy to oblige (if you read German):

              https://www.netzwerk-datenschutzexpertise.de/sites/default/files/gut_2020tesla.pdf

              And if you don't, download the car's manual and read the Privacy section.

              But basically, there are some practices that are concerning (to me, at least) but it's surprisingly not that much. The risk I see is that if they ever decide that they are going to start "monetising" all this data by selling to third parties, etc., then things could get super nasty.

              1. MachDiamond Silver badge

                Re: Flash wear

                " if they ever decide that they are going to start "monetising" all this data by selling to third parties, etc., then things could get super nasty."

                If the data is there it could also be subpoenaed by a court. Your car might rat you out.

          3. joe bixflics

            Re: Flash wear

            Most affected vehicles are pre-AP2 and are no longer being actively monitored.

        4. SteveCoops

          Re: Flash wear

          Yep I've seen a good number of failed firewalls after logging was configured to store on the local storage after a couple years. Never used to happen when you had real hard drives inside the firewalls :)

    2. The Man Who Fell To Earth Silver badge
      FAIL

      Tesla has always operated in a reality distortion field

      That's why since it's founding it's made ridiculous statements about how quickly electric vehicles will take over the roads despite the fact that hundreds of millions of vehicles were registered in the US (273.6 million in 2018) with the average age of 12 years with over 25% over 16 years old, and the miniscule electric car production rate.

      Why would anyone think given Tesla's failure to recognize that, that their engineers designing the brains in their cars to last more than a few years?

      It's doubtful that even with top notch modern SSD's with their sophisticated wear leveling algorithms that you'd be able to put the decades of operational hours on them that a vehicle has a good chance of experiencing. One has to wonder about all the conventional vehicles whose computers & infotainment systems are the weak-link in those vehicle's operational lifetimes.

      Math - it's a thing, you know?

      1. Sgt_Oddball

        Re: Tesla has always operated in a reality distortion field

        I'd disagree with that, I've got some 128gb SSD's that are now 8 years old and still show over 80% life span left.

        Really they should have invested in a larger eMMC chips or proper SSD chipage.

        So long as its not shifting 10s of gigs in a day or should easily last 10 years.

        Or they could have made the part a maintenance part, have it report it's expected remaining life and replace when worn like any other mechanical part.

        1. The Man Who Fell To Earth Silver badge

          Re: Tesla has always operated in a reality distortion field

          Your mileage may vary depending on the system your SSD is in and how much write activity was taking place during those 8 years. Modern cars are continuously logging a lot of data, and its not being stored on spinning rust.

          I agree they should design from the getting to plan for the storage to have to be replaced at some point and design for that to be easy.

          1. Anonymous Coward
            Anonymous Coward

            Re: Tesla has always operated in a reality distortion field

            Most modern cars with electrical faults are right offs already, investigation, parts, installation & re-pairing often cost north of £1k.

            Our toddler fed coins into other half’s mini’s CD player, stopped the radio unit working which also controlled reversing sensors, indicators, Bluetooth phone, car info etc etc.

            Dealership wanted £250 for investigation, £700 for new radio unit including installation and pairing all plus vat.

            I downloaded the removal instructions and spent a few hours disassembling the dash to get the cd unit out, opening it up removing the coins and finally finding the fuse had blown. An hour finding a garage on a Sunday evening that sold fuses, replace the fuse and on testing it all worked saving her (me) over £1k for the sake of a few hours and a £1 pack of fuses. Would the dealership had done us a solid and just charged us for the fuse, no chance, possibly £302 inc vat for the investigation and testing of the cd unit but most likely the full £1k+ with the ready excuse that other unknown damage could have been done to the existing unit and a new unit would save them having to investigate again likely at their cost and the hassle of an upset customer of having to return again for something they said was fixed.

            There is little incentive to fix anything cheaply when new is always more lucrative.

            1. ThatOne Silver badge
              Devil

              Re: Tesla has always operated in a reality distortion field

              > Would the dealership had done us a solid and just charged us for the fuse

              You're kidding, that's way too technical for them. First of all, the would have to know what a "fuse" is.

              All they know is screw/unscrew parts and replace them with new ones till the problem goes away. You spot the good dealerships because they start replacing random parts close to where the problem probably is, as opposed to starting at the opposite side of the car.

            2. hoola Silver badge

              Re: Tesla has always operated in a reality distortion field

              Much of the problem also stems from the "plug the diagnostic tool" in. In other words a laptop, expensive cable and matching software. This then reads a load of binary shite and interprets it into a part number to be replaced. Job done.

              Anything that requires real investigation is beyond most garages now as the skill no longer exist. All the technicians (if that is what they are called) do is read a screen, order a part, fit it & repeat. On cars that are under warranty you are often forced into using a dealership to maintain the warranty, even if the part that is knackered has nothing to do with the warranty on the engine, gearbox or anti-corrosion.

              They will know that someone had gone and replaced part "x", what it number is and whether it is a manufacturers original. Fuses are possibly the only thing that is not covered.

              1. MachDiamond Silver badge

                Re: Tesla has always operated in a reality distortion field

                "Anything that requires real investigation is beyond most garages now as the skill no longer exist"

                There are independent EV shops that are very good. Basic wrench turners are askered of EVs and you see many more tech minded people at those shops. Some EV shops have a black ops department in the back where they reverse engineer stuff since companies like Tesla won't provide docs.

            3. MachDiamond Silver badge

              Re: Tesla has always operated in a reality distortion field

              "Dealership wanted £250 for investigation, £700 for new radio unit including installation and pairing all plus vat."

              I think an OEM replacement infotainment unit for a Model S is around $3000. It's replaced as a complete unit even if it's just a broken display. I'm not sure if the module is registered to the main computer of the car. If it is, you have to find somebody that can program a used unit you get online for a reasonable price.

              For a Model 3, you are dead in the water if the display dies so you have to pay Tesla whatever they ask to have the car work again.

        2. a_builder

          Re: Tesla has always operated in a reality distortion field

          I have similar experience with SSDs used as change in a high power NAS.

          As part of its function it backs up an office with TimeMachine so plenty of cycles.

          TBH I thought they would have failed by now: no sign of it.

        3. MachDiamond Silver badge

          Re: Tesla has always operated in a reality distortion field

          "I'd disagree with that, I've got some 128gb SSD's that are now 8 years old and still show over 80% life span left."

          Yes, but you aren't writing log files on a continuous basis all of the time your computer is on to that drive.

      2. Anonymous Coward
        Anonymous Coward

        Re: Tesla has always operated in a reality distortion field

        Electric cars will take over the roads, the kids who have been taught that fuel is bad electric is good will enter the car buying market soon and will expect to buy electric cars, coupled with the fact that major economies are mandating electric car sales only from 2030 and suddenly Musk’s early entry to the electric car market makes perfect sense.

        Musk also knows the money and fame comes from the technology of which he has a huge head start over the rest.

        Remember all those pc ads that stated “Intel inside”, the next evolution in Carr’s will feature no/new brand endeavours that will state “powered by Tesla” or something similar.

        1. Anonymous Coward
          Anonymous Coward

          Re: Tesla has always operated in a reality distortion field

          "and suddenly Musk’s early entry to the electric car market makes perfect sense."

          Not necessarily. Early pioneers rarely become the #1 because they make all the mistakes others can learn from and then surpass the pioneer. The Model S might still be the fastest accelerating road car but build quality control problems have dogged all of Teslas vehicles and having sat in a Tesla, well you either love the big touchscreen or think it looks like a naff after market tablet nailed to the dash that makes it hard to see some important information when driving and MUCH harder to do simple tasks that require input such as setting the radio the radio or air con. Plus IMO the whole interior feels far too cheap for the price of the vehicles.

          These points may sound trivial but its not to a lot of car buyers who arn't tech heads and just want a nice, good looking inside and out reliable car to drive and don't care about doing 0-60 in 3 seconds or self driving (when it works).

          1. Anonymous Coward
            Anonymous Coward

            Re: Tesla has always operated in a reality distortion field

            Your description of the S exactly matches my impression (and that of someone else who was with me).

            Sadly, I want an electric car that I can drive long-distance with and Tesla is right now the only option out there with proper range and which doesn't feel like a golf cart. I'll hire one for a few days see if I can convince myself to buy a non-German means of conveyance.

            1. Slef

              Re: Tesla has always operated in a reality distortion field

              You could always check out the KIa e niro!

              1. Anonymous Coward
                Anonymous Coward

                Re: Tesla has always operated in a reality distortion field

                Thanks for the pointer!

                Sadly, a quick check on the internet seems to indicate a range of only 280 miles, a design around an ICE platform, and a golf cart like driving experience. For the record, I'm a slow, always below the limit driver (on public roads). I just like getting to the limit very quickly >:)

            2. MachDiamond Silver badge

              Re: Tesla has always operated in a reality distortion field

              "Sadly, I want an electric car that I can drive long-distance with and Tesla is right now the only option out there with proper range and which doesn't feel like a golf cart."

              Get out and test drive some more models. Search YouTube for EV owners doing long distance road trips.

              If you are 20 something and have an iron bladder and no passengers, you might be able to drive 6 hours in one go. Get to 40-50 and bio breaks are mandatory in far less time and your traveling mate is going to be boggin for the loo long before the battery goes flat. If you can go 4 hours, that's often all you need if there is a DC fast charger at the end of that segment. You likely will want a meal, a run to the potty and a stretch. Another consideration is how often you take a trip that's longer than the range of the car. A little extra time a couple of times a year is balanced with never having to stop at a petrol station if you can charge at home the rest of the time. I've calculated that for the one really long trip I do every couple of years, I add about 2 hours to a full day of driving. With newer cars able to charge faster, that might be down to an hour by the time I buy an EV. I was going to get one in 2020, but ...... it's been a tough year.

              I don't think I've seen any current model EV that can be properly described as a "golf cart" like experience. They tend to be fairly posh.

        2. Anonymous Coward
          Anonymous Coward

          Re: Tesla has always operated in a reality distortion field

          > the kids who have been taught that fuel is bad electric is good

          I can see you've never driven an electric car (a Tesla, at any rate). Go and try one, then we'll talk.

          1. Michael Wojcik Silver badge

            Re: Tesla has always operated in a reality distortion field

            When Tesla make one without a fucking touchscreen, maybe I will.

            1. Anonymous Coward
              Anonymous Coward

              Re: Tesla has always operated in a reality distortion field

              > When Tesla make one without a fucking touchscreen, maybe I will.

              I was concerned about that as well, but in the flesh it's surprisingly unobtrusive. Another poster who is a Tesla owner (I'm not, I've only test driven the thing) reckons that the car is operated largely via the steering wheel controls.

              What I haven't done is drive at night, so I don't know how much the screen will affect your night vision.

              And with that said, I agree that the integration of the screens (in all models) with the interior design is terrible. It's like someone's nailgunned your computer monitor onto the dashboard.

            2. MachDiamond Silver badge

              Re: Tesla has always operated in a reality distortion field

              "When Tesla make one without a fucking touchscreen, maybe I will."

              A touch screen isn't a bad thing. When everything is accessed from the touch screen, it's a problem. I like the Kia and Hyundai approach of having traditional controls for the things that get used all of the time. It's not a problem to fiddle with the screen to dig out trip and car data if I'm all that interested. To change the fan setting, a touch screen is a nuisance.

        3. bigphil9009

          Re: Tesla has always operated in a reality distortion field

          I'm hoping that they'll come out with a nice new water biscuit, maybe something flavoured with mint.

    3. FlamingDeath Silver badge

      Re: Not Fit For Purpose

      Mature engineering and capitalism are incompatible

      1. MachDiamond Silver badge

        Re: Not Fit For Purpose

        "Mature engineering and capitalism are incompatible"

        I disagree. It takes mature engineers that are less focused on gadgets and more focused on functionality.

        I had a manufacturing company for years and could see that making a great product meant not losing money on warranty issues. Part of that was the nature of the product, but I'd rather spend time making and selling new product than dealing with returns. It kept more money in the bank account.

    4. bazza Silver badge

      Re: Not Fit For Purpose

      I've since read elsewhere that there is a free replacement program at least for broken vehicles (good), and for a small fee you can turn that into an upgrade to a later version of the ICE system. That seems like a pretty good deal.

      Wish the manufacturer of my car offered an ICE system upgrade. With most the only way to get the latest and greatest is to buy a new car...

      1. Steve the Cynic

        Re: Not Fit For Purpose

        "ICE system" is marvellously ambiguous, and I have to fight an urge to interpret "ICE" as "Internal Combustion Engine"...

      2. Geoff Campbell Silver badge

        Re: Not Fit For Purpose

        Yes, quite so, there's now an option to upgrade to the MCU2 from MCU1. I can't recall the price, but I didn't feel it was outrageous when I saw it a while ago.

        I probably won't bother, because MCU1 does everything I need it to in my six year old pre-AP Tesla, but it's nice to know the option is there.

        GJC

        1. MachDiamond Silver badge

          Re: Not Fit For Purpose

          "I probably won't bother, because MCU1 does everything I need it to in my six year old pre-AP Tesla, but it's nice to know the option is there."

          From what I remember, you lose the radio with the upgrade.

    5. fidodogbreath

      Re: Not Fit For Purpose

      I wonder what kind of storage chips they use in the Dragon crew vehicle.

  4. Piro Silver badge

    Would it have killed them to have a factory ROM failsafe?

    That way if the system fails to access the NAND, it can simply reboot to the original factory image, from ROM. You lose all your fancy updates, but the car will work. It's not like a mask ROM is going to 'die' any time soon unless the damn thing is physically damaged.

    1. David Pearce

      Who makes ROM these days?

      Masked ROM is almost extinct. A second written once NAND is more practical these days. It still self erases, but should be easily good for >10 years

      1. Anonymous Coward
        Anonymous Coward

        Re: Who makes ROM these days?

        "Masked ROM is almost extinct"

        It most certainly is not. Metal/via or diffusion programmable ROM compilers are widely available and used on ASICs.

        1. Anonymous Coward
          Anonymous Coward

          Re: Who makes ROM these days?

          Witness the Apple T2 chip, which has mask-programmed ROM - and unfortunately contains a security bug that cannot be fixed as a result.

      2. Natalie Gritpants Jr

        Re: Who makes ROM these days?

        Some SOCs have an OTP for storing serial numbers, crypto keys etc. Not a mask ROM but similar in that it is small but very stable. It's write-once non-volatile RAM, usually programmed on the chip tester or at packaging.

      3. The Man Who Fell To Earth Silver badge
        Boffin

        Re: Who makes ROM these days?

        >10 years isn't good enough. The average age of vehicles on US roads is 12 years, with over 25% of vehicles on US roads older than 16 years.

        In case anyone cares, the best review of long term storage persistence is still

        https://www.imaging.org/site/PDFS/Reporter/Articles/REP26_3_4_ARCH2011_Lunt.pdf

        1. Buzzword

          Re: Who makes ROM these days?

          You've linked to an article written by researcher Barry M. Lunt of Provo, Utah; who discovers that the best long-term archival medium is the Millenniata M-Disc. By an incredible coincidence, that same Barry M. Lunt is also the co-founder of the Millenniata company.

          1. Anonymous Coward
            Anonymous Coward

            Re: Who makes ROM these days?

            Unfortunately, the claimed 1000-year data retention life of the discs was not matched by the life of the company, which went bankrupt in 2016. https://en.wikipedia.org/wiki/M-DISC

            The drives are still available, as are the discs (4.7GB, 25GB or 100GB - even the largest size doesn't sound that much these days). They can be read by standard blu-ray players, although whether those are still available to buy in 10 years time, let alone 1000 years, remains to be seen.

      4. MachDiamond Silver badge

        Re: Who makes ROM these days?

        Some things should never change so a masked ROM is a really cheap way to implement those functions. It's also going to be nearly bullet proof.

    2. Neil Barnes Silver badge

      I have to confess that I have been concerned about bitrot in the EPROM on my 25-year-old Fiat's ECU... enough to make a copy so I can blow it again if necessary. But not doing a basic life expectancy of your long term storage? Inexcusable.

      I see a class action coming head on. I wonder if the autopilot will recognise it?

    3. Persona Silver badge

      If only they had a battery available they could have used RAM ........

      1. sgp

        Batteries degrade too, I would opt for a small petrol power generator.

        1. Will Godfrey Silver badge
          Angel

          But what if the petrol runs out?

      2. MachDiamond Silver badge

        If only they had a battery available they could have used RAM ........

        Oh great. It's bad enough to have to reprogram radio stations back and reset the clock in when I need to disconnect the battery.

  5. Mike 137 Silver badge

    "vehicles built between 2012 and 2018 are at risk"

    My vehicle was built in the mid '90s and is still running glitch free. It contains no NAND flash. Is it actually necessary?

    1. Anonymous Coward
      Anonymous Coward

      Re: "vehicles built between 2012 and 2018 are at risk"

      Indeed. The only reason why my parents' vehicles from the late 1980's aren't still running is rust (British maritime climate) ...

    2. Anonymous Coward
      Happy

      Re: "vehicles built between 2012 and 2018 are at risk"

      My vehicle was built in the mid '90s and is still running glitch free. It contains no NAND flash. Is it actually necessary?

      The engine management system often tunes itself to the behaviour of the engine, and things like odometer and maintenance reminder resets also need to be stored permanently. You wouldn't want this stuff reset when a battery is disconnected. The amount of data is small though.

      But you can get around the limited write cycle life of flash memory blocks by using an overly large flash memory with wear levelling, and by maintaining the current data in RAM and only preserving it to flash during the system power down interrupt.

      1. Mike 137 Silver badge

        Re: "vehicles built between 2012 and 2018 are at risk"

        The odometer is mechanical and I remember the service schedule using my brain and a diary, so nothing resets when I disconnect the battery. Only the dash clock has to be adjusted when it's reconnected. Where's the need for flash memory?

        1. John Robson Silver badge

          Re: "vehicles built between 2012 and 2018 are at risk"

          Well if you're going to go like that: "Where's the need for flash memory?"

          You could equally say that we managed with horses for centuries, where's the *need* for a car.

        2. Anonymous Coward
          Anonymous Coward

          Re: "vehicles built between 2012 and 2018 are at risk"

          Where's the need for flash memory?

          A car only needs three wheels, a single speed transmission and a simple tiller for steering to perform its basic "transporting people" function, so it doesn't really need seats or a windscreen either.

          But, for one thing, the fuel efficiency of modern engines demands persistent memory. See the dark green line here for how fuel efficiency of European cars has changed since the 1990s.

          An early 1990s car typically had a mechanical distributor/spark gap ignition system and a carburettor. You can't get modern fuel economy using that.

          A modern car has a sophisticated engine management system that precisely controls the amount of fuel fed into the engine, and typically injects fuel at several points during the ignition portion of the cycle, as well as precisely controlling the spark timing. This is all mediated by sensor inputs, such as the throttle setting, the oxygen level sensor in the exhaust, the engine torque and the engine speed etc and there are calibration settings that need to be automatically updated with engine wear.

          And customers and legislation demand other features that require persistent memory - multiple trip recorders, fuel economy readouts, infotainment systems, navigation systems, cruise control, automatic tyre pressure monitoring, sensor activated non-Bulgarian airbags, ABS and so on. Most of these have persistent data, such as sensor calibration settings, stored preferences, and all of them have software (and sometimes data like maps) that needs updating with the minimum of effort.

          1. Michael Wojcik Silver badge

            Re: "vehicles built between 2012 and 2018 are at risk"

            An early 1990s car typically had a mechanical distributor/spark gap ignition system and a carburettor

            Citation, please. AFAIR, by the early 1990s most passenger cars used fuel injection. Wikipedia agrees, for what that's worth.

            I checked a couple of easily-accessible possible authoritative sources (such as the annual US EPA Fuel Mileage guides), but they didn't include carburetor-versus-injection information.

        3. MachDiamond Silver badge

          Re: "vehicles built between 2012 and 2018 are at risk"

          " Where's the need for flash memory?"

          It's all of the gadgets that cars have now. Seat position memory, custom radio station/music subscriptions for each driver, car data, Addresses in SatNav, databases of charging stations. preferences for various features, ad nauseum and lets not forget all of the spyware that rats out how many times you've been over the limit or the G forces haven't been on the high side.

      2. Anonymous Coward
        Anonymous Coward

        Re: "vehicles built between 2012 and 2018 are at risk"

        But tuning stuff should be in a FRAM.

    3. Blank Reg

      Re: "vehicles built between 2012 and 2018 are at risk"

      I had a a late 80's era Dodge that had it's "computer" fail after almost a decade. It cost around $150 to replace, parts and labour. Now it would likely be an arm and both legs.

  6. macjules

    Nail meet head

    Got rid of my 2014 Model S in late 2017 when lots of things started to go wrong with it, including loss of rear view camera, indicators not working properly and the entire Multimedia system repeatedly going into reset (not good on a long journey with 3 whining teenagers in the back of the car).

    The part exchange reason I gave (to Audi) was "too many gremlins". Thanks for clarifying what it really was.

    1. Anonymous Coward
      Anonymous Coward

      Re: Nail meet head

      What do you drive these days?

      1. macjules

        Re: Nail meet head

        Gas-guzzling Audi Q7 SUV. Done my tree hugging bit by buying a Tesla. Now it's my turn to reduce the amount of fossil fuel left in the ground,

        1. Anonymous Coward
          Anonymous Coward

          Re: Nail meet head

          Well, it's one way to help with the transition to clean energy! :-)

  7. John Sturdy

    We'll get used to it

    Perhaps this will be the equivalent of clutch / head gasket replacement for electric cars.

    1. Anonymous Coward
      Anonymous Coward

      Re: We'll get used to it

      No, it's a sign of poor engineering and lack of (experienced) design review.

      What could be considered as similar is having to replace an ECU when it's flash "self erases". Consumer grade devices can do this quite quickly (a few years), but "proper" embedded ones should be good for much longer (for example, the XMC4500 is good for 20 years with up to 100 erase/program cycles).

      1. Geoff Campbell Silver badge

        Re: We'll get used to it

        There are plenty of examples of poor engineering leading to large numbers of component replacements on internal combustion vehicles. Head gaskets on Rover K series engines, for example, or timing belts on Land Rover TDi engines, or camshafts on the Rover SD1 V8s.

        Funny how so many of the examples that spring to mind are Rover...

        GJC

        1. Ozumo

          Re: We'll get used to it

          GM Northstar head gasket failures?

          1. Geoff Campbell Silver badge

            Re: We'll get used to it

            Outside my experience, but I wouldn't be at all surprised.

            GJC

        2. Peter Gathercole Silver badge

          Re: We'll get used to it

          Let's be a little forgiving of the Rover engineers. Most of the time, they knew what was wrong, knew how to fix it, but just weren't allowed by the management to spend the money to change the production line. See the AROnline site for the story.

          The example of the K-series engine is interesting, because SAIC took the PowerTrain molds and tooling, and NAC took the design, and both did what the engineers said, changing the material of the head bolts, designing a new head gasket, stiffening the various components, putting metal dowels in and changing the location of the thermostat, and ended up with the N-series and Kavachi engines, both of which I believe have gained quite a reputation for being a lot more reliable than the original K-series.

          I loved my 1.8 Rover 45 Connoisseur saloon, right up to the time it overheated for the first time. I really wish I had spent some money keeping it on the road and applied the N-series mods to it.

          1. Geoff Campbell Silver badge

            Re: We'll get used to it

            Oh, for sure - but the original engineers doing the original design overlooked some obvious flaws, just as the Tesla engineers are being lambasted for now.

            Bottom line, it happens, and will always happen, and the measure of the company is how it responds. As an owner of one of the first Teslas to be imported into this country, I'm quite happy with how this one has been handled.

            GJC

            1. Anonymous Coward
              Anonymous Coward

              Re: We'll get used to it

              I'm not sure I agree. This Tesla failure is a component failure, not so much a design failure.

              It's also a problem long solved by the automotive industry and flash manufacturers. There is plenty of info and advice on avoiding this issue, as well being easily detectable in MTBF type tests.

              How is Tesla confident of their warranty commitments? Brand damage due to reliability concerns. These are company assets that are potentially being damaged.

              Musk cleverly uses Twitter stunts to distract from these operational concerns.

              1. Mike 16

                Re: We'll get used to it

                @AC -- a component failure, not so much a design failure.

                The design failure is running a toxic waste channel through a recreational area.

                OK, actually putting safety-critical functions in the "infotainment" system. This was a WTF moment when another manufacturer did it a decade or more ago. There is no excuse to repeat that mistake today.

        3. david bates

          Re: We'll get used to it

          Head gaskets on the K were fine until BMW stretched the engine too far and replaced the dowels with plastic versions....

          1. Peter Gathercole Silver badge

            Re: We'll get used to it @david

            Not sure that is really the case. Sure, the plastic dowels did not help, but the primary problem was thermal shock, because the thermostat was on the water-in side of the block which caused the block to overheat before the thermostat opened and flooded the engine with cold water, compounded with the thermal co-efficient of expansion on the head bolts being greater than the block and head (remember, they were long thru-block bolts going right down to the sump) which allowed the head to move about on the block when the engine was very hot, peeling the mastic off of the gasket.

            The aluminum alloy used for the head would also become gas pourous when it got too hot, which meant that once cooked, no amount of changing the gasket, skimming etc. would allow the engine to return to it's former glory, and it would pressurize one or both of the water and oil ways.

            This was documented long before BMW got their hands on the company. I believe that some of the problems stemmed from the original decisions selecting the metal alloys for the DOHC (16 valve) engines. Late in the design process for that particular variant, a cheaper alloy was substituted for the head as a cost saving. The original 8 valve 1.1 version was much more reliable.

            Before my 45, I had a 1.6 K-series Rover 400. That suffered a mild head gasket failure (without being cooked), and that was repaired with an MLS gasket and aftermarket bolts (it already had metal dowels), which survived close on 100,000 miles without the slightest sign of problems before I turned it on it's side during some particularly nasty icy weather (long story involving a sharp turn on a downward hill and sloped banks on a rural road in Somerset).

            1. Anonymous Coward
              Anonymous Coward

              Re: We'll get used to it @david

              I had a Rover 800, but I made sure it had the 2.7 Honda engine. Never had a single problem with the engine unlike the scrap iron with a flywheel Longbridge tossed over the gates. Sadly the rest of the car was typical Rover reliability and despite the engine being fine it had to go.

            2. Anonymous Coward
              Anonymous Coward

              Re: We'll get used to it @david

              Had the 1.8 vvt version in my Elise - they were notorious for head gaskets failure, but I never had any problems in the 8 years and 50k miles I owned it - unlike the Toyota engined Exige that followed it.

        4. Steve K

          Re: We'll get used to it

          Funny how so many of the examples that spring to mind are Rover...

          No coincidence that the motoring press called the Rover 75 the "Last Chance Saloon"....

          1. Peter Gathercole Silver badge

            Re: We'll get used to it @Steve

            The 75 was quite nice, if you could get past the slightly retro look (although the look appears to be coming back a bit in Audi and some other makers who still make saloons).

            But for good or bad, many of the parts were from the BMW parts bin. I had a face-lift 2.0 cdti which had a roverised BMW diesel (M47R), which gave me no end of minor troubles (thermostat failure, coking up, injector and sensor wiring) and eventually a total failure, probably caused by the high pressure fuel rail sensor (that was probably a Rover thing, as the BMW M47 engines were not common rail injection when Rover were told to use the block). I spent so much time trying to fix it that by the time I got to look at this sensor, the work looked more than I was prepared to put in as it was such a bitch to work on.

            In addition, the Bosh locking system gave me regular problems, the JTAC auto box kept reporting errors, the hand brake was troublesome, the electric door mirrors didn't work properly, the door trim kept falling off on the back doors, and it leaked. And you couldn't read the faults from the ECU, because Rover did their own thing with th OBD socket. It was a good drive when it was working, but I wouldn't have another.

            1. MJI Silver badge

              Re: We'll get used to it @Steve

              For Rover I think the BMW engines were actually a disadvantage.

              Rover group had a post L series Diesel, better than the BMW, never got used by them.

              Land Rover had a range of engines, 2 of them were killed before lauch, the surviving middle one is an excellent lump.

              Imagine a 75 with a G series, a Freelander with a TD4, or a L322 with a TD6.

        5. MJI Silver badge

          Re: We'll get used to it

          I have a car with Rover in the name.

          Failed parts on it are from Denso, VDO, and a few others.

          That said, the supplied chassis rusts for England, tempted to put a galvanised one under it.

          Bu the factory built parts are fine.

    2. John Robson Silver badge

      Re: We'll get used to it

      Nope - it's nothing like that... partly because all cars use flash memory nowadays.

      If they can come up with a way to make m.2 sufficiently vibration tolerant then I can see them just falling back on "standard" storage, and replacing it every ten years.

  8. anthonyhegedus Silver badge

    It is inexcusable when the correct thing to do would be to use higher quality storage and/or a failsafe. It would add less than a few tens of dollars to the cost of manufacture. But all that battery-bothering nutjob Musk is interested in doing is putting out silly updates with games or other 'fun things' that have nothing to do with safety. There really will come a time when you can't leave for work because your car's stuck doing updates.

    1. Ochib

      A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.

      1. Geoff Campbell Silver badge
        Facepalm

        The poster-boy case for this was the Ford Pinto, which had a nasty habit of self-immolation when rear-ended.

        Which is why the recall system is now run by external bodies, as I understand it. I doubt very much any company still does the sort of calculations you mention, and if you have evidence that I am wrong, I would strongly suggest that you investigate the whistle-blower laws in your state/country/whatever and dob them in.

        GJC

        1. Ochib

          Not a fan of the film staring Brad Pitt, Edward Norton, Meat Loaf and Zach Grenier.

        2. Anonymous Coward
          Anonymous Coward

          Yep, just doing the calculation at all would massively increase a jury award if you got caught doing it. You'd almost certainly get fired for even suggesting it.

          Life threatening problem = recall these days, it's guaranteed to be cheaper than the lawsuit payouts.

        3. TimMaher Silver badge
          FAIL

          Pinto

          Didn’t that feature in “Unsafe at Any Speed”?

          1. Anonymous Coward
            Anonymous Coward

            Re: Pinto

            First time I was in California the local car rental was offering great deals on Pintos - took me a while to find out why. Luckily they then switched to special deals on Mustangs because gas hit $1 a gallon and the Angelenos were in shock.

    2. Roland6 Silver badge

      >There really will come a time when you can't leave for work because your car's stuck doing updates.

      The time is already here where you can't be sure if the Tesla you intend to drive to work in, will behave the same as it did when you parked it on the drive last night; it caught James May out...

      1. MachDiamond Silver badge

        "There really will come a time when you can't leave for work because your car's stuck doing updates."

        Tesla is really big on updates. There was a braking issue and they released an update in a day or two. A Model S catches fire sitting in a parking garage in China, next day there is an update. I have to wonder if they knew about these problems before hand or if they rushed something out before they did any proper testing. The fire in China hadn't been investigated and I have a hard time thinking that a braking system could be recalibrated and thoroughly tested in two days.

        The hoi poloi are all smitten with OTA updates, but I've been burned enough times to like to wait and see how other fair with updates before I update anything of mine.

  9. Howard Sway Silver badge

    Autopilot is part of the "infotainment system"?

    Surely any respectable engineer would realise that the mp3 player and the steering-and-driving-the-actual-car system should be completely seperate systems, with no physical way for them to affect each other, ever. I mean, they don't connect in flight entertainment systems on planes to the autopilot do they? I would be interested to hear how modular these car systems are, in terms of ease of replacement and cost for failed parts.

    1. Anonymous Coward
      Anonymous Coward

      Not sure where Tesla are...

      But quite a few systems are moving towards platforms where a (hardened) hypervisor* is used (ARINC, Linx, QNX, other posix derivatives) to run a lot of the higher-level functionality as isolated components as it saves space, reduces weight and has the potential to reduce costs. This is happening within automotive and aerospace.

      So, that gives modular at the software level, and possibly at the hardware level.

      *In theory, there are no connections between components, but security vulnerabilities are likely to exist (and eventually be found).

    2. Geoff Campbell Silver badge

      Re: Autopilot is part of the "infotainment system"?

      The cars can be driven with a complete failure of the MCU.

      There are a few edge cases, like for example the PIN-to-drive system can leave you unable to turn the drive system on once the MCU has failed, but all the basic vehicle driving controls work fine. I drove 250 miles home after the MCU had failed (whereupon I discovered that the MCU also controls AC charging, and I was too far from a DC charger to get there, but at least I was home).

      GJC

      1. Anonymous Coward
        Anonymous Coward

        Re: Autopilot is part of the "infotainment system"?

        What do you drive and how do you like it?

        1. Geoff Campbell Silver badge
          Go

          Re: Autopilot is part of the "infotainment system"?

          2014 Model S85. I bought it from Tesla in 2017 with a full warranty, because I'm brave rather than stupid :-)

          I've done 50,000 miles in it, bringing the total up to 103,000 miles, in those three years, and I still love it, despite a few minor reliability issues such as the eMMC under discussion. It's too big for UK roads, but it's fast (despite being the slowest model they've produced), smooth, quiet, costs pretty much nothing to run, and still brings a smile to my face every time I drive it.

          Perhaps more importantly, the UI is superb. Forget the big touchscreen, that's mostly irrelevant during actual driving, everything can be done from a few buttons and two scroll wheels on the steering wheel.

          Every time I get back into a petrol or diesel car, I'm stunned to remember how badly laid out they are, with buttons and switches all over the place, and it takes me weeks to work out how to do the simplest operations. It doesn't help that hire cars tend not to include the user manual, weirdly.

          GJC

          1. Anonymous Coward
            Anonymous Coward

            Re: Autopilot is part of the "infotainment system"?

            Thanks, interesting point about the human-machine interface, that's precisely the kind of things that I cannot evaluate properly in a short ride.

            Doesn't the plastic fantastic interior quality bother you? Personally, I dislike pretentiousness so I stay away from leather, brothel-like interior lightning, silly gimmicks, and shit like that, but I do like a solid, durable interior (for comparison, I drive a Bavarian brand which is not BMW).

            Also, the lights, especially the lack of adaptive lights, bothers me intensely.

            1. Geoff Campbell Silver badge

              Re: Autopilot is part of the "infotainment system"?

              I don't find the interior offensive, but then I've always been puzzled by the whole concept of "build quality". After six years and 103,000 miles, nothing has fallen off, and there are no squeaks or rattles, so that's good quality to me. The seats are leather, and pretty hard-wearing with just a bit of creasing on the side of the driver's seat where I get in and out, and there's nothing that I find offensive to look at or touch. What more could I want?

              I'm not sure what you mean by adaptive lights, which is perhaps why I don't miss them, whatever they are :-)

              GJC

              1. Anonymous Coward
                Anonymous Coward

                Re: Autopilot is part of the "infotainment system"?

                > After six years and 103,000 miles, nothing has fallen off, and there are no squeaks or rattles, so that's good quality to me

                Rattles and squeaks are one thing that really does get on my tits and my main concern, along with stuff just becoming brittle and breaking (although I do garage my cars).

                > I'm not sure what you mean by adaptive lights, which is perhaps why I don't miss them, whatever they are

                This: https://www.youtube.com/watch?v=Nu9Unar4tEw.

                It is, literally, the difference between night and day. On dark nights, quite often I end up with cars just sitting behind mine in spite of my slow driving, quite obviously because of the vastly improved visibility.

                I know Tesla has automatic high beams, but that's nowhere near the same thing. Apparently, the issue might be that adaptive lights are not approved in the US and their car design still seems to be very US-centric.

                One last question if I may, do the over the air updates actually "improve" the car in any way, or are they just a convenience to save you a trip to the garage for a firmware update?

                Once again, thanks for sharing your experience! It does give me quite a bit more confidence!

                1. Geoff Campbell Silver badge

                  Re: Autopilot is part of the "infotainment system"?

                  My pleasure - always good to chat.

                  Adaptive lights do look like a lot of fun, certainly.

                  OTA updates often add new features. Case in point, a couple of years ago when there was a spate of radio-relay thefts of high-end cars, Tesla very quickly added and rolled out a PIN-to-drive feature, so that the car couldn't be driven without first entering a PIN on the touchscreen. Teslas stopped being stolen, Range Rovers and the like carried on disappearing.

                  Cars with Autopilot/Full self driving benefit more from the updates, as new capabilities are being added all the time, however they do keep updating the older cars alongside the new ones, as well.

                  GJC

                  1. Anonymous Coward
                    Anonymous Coward

                    Re: Autopilot is part of the "infotainment system"?

                    Thanks!

                    Yeah I recall those thefts. I understand that the key fob firmware is also upgradeable, which is nice.

                    The big question mark is what degree of automatic driving will be allowed in Europe (where I live) and when. The more automation the better: it's not a replacement for the human driver but a complement, and as a former pilot I think I'm well trained to operate a highly automated vehicle.

                    1. Geoff Campbell Silver badge

                      Re: Autopilot is part of the "infotainment system"?

                      Absolutely - we live in interesting times. Some of the videos coming out on YouTube with the newly released full self-driving beta are looking very encouraging, but as you say the legislation needs to catch up.

                      GJC

              2. MachDiamond Silver badge

                Re: Autopilot is part of the "infotainment system"?

                "After six years and 103,000 miles, nothing has fallen off, "

                There are plenty of owners that say the same thing, but there are plenty of owners with big horror stories about endless problems and the ages it takes to get them fixed.

                Every EV is cheap to operate. The cost and time to repair a Tesla can be huge. It's not a bad idea to research things like the cost to repair a minor bumper ding or headlight/taillight replacement. The headlight on a Model 3 is $880 from Tesla. I'd love to get my hands on one as it has a circuit board inside and I'm wondering if it's loaded with DRM to fend off third party replacements.

    3. Anonymous Coward
      Anonymous Coward

      Re: Autopilot is part of the "infotainment system"?

      "Autopilot is part of the "infotainment system"?"

      Well, (on a particularly shallow level) that seems appropriate, since at least in the UK, the driver is fully responsible for what the vehicle does even if autopilot is turned on, although admittedly, autopilot is probably safer than using a radio to chose when to apply the brakes. I mean, you can't count on the right bit of MC hammer or Vanilla Ice to play at just the right moment.

    4. mevets

      Re: Autopilot is part of the "infotainment system"?

      Aircraft autopilots actually work, with well defined boundaries and abilities. Tesla's is a joke, so yes, entertainment is quite appropriate.

      1. Anonymous Coward
        Anonymous Coward

        Re: Autopilot is part of the "infotainment system"?

        > Aircraft autopilots actually work

        Former commercial pilot here. Unless you are (or were at the time) working for Ryanair, it was a rare day when you flew a ship that didn't have at least one failed system. Including the F/D or the autopilot (you can actually despatch without, there is something called the Minimum Equipment List and if the bits that are not working are not there and the engineers haven't grounded the aircraft you're good to fly).

        Please consider contributing something useful next time.

      2. MachDiamond Silver badge

        Re: Autopilot is part of the "infotainment system"?

        "Aircraft autopilots actually work"

        There big differences between Tesla FSD and an aircraft autopilot. There are air traffic controllers that are telling pilots where they need to be and what speed to be going. Aircraft are also spaced far enough apart that they have plenty of wiggle room. A car needs far greater accuracy. I don't recall a tree that has ever blown over into a sky lane. Nor is it common for a plane to pull out in front of another plane in the flight levels. A car that's 5ft to one side is going to be in the other lane or half on the pavement where bad things are likely to happen. Planes: 5ft is no big deal.

        There is an automated ore train in Western Australia. I think they figured out that with some extra sensors at level crossings, the need for a driver is zero. The trains are so heavy that a driver seeing something wrong up ahead might only mean hitting it 10mph slower than if it wasn't seen if they could slow down that much. That "autopilot" is in a wholly different situation and there aren't good comparisons with aircraft or cars. The same applies to aircraft autopilots and self-driving cars. The issues with self-driving cars is massively complex. The simplest approach would be separate lanes and external navigation. For long trips, a separate lane on the motorway would mean you could switch the car over to the "bug" and have a nap for the next 4-6 hours until your car needs a charge up. City centers are a bigger problem and I've always found so many re-routes and obstacles that self driving would take a science fiction grade AI to sort out the driving.

    5. MachDiamond Silver badge

      Re: Autopilot is part of the "infotainment system"?

      " I would be interested to hear how modular these car systems are, in terms of ease of replacement and cost for failed parts."

      Not very modular and many published hacks show the exploit going through the infotainment system as it usually has the most exposure to the outside world.

  10. Geoff Campbell Silver badge
    Linux

    Just for the record:

    The Tesla response to this can be found on their website. I'm not sure if you need an account to get to it, though. In summary, warranty has been extended to 8 years or 100,000 miles, any failures will replace the 8GB eMMC with a more recent 64GB eMMC chip, and they will consider claims for reimbursement of third party repairs that have already been performed to do the same replacement (which I have had done, so I may put in a claim when the process is clearer).

    https://www.tesla.com/support/warranty-adjustment-program

    GJC

  11. mark l 2 Silver badge

    While mechanically EVs are much simpler than internal combustion engine vehicles, all modern cars have so many more electronic components added compared to ones of 10 to 15 years ago, which means that there is a potential for them to go wrong. So no doubt when a lot of these cars are approaching their first decade of use things will start to fail and need replacing. And will spare parts be available?

    1. Anonymous Coward
      Anonymous Coward

      Just hope the media entertainment system on Dragon doesn't contain any NAND chips ...

  12. DrXym

    3-4 years??

    That suggests some heavy duty writing. If this is such an issue then the eMMC should be an easily replaceable part, e.g. an SD card under the dash and Tesla should figure out how to use caching for non essential information instead of hammering it.

    1. Anonymous Coward
      Anonymous Coward

      Re: 3-4 years??

      It suggests some heavy duty stupidity on behalf of the developers.

      1. DrXym

        Re: 3-4 years??

        Tesla logs a lot of stuff and continuously records video for stuff like sentry mode and its own ass-covering. I have no issue with that - parts in cars wear out.

        But if they're going to clobber the storage then it should be easily replaceable. If it fails just like other bits on a car that people have to change from time to time. They could also replace the thing during servicing or make it so somebody can reach under the dash or the frunk and swap it out.

    2. Jellied Eel Silver badge

      Re: 3-4 years??

      ...If this is such an issue then the eMMC should be an easily replaceable part, e.g. an SD card under the dash...

      Yup, that would make sense. ISTR seeing Rich Rebuilds showing a previous issue where the flash would just fill up with log files and fail, which was fixable via an OTA update. I agree that it would make sense for wear parts to be easily accessable, and replaceable. So rather than having to replace an entire ECU/MCU due to being SOC, just replace the drive.

      I guess that's part of the cost modelling done by the mature automakers, ie labour cost to get to a part, then cost of just swapping a module vs swapping a component. Would seem kinda important, especially if done under warranty. And I guess given Teslas are mobile compute devices, there may be some benefits to swapping the entire 'brain' with the latest tech so it could support new features.

      1. MachDiamond Silver badge

        Re: 3-4 years??

        "would just fill up with log files and fail, which was fixable via an OTA update."

        If it is the same episode I remember, it wasn't an update fix. The module had to be replaced and the guy Rich was talking to showed a collection of failed flash sticks.

  13. Anonymous Coward
    Anonymous Coward

    Beta

    These are problems long solved in this industry. They need to give Toyota a call.

    It's said "Cheap, quick to market, or good quality, pick two". Tesla is clearly the first two. A which review of car brands found most owners finding lots of trouble like leaking door seals, trim and electrics issues.

    This is their MVP - minimum viable product.

    I guess in Silicon valley spirit, Tesla cars are still in beta.

    1. Martin
      WTF?

      Re: Beta

      It's said "Cheap, quick to market, or good quality, pick two". Tesla is clearly the first two.

      In what universe are Tesla's cheap?

      1. Claptrap314 Silver badge
        Boffin

        Re: Beta

        The one in which "inexpensive" is different from "cheap".

  14. This post has been deleted by its author

  15. MarkMLl

    Old news

    This has been knocking around for a year, see https://hackaday.com/2019/10/17/worn-out-emmc-chips-are-crippling-older-teslas/

  16. TWB

    I blame...

    ...infotainment systems - they are too prevalent now, bring back proper switches and controls I can use without looking which don't all depend on some all governing 'CPU'

    1. hoola Silver badge
      WTF?

      Re: I blame...

      This current fashion for a completely software based dashboard including the major dials is just stupid. We have reached the point that the basic functionality of a car cannot be improved so we are saddled with long haired geeks and arts graduates designing things to sell the car.

      There is simply no way that many of the current vehicles ever stand a chance of being "vintage" as so much is based on tech that has a lifespan of 5 years if you are lucky. It is consumerism gone mad.

      It is bad enough when a £1000 mobile is obsolete after a 5 years or cannot be repaired by £50k cars?

      WTF

      1. MachDiamond Silver badge

        Re: I blame...

        "so much is based on tech that has a lifespan of 5 years if you are lucky."

        The displays may be a big weak link. 5 years down the road when the warranty runs out (battery and drive train are all that get the long warranty), will the LCD still be manufactured? Will the whole module have to be replaced to support a new LCD?

        Tesla motor packs out of warranty run about $6k. A Honda or Toyota 4cyl used engine is about $1,500 installed. It's important to know what repairs might cost down the road if you hang on to your cars for a long time or are looking to buy used. If I could find a used Honda in good condition but having transmission problems really cheap, it would be totally worth buying. A Tesla Model S with bearing noise coming from the motor pack could mean more money than I'd want to spend on repairs.

  17. FlamingDeath Silver badge

    Made by humans

    Not corporations, shareholders or stakeholders

    Next time you think of a Tesla car or Boeing aeroplane, dont think Tesla, or Boeing etc. Think people

    These corporations have a lot of hot fucking air to blow at their stakeholders, and take all the credit, quite fucking literally

  18. FlamingDeath Silver badge

    Once upon a time

    Cities built aqua ducts without sewage systems

    True story

    They had their heads so far up their own arses, they thought chucking filthy waste water into the street was the answer

    Humans, fucking muppets

  19. Grease Monkey Silver badge

    Always a great idea to build critical bits of the automotive control systems into what's basically a cheap tablet.

    You wouldn't so much mind if Tesla were some cheap and nasty Malaysian brand, but they want us to believe they are up with the best in the world and they're using the same tech as those cheap chinese phones that last a couple of years max before they start to die horribly and slowly.

    Did they really not see this coming?

  20. Grease Monkey Silver badge

    Always a great idea to build critical parts of the automotive control system into what's basically a cheap tablet.

    You wouldn't mind so much if they were a cheap Malaysian brand, but they want us to believe that they are up there with the best in the world.

    I can't believe they didn't see this coming.

    1. MachDiamond Silver badge

      "Always a great idea to build critical parts of the automotive control system into what's basically a cheap tablet."

      Yes, but it breadboards out just fine. It's just that they didn't take it to the next step by simulating what the conditions are in the real world. Nobody thought to go through the limitations of the component parts as part of a lifecycle analysis. Some of that might be down to the department being under pressure to deliver and having huge turnover so individual projects are frequently handed off.

      I'm working on a project right now with off the shelf parts from the DIY store and Chinesium module purchased off of eBay. It's just a proof of concept and I hope to have a rig fielded for a real test in mid-December under actual conditions. Nothing with life or property in the balance. What I'm building now is not what's going to wind up in the final product. I'll be sitting down and doing an optimized version with parts from well established suppliers once I'm satisfied with how it works. If things go well enough, I might even have about half of it constructed with beta circuit boards before the big test. I've done plenty of projects where the first go was something of a bodge. It's a cheap and quick way to sort out all of the assumptions I didn't know I was making before spending real money. I would never just glue an iPad to something attached with an eBay cordset and call it ready for production.

  21. Man inna barrel

    Entertainment and vehicle info share NVRAM?

    Others have mentioned this, but I think it bad that vehicle sensors etc. could malfunction due to NVRAM wearing out in the entertainment system. I have heard of something like this before, where entertainment and vehicle sensors and control share a vehicle-wide bus. The entertainment system then provides a point of entry for various hacks.

    I presume this is all about saving costs, as it is cheaper to make one processor do multiple tasks, rather than have separate processing centers for different tasks. The common communication bus saves on vehicle wiring. A single LCD can display infotainment data and vehicle data (don't know if Tesla do that).

    As far as processors are concerned, 32 bit microcontrollers are cheap these days. We are talking about a few quid in modest quantities. A custom LCD is more costly. The shared bus could be fairly secure, if everything connected to it treats it as an untrusted resource, like the Internet. Just speculating here.

  22. Chronos
    WTF?

    EMMC wearing out?

    What the giddy frig are they storing? GPS tracks? Or is it that there's a software upgrade every time some intern at Tesla moves their mouse?

    1. MachDiamond Silver badge

      Re: EMMC wearing out?

      Lots and lots of log files from all sorts of stuff that happens while the car is on.

      If they are just going from 8gb to 64gb and not fixing the underlying problem, they are just kicking the can down the road so the next round of fixes will be long past the warranty period.

      A judge just upheld an odd defense from GM over cracked Corvette wheels. The warranty covers defects in materials and workmanship, but not fundamental design flaws. The cracking is down to a poor design but manufactured correctly so no warranty on them.

  23. jonnycando
    Facepalm

    So a car with the price of a Tesla

    ...is put together with the same memory that goes in cheap laptops.....somehow...I don't want one...

    1. Chronos
      Thumb Up

      Re: So a car with the price of a Tesla

      It's an 8Gb(it) NAND flash, so the same memory used in, e.g. 2000's mass-produced landfill Android tablets. At least when those crash, nobody dies.

  24. NelliBalzer

    Hi....the unreasonable logging quicken their demises. RichRebuild effectively called attention to this like 2-3 years prior. There are outfits represented considerable authority in fixing this for out of guarantee Teslas.

    Other OEMs that began to have more NANDs in their vehicles will in the end have similar issue if their developers are not cautious by they way they log occasions. Tesla is more awful on the grounds that they attempted to gather all of data for their R&D/FSD reason. What's more, the guard camera recording additionally quicken the passing.

    Furthermore, in the event that they duplicate Apple and Samsung and begin to battle option to fix, it's will be difficult to keep old vehicles out and about.

    We are gradually moving to a future where you don't possess anything in any case, you are just paying for the option to utilize it.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like