back to article 'Incorrect software parameter' sends Formula E's Edoardo Mortara to hospital: Brakes' fail-safe system failed

Swiss Formula E driver Edoardo Mortara ended up in hospital after a software error left him driving into a safety wall at the ABB FIA Formula E World Championship in Diriyah, Saudi Arabia, on Saturday. In a statement issued via Twitter, the Mercedes-EQ Formula E Team blamed the code governing the race car's braking system. " …

  1. tip pc Silver badge
    Black Helicopters

    scary about the car not braking and the missile interception

    how often does the missile intercept happen and where did the missile come from?

    1. Anonymous Coward
      Big Brother

      Re: scary about the car not braking and the missile interception

      > how often does the missile intercept happen

      As often as the government's popularity polls start to show that the populace might be feeling insufficiently grateful for the important work the government does protecting them.

      1. Anonymous Coward
        Anonymous Coward

        Re: scary about the car not braking and the missile interception

        Since when did Saudi rulers care about popularity polls?

  2. stratcat

    Fail safe systems...

    I can't remember the source of this quote, but it generally holds true...

    "Fail safe systems always fail by failing to fail safe"

    1. David 132 Silver badge

      Re: Fail safe systems...

      "In theory, there's no difference between theory and practice.

      In practice, there is."

      1. Electronics'R'Us
        Holmes

        Re: Fail safe systems...

        My response to that has been (for many years now):

        If the theory and practice do not match, then the theory is incomplete.

        1. Evil Auditor Silver badge
          Coat

          Re: Fail safe systems...

          If theory and practice to not match, then the practice must be changed.

          Then again, I'm an auditor, I don't do practice.

          1. Trigun
            Joke

            Re: Fail safe systems...

            "Then again, I'm an auditor"

            From the Laundry? Scarey! Must make sure my paperclips are in order and accounted for!

    2. Empty1

      Re: Fail safe systems...

      We used to get more downtime due to UPS failures than power outages.

    3. Cuddles

      Re: Fail safe systems...

      It's a nice soundbite, but more common is fail safe systems failing because they're not actually fail safe at all. The point of a fail safe is that it fails into a safe state. The common example being fire doors held open by an electromagnet. You can close the door manually or by pushing a button or whatever, but if everything goes horribly wrong and power is lost entirely, the door simply swings closed under its own weight. The point is that in the event of a failure, nothing should need to be actively done; rather, you need to actively prevent it from entering the safe state while things are working. Obviously there can always be other failure modes that can't be safely handled - a self-closing fire door won't work if an earthquake distorts the building structure - but the failures your system is designed to handle should be handled with no active intervention required.

      If we look at this case, it's clear that there was no fail safe involved at all. If the front brakes fail, the system needs to detect the problem and actively apply the rear brakes. That's not failing into a safe state. A fail safe brake system would mean something like actively holding the brakes away from the wheels, so in the event of a failure they drop into place and stop the car. But since randomly slamming the brakes on could be just as dangerous as not having any brakes, I don't think the concept of fail safe even makes sense to think about when it comes to braking systems in racing cars.

      1. Tempest 3K

        Re: Fail safe systems...

        'A fail safe brake system would mean something like actively holding the brakes away from the wheels, so in the event of a failure they drop into place and stop the car.'

        You've just described the exact system used on the railway - Air is used to force the brakes open; A brake application or failure (i.e. damaged pipe) causes a loss of air and the brakes apply automatically.

        1. John Robson Silver badge

          Re: Fail safe systems...

          And many HGV braking systems as well...

          In the case of a loss of service air from the cab, the whole rig will stop, and following traffic should be able to deal with that.

          On a race track the following vehicles aren't expecting random braking, just braking for corners, that's why F1 has flashing red lights not for braking, but for "This car has suddenly lost 160bhp".

          There is no way a racing driver could react to brake lights.

          1. Anonymous Coward
            Anonymous Coward

            Re: Fail safe systems...

            > In the case of a loss of service air from the cab, the whole rig will stop, and following traffic should be able to deal with that

            And might as well mention, a lightly loaded lorry or bus has significantly higher braking power than most cars. Keep your distance!

            I did drive HGV + PCV for a while. In non-emergency vehicles we are trained to brake gently for pax comfort / load safety. What you see on a regular day is *not* what the machine can actually do. As my colleagues from the command vehicle in the fire service found out when following our new truck a bit too closely. :)

      2. Yet Another Anonymous coward Silver badge

        Re: Fail safe systems...

        In this case it looks very similar to the A400 transport plane crash.

        Config file for the brakes bad/missing so the failsafe disables the system = in this case the system being brakes.

        In the transport plane the config file for the engine was missing so the failsafe system disabled the system safely = in that case all 4 engines

        1. Anonymous Coward
          Anonymous Coward

          Re: Fail safe systems...

          3 out of 4 engines were in this state, which was only reached through exactly the sequence of power lever movements you would expect when an engine quits and won't feather thus giving a massively draggy disc which the one good engine couldn't overcome.

        2. Anonymous Coward
          Anonymous Coward

          Re: Fail safe systems...

          Care to throw in a reference to the relevant accident report?

  3. TonyK

    The stewards in that video looked a bit nonchalant to me. Is that because they were shiftless ne'er-do-wells, or is it just because E-cars don't tend to burst into flames like proper cars do?

    1. Anonymous Coward
      Anonymous Coward

      Perhaps they were waiting for the man with the rubber gloves to arrive and disconnect the battery?

    2. Ol'Peculier

      I think Richard Hammond may disagree with you there...

      https://www.youtube.com/watch?v=j9W8N24CCCI

  4. captain semtex
    Facepalm

    Just because...

    "In this instance, an incorrect software parameter that meant the rear brake system didn't activate as intended and the fail-safe did not kick in."

    You have to wonder why such a parameter even exists?! It should be impossible for a fail-safe to be disabled, otherwise it's not a fail safe!!

    1. big_D

      Re: Just because...

      Maybe the programmers should be forced to drive at a brick wall and test their own code, before it is let loose on the "end users"...

    2. Anonymous Coward Silver badge
      Big Brother

      Re: Just because...

      Because it needs to know what constitutes a failure. Presumably it's sensing how much the front wheels are slowing down relative to braking pressure, and that probably varies depending on choice of tyres and racing surface etc.

  5. Potemkine! Silver badge
    WTF?

    So it's the fault of the software... I was afraid a human was involved at some point.

    1. Paul Crawford Silver badge

      It is the result of a human mistake.

      That mistake was to assume software is reliable enough. Where is the back-up hardware or hydraulics that can be 100% tested?

  6. David M

    A failure of testing

    The fault was primarily in the tests. Software always has bugs, and the tests are there to catch them. The more critical the software, the more comprehensive the testing should be. So the most important issue right now isn't finding the bug, it's working out how it got past the tests.

    1. big_D

      Re: A failure of testing

      Maybe if the programmers were forced to test their own code, in-situ, they'd be a little more careful...

      I never worked on automotive systems, but I did work on nuclear reactors, traffic systems and manufacturing systems and the code was always thoroughly tested before release. We'd probably spent about 2/3 of project testing, before it was released into production - and generally, you did your own unit testing, but system and integration testing had a whole dedicated team.

      1. David M

        Re: A failure of testing

        Having a dedicated testing person or team is really important. For one thing a programmer testing their own code will only test that it does what they think it should do, which doesn't help if they've misunderstood the specification, or missed part of it. But more importantly, test engineering is a whole different mindset - the best programmers often aren't the best test engineers, and vice-versa.

        1. big_D

          Re: A failure of testing

          That was one of the most important things that was drummed into us in college and at my first employer.

          You test what you expect to happen. Therefore you take the specification and the first thing you do is write your test cases based on the specification. Once you have your test cases finished, you can start with the programming. That doesn't make them perfect, but it means they are based on the specification and not the code you have written. That picks up a lot of common errors that would otherwise have been missed.

          Unfortunately, many projects I've worked on recently have not had proper specifications and testing was cut to the bone and you had to turn out code first and write test scripts and test cases if you had time afterwards. Time scales were not realistic and the budget for the coding were much too low to do the job properly. But, hey, they are only internet facing shopping and payment systems, so doing it properly and safely isn't really necessary, after all, what could possibly go wrong?

          My "best" project (as in closest to being perfectly executed) was a VisualBasic 6 system written for a large agrochemical company to collect its world-wide budgeting and sales forecasts and pushing the data into Hyperion Essbase.

          The specification was pretty comprehensive and included use cases and all the business rule errors that could occur. The unit tests and system tests were written based on the spec, then we got around to coding. We had around 75,000 lines of code, when it went live. 3 programmers, 4 months of coding and testing. The system was released world-wide to over 60 national subsidiaries.

          In 2 years, we had 2 bug reports. The first was a bug in Windows 98 International English edition*. The other was a simple problem with cell editing in a grid container. That is the only system I've worked on that had so few bugs reported. A shame other projects didn't run so smoothly - although I was generally assigned to projects that had already gone pear-shaped and had to help bring them back on track.

          * The bug in W98 International English edition was the Win32 API for getting the localized month names. We tested it in UK English, German and French. All worked flawlessly. But a lot of Asian and South American countries were using International English editions. You gave the API function the month number and it returned the name of the month (E.g. 1 = January (EN), Janvier (FR) and Januar (DE), 2=February, Février, Februar etc.). Only in International English 1=January, 2=January, 3=January, 4=January, 5=January...12=January. In the end, HQ decided that we would just hard-code the English month names for everybody, as English was the corporate language, so we didn't have to be "nice" to the users.

          1. yetanotheraoc Silver badge

            Re: A failure of testing

            I'm maths challenged. Help me out.

            "We had around 75,000 lines of code, when it went live. 3 programmers, 4 months of coding and testing."

            At 22 days/month, that's 284 lines of code per developer per day, not counting testing. At 30 days/month, it's 208 lines. It seems a *little* high.

            1. big_D

              Re: A failure of testing

              VB auto-generates a lot of code for the form layout.

              And well commented code, with templates for file, class and function/property/method level comments.

    2. Sam not the Viking Silver badge

      Re: A failure of testing

      Richard Feynman's annex to the Challenger Report is always a good read for anyone involved in system reliability. His section on 'Avionics' is, in my opinion, outstanding. It's relevance to software-safety is exemplary.

      https://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt

      1. Yet Another Anonymous coward Silver badge

        Re: A failure of testing

        In the shuttle case it would seem to be bad project management at the top.

        If you spend all your budget making insanely safe software at 1000x the cost of normal software you haven't really benefitted anyone.

        The shuttle was still dangerous because of solid fuel boosters and nobody in industry benefits because none of the lessons of wiring one line of code/person/year with nines-nines safety were usable in the world.

        1. Anonymous Coward
          Anonymous Coward

          Re: A failure of testing

          You know you're in trouble when, in a critical engineering meeting, you are asked to "remove your engineering hat and replace it with your management hat". This is exactly what happened at Morton-Thiokol.

  7. Blackjack Silver badge

    Evidently

    It didn't fail in a safe way.

    One would think coding in racecars would cause braking in case of errors instead of not.

    Just like you know, airplanes should listen to manual overrides no matter what their software is saying.

  8. hoola Silver badge

    Looking to the future

    A lot is made of the "Trickle Down Effect" of technology from racing and this is something that is pushed particularly hard in Formula-E.

    If this is the sort of "improvements" we can see trickling down into standard cars the outlook is not particularly good. This stinks of Boeing type QA.

    1. Intractable Potsherd

      Re: Looking to the future

      This feels like re-inventing the... braking system. Automotive brakes have been effectively fool-proof for decades. Dual-circuit hydraulic brakes virtually never failed, and the most common failure mode was visually obvious (a puddle under a wheel) or tactile (the brakes didn't feel "right"). Then ABS came along, and brake failure went through the roof (or, more accurately, brake sensor failure that didn't actually affect braking performance but lit up a warning on the dash that then needed £££ to fix went through the roof). Now fly-by-wire brakes that can fail with no warning at all. This is not a good trajectory. Let's go back to simple dual-circuit hydraulics that work.

      1. Yet Another Anonymous coward Silver badge

        Re: Looking to the future

        >Let's go back to simple dual-circuit hydraulics that work.

        But not use them in 1000bhp cars doing 200mph around a tight race course with other cars a few inches away.

  9. stungebag

    I look foreward to the Who, Me?

  10. Sgt_Oddball
    Holmes

    This would imply...

    That the brakes are completely fly-by-wire with no direct actuation from the drivers foot. That seems a tad short-sighted to me as even if the brake booster failed, a car normally can still be stopped by simply jamming your foot down harder (as people are want to do in an emergency).

    1. Dave314159ggggdffsdds Silver badge

      Re: This would imply...

      They are, yes. These are high-end electric race cars, to which regenerative braking is very important, and they're incredibly safe, not very fast, and running on safe circuits, so they've made trade-offs you wouldn't make in a road car.

      1. Anonymous Coward
        Anonymous Coward

        Re: This would imply...

        "and they're incredibly safe"

        Except when the braking software fails and they headbutt a wall. But yes, apart from that, incredibly!

        Seriously, using regen braking is no excuse for not having some kind of mechanical fall back system. Road cars manage it. While there is a human in the cockpit system safety critical systems should not be drive by wire unless there is no option, eg ABS.

        1. Dave314159ggggdffsdds Silver badge

          Re: This would imply...

          "Except when the braking software fails and they headbutt a wall. But yes, apart from that, incredibly!"

          Including, even especially when the brakes fail - which more commonly happens for other reasons. These cars (and track safety systems) are designed to let drivers walk away from high speed impacts. They can therefore accept risks of things like brake failure that are utterly unacceptable in another situation.

          Having a mechanical fall-back system is pointless if there's no chance anyone could activate it. (It's even more pointless if the brake failure is mechanical, which is far more common; you can yank a mechanical e-brake all you like, it won't do anything if your brake discs fell apart.)

          Perhaps you haven't considered that the nature of racing means that drivers are _always_ attempting to extract the maximum performance by braking as late as possible and then hitting the brakes as hard as possible without locking the wheels. If the brakes fail under braking, even ignoring reaction time it's still too late to take alternative action. On a lot of tracks momentarily locking the inside front wheel under braking can lead to hitting the barriers, let alone losing braking entirely.

          1. Yet Another Anonymous coward Silver badge

            Re: This would imply...

            >It's even more pointless if the brake failure is mechanical,

            I was thinking of a cartoon style anchor

          2. Anonymous Coward
            Anonymous Coward

            Re: This would imply...

            "These cars (and track safety systems) are designed to let drivers walk away from high speed impacts"

            Thats what everyone thought just before Senna died.

            "They can therefore accept risks of things like brake failure that are utterly unacceptable in another situation."

            I suspect if you asked a racing driver whether they accept brake failure as an occupational hazard the answer would be a resounding no. They take risks but they expect their safety equipment to work - any failure should be on their part, not the machines! Using your logic you could say "Hey, so he got badly burnt, he shouldn't expect Nomex to work 100% of the time!"

            "Having a mechanical fall-back system is pointless if there's no chance anyone could activate it"

            God knows what thats supposed to mean.

            "Perhaps you haven't considered that the nature of racing"

            Haven't I? Oh ok, thanks for the heads up there Murray.

            Being patronising adds precisely nothing to your already flimsy argument.

        2. Brewster's Angle Grinder Silver badge

          "Except when the braking software fails and they headbutt a wall."

          You're equating "safe" with "not crashing". That's not what it means. "Safe" means that when the car headbutts the wall or becomes involved in any of those other "incidents" that can happen in motor racing, the driver gets out and walks away.

          1. Anonymous Coward
            Anonymous Coward

            Except they don't always - Senna being a case in point. Yes it was 25 years ago but I'm sure they thought open seat racing was really safe back then too. Anyone who says its 100% safe now is a shill or a fool.

            1. werdsmith Silver badge

              Nobody suggests it’s 100% safe, Jules Bianchi is a more recent tragedy.

              But it is amazing and beneficial to us all that so much engineering effort goes into racing car safety that an F1 car can be sliced in two as it pierces and passes through a metal barrier, burst into a fireball and the driver can step out and hop over a fence.

              Senna’s accident had an element of misfortune about it, that the high energy suspension component should impact where it did. And the Jules Bianchi incident involved a construction vehicle on the track boundary where there ought to have been, in normal circumstances, an energy absorbing barrier.

            2. Anonymous Coward
              Anonymous Coward

              Senna is a bad example

              They knew exactly how dangerous open-wheel (F1) racing was at the time when Senna died. Roland Ratzenberger died during qualifying the day before.

              Senna's and Ratzenberger's deaths were at the end of what Martin Brundle has described as F1's "light-bulb" years: if a driver died, they slotted another one in and carried on.

              At least two marshals have died at F1 race weekends since Senna as well as Bianchi (and test driver Maria de Villota, whose death was recorded as due to her injuries sustained in a crash during testing for Marussia). Anthoine Hubert was killed in an F2 race at Spa in 2019.

              1. werdsmith Silver badge

                Re: Senna is a bad example

                Maria de Villota's accident didn't happen on an F1 circuit, it was at Duxford Airfield. She had not been trained to drive the car properly and was unfamiliar with its anti-stall system.

  11. Grease Monkey Silver badge

    "The braking system on Formula E cars is designed so that if the front brakes fail, the rear brake system is activated as a fail-safe," the Mercedes-EQ Team said. "In this instance, an incorrect software parameter that meant the rear brake system didn't activate as intended and the fail-safe did not kick in."

    So software stopped the failsafe kicking in to apply the rear brakes? Jolly good. But I has questions:

    1. How safe is the failsafe? Try stopping a car from speed using the rear brakes only. Easy enough to do just whip the handbrake on. Doesn't do a very good job of stopping the car does it? So you're piling down towards one of those hairping bends formula E track designers are so keen on and at your braking point you discover your front brakes have failed. How likely is the "failsafe" to stop you from hitting the wall?

    2. Presumably the team are so keen to discuss the failure of the failsafe at length because that distracts from the total failure of the front brakes. So why did the front brakes fail so catastrophically?

  12. Eponymous Bastard

    1988 Airbus 320 . . .

    Planes crashed.

  13. yetanotheraoc Silver badge

    Testing in production

    Okay not properly tested, but if you change a software parameter, how exactly would you test that it still fails safely when the front brakes fail? Sounds like a destructive test to me, very expensive when the widget is a racecar.

    1. jtaylor

      Re: Testing in production

      how exactly would you test that it still fails safely when the front brakes fail?

      Big runoff area. And telemetry.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like