Brit Watchkeeper drone fell in the sea because blocked sensor made algorithms flip out A British Army Watchkeeper drone stalled itself and crashed into the sea on a bad weather flight test, military investigators have said – though most of the wreckage was never found. The unmanned aircraft, tail number WK042, fell from the sky in February 2017 while trialling a new ice detection system. The drone was being …
Not sure that GPS guidance on drones is flavour of the month, after the Iranians nicked one and the Russians have a habit of jamming the signal...
Still, my mobile phone has a gyroscope that should allow it to stay stable, but even so, with no remote control, how in Satan’s Glorious Name did the designers hope to land one of these if it can be confused enough misjudge it’s own airspeed and stall into oblivion? Are Thales making some extra cash making fail videos?
You'd think that people would think about the problem...
First GPS... unless you're using the military precision signals along with a good connection to a few of the sats, along with a really good clock on the plane itself... you're not going to be accurate enough to know your specific position. (You can add a radio signal from a known ground point to also help too.) Commercial GPS is only accurate within 1.5meters at best. This is in part due to the maps, but also the clock signals and the quality of the clock in the GPS device itself. Surveyors that get down to 1cm precision have to set up and keep the units still for 24 hours before using because even throughout the day, the sun and atmospheric conditions can impact the signals.
But I digress, the issue isn't position, but one of flight controls... airspeed, altitude, attitude (AOA) etc ...
So GPS will have nothing to do with it.
I guess you could put some accelerometers ?sp? throughout the plane to help identify some information, but not all of it. So if you have a sensor failure... you are SOL and will lose the aircraft. This is where flight control systems need to be improved.
While I am an engineer, I am not an aeronautical engineer so I don't know all of the stresses and issues with flight systems.
I have to wonder about adding redundancy to the sensors to help and to also think about other things that they could do to measure AoA (pitch) etc ...
... it could never happen on a commercial aircraft!
I'd hate to think a new Boeing, say, could get into a state in which it behaved erratically and crashed just because it couldn't get an accurate Angle of Attack reading. That would be really dangerous.
Fortunately passenger airlines are supposed to be built to higher standards, with multiply redundant systems, and so on, so they are bound to be safe ... right?
Let me clarify, because people seem to think I'm wrong.
It is indeed never too soon to discuss correct design and criticise failures.
But right now is too soon to be making flippant jokes about an aircraft crash that claimed hundreds of lives only a month ago.
There is an obvious comparison to be drawn between the two events, and I'm right with you in making it, but pleae just be a bit more careful about making your point by turning it into a snarky joke, because a month after the Ethiopian crash, that comes across as being in poor taste.
With Air France flight 447 it was a problem with the pilots losing situational awareness rather than an error associated with a flight control algorithm. When the airspeed measurements became unreliable, the autopilot disengaged, but after the crew took over they stalled the aircraft. Nothing at all to do with dodgy firmware.
Air France Rio to Paris pitot tube blocked was flown into the Atlantic some years back and a Peruvian airline now defunkt had a intermittent problem with its altimeter and it too was flown into the Pacific. In both cases and apparently now with Boeing is that the GIGO rule has been forgotten and flight programs are light in such fault decoding.
One thing crossing my mind is how easy it must be to steal one of these if you are a foreign power. Knock it out of the sky, nobody trusts the readings and then collect the thing leaving just a panel.
But then considering how great these things seem to have been so far I cant see why someone would want to steal one.
Depends on how you consider them to be different. Both movies originated from the same screenplay written by Ian Fleming, Kevin McClory and Jack Whittingham before the 'official' James Bond movies started. Ian Fleming wrote the novel Thunderball based on that script without acknowledging his co-authors, which led to some legal haggling over the years, which resulted in McClory getting producer credit on Thunderball (movie), and retaining enough rights to make a Bond movie with that specific story later on (more or less).
Iran did this with a US Sentinel, except instead of faking a crash they made it land safely and were all "nahaha we've got your drone we're going to copy it" and then they copied it.
Then they gave a panel of the copy to Netanyahu and he was all "nahaha we've got your drone" and presumably set about copying it.
I think this process might be how the machines will 'self-replicate' in future?
Surveillance drones? Meh.
Well ok, it depends on which one. What the Iranians got isn't that terribly advanced. You don't need them to be that advanced in most situations.
Now China got a lot of advanced tech, and they didn't even need to leave their cubicles to get it. But that's a different story.
Not true, Watchkeeper is supposed to be an all-weather version of the Hermes 450. Hence pitot heaters, a redesigned icing resistant wing etc. In this case as far as I could make out, although the aircraft was obviously in icing conditions the operators didn't select the heating on as there was no indication from the aircraft, despite all the warning signs being there.
Incidentally it was being flown by Thales, the manufacturer rather than the Army, so I'd blame them for this one.
Update having skimmed the report, it appears due to occasional issues with water ingress to the pitot system the aircraft are always operated with the pitot heaters on, including in this case. Plus they purge the system as part of the maintenance. What they didn't do was activate the wing de-icing but examination of the air frame with the camera pod didn't show any icing.
Two different designs of pitot are used, but the conclusion seems to be that moisture accumulated in both during the flight leading to errors in speed as altitude changes. It could also affect Angle of Attack and Angle of Slip measurement as they're done via the pitot system on Watchkeeper, rather than a rotating vane as on most airliners, including the 737.
Pitots are pretty much static solid state devices. It's just that there is a tube leading from the pitot tube to the pressure transducer. You don't want to put that transducer right in the airflow because of the icing and moisture problems (Temperature swings would also be very difficult to compensate for). That leaves you with a pitot probe that CAN have icing and water ingress problems. The fact they have so much problems when it is a pretty much solved problem in aviation (the problem is well understood and researched) is beyond me though. To me it indicated serious incompetence or willful ignorance. (Both unfortunately run rampant in government contracts in my limited experience)
"I think the use case for these drones has always been assumed to "somewhere dry, hot and dusty" so the wet and icy weather to be found off the coast of Wales was never in the design spec, so no heated pitots."
You need to watch the Long Johns (Johns Bird and Fortune) interview on Bremner, Bird and Fortune around the time of the Gulf War where John Bird is being interviewed as a UK army general who continually talks up the capabilities of the army equipment before explaining that it was intended for use in a war in northern europe and won't work in sand/heat/dry conditions in Iraq ... he ends with the suggestion that they write to Sadaam Hussein and suggest that as he clearly wants a war why doesn't he come over to northern europe to fight it.
From the article: The unmanned aircraft, tail number WK042, fell from the sky in February 2017 while trialling a new ice detection system.
From John H Woods' post: wouldn't one expect these to be prone to freezing?
In a word... yes, which is why pitot tubes have heaters to prevent icing; I am less certain about the accompanying static vents but ISTR an aircraft falling out of the sky because a static vent was blocked by ice.
I find myself wondering if (functioning!) heated pitot tubes were specified; if part of the testing procedure involved flying around looking for icing then having unheated tubes would seem negligent beyond belief.
Perhaps it really was a software problem, but applying Occam's Razor points to the possibility of an altogether simpler failure in what is, furthermore, a known failure mode.
I wonder if "Pitot Heaters On" (if the heaters actually exist and are supposed to work) is a function buried in all the other data the drone is feeding back.
Trouble is that even if the wreckage had been found there would have been no physical evidence that a frozen pitot or static vent had caused the crash.
This post has been deleted by its author
One of my friends that works at a large US DoD company - his job is basically asking "what if [sensor] takes a shit?" and sitting back to see all the deer-in-a-headlight stares.
"oh yeah if that sensor stops working the computer will ignore it"
"how does the computer know it's stopped working/faulty/incorrect?"
"er. uh. um. well. I will have to get back to you on that"
Netlfix goes one further, it invented "chaos monkey" - an app that randomly shuts down servers, kills process, drops network connections, corrupts packets etc - just to test what would happen.
If your bit of the system doesn't degrade gracefully - you fix it.
Of course the engineering time and expense necessary to ensure that Game of Thrones isn't interrupted wouldn't be appropriate to building an aircraft
Or even one sensor so long as you can detect when it fails.
Don't write code which performs perfectly well when the sensor outputs a normal range of values but when a millisecond later it sends you 0xFFFF and you just decide that this means the plane is currently flying upside down and so your autopilot needs to do a barrel roll in a 737.
Yes, this is a start. But, as was discovered on the US Space Shuttle program, even different manufacturers using different parts will have systems performing the same function designed in broadly the same way with potentially the same logical flaws. The answer of course is to farm out at least one of your triplex redundant systems to a non-human entity for design and manufacture. Of course in the absence of an acknowledged ETI presence maybe us humans could try AI?
"Install THREE sensors for every required measurement. Read all three and believe two of them that have a similar report."
Standard operating practice in commercial airliners. Usually a good idea, but sometimes it's not enough on its own, e.g. AF447 crashed because other Bad Things happened even with three sensors of two dissimilar desgns, before and after the identical two of its three pitot tubes failed identically at the same time (as had occurred on other aircraft on previous occassions in similar conditions, fortunately with less drastic consequences, but the implications had been recognised sufficiently for modification programmes to be required). It's well documented in various places.
When that happens, the two failed sensors can outvote the one which is behaving sensibly, but sensible flight crew may be able to either reduce the risk of it happening (don't fly through icing conditions) or recognise and resolve the issue if two did fail identically..
On the AF447 flight in question, the two identical pitot failures combined with various other unrelated failures (e.g. flying through icing conditions rather than around, less than ideal reaction from the crew, etc) led to the loss of all on board.
My (diving) rebreather has three O2 sensors.
Prior to a dive you calibrate the unit. During the calibration, and during the dive, it uses an algorithm (closely guarded secret, of course) to determine if any one of them is faulty and ignores it.
Of course, it doesn't rule out two could be faulty but that's why you watch the calibration process and rotate the cells out at different times, so they're not all three from the same batch.
One of my friends that works at a large US DoD company - his job is basically asking "what if [sensor] takes a shit?" and sitting back to see all the deer-in-a-headlight stares.
I've watched too many "aircrash investigation" episodes (well, mostly partial episodes as my wife and son seem to be hooked on them and have to occasionally endure it!) and invariably in the program as the invesitgators seem totally puzzled by what caused the crash there's a point(*) where one of the actors will suddenly pause appearing to have had a "lightbulb moment" say something like "bu what if the sensor stopped working", rush to a whiteboard where he'll draw a line that shows the plane flying along then plummeting to the ground while all the others watch and start to murmer in a "why didn't we think of this earlier" sort of way.
(*) a similar inevitable point to the one in Grand Designs episodes where Kevin McCloud puts on his most serious voice to say "today is the day when the steel work arrives - but are the plans accurate as if the steel doesn't fit there's no chance the buildiong will be complete before Christmas/the baby/end of lease on current house arrives.
Given the drones seem to fare badly in rain, wind and ice, what is the point of them in British Weather?
Not much use if they can only function on warm, dry, still days.
All the enemy has to do is wait for some (all too frequent) crap British weather and be safe in the knowledge the drones are helpless
All the enemy has to do is wait for some (all too frequent) crap British weather and be safe in the knowledge the drones are helpless
Waiting for a moment when you can expect little or no resistance is a common tactic in warfare, Has been so at least since the Romans, who simply waited for the Brits to take their tea break.
Can we test our software properly before we let control things that fly over people, put people inside things it controls, put it inside always-connected devices in our houses, let it add up our money, etc... instead of chopping development time by half, using agile as if it were a development methodology instead of a way ticking off items on a list, cutting back QA, and doing everything else which means marketing/upper management get their bonuses quicker so they can snort it away sooner?
Every time this happens, the only answer is to develop better code and test it properly, which requires more time.
the response will be to add another set of meetings to sign off on another requirements doc to ensure that an appendix is issued to all airlines - with a rigorous enforcement regime to ensure it is properly filled in the back of each operation manual 3 ring binder.
Remember if it's documented it isn't a bug it's a feature.
"Can we test our software properly"?
Costs money, you know, even if or especially if you are employing engineer-qualified programmers rather than the cheapest coder off the street. MOD/Treasury are always insisting on the cheapest contracts, so when testing time arrives most of the dosh is already spent.
The short answer is no, we apparently can't write good code anymore. No one gives a shit if their code works.
Look at all of the crap code we have to deal with everyday like Microsoft patches that blue-screen your PC. I think I/we had some idea that aerospace software was somehow different. That when human lives were involved, that the people writing code took a little more care with the code that they write? Maybe in the past this was the case?
Somehow all of this has changed. People coming out of school now are taught that if it compiles, it's good to go. I think the younger generations are just terminally lazy. Now everything is just "good enough". The "move fast and break things" mentality is actually killing people.
We have been working with an outside web development company run by a couple of young hipsters, and most of their work is very sloppy. No attention to detail. I keep telling one of the VPs that we should fire their ass. He's not sure a replacement would be any better. All we would do is set the project back, and end up with someone else that writes crap code. He's probably right?
I wonder if any serious research is going into a successor to pitot probes the design dates back to the 1730s and seems to have a long and sordid relationship with air disasters in general :
That they never actually built any of the drones, they just mashed a few Airfix kits together, removed the strings in MS-Paint and pocketed the money
Now the auditors are coming they have to explain where all the hardware is.
Investigators concluded that one of its pitot probes used for reading the aircraft's speed and angle of attack (AOA) became blocked, causing the Watchkeeper's onboard flight control logic to enter an erratic series of climbs and dives until it stalled itself and flopped into the sea.
I'm pretty sure that's Boeing's IP. I hope Thales paid royalties.
Finally! A task for machine learning that isn't creepy.
They should program an a.i. to analyse flight data over tens of thousands of aircraft to build a model of what "normal" looks like, and use that to help handle scenarios when the instruments are malfunctioning.
For example, some combinations of say airspeed and climb rate simply wouldn't hold up, so if one said you were slowing down, and the other said you were diving, then you can kick in a default of "fly level" or "check other instruments"
Perhaps because the full-time datalink back to the ground station provides sufficient realtime system monitoring to make adding a black box an undesirable weight/cost penalty vs the benefits it would offer for incident analysis if a WK crashes after the datalink has failed?
Perhaps because under normal operating conditions, if a WK is lost, there's a good chance it'll happen somewhere over hostile territory where retrieving the black box may not be possible?
Perhaps because, unlike civil airliners where there's a very real desire to learn as much as possible from any incident in order to reduce the risk of it happening again, if a WK decides to fall out of the sky for no good reason, TPTB will just shrug their shoulders and remind themselves to +1 the quantity of WK's in next years "spares and replacements" shopping basket?
I really dont get it, The army have a pilotless drone, the airforce fly things with pilots, including drones, why isnt the Watchkeeper a RAF project? They already hacve a relationship with Thales who make RAF simulators. Why do the Army have helicopters and drones, I know the RAF have thier own troops, maybe the army dont do flying or protecting airfields well enough. How about me employing painters and decorators to cable our installations and bankers to install servers?
Why do the Army have boats? Why does the RN have aircraft? Why does the RAF have ground vehicles? You wouldn't employ a banker to install a server, but what about a server specialist from the bank IT department who's doing some freelance work on the side? Army and Navy pilots might not wear RAF uniforms, but make no mistake, they *are* fully qualified pilots who've earned their wings and the right to fly multi-million pound military aircraft.
Historically, worldwide, different military services have different priorities and viewpoints.
Either you have a unitary military structure, like the PLA, responsible for everything, or you have multiple services that want very different things.
Classic example - the US air force doesn't like subsonic, low altitude close support aircraft... which is why they have been trying to retire the A10 for decades, so they can buy more shiny supersonic fighters that can do more 'air forcey' things than close support.
The US army does not have fixed wing aircraft because the US decided to give those all to the Air Force, who wanted a monopoly on that budget and function, save for the ones that fly off ships or spend all their time at low speed and altitude looking for submarines.
The US army doesn't trust air force priorities for aircraft design, numbers, and training, but can't have their own fixed wing support aircraft, hence the large number of attack helicopters in army units. Compared to close support fixed wing aircraft they are slow, vulnerable, overly complex, and carry small weapons loads... but they will be built, and they will do what the army considers important. If the army could have their own A10s, they'd love to have them instead of gunships.
Similar things tend to happen within services between branches. I expect that in the RN, submarine captains want more underwater boats, while surface captains want more surface warships to command - IIRC this is important for their chance of promotion.
In the US navy the naval aviators have huge influence, hence large numbers (10, more if they can get them) of very expensive carrier battle groups built around huge nuclear powered carriers, each carrying 100+ aircraft. The nuclear sub crew are also influential, and the blue water surface captains have little interest in ships designed for close to shore missions, which they consider a waste of money better spent on more impressive deep ocean combatants.
Air forces tend to pay more attention to air superiority, strike, recon, and bombers than to maritime patrol, anti-sub, or close support roles.
It also shows up in role differences. The strategic deterrence guys want long range nukes, the tactical guys want more support, strike, and air defence units...
Occasionally someone forces common weapons and acquisition on these disparate groups, and you get mediocre or compromised equipment like the F35, where design changes to fit three or four very different roles reduced the capability in all of them, while driving up costs and development times.
The US can probably live better with this than most prospective users as they also have several dedicated air superiority designs, a couple of good to excellent close support models, dedicated EW planes, strategic stealth aircraft big enough to actually resist VHF radar detection, and so on.
Unfortunate services trying to do all this with one aircraft type that happens to be partially stealthy in some directions to some types of detection as long as it limits its load to a few internally carried weapons, and which can perform adequately as long as it doesn't have to go fast, or maneuver, or try to shoot at something with its gun, or operate without massive computer support at its base, or fly more than one mission every three days per plane... might wish they had something trying less hard to be all things to all people.
That's why each service wants its own custom designed toys under their control and planning structure.
This is just another case like the Apache. The government gets to announce that they are bringing X percent of the money to UK companies by developing our UK version of the Apache software and buying the aircraft from the USA. Then what happens, they discover the software is actually more expensive to make per aircraft than the hardware.
In this case, they decided to bring Drone development to UK industry, and to save money some accountant opted not to have the ability of the manual remote flight control. Any engineer can tell you that a real flight trial is going to have an issue of some sort, and autopilot is going to go wrong some time!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
