One man's mistake, missing backups and complete reboot: The tale of Europe's Galileo satellites going dark Key details about the failure of Europe's Galileo satellite system over the summer have started to emerge – and it's not pretty. While one key official has sought to blame a single individual for the system going dark, insiders warn that organizational chaos, excessive secrecy and some unusual self-regulation is as much to …
So you have the an alphabet soup of agencies who are key stakeholders, on which the GNSS Agency seems to have dependencies. An unnamed single person, who may just be a fall guy, who can walk in and introduce a creeping, debilitating defect in the Galileo satellites. No backup to restore the past, stable configuration for the Galileo system once the defect is introduced. The addition and creation of new stakeholder agencies and offices to an already complicated org chart. A satellite system that takes days to reboot, if you are compelled to seek that option. Then to top it off you have a culture of secrecy and defensiveness that hid and is probably still hiding what actually happened.
Its unfortunate that I can't buy stock in the GPS system.
It is worse than "authority without responsibility" as nobody is in a position of overall authority - it descends in to finger pointing and impotence because no one knows all that is happening, nor could they make any useful impact if they did.
As for the "single mistake" at fault that is utter pish! If something the size, complexity and cost of Galileo can be brought down by a fat-fingered mistype in some console then the upper layer of managers should be sacked for not having a proper system of tested procedures (e.g. a scripted system wide restart), verified backup/snapshots, and a test system where changes are (whenever practical) run through before live deployment.
If something the size, complexity and cost of Galileo can be brought down by a fat-fingered mistype in some console then the upper layer of managers should be sacked for not having a proper system of tested procedures.
That most likely explains it: they're a collection of PHBs and MBAs who don't understand what they're managing and so are incapable of organising either an effective backup system or timely recovery from a system crash. However, they will never admit this or permit the true story to be published because they know damn well they'll be blamed for allowing the situation to occur and will be sacked as a result.
In short, they seem very similar to the upper levels of the British Civil Service.
It's not just typical bureaucracy though. It's a classic example of the way Europe works for grand and not-so-grand projects. Projects have to be divvied up amongst various bits of Europe to keep everyone happy. In a less politically rivalrous entity, the dividing would more likely be between projects rather than within projects to retain some semblance of practical organisation. One would hope that the EU would 'learn lessons' but I am of the view that the politics won't let that happen. I don't believe that 'ever closer union', even if it is achieved, will improve matters either.
Yes, very similar to how it works in the USA. To get a Congress-critter to vote your way you have t make sure his/her state gets some of the pork. Hence bits of rockets and space shuttles being moved about the country to where they are needed for assembly.
Hence bits of rockets and space shuttles being moved about the country to where they are needed for assembly.
To be fair, SpaceX also move rockets from the factory in California to their test centre in Texas and eventually to Florida (usually, sometimes California) for launch and recovery (and then back to California for refurb).
Transport in itself is relatively cheap. The problem is when those rockets are being built from bits supplied by 20 major Congress-selected sub-contractors in the appropriate districts, in some cases using components selected by Congress (i.e. SLS being largely cobbled together from old Shuttle components by former Shuttle suppliers). Any change request has to pass a dozen-plus desks. Far slower than wandering across the single design office in Hawthorne to the relevant department, having a chat with the team(s) your change might impact and sorting it out in half an hour over a coffee.
That's how they get *built* not how they are managed and maintained once in orbit.
That comes down to exactly two agencies - NASA and [REDACTED] - the heads of which can be fired at any time for anything as massive a fuck-up as what happened to Galileo.
And the major U.S. DoD space contractors do back-channel with each other on certain things so it's not like they're out-of-the-loop if something goes awry.
I used to work for Raytheon and we'd regularly chat with our peers over in other defense companies, especially concerning satcoms issues. Nothing company-sensitive of course, but the channels are there to prevent things like this from happening to U.S.-based orbital assets.
@Paul Crawford "authority without responsibility", more than a little like Trump's White House and the people he has appointed, especially in the judiciary.
I thought I should point that out as the EU/ESA is taking a lot of, well deserved, flack; but, it is not a unique experience. A better example, in this case, might be NASA and their inability to produce rockets due to the pork required by the Congress and Senate to keep meaningful employment in certain geographic areas.
Already been suggested...
Siberian Hamster
Reply Icon
Re: Time enough
I have the inkling that there's a new internet law occurring...
As the length of a comments section increases the probability that someone states that the problem/issue is due to or can be fixed by Brexit tends to 1.
In deference to my favourite author I propose it is termed the PTerry Law.
An unnamed single person, .... who can walk in and introduce a creeping, debilitating defect in the Galileo satellites.
At least that part is not surprizing at all, isn't it? Every complicated setup eventually comes down to a few, key people who actually know (or possibly think they know, without really knowing - in which case your organization is really in trouble) how the things are set up and work in practice. When one of them makes a mistake which is not caught in time, the consequences and the recovery can be "interesting".
I believe we've all been there; I know I was.
I've been that person with the understanding to make things work on occasion and can attest at least JPL will put a stop to that. Engineers and programmers are not let anywhere near live systems: there are separate groups that compile programs, install them on machines, and operate said machines. As a developer we were only ever allowed to observe then make corrections on our development system eventually to be distributed to the groups that were allowed to touch live systems. And that was for ground-based systems; I'm sure the systems that flew had a lot more hoops to jump through.
I was peripherally involved in a project supplying ITER. It was completely open - anyone could bid and anynkit was acceptable
Anyone, that is, so long as you were French and using any kit, so long as it was a certain German PLC. Yeah, right. European open organisation. Almost as bad as the Yanks.
Anon 'cos I'm still pissed.
These friendly vendors, would they be ones in which the managers making decisions, happen to end up with non-executive directorships in, by any chance would they?
Those managers don't make decisions. They call in management consultants to do that for them, consultants who belong to one of the Big Four and who have a healthy rotating-door policy with regard to industry executives looking for a two-year placement position.
I told him to make sure he made the DST adjustment, but the berk just left it to go home early that evening, now 24 hours have passed we'll have to wait until end of Summertime next year.
With the bureaucratic decision to remove DST in the EU altogether from March the whole project is royally f'ed and all the satellites will not be allowed to resync at all.
We all hate Phil and his croissant stuffing face, he had one job in the time sync department, but all he does is bloody clock watch.
We USAians are beginning to realise it's stupid as well. For example: Daylight "Saving" Time occurs in the summer. Think about that for a minute or two (I'll wait).
Why are we "saving" daylight in the summer, when there's more daylight than in any other part of the year? Conversely, Standard Time is used in the winter, when there is the least amount of daylight.
Nobody seems to be able to justify the silly thing anyway. Safety, energy conservation, farming considerations...none of them actually hold up under critical scrutiny. Time to dump it.
Nobody seems to be able to justify the silly thing anyway. Safety, energy conservation, farming considerations...none of them actually hold up under critical scrutiny. Time to dump it.
It did when it was introduced, as it meant that people could do the things they needed to after work without needing artificial lighting, thus lowering the energy grid requirements during the war, (when it was introduced, I believe).
These days, however, you are quite right, and it serves little functional purpose.
How does that solve the problem?
You could argue that a European GPS is an unnecessary vanity project but the UK, for example, wasn't forced to take part: we thought it was a good idea at the time and it was the UK that wanted it kept to EU members.
If you argue that it is necessary, there is no way that it could be funded either by a single country on its own (despite the UK's aspirations for a local system for local people), nor that the other European countries would accept such a system being under the sole control of another nation's government.
It's inevitable that a project of this kind will be a multinational operation with the consequent vested interests of multiple governments wanting their piece of the action, regardless of the umbrella under which those governments gather.
"a federal government" and that is the difference. The EU doesn't have a federal government, yet.
A friend of mine worked for the European Bank of Reconstruction and Development. Over a beer (well crying over a beer) he once described to me their structure and how they have to work to get things done. It was similar in ways to the structure governing Galileo, and 28 "equal" (in their eyes) bosses each had their own "creative" content to add to the party.
Big ones. Similar in size to the EU.
While the US and China are of the similar "size" (economically; that's the only size that really matters) to the EU, Russia's is not even of the same order of magnitude:
From https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal):
Entity GDP[1012 USD]
1. USA 21.4
-. EU 18.7
2. China 14.1
3. Japan 5.2
4. Germany 3.9
5. India 2.9
6. UK 2.9
7. France 2.7
8. Italy 1.9
...
11. Russia 1.6
[IMF values; The numbers vary by the agency, but the ranking remains similar enough]
Note that the EU members Germany, UK, France, and Italy each have the economy substantially larger than Russia's; if Russians can afford to run their own GNSS system, there is obviously no overwhelming economic reason why either one of these three can't. And it's not like the Russians are even spending hand over fist on their space program: it costs about 1/4th of one percent of their GDP; to put that number in perspective, just the annual cost of implementing the Universal Credit programme is about 10% of the cost of the Russian space program; the total amount sunk into UC would be enough to run Russia's space program for the full year.
.. And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980 and hardly managed to build any new hardware - remember Kliper shuttle? Got absolutely nowhere but made a lot of noise.
Same with the Russian military hardware, they're ambitious and make a lot of noise, but their main bulk of aircraft produced right now is just upgraded versions of Soviet designs. Their advanced aircraft are nothing but techonolgy demonstrators with a handful of production samples.
"And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980 and hardly managed to build any new hardware"
Like Soyuz, that is used to carry US and other nations astronauts to the ISS
https://en.wikipedia.org/wiki/Soyuz_(spacecraft)
.. And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980
Yes. Their Soyuz rocket is a bit shit in terms of design in comparison to wonderful bits of kit like the space shuttle.
However, bits of American kit like the space shuttle are so frigging expensive (>$1 billion per launch) that there was a total of 131 shuttle launches before it was grounded the last time. The currently used soyuz rocket costs something like $20-40 million per launch, and as a result has done 727 launches, 708 of were successful at a cost of $15-$30 billion. The shuttle projects 131 launches cost $209 billion.
Honestly, on this one the Russians win with more launches and more payload delivered to orbit with less cost. There is frankly no point them developing something else; when you have a business model of producing lots of cheap rockets then your R&D goals are to optimise the production to minimise production costs, not design something fancy, new and overly sophisticated that pushes up the production cost.
Actually, there are several alternatives cited in other answers. Or, if you have unique requirements, you can possibly design and launch your own GPS/GLOSNASS,Beidou,IRNSS (pick one) compatible satellites as Japan seems to be proposing with QZSS.
^ Do people seriously believe it would be any better if we ever do embark on our own post-brexit British Satellite Positioning System? Or any other British project for that matter.
It will be an omnishamble clusterfuck like every other, no doubt with Crapita and all the usual suspects involved.
The EU doesn't have a monopoly on those.
"It will be an omnishamble clusterfuck like every other, no doubt with BAE and all the usual suspects involved."
FTFY - it will be classed as a defence project rather than just a run of the mill omnishambles.
While I'm not suggesting Crapita could do a better job, they could likely provide a similar end product for significantly less. But it wouldn't be "BAE secure"...
Britain's answer to GPS will be sending Phileas Fogg up in a balloon with a compass and a megaphone to tell you where you are.
(seriously though, I haven't forgotten Decca, which served quite well from the 1940s until satellite GPS took over).
While EU is pretty bureaucratic it is also, by comparison, remarkably efficient. Or perhaps that should be suprisingly less inefficient than many national bureaucracies.
Setting up complex structures with lots of competing committees in order to piss money against the wall is what bureaucracies do.
Having had personal experience of this positional persiflage by not being able to find his ditch, Boris-UK will make its own navigational system. Written in Latin, calibrated in fathoms, with an origin of the London Garden Bridge it will be staffed by Etonians and be 'on-time' for fifteen years.
government project where the poor guys trying to run the damn thing are micromanaged by 6 dozen competing agencies all trying to assert their control of said flagship project.
But then having the EU ask ESA to build and run it as a seperate agency to ESA, responsible only to ESA is far too simple (and doesn't employ enough middle managers/failed politicians/career minded bureaucrats)
As for the 'one tech guy fekked up' story.... how many of us have heard the same thing from senior manglement when things have gone wrong ... especially when said senior manglement are the guys who made the decision not to have a backup system ..... (London ambulance computer dispatch system springs to mind as a fine example of this)
"government project where the poor guys trying to run the damn thing are micromanaged by 6 dozen competing agencies all trying to assert their control of said flagship project."
The first thing to specify on a project like that are the locks. The ones on the doors that keep the micromanagers out.
I was part of a team brought in to upgrade a large multi-terabyte company database. We did have multiple upgrade rehearsals but these took most of a weekend each so to save time the managers decided (against our advice) to use an old snapshot of their database rather than copying the live system, which would have added 3 hours to each rehearsal. This meant we were testing against the same old static data over and over again. When the real upgrade happened it failed because the live data had something unexpected which wasn't in the test data! Guess who got the blame?
As for the 'one tech guy fekked up' story.... how many of us have heard the same thing from senior manglement when things have gone wrong ... especially when said senior manglement are the guys who made the decision not to have a backup system ..... (London ambulance computer dispatch system springs to mind as a fine example of this)
Especially when the "one guy" has probably jumping up and down for the last couple of years scream at how totally broken the system is and that we're one small step away from disaster.
The senior mangler is now blaming them "coz it would never have died if you hadn't said it was broken"
Sounds familiar.
In a previous job, the thing I had been warning for months would happen, happened, to within a few days of when I predicted. Had all the email I’d sent about it as evidence.
The moronic manager who should have acted and didn’t blamed me anyway. Said I hadn’t made the case strongly enough!
That company no longer exists and said manager went on to have a successful career at one of the outsourcing companies that’s always appearing in El Reg. Successful for him, not any of his clients, naturally.
From me just a big thanks for publishing you report linked from the article. I assume that was a LOT of work!
Yes, this took a lot of work - especially to gather all the relevant documents, but I am so happy many people get to read about it now! I hope this contributes to some new transparency. Because the good news is that fundamentally Galileo appears to be a good system, once the kinks get worked out.
Having a precision, 24/7, multi-billion $, purportedly mission-critical electronic system into which a single person can alledgedly introduce a debilitating flaw, without adequate or accurate release testing and with no backup to restore the system to a working, previous build is one hell of a set of "kinks".
I certainly hope that all these can be dealt with within the limitations of not having physical access to the existing satellites, but avoiding any one of the kinks I mention above is systems administration 1A stuff for a high-profile, mission-critical system. I'm not even a technologist, and I know that.
> without adequate or accurate release testing and with no backup to restore the system to a working, previous build is one hell of a set of "kinks".
as if most IT systems couldn't be brought down with one wrong person getting sudo access on a domain controller or such – yes, you can protect against maid pulling the power cable of the root DNS server, can't protect against the authoritative root server advertising empty or wrong root zone – there needs to be a single source of truth, but you need a good process/syste how it is updated, which appears to be lacking in Galileo. So, yes, that's a minor kink compared to overall complexity of the system.
and it does look like they did have backups to restore, but because of the precision necessary, and given that the satellites are in in non-inertial reference frames, it does mean that getting it all back synchronised isn't a matter of seconds
Thanks - well, at a guess, only people that have experienced the pain of complex systems will demand simplicity. Complexity is just like entropy - it happens, without anyone asking for it. You only get someone simple if you have battle hardened cynics in the room that keep fighting against complexity. And I guess those people did not win the day.
This is true of almost any collaborative project, irrespective of size (I've only been involved with small ones) or industry.
The cynical voices asking "what if" and "how can we be sure" will be silenced by senior managers from the different quarters,who are committed to get said project completed within a promised time frame that was chosen to please their managers, the council, the board or whoever, rather than a realistic estimate. And within a budget, ditto.
So that when it's completed there will be doors hanging off, IT missing because the figures were fudged so that, say electronic security was built in, but without enough finding budget to pay for it, the fire exit will open onto the bin areas, so that no one can get out because they promised to sort out the bins later ( this not being an important aspect, it's only bins after all, until, of course reality sets in), the cupboards won't be big enough to hold the kit, there's a water pipe where the toilet is meant to go, the green heating/cooling system will be inadequate, because no one listened when a staff member pointed out that the large doors do actually have to be opened several times a day, and so on.
US DoD - the reason we don't want to release the hi-res GPS feed is because we've spent a lot of time and money getting it to work and we don't see why everyone else should get it for free
EU - Pah. It's just clocks and maths. How hard can it be ?
US - Be my guest.
"because we've spent a lot of time and money getting it to work"
It is, of course, perfectly possible that this statement is entirely true. Specifically, we don't know how many times the high-precision parts of the US system fell over in their early stages, because we didn't have access to them. But it is only clocks and maths, so I expect Europe will get there eventually, just like the US did.
This post has been deleted by its author
That's because the lone employee is probably an incompetent manager who has jumped from one project to another whilst always pinning the blame on whoever was closest to the cock-up rather than whoever was responsible for it.
See also https://en.wikipedia.org/wiki/Proximate_and_ultimate_causation
When has a simple answer for a complicated question ever worked?
You have no idea what to do next after leaving and if you think you do it can be shot down in 57 ways in the first five minutes.
But carry on, I guess learning the hard way will work if nothing else does.
>> in which no one wants to take the blame for problems caused by unnecessary organizational complexity
Hmm, that smells a lot like how the Brits do things as well... From recent memory..
Unified Credit Benefits, National Rail and Brexit itself.
Throwing stones and glasshouses?
In short, the UK is a classic British venture: a great idea with (some) talented people that has turned into a bureaucratic mess in which no one wants to take the blame for problems caused by unnecessary organizational complexity.
BETTER OFF OUT!
Sounds like we need UKxit by the same logic..
Discuss.
Yes, this is typical EU. But, all things considered, the fact that "typical Europe" now means "coming up with awesome shared projects and then fucking them up due to political bickering" is a massive improvement. I mean, "typical Europe" before the EU used to mean "war". Had been since literally forever. If it takes another hundred years to get a EU that really works, the process will still look pretty good in a history book.
I used to work in a related area where high accuracy, uptime and reliability are critical. Anything done on the live system had to be rehearsed on the test system first. It had to be done according to a written procedure which had been reviewed and approved beforehand. Part of the review was a risk assessment (i.e. ask "what could possibly go wrong?"). There also had to be a reversion procedure (i.e. "We screwed up; put it back the way it was").
We did have occasional outages, including one particularly embarrassing incident where "Do routine thing" was next to "Shut down the system" on a menu. But they were rare, and both management and engineering took justifiable pride in that.
One thing we would *NEVER* do is blame the engineer holding the mouse (short of actual malice). If they made a mistake, its because the system upstream of them enabled that mistake and set them up to fail. You don't shoot the engineer, you fix the system.
@Paul Johnson 1
In an ideal world I should be down voting you for stating the bleedin' obvious, but the reality is I have to upvote you for banging the drum for automation and repeatability.
I hope everyone here is scripting their changes - not just following a written procedure - although that in itself is better than doing it ad-hoc. If not you need to start asking yourself why not. Not working for ESA is not a valid reason.
"The EU is planning to create a new European Space Agency, called EUSA, which will largely be a renaming of the existing Global Navigation Satellite System agency. Yet another space entity, GSA, will become the EU Agency for the Space Program, and the EC will soon have a new director general position in charge of the “defense industry and space.” In short, there are a lot of political maneuverings and that is causing all kinds of other problems."
Really ????? WTF ? Why ? Why creating issues whereas a very good company, called ESA, had already so many successes ? Sure, this mishap is embarrassing, but everyone went there, I'm sure the US had some glitches as well, back in the time, but since it was new and military restricted, no-one got the message.
I'm baffled and sad. This is all gonna destroy what was done ...
As for ESA, I'm sure they'll figure out how to setup rollbacks processes for this. It is surely not simple.
Because, actually the ESA isn't an EU organisation. Not all EU countries are in the ESA, though the the EU as a whole contributes. Also there are three (I think) non-EU countries in the ESA.
It's explained accurately in this otherwise fictional account of Aliens visiting earth. "The Solar Alliance" by Ray McCarthy. The Aliens quickly figure out that the UN is mostly a talking shop. The ESA and EU take them longer to understand. A new Earth agency is setup by the countries with space launch facilities, or willing to make a major commitment, but who is REALLY running the Solar Alliance, with its HQ in Shannon Ireland, an Israeli Finance director and an ex KGB/SVR Russian in charge of Security?
No, amfM1, I don't think so. though I don't know how random the pseudo codes are, so in stego theory, yes, but that's only a downlink - so no mesh.
I think it (Galileo) primarily came from the fact that L1 was unreliable due SPOF, serious CIP required a backup, "A-GPS," where A should be as many things as possible. Why not build one's own!
And there will always remain a serious question mark over your statement outside of ...3P C&C, to which I just answer, really?
But far too complicated for them to understand :)
Wasn't the UK going to build and launch its own system as soon as we left? I suppose it's all Parliament's fault that that hasn't happened yet. Still arguing about whether the clocks should be set to 1953, 1940, or 1887.
Who uses Galileo GPS? Genuine question - I realise that the original GPS system was USA based and that this was to reduce reliance on the Yanks (don't Russia have their own system?), but I don't I have ever seen a GPS or smart phone that has a "Compatible with Galileo GPS!" type sticker on it.
The IT Crowd reference -
So it was fixed by switching it off and on again? Was it running on Vista....?
It's built into most GNSS chips as standard now. There are over 150 Android smartphones on the market which can receive Galileo (in addition to GPS and GLONASS) and it's been available on the iPhone since the X/8/8Plus were released in 2017.
There's no sticker, it's just baked into the chips by default, same as your handset being quad-band and being able to connect to networks using 2/3/4G in multiple countries, allowing for different regional spectrum allocations.
One of the great issues of any GPS system is you don't know exactly where anything is at any given time. You know were it was and you can predict where it is going to be but its only a very good guess. The satellites are being tracked but there is a delay between signal tracking them and getting the info into a computer half a continent away. The weather is going to delay signals in odd ways that usually allows compensation using different frequencies but only some times. The clocks are ticking away with some very high degree of accuracy yet subject to all the oddness that relativity in a gravity well has to offer. The ground stations are busy floating on land that is cruising in different directions at a few cm a year which was considered slow and stable until better GPS systems showed drift rate can vary over the months yet maintain a rock solid annual average. Yet in all that chaos, my phone still can display a map of where it is down to a few meters. I guess this problem demonstrated just how related the chaos of all guesses can be.
Can you imagine if the official response was "turn it off and on again", in the middle of an actual conflict, 15+ years down the road?
I'm going to hazard a guess that were this military only, it would cost 4 to 8 times more than it currently does, but at least the org chart would be a bit more vertical, if porky.
I think in the middle of an actual conflict 15 years hence, there would be no point in turning it back on again before you'd shot down the several dozen "interference satellites" in low orbit beaming noise on the relevant frequencies at several thousand times the GPS signal strength.
You know, I think a lot of people wanted to move from GPS to Galileo due to it being a lot less boggled down in politics and better accuracy. However neither Galileo going down or the spat between ESA and EU because of politics is really convincing any manufacturer to actually adopt Galileo rather than sticking with just GPS.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
