One man's mistake, missing backups and complete reboot: The tale of Europe's Galileo satellites going dark Key details about the failure of Europe's Galileo satellite system over the summer have started to emerge – and it's not pretty. While one key official has sought to blame a single individual for the system going dark, insiders warn that organizational chaos, excessive secrecy and some unusual self-regulation is as much to …
How does that solve the problem?
You could argue that a European GPS is an unnecessary vanity project but the UK, for example, wasn't forced to take part: we thought it was a good idea at the time and it was the UK that wanted it kept to EU members.
If you argue that it is necessary, there is no way that it could be funded either by a single country on its own (despite the UK's aspirations for a local system for local people), nor that the other European countries would accept such a system being under the sole control of another nation's government.
It's inevitable that a project of this kind will be a multinational operation with the consequent vested interests of multiple governments wanting their piece of the action, regardless of the umbrella under which those governments gather.
"a federal government" and that is the difference. The EU doesn't have a federal government, yet.
A friend of mine worked for the European Bank of Reconstruction and Development. Over a beer (well crying over a beer) he once described to me their structure and how they have to work to get things done. It was similar in ways to the structure governing Galileo, and 28 "equal" (in their eyes) bosses each had their own "creative" content to add to the party.
Big ones. Similar in size to the EU.
While the US and China are of the similar "size" (economically; that's the only size that really matters) to the EU, Russia's is not even of the same order of magnitude:
From https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal):
Entity GDP[1012 USD]
1. USA 21.4
-. EU 18.7
2. China 14.1
3. Japan 5.2
4. Germany 3.9
5. India 2.9
6. UK 2.9
7. France 2.7
8. Italy 1.9
...
11. Russia 1.6
[IMF values; The numbers vary by the agency, but the ranking remains similar enough]
Note that the EU members Germany, UK, France, and Italy each have the economy substantially larger than Russia's; if Russians can afford to run their own GNSS system, there is obviously no overwhelming economic reason why either one of these three can't. And it's not like the Russians are even spending hand over fist on their space program: it costs about 1/4th of one percent of their GDP; to put that number in perspective, just the annual cost of implementing the Universal Credit programme is about 10% of the cost of the Russian space program; the total amount sunk into UC would be enough to run Russia's space program for the full year.
.. And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980 and hardly managed to build any new hardware - remember Kliper shuttle? Got absolutely nowhere but made a lot of noise.
Same with the Russian military hardware, they're ambitious and make a lot of noise, but their main bulk of aircraft produced right now is just upgraded versions of Soviet designs. Their advanced aircraft are nothing but techonolgy demonstrators with a handful of production samples.
"And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980 and hardly managed to build any new hardware"
Like Soyuz, that is used to carry US and other nations astronauts to the ISS
https://en.wikipedia.org/wiki/Soyuz_(spacecraft)
.. And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980
Yes. Their Soyuz rocket is a bit shit in terms of design in comparison to wonderful bits of kit like the space shuttle.
However, bits of American kit like the space shuttle are so frigging expensive (>$1 billion per launch) that there was a total of 131 shuttle launches before it was grounded the last time. The currently used soyuz rocket costs something like $20-40 million per launch, and as a result has done 727 launches, 708 of were successful at a cost of $15-$30 billion. The shuttle projects 131 launches cost $209 billion.
Honestly, on this one the Russians win with more launches and more payload delivered to orbit with less cost. There is frankly no point them developing something else; when you have a business model of producing lots of cheap rockets then your R&D goals are to optimise the production to minimise production costs, not design something fancy, new and overly sophisticated that pushes up the production cost.
Actually, there are several alternatives cited in other answers. Or, if you have unique requirements, you can possibly design and launch your own GPS/GLOSNASS,Beidou,IRNSS (pick one) compatible satellites as Japan seems to be proposing with QZSS.
^ Do people seriously believe it would be any better if we ever do embark on our own post-brexit British Satellite Positioning System? Or any other British project for that matter.
It will be an omnishamble clusterfuck like every other, no doubt with Crapita and all the usual suspects involved.
The EU doesn't have a monopoly on those.
"It will be an omnishamble clusterfuck like every other, no doubt with BAE and all the usual suspects involved."
FTFY - it will be classed as a defence project rather than just a run of the mill omnishambles.
While I'm not suggesting Crapita could do a better job, they could likely provide a similar end product for significantly less. But it wouldn't be "BAE secure"...
Britain's answer to GPS will be sending Phileas Fogg up in a balloon with a compass and a megaphone to tell you where you are.
(seriously though, I haven't forgotten Decca, which served quite well from the 1940s until satellite GPS took over).
While EU is pretty bureaucratic it is also, by comparison, remarkably efficient. Or perhaps that should be suprisingly less inefficient than many national bureaucracies.
Setting up complex structures with lots of competing committees in order to piss money against the wall is what bureaucracies do.
Having had personal experience of this positional persiflage by not being able to find his ditch, Boris-UK will make its own navigational system. Written in Latin, calibrated in fathoms, with an origin of the London Garden Bridge it will be staffed by Etonians and be 'on-time' for fifteen years.
government project where the poor guys trying to run the damn thing are micromanaged by 6 dozen competing agencies all trying to assert their control of said flagship project.
But then having the EU ask ESA to build and run it as a seperate agency to ESA, responsible only to ESA is far too simple (and doesn't employ enough middle managers/failed politicians/career minded bureaucrats)
As for the 'one tech guy fekked up' story.... how many of us have heard the same thing from senior manglement when things have gone wrong ... especially when said senior manglement are the guys who made the decision not to have a backup system ..... (London ambulance computer dispatch system springs to mind as a fine example of this)
"government project where the poor guys trying to run the damn thing are micromanaged by 6 dozen competing agencies all trying to assert their control of said flagship project."
The first thing to specify on a project like that are the locks. The ones on the doors that keep the micromanagers out.
I was part of a team brought in to upgrade a large multi-terabyte company database. We did have multiple upgrade rehearsals but these took most of a weekend each so to save time the managers decided (against our advice) to use an old snapshot of their database rather than copying the live system, which would have added 3 hours to each rehearsal. This meant we were testing against the same old static data over and over again. When the real upgrade happened it failed because the live data had something unexpected which wasn't in the test data! Guess who got the blame?
As for the 'one tech guy fekked up' story.... how many of us have heard the same thing from senior manglement when things have gone wrong ... especially when said senior manglement are the guys who made the decision not to have a backup system ..... (London ambulance computer dispatch system springs to mind as a fine example of this)
Especially when the "one guy" has probably jumping up and down for the last couple of years scream at how totally broken the system is and that we're one small step away from disaster.
The senior mangler is now blaming them "coz it would never have died if you hadn't said it was broken"
Sounds familiar.
In a previous job, the thing I had been warning for months would happen, happened, to within a few days of when I predicted. Had all the email I’d sent about it as evidence.
The moronic manager who should have acted and didn’t blamed me anyway. Said I hadn’t made the case strongly enough!
That company no longer exists and said manager went on to have a successful career at one of the outsourcing companies that’s always appearing in El Reg. Successful for him, not any of his clients, naturally.
From me just a big thanks for publishing you report linked from the article. I assume that was a LOT of work!
Yes, this took a lot of work - especially to gather all the relevant documents, but I am so happy many people get to read about it now! I hope this contributes to some new transparency. Because the good news is that fundamentally Galileo appears to be a good system, once the kinks get worked out.
Having a precision, 24/7, multi-billion $, purportedly mission-critical electronic system into which a single person can alledgedly introduce a debilitating flaw, without adequate or accurate release testing and with no backup to restore the system to a working, previous build is one hell of a set of "kinks".
I certainly hope that all these can be dealt with within the limitations of not having physical access to the existing satellites, but avoiding any one of the kinks I mention above is systems administration 1A stuff for a high-profile, mission-critical system. I'm not even a technologist, and I know that.
> without adequate or accurate release testing and with no backup to restore the system to a working, previous build is one hell of a set of "kinks".
as if most IT systems couldn't be brought down with one wrong person getting sudo access on a domain controller or such – yes, you can protect against maid pulling the power cable of the root DNS server, can't protect against the authoritative root server advertising empty or wrong root zone – there needs to be a single source of truth, but you need a good process/syste how it is updated, which appears to be lacking in Galileo. So, yes, that's a minor kink compared to overall complexity of the system.
and it does look like they did have backups to restore, but because of the precision necessary, and given that the satellites are in in non-inertial reference frames, it does mean that getting it all back synchronised isn't a matter of seconds
Thanks - well, at a guess, only people that have experienced the pain of complex systems will demand simplicity. Complexity is just like entropy - it happens, without anyone asking for it. You only get someone simple if you have battle hardened cynics in the room that keep fighting against complexity. And I guess those people did not win the day.
This is true of almost any collaborative project, irrespective of size (I've only been involved with small ones) or industry.
The cynical voices asking "what if" and "how can we be sure" will be silenced by senior managers from the different quarters,who are committed to get said project completed within a promised time frame that was chosen to please their managers, the council, the board or whoever, rather than a realistic estimate. And within a budget, ditto.
So that when it's completed there will be doors hanging off, IT missing because the figures were fudged so that, say electronic security was built in, but without enough finding budget to pay for it, the fire exit will open onto the bin areas, so that no one can get out because they promised to sort out the bins later ( this not being an important aspect, it's only bins after all, until, of course reality sets in), the cupboards won't be big enough to hold the kit, there's a water pipe where the toilet is meant to go, the green heating/cooling system will be inadequate, because no one listened when a staff member pointed out that the large doors do actually have to be opened several times a day, and so on.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
