back to article One man's mistake, missing backups and complete reboot: The tale of Europe's Galileo satellites going dark

Key details about the failure of Europe's Galileo satellite system over the summer have started to emerge – and it's not pretty. While one key official has sought to blame a single individual for the system going dark, insiders warn that organizational chaos, excessive secrecy and some unusual self-regulation is as much to …

Page:

    1. maffski

      Re: if there are any questions

      Thanks for your work.

      Do they not have anything like a 'live' and 'beta' data stream so they can test their updates on the system before publishing them?

  1. J4

    Chortling sound coming over the ocean

    US DoD - the reason we don't want to release the hi-res GPS feed is because we've spent a lot of time and money getting it to work and we don't see why everyone else should get it for free

    EU - Pah. It's just clocks and maths. How hard can it be ?

    US - Be my guest.

    1. Anonymous South African Coward Bronze badge

      Re: Chortling sound coming over the ocean

      Plus a poorly tested backup method, and no rollback or fallback plan.

    2. Ken Hagan Gold badge

      Re: Chortling sound coming over the ocean

      "because we've spent a lot of time and money getting it to work"

      It is, of course, perfectly possible that this statement is entirely true. Specifically, we don't know how many times the high-precision parts of the US system fell over in their early stages, because we didn't have access to them. But it is only clocks and maths, so I expect Europe will get there eventually, just like the US did.

  2. herman Silver badge

    Hmm, Navstar is not immune to cock-ups either, but it usually resolves quite quickly.

  3. Will Godfrey Silver badge
    FAIL

    Oh yeah?

    That "Lone employee" sure gets around, eh.

    1. a pressbutton

      Re: Oh yeah?

      Last seen on a grassy knoll?

      1. Clunking Fist

        Re: Oh yeah?

        Last seen entering a federal penitentiary in New York in a blue dress.

    2. It's just me

      Re: Oh yeah?

      Probably the same rogue engineer that worked for some auto manufacturers.

      1. Anonymous Coward
        Anonymous Coward

        Re: Oh yeah?

        His great-great grandfather was responsible for buoyancy control and passenger evacuation systems on the Titanic.

      2. A. Coatsworth Silver badge
        Holmes

        Re: Oh yeah?

        Yup, he got this gig after leaving Google. Back in 2012 he worked in the comms systems for street view cars.

        He got massive good recommendations from his PHB at Google, too.

      3. This post has been deleted by its author

    3. Ken Hagan Gold badge

      Re: Oh yeah?

      That's because the lone employee is probably an incompetent manager who has jumped from one project to another whilst always pinning the blame on whoever was closest to the cock-up rather than whoever was responsible for it.

      See also https://en.wikipedia.org/wiki/Proximate_and_ultimate_causation

  4. The Average Joe

    Windows XP! DOH! they should have upgraded to a newer OS!

    so the Windows XP is out of support, better send a helpdesk guy up there to repair the OS. It is always cheaper to use windows for mission critical processes, just look at the factory floor...

  5. Anonymous Coward
    Anonymous Coward

    It's a glorious metaphor.

    In short, the EU is a classic European venture: a great idea with (some) talented people that has turned into a bureaucratic mess in which no one wants to take the blame for problems caused by unnecessary organizational complexity.

    BETTER OFF OUT!

    1. Dan 55 Silver badge

      Re: It's a glorious metaphor.

      When has a simple answer for a complicated question ever worked?

      You have no idea what to do next after leaving and if you think you do it can be shot down in 57 ways in the first five minutes.

      But carry on, I guess learning the hard way will work if nothing else does.

    2. Anonymous Coward
      Anonymous Coward

      Re: It's a glorious metaphor.

      >> in which no one wants to take the blame for problems caused by unnecessary organizational complexity

      Hmm, that smells a lot like how the Brits do things as well... From recent memory..

      Unified Credit Benefits, National Rail and Brexit itself.

      Throwing stones and glasshouses?

    3. Anonymous Coward
      Anonymous Coward

      Re: It's a glorious metaphor.

      In short, the UK is a classic British venture: a great idea with (some) talented people that has turned into a bureaucratic mess in which no one wants to take the blame for problems caused by unnecessary organizational complexity.

      BETTER OFF OUT!

      Sounds like we need UKxit by the same logic..

      Discuss.

    4. batfink

      Re: It's a glorious metaphor.

      ...so the UK can have its own proper British fuck-ups.

      1. GrapeBunch

        Re: It's a glorious metaphor.

        The story is an abject example of how the lessons of this day, November 11th, have only partly been learned. Shouldn't we borrow from the Americans and make it "three strikes and you're out" for political meddling?

  6. CAPS LOCK

    Handily eLoRaN base stations always know...

    ... where they are.

  7. Filippo Silver badge

    Yes, this is typical EU. But, all things considered, the fact that "typical Europe" now means "coming up with awesome shared projects and then fucking them up due to political bickering" is a massive improvement. I mean, "typical Europe" before the EU used to mean "war". Had been since literally forever. If it takes another hundred years to get a EU that really works, the process will still look pretty good in a history book.

    1. Laurent 💔 🐧

      The best comment so far for this article, without contest.

    2. David Beck

      Ask a Bosnian how that's going.

      1. Anonymous Coward
        Anonymous Coward

        QED

        Last I heard Bosnia is not yet a fully fledged member of the EU.

        1. IDoNotThinkSo

          Re: QED

          But it is in Europe. The EU did nothing. It took NATO to sort it out.

  8. Paul Johnson 1
    FAIL

    Where were the risk assessment and reversion procedure?

    I used to work in a related area where high accuracy, uptime and reliability are critical. Anything done on the live system had to be rehearsed on the test system first. It had to be done according to a written procedure which had been reviewed and approved beforehand. Part of the review was a risk assessment (i.e. ask "what could possibly go wrong?"). There also had to be a reversion procedure (i.e. "We screwed up; put it back the way it was").

    We did have occasional outages, including one particularly embarrassing incident where "Do routine thing" was next to "Shut down the system" on a menu. But they were rare, and both management and engineering took justifiable pride in that.

    One thing we would *NEVER* do is blame the engineer holding the mouse (short of actual malice). If they made a mistake, its because the system upstream of them enabled that mistake and set them up to fail. You don't shoot the engineer, you fix the system.

    1. Anonymous Coward
      Anonymous Coward

      Re: Where were the risk assessment and reversion procedure?

      @Paul Johnson 1

      In an ideal world I should be down voting you for stating the bleedin' obvious, but the reality is I have to upvote you for banging the drum for automation and repeatability.

      I hope everyone here is scripting their changes - not just following a written procedure - although that in itself is better than doing it ad-hoc. If not you need to start asking yourself why not. Not working for ESA is not a valid reason.

  9. dnicholas
    Paris Hilton

    We've all been there...

    ...(over)confidently bashing a command in the ...(wrong)terminal window

  10. Anonymous Coward
    Anonymous Coward

    Amazed, trully

    "The EU is planning to create a new European Space Agency, called EUSA, which will largely be a renaming of the existing Global Navigation Satellite System agency. Yet another space entity, GSA, will become the EU Agency for the Space Program, and the EC will soon have a new director general position in charge of the “defense industry and space.” In short, there are a lot of political maneuverings and that is causing all kinds of other problems."

    Really ????? WTF ? Why ? Why creating issues whereas a very good company, called ESA, had already so many successes ? Sure, this mishap is embarrassing, but everyone went there, I'm sure the US had some glitches as well, back in the time, but since it was new and military restricted, no-one got the message.

    I'm baffled and sad. This is all gonna destroy what was done ...

    As for ESA, I'm sure they'll figure out how to setup rollbacks processes for this. It is surely not simple.

    1. Mage Silver badge

      Re: The EU is planning to create a new European Space Agency

      Because, actually the ESA isn't an EU organisation. Not all EU countries are in the ESA, though the the EU as a whole contributes. Also there are three (I think) non-EU countries in the ESA.

      It's explained accurately in this otherwise fictional account of Aliens visiting earth. "The Solar Alliance" by Ray McCarthy. The Aliens quickly figure out that the UN is mostly a talking shop. The ESA and EU take them longer to understand. A new Earth agency is setup by the countries with space launch facilities, or willing to make a major commitment, but who is REALLY running the Solar Alliance, with its HQ in Shannon Ireland, an Israeli Finance director and an ex KGB/SVR Russian in charge of Security?

  11. amanfromMars 1 Silver badge

    Hiding in plain sight and disguised as another Global Navigation Satellite System

    Is Galileo really just a private mesh communication system outside of wacky wild west third party command and control?

    1. David Shaw

      Re: Hiding in plain sight and disguised as another Global Navigation Satellite System

      No, amfM1, I don't think so. though I don't know how random the pseudo codes are, so in stego theory, yes, but that's only a downlink - so no mesh.

      I think it (Galileo) primarily came from the fact that L1 was unreliable due SPOF, serious CIP required a backup, "A-GPS," where A should be as many things as possible. Why not build one's own!

      And there will always remain a serious question mark over your statement outside of ...3P C&C, to which I just answer, really?

  12. beast666

    In short, the EU project is a classic European misadventure: a tyrannical idea with corrupt people that has turned into a bureaucratic mess in which no one wants to take the blame for problems caused by unnecessary organizational complexity.

    1. Intractable Potsherd

      In short, the UK project is a classic English misadventure: a tyrannical idea with corrupt people that has turned into a bureaucratic mess in which no one wants to take the blame for problems caused by unnecessary organizational complexity.

      FTFY

      1. Oh Matron!

        Those who govern are, ironically, those least suitable to do so.

  13. smudge
    Trollface

    A gift for the Brexiteers

    But far too complicated for them to understand :)

    Wasn't the UK going to build and launch its own system as soon as we left? I suppose it's all Parliament's fault that that hasn't happened yet. Still arguing about whether the clocks should be set to 1953, 1940, or 1887.

  14. fuserly

    Why does the author assume that it was a man?

    1. rg287 Silver badge
      Joke

      Why does the author assume that it was a man?

      Because a woman would have read the instruction manual, or stopped and asked for directions when she became unsure - instead of insisting that she knew where she was and "it's just around here somewhere in one of these menus".

  15. Anonymous Coward
    Anonymous Coward

    Who uses it?

    Who uses Galileo GPS? Genuine question - I realise that the original GPS system was USA based and that this was to reduce reliance on the Yanks (don't Russia have their own system?), but I don't I have ever seen a GPS or smart phone that has a "Compatible with Galileo GPS!" type sticker on it.

    The IT Crowd reference -

    So it was fixed by switching it off and on again? Was it running on Vista....?

    1. rg287 Silver badge

      Re: Who uses it?

      It's built into most GNSS chips as standard now. There are over 150 Android smartphones on the market which can receive Galileo (in addition to GPS and GLONASS) and it's been available on the iPhone since the X/8/8Plus were released in 2017.

      There's no sticker, it's just baked into the chips by default, same as your handset being quad-band and being able to connect to networks using 2/3/4G in multiple countries, allowing for different regional spectrum allocations.

  16. -tim
    Boffin

    The lovely problem of a complex problem

    One of the great issues of any GPS system is you don't know exactly where anything is at any given time. You know were it was and you can predict where it is going to be but its only a very good guess. The satellites are being tracked but there is a delay between signal tracking them and getting the info into a computer half a continent away. The weather is going to delay signals in odd ways that usually allows compensation using different frequencies but only some times. The clocks are ticking away with some very high degree of accuracy yet subject to all the oddness that relativity in a gravity well has to offer. The ground stations are busy floating on land that is cruising in different directions at a few cm a year which was considered slow and stable until better GPS systems showed drift rate can vary over the months yet maintain a rock solid annual average. Yet in all that chaos, my phone still can display a map of where it is down to a few meters. I guess this problem demonstrated just how related the chaos of all guesses can be.

  17. Zarno
    Mushroom

    I sure hope they don't use this for military purposes...

    Can you imagine if the official response was "turn it off and on again", in the middle of an actual conflict, 15+ years down the road?

    I'm going to hazard a guess that were this military only, it would cost 4 to 8 times more than it currently does, but at least the org chart would be a bit more vertical, if porky.

    1. Ken Hagan Gold badge

      Re: I sure hope they don't use this for military purposes...

      I think in the middle of an actual conflict 15 years hence, there would be no point in turning it back on again before you'd shot down the several dozen "interference satellites" in low orbit beaming noise on the relevant frequencies at several thousand times the GPS signal strength.

  18. Clunking Fist

    Brexit

    Remain, they say, you'll be part of Galileo, they say. I guess this is one more reason why it is safe to leave.

  19. Anonymous Coward
    Anonymous Coward

    This really isn't helping anyone.

    You know, I think a lot of people wanted to move from GPS to Galileo due to it being a lot less boggled down in politics and better accuracy. However neither Galileo going down or the spat between ESA and EU because of politics is really convincing any manufacturer to actually adopt Galileo rather than sticking with just GPS.

    1. Spikehead

      Re: This really isn't helping anyone.

      There isn't an either-or decision. GNSS receiver chipset manufacturers have all available GNSS built into their chips.

      My phone is able to track GPS, Galileo, GLONASS and BeiDou.

  20. HKmk23

    I know which one I choose...

    Would you trust getting to your destination to the makers of Renaults and Citroens or Jeeps and Hummers?

  21. Turgut Kalfaoglu

    Are they running Windows?

    That's what it looks like..

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like