back to article One man's mistake, missing backups and complete reboot: The tale of Europe's Galileo satellites going dark

Key details about the failure of Europe's Galileo satellite system over the summer have started to emerge – and it's not pretty. While one key official has sought to blame a single individual for the system going dark, insiders warn that organizational chaos, excessive secrecy and some unusual self-regulation is as much to …

    1. Warm Braw Silver badge

      Re: And we wonder why people want to exit the EU

      How does that solve the problem?

      You could argue that a European GPS is an unnecessary vanity project but the UK, for example, wasn't forced to take part: we thought it was a good idea at the time and it was the UK that wanted it kept to EU members.

      If you argue that it is necessary, there is no way that it could be funded either by a single country on its own (despite the UK's aspirations for a local system for local people), nor that the other European countries would accept such a system being under the sole control of another nation's government.

      It's inevitable that a project of this kind will be a multinational operation with the consequent vested interests of multiple governments wanting their piece of the action, regardless of the umbrella under which those governments gather.

      1. Tim Worstal

        Re: And we wonder why people want to exit the EU

        "there is no way that it could be funded either by a single country on its own"

        The alternative system that actually works was built by whom with the aid of which countries?

        1. Persona Silver badge

          Re: And we wonder why people want to exit the EU

          Alternative systems. As well as GPS there is GLONASS which works too as does BeiDou albeit currently restricted to Asia till next year.

        2. Adrian Midgley 1

          Re: And we wonder why people want to exit the EU

          Big ones. Similar in size to the EU.

          The USA one was built by the federal government of a grouping of 50 states.

          1. Persona Silver badge

            Re: And we wonder why people want to exit the EU

            "a federal government" and that is the difference. The EU doesn't have a federal government, yet.

            A friend of mine worked for the European Bank of Reconstruction and Development. Over a beer (well crying over a beer) he once described to me their structure and how they have to work to get things done. It was similar in ways to the structure governing Galileo, and 28 "equal" (in their eyes) bosses each had their own "creative" content to add to the party.

          2. Anonymous Coward
            Anonymous Coward

            Re: And we wonder why people want to exit the EU

            Big ones. Similar in size to the EU.

            While the US and China are of the similar "size" (economically; that's the only size that really matters) to the EU, Russia's is not even of the same order of magnitude:

            From https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal):

            Entity GDP[1012 USD]

            1. USA 21.4

            -. EU 18.7

            2. China 14.1

            3. Japan 5.2

            4. Germany 3.9

            5. India 2.9

            6. UK 2.9

            7. France 2.7

            8. Italy 1.9

            ...

            11. Russia 1.6

            [IMF values; The numbers vary by the agency, but the ranking remains similar enough]

            Note that the EU members Germany, UK, France, and Italy each have the economy substantially larger than Russia's; if Russians can afford to run their own GNSS system, there is obviously no overwhelming economic reason why either one of these three can't. And it's not like the Russians are even spending hand over fist on their space program: it costs about 1/4th of one percent of their GDP; to put that number in perspective, just the annual cost of implementing the Universal Credit programme is about 10% of the cost of the Russian space program; the total amount sunk into UC would be enough to run Russia's space program for the full year.

            1. Mage Silver badge

              Re: if Russians can afford to run their own GNSS system

              Possibly they can't afford it. The reason to have it would be both political and military.

            2. hittitezombie

              Re: And we wonder why people want to exit the EU

              .. And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980 and hardly managed to build any new hardware - remember Kliper shuttle? Got absolutely nowhere but made a lot of noise.

              Same with the Russian military hardware, they're ambitious and make a lot of noise, but their main bulk of aircraft produced right now is just upgraded versions of Soviet designs. Their advanced aircraft are nothing but techonolgy demonstrators with a handful of production samples.

              1. batfink Silver badge

                Re: And we wonder why people want to exit the EU

                And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980

                You left out the words "mostly successfully" from your sentence...

              2. tip pc Silver badge

                Re: And we wonder why people want to exit the EU

                "And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980 and hardly managed to build any new hardware"

                Like Soyuz, that is used to carry US and other nations astronauts to the ISS

                https://en.wikipedia.org/wiki/Soyuz_(spacecraft)

                1. Augie
                  Thumb Up

                  Re: And we wonder why people want to exit the EU

                  Yeah that bit often gets overlooked.. designed in the 60's.. still going while other designs have come and go.

              3. Peter2 Silver badge

                Re: And we wonder why people want to exit the EU

                .. And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980

                Yes. Their Soyuz rocket is a bit shit in terms of design in comparison to wonderful bits of kit like the space shuttle.

                However, bits of American kit like the space shuttle are so frigging expensive (>$1 billion per launch) that there was a total of 131 shuttle launches before it was grounded the last time. The currently used soyuz rocket costs something like $20-40 million per launch, and as a result has done 727 launches, 708 of were successful at a cost of $15-$30 billion. The shuttle projects 131 launches cost $209 billion.

                Honestly, on this one the Russians win with more launches and more payload delivered to orbit with less cost. There is frankly no point them developing something else; when you have a business model of producing lots of cheap rockets then your R&D goals are to optimise the production to minimise production costs, not design something fancy, new and overly sophisticated that pushes up the production cost.

            3. Ken Hagan Gold badge

              Re: And we wonder why people want to exit the EU

              "the total amount sunk into UC would be enough to run Russia's space program for the full year."

              Perhaps it was? There's eff-all evidence that it went to the intended recipients.

        3. vtcodger Silver badge

          Re: And we wonder why people want to exit the EU

          Actually, there are several alternatives cited in other answers. Or, if you have unique requirements, you can possibly design and launch your own GPS/GLOSNASS,Beidou,IRNSS (pick one) compatible satellites as Japan seems to be proposing with QZSS.

        4. codejunky Silver badge
          Coffee/keyboard

          Re: And we wonder why people want to exit the EU

          @Tim

          You literally made me laugh into my coffee! I was just thinking of that then saw your comment.

      2. Stoneshop Silver badge
        Trollface

        (despite the UK's aspirations for a local system for local people)

        That would be a single transmitter on top of Ben Nevis? Oh wait, that would be scuppered by ScotExit. Scafell Pike then, or just fire up those DECCA transmitters again.

    2. Jason Bloomberg Silver badge

      Re: And we wonder why people want to exit the EU

      ^ Do people seriously believe it would be any better if we ever do embark on our own post-brexit British Satellite Positioning System? Or any other British project for that matter.

      It will be an omnishamble clusterfuck like every other, no doubt with Crapita and all the usual suspects involved.

      The EU doesn't have a monopoly on those.

      1. Anonymous Coward
        Anonymous Coward

        Re: And we wonder why people want to exit the EU

        "It will be an omnishamble clusterfuck like every other, no doubt with BAE and all the usual suspects involved."

        FTFY - it will be classed as a defence project rather than just a run of the mill omnishambles.

        While I'm not suggesting Crapita could do a better job, they could likely provide a similar end product for significantly less. But it wouldn't be "BAE secure"...

      2. John Brown (no body) Silver badge

        Re: And we wonder why people want to exit the EU

        "British Satellite Positioning System?"

        Untied Kingdom Independent Positioning System. UKIPS.

        1. TRT Silver badge

          Re: And we wonder why people want to exit the EU

          Federated

          United

          Kingdom &

          Independant

          Nations

          Global

          Position

          Indicating

          Satellite

          System

          Unification

          Project

        2. PerlyKing Silver badge
          Go

          Backronyms ahoy!

          British Alternative Positioning System :-)

          1. Simon Harris Silver badge

            Re: Backronyms ahoy!

            That's a bit close to BAAPS.

            Appropriately the British Association of Aesthetic Plastic Surgeons.

      3. Persona Silver badge

        Re: And we wonder why people want to exit the EU

        I don't believe we need our own. With 3 competing superpowers having working systems even if we really piss of any two of them the third would fall over themselves to help us get maximum benefit from their system.

        1. eldakka Silver badge
          Coat

          Re: And we wonder why people want to exit the EU

          With PMs like Boris Johnson, the chance of pissing all three of them off, at the same time, not to mention the entire rest of the world, is pretty high in my opinion.

          1. Simon Harris Silver badge

            Re: And we wonder why people want to exit the EU

            Britain's answer to GPS will be sending Phileas Fogg up in a balloon with a compass and a megaphone to tell you where you are.

            (seriously though, I haven't forgotten Decca, which served quite well from the 1940s until satellite GPS took over).

    3. Tomato42 Silver badge

      Re: And we wonder why people want to exit the EU

      yes, no big, technologically complex project started by UK government ever run out of budged and all of them were delivered on time and with complete feature scope

      and we can read about this successes daily on this very website /s

    4. John Jennings Bronze badge

      Re: And we wonder why people want to exit the EU

      https://www.jerrypournelle.com/reports/jerryp/iron.html

      Its a factor in all Bureaucracies - I have seen it in the UK Civil Service - the EU seems more keen on it than most, though

      1. Charlie Clark Silver badge

        Re: And we wonder why people want to exit the EU

        While EU is pretty bureaucratic it is also, by comparison, remarkably efficient. Or perhaps that should be suprisingly less inefficient than many national bureaucracies.

        Setting up complex structures with lots of competing committees in order to piss money against the wall is what bureaucracies do.

        1. codejunky Silver badge
          Thumb Up

          Re: And we wonder why people want to exit the EU

          @Charlie Clark

          That is a damned well reasoned comment for why we should leave.

    5. Charlie Clark Silver badge

      Re: And we wonder why people want to exit the EU

      Prety certain nobody was thinking about this in June 2016. Since then there has been more than sufficient evidence of how shit the UK's own bureaucracy is.

    6. Peter Prof Fox

      Boris will sort it out

      Having had personal experience of this positional persiflage by not being able to find his ditch, Boris-UK will make its own navigational system. Written in Latin, calibrated in fathoms, with an origin of the London Garden Bridge it will be staffed by Etonians and be 'on-time' for fifteen years.

      1. Charlie Clark Silver badge
        Pint

        Re: Boris will sort it out

        Have one of these, sir

      2. Simon Harris Silver badge

        Re: Boris will sort it out

        Boris Johnson's personal GPS seems to have failed spectacularly in the past.

        When directing it to guide him to Parliament for the Heathrow expansion vote, somehow he ended up in Afghanistan.

    7. TRT Silver badge

      Re: And we wonder why people want to exit the EU

      Well, we would have left the EU by now... only for some reason we couldn't seem to find the road to the way out.

  1. AlgernonFlowers4

    The Road to Bohemia is paved with good intentions!

    (Galileo) Galileo.

    (Galileo) Galileo,

    Galileo Figaro

    Magnifico-o-o-o-o.

    Oh oh oh oh

    No, no, no, no, no, no, no

    Oh, mama mia, mama mia (Mama mia, let me go.)

    Beelzebub has a devil put aside for me, for me, for me.

    1. a pressbutton Silver badge

      Re: The Road to Bohemia is paved with good intentions!

      ... but you wont be able to know where you are on it.

  2. Boris the Cockroach Silver badge
    Facepalm

    Sounds a typical

    government project where the poor guys trying to run the damn thing are micromanaged by 6 dozen competing agencies all trying to assert their control of said flagship project.

    But then having the EU ask ESA to build and run it as a seperate agency to ESA, responsible only to ESA is far too simple (and doesn't employ enough middle managers/failed politicians/career minded bureaucrats)

    As for the 'one tech guy fekked up' story.... how many of us have heard the same thing from senior manglement when things have gone wrong ... especially when said senior manglement are the guys who made the decision not to have a backup system ..... (London ambulance computer dispatch system springs to mind as a fine example of this)

    1. Doctor Syntax Silver badge

      Re: Sounds a typical

      "government project where the poor guys trying to run the damn thing are micromanaged by 6 dozen competing agencies all trying to assert their control of said flagship project."

      The first thing to specify on a project like that are the locks. The ones on the doors that keep the micromanagers out.

    2. Sapient Fridge

      Re: Sounds a typical

      I was part of a team brought in to upgrade a large multi-terabyte company database. We did have multiple upgrade rehearsals but these took most of a weekend each so to save time the managers decided (against our advice) to use an old snapshot of their database rather than copying the live system, which would have added 3 hours to each rehearsal. This meant we were testing against the same old static data over and over again. When the real upgrade happened it failed because the live data had something unexpected which wasn't in the test data! Guess who got the blame?

    3. Dazed and Confused Silver badge

      Re: Sounds a typical

      As for the 'one tech guy fekked up' story.... how many of us have heard the same thing from senior manglement when things have gone wrong ... especially when said senior manglement are the guys who made the decision not to have a backup system ..... (London ambulance computer dispatch system springs to mind as a fine example of this)

      Especially when the "one guy" has probably jumping up and down for the last couple of years scream at how totally broken the system is and that we're one small step away from disaster.

      The senior mangler is now blaming them "coz it would never have died if you hadn't said it was broken"

      1. Gaius

        Re: Sounds a typical

        Sounds familiar.

        In a previous job, the thing I had been warning for months would happen, happened, to within a few days of when I predicted. Had all the email I’d sent about it as evidence.

        The moronic manager who should have acted and didn’t blamed me anyway. Said I hadn’t made the case strongly enough!

        That company no longer exists and said manager went on to have a successful career at one of the outsourcing companies that’s always appearing in El Reg. Successful for him, not any of his clients, naturally.

  3. bert hubert

    if there are any questions

    Hi - I'm the "Bert Hubert" mentioned in the article, if anyone has any questions, I'd be most pleased to answer them!

    1. GrumpenKraut Silver badge
      Pint

      Re: if there are any questions

      From me just a big thanks for publishing you report linked from the article. I assume that was a LOT of work!

      1. bert hubert

        Re: if there are any questions

        Yes, this took a lot of work - especially to gather all the relevant documents, but I am so happy many people get to read about it now! I hope this contributes to some new transparency. Because the good news is that fundamentally Galileo appears to be a good system, once the kinks get worked out.

        1. Marketing Hack Silver badge
          IT Angle

          Re: if there are any questions

          Having a precision, 24/7, multi-billion $, purportedly mission-critical electronic system into which a single person can alledgedly introduce a debilitating flaw, without adequate or accurate release testing and with no backup to restore the system to a working, previous build is one hell of a set of "kinks".

          I certainly hope that all these can be dealt with within the limitations of not having physical access to the existing satellites, but avoiding any one of the kinks I mention above is systems administration 1A stuff for a high-profile, mission-critical system. I'm not even a technologist, and I know that.

          1. Tomato42 Silver badge

            Re: if there are any questions

            > without adequate or accurate release testing and with no backup to restore the system to a working, previous build is one hell of a set of "kinks".

            as if most IT systems couldn't be brought down with one wrong person getting sudo access on a domain controller or such – yes, you can protect against maid pulling the power cable of the root DNS server, can't protect against the authoritative root server advertising empty or wrong root zone – there needs to be a single source of truth, but you need a good process/syste how it is updated, which appears to be lacking in Galileo. So, yes, that's a minor kink compared to overall complexity of the system.

            and it does look like they did have backups to restore, but because of the precision necessary, and given that the satellites are in in non-inertial reference frames, it does mean that getting it all back synchronised isn't a matter of seconds

    2. Anonymous South African Coward Silver badge

      Re: if there are any questions

      Hi Bert

      Why was the system built as a complex system, and not as a simple system?

      Just curious to hear.

      Keep up the good work.

      1. bert hubert

        Re: if there are any questions

        Thanks - well, at a guess, only people that have experienced the pain of complex systems will demand simplicity. Complexity is just like entropy - it happens, without anyone asking for it. You only get someone simple if you have battle hardened cynics in the room that keep fighting against complexity. And I guess those people did not win the day.

        1. Anonymous Coward
          Anonymous Coward

          Re: if there are any questions

          This is true of almost any collaborative project, irrespective of size (I've only been involved with small ones) or industry.

          The cynical voices asking "what if" and "how can we be sure" will be silenced by senior managers from the different quarters,who are committed to get said project completed within a promised time frame that was chosen to please their managers, the council, the board or whoever, rather than a realistic estimate. And within a budget, ditto.

          So that when it's completed there will be doors hanging off, IT missing because the figures were fudged so that, say electronic security was built in, but without enough finding budget to pay for it, the fire exit will open onto the bin areas, so that no one can get out because they promised to sort out the bins later ( this not being an important aspect, it's only bins after all, until, of course reality sets in), the cupboards won't be big enough to hold the kit, there's a water pipe where the toilet is meant to go, the green heating/cooling system will be inadequate, because no one listened when a staff member pointed out that the large doors do actually have to be opened several times a day, and so on.

      2. Roger Greenwood

        Re: if there are any questions

        Q. "Why was the system built as a complex system, and not as a simple system?"

        A. How about "mission creep"? With so many folks likely involved at so many meetings it seems plausible.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020