back to article One man's mistake, missing backups and complete reboot: The tale of Europe's Galileo satellites going dark

Key details about the failure of Europe's Galileo satellite system over the summer have started to emerge – and it's not pretty. While one key official has sought to blame a single individual for the system going dark, insiders warn that organizational chaos, excessive secrecy and some unusual self-regulation is as much to …

  1. Doctor Syntax Silver badge

    Who, me?

    Go on. You know who you are and you know you want to.

  2. Anonymous Coward
    WTF?

    Doesn't inspire confidence....

    So you have the an alphabet soup of agencies who are key stakeholders, on which the GNSS Agency seems to have dependencies. An unnamed single person, who may just be a fall guy, who can walk in and introduce a creeping, debilitating defect in the Galileo satellites. No backup to restore the past, stable configuration for the Galileo system once the defect is introduced. The addition and creation of new stakeholder agencies and offices to an already complicated org chart. A satellite system that takes days to reboot, if you are compelled to seek that option. Then to top it off you have a culture of secrecy and defensiveness that hid and is probably still hiding what actually happened.

    Its unfortunate that I can't buy stock in the GPS system.

    1. Mark 85

      Re: Doesn't inspire confidence....

      This all lead to a clusterfuck of epic proportions caused by authority without responsibility. More or less expected as a result of the culture.

      1. Paul Crawford Silver badge

        Re: Doesn't inspire confidence....

        It is worse than "authority without responsibility" as nobody is in a position of overall authority - it descends in to finger pointing and impotence because no one knows all that is happening, nor could they make any useful impact if they did.

        As for the "single mistake" at fault that is utter pish! If something the size, complexity and cost of Galileo can be brought down by a fat-fingered mistype in some console then the upper layer of managers should be sacked for not having a proper system of tested procedures (e.g. a scripted system wide restart), verified backup/snapshots, and a test system where changes are (whenever practical) run through before live deployment.

        1. Martin Gregorie

          Re: Doesn't inspire confidence....

          If something the size, complexity and cost of Galileo can be brought down by a fat-fingered mistype in some console then the upper layer of managers should be sacked for not having a proper system of tested procedures.

          That most likely explains it: they're a collection of PHBs and MBAs who don't understand what they're managing and so are incapable of organising either an effective backup system or timely recovery from a system crash. However, they will never admit this or permit the true story to be published because they know damn well they'll be blamed for allowing the situation to occur and will be sacked as a result.

          In short, they seem very similar to the upper levels of the British Civil Service.

          1. John Sager

            Re: Doesn't inspire confidence....

            It's not just typical bureaucracy though. It's a classic example of the way Europe works for grand and not-so-grand projects. Projects have to be divvied up amongst various bits of Europe to keep everyone happy. In a less politically rivalrous entity, the dividing would more likely be between projects rather than within projects to retain some semblance of practical organisation. One would hope that the EU would 'learn lessons' but I am of the view that the politics won't let that happen. I don't believe that 'ever closer union', even if it is achieved, will improve matters either.

            1. John Brown (no body) Silver badge

              Re: Doesn't inspire confidence....

              Yes, very similar to how it works in the USA. To get a Congress-critter to vote your way you have t make sure his/her state gets some of the pork. Hence bits of rockets and space shuttles being moved about the country to where they are needed for assembly.

              1. rg287 Silver badge

                Re: Doesn't inspire confidence....

                Hence bits of rockets and space shuttles being moved about the country to where they are needed for assembly.

                To be fair, SpaceX also move rockets from the factory in California to their test centre in Texas and eventually to Florida (usually, sometimes California) for launch and recovery (and then back to California for refurb).

                Transport in itself is relatively cheap. The problem is when those rockets are being built from bits supplied by 20 major Congress-selected sub-contractors in the appropriate districts, in some cases using components selected by Congress (i.e. SLS being largely cobbled together from old Shuttle components by former Shuttle suppliers). Any change request has to pass a dozen-plus desks. Far slower than wandering across the single design office in Hawthorne to the relevant department, having a chat with the team(s) your change might impact and sorting it out in half an hour over a coffee.

                1. HdwJunkie

                  Re: Doesn't inspire confidence....

                  I met a senior manager in the old US B1B program in the DOD who proudly proclaimed that 1) there were plane components built in every one of the 435 legislative districts of the US, and 2) that this fact had no effect on program efficiency.

              2. Antron Argaiv Silver badge
                Thumb Up

                Re: Doesn't inspire confidence....

                "...To get a Congress-critter to vote your way..."

                You simply have to make a suitable "campaign contribution", and/or employ some of his/her relatives in well-paid no-show jobs.

              3. NonyaDB

                Re: Doesn't inspire confidence....

                That's how they get *built* not how they are managed and maintained once in orbit.

                That comes down to exactly two agencies - NASA and [REDACTED] - the heads of which can be fired at any time for anything as massive a fuck-up as what happened to Galileo.

                And the major U.S. DoD space contractors do back-channel with each other on certain things so it's not like they're out-of-the-loop if something goes awry.

                I used to work for Raytheon and we'd regularly chat with our peers over in other defense companies, especially concerning satcoms issues. Nothing company-sensitive of course, but the channels are there to prevent things like this from happening to U.S.-based orbital assets.

          2. vtcodger Silver badge

            Re: Doesn't inspire confidence....

            I thought at first that the organization chart might be similar to xkcd's Python Environment chart https://xkcd.com/1987/ But on further analysis, I think the Python thing might be clearer and simpler.

        2. Reg Reader 1

          Re: Doesn't inspire confidence....

          @Paul Crawford "authority without responsibility", more than a little like Trump's White House and the people he has appointed, especially in the judiciary.

          I thought I should point that out as the EU/ESA is taking a lot of, well deserved, flack; but, it is not a unique experience. A better example, in this case, might be NASA and their inability to produce rockets due to the pork required by the Congress and Senate to keep meaningful employment in certain geographic areas.

      2. fnusnu

        Re: Doesn't inspire confidence....

        Just like the EU itself. Good job we're leaving.

        1. Anonymous Coward
          Anonymous Coward

          Re: Doesn't inspire confidence....

          I am starting to think that Brexit has become a subset of Godwin's Law:

          "If an online discussion (regardless of topic or scope) goes on long enough, sooner or later someone will compare someone or something to Brexit".

          1. IceC0ld

            Re: Doesn't inspire confidence....

            I am starting to think that Brexit has become a subset of Godwin's Law:

            Say what you want about Hitler, but at least he killed Hitler ...................

            1. Anonymous Coward
              Anonymous Coward

              Re: Doesn't inspire confidence....

              say what you want about Brits, but at least they kicked out the naysayers out of the Union

              1. Anonymous Coward
                Anonymous Coward

                Re: What about auto-updates?

                >> at least they kicked out the naysayers out of the Union

                could you clarify which Union? :)

                1. seven of five Silver badge

                  Re: What about auto-updates?

                  all of them, sooner or later.

                2. 's water music
                  Trollface

                  Re: What about auto-updates?

                  could you clarify which Union? :)

                  Best to nuke 'em all from orbit, just to be certain

                3. Anonymous Coward
                  Anonymous Coward

                  Re: What about auto-updates?

                  He probably mis-spelled onion.

                  (Because he's still crying.)

          2. Siberian Hamster

            Re: Doesn't inspire confidence....

            Already been suggested...

            Siberian Hamster

            Reply Icon

            Re: Time enough

            I have the inkling that there's a new internet law occurring...

            As the length of a comments section increases the probability that someone states that the problem/issue is due to or can be fixed by Brexit tends to 1.

            In deference to my favourite author I propose it is termed the PTerry Law.

            1. Evil Scot Bronze badge

              Re: Doesn't inspire confidence....

              Sir pTerry to the likes of you.

          3. Simon Harris

            Re: Doesn't inspire confidence....

            ... and the ensuing argument about Brexit is then sure to invoke Godwin's Law itself.

          4. batfink

            Re: Doesn't inspire confidence....

            AC's Law?

        2. Anonymous Coward
          Anonymous Coward

          Re: Doesn't inspire confidence....

          And you're leaving in such an orderly, organized, efficient way, you put the EU to shame.

          Don't let the door hit you on your way out. Whenever you manage to find it. And understand how those things work.

          1. rskurat

            Re: Doesn't inspire confidence....

            hint: it has a knob on it

            1. Kane
              Thumb Up

              Re: Doesn't inspire confidence....

              And it's not a Wizzard's Staff.

              1. Evil Scot Bronze badge

                Re: Doesn't inspire confidence....

                or his Potatoe

          2. Alan Brown Silver badge

            Re: Doesn't inspire confidence....

            https://www.reddit.com/r/ukpolitics/comments/b9vhfh/der_englische_patient/

    2. Anonymous Coward
      Anonymous Coward

      Re: Doesn't inspire confidence....

      An unnamed single person, .... who can walk in and introduce a creeping, debilitating defect in the Galileo satellites.

      At least that part is not surprizing at all, isn't it? Every complicated setup eventually comes down to a few, key people who actually know (or possibly think they know, without really knowing - in which case your organization is really in trouble) how the things are set up and work in practice. When one of them makes a mistake which is not caught in time, the consequences and the recovery can be "interesting".

      I believe we've all been there; I know I was.

      1. Kubla Cant

        Re: Doesn't inspire confidence....

        An unnamed single person, .... who can walk in and introduce a creeping, debilitating defect in the Galileo satellites.

        It's that rogue engineer again.

        1. Myvekk

          Re: Doesn't inspire confidence....

          It's that rogue engineer again.

          If he was from a former Eastern Bloc country, would that make him a rouge engineer?

          Or would that be if he was French, and embarrassed about what he did?

          1. bork

            Re: Doesn't inspire confidence....

            A french dancing engineer?

      2. trindflo Silver badge

        Re: Doesn't inspire confidence....

        I've been that person with the understanding to make things work on occasion and can attest at least JPL will put a stop to that. Engineers and programmers are not let anywhere near live systems: there are separate groups that compile programs, install them on machines, and operate said machines. As a developer we were only ever allowed to observe then make corrections on our development system eventually to be distributed to the groups that were allowed to touch live systems. And that was for ground-based systems; I'm sure the systems that flew had a lot more hoops to jump through.

  3. Anonymous Coward
    Anonymous Coward

    Here we go again

    I was peripherally involved in a project supplying ITER. It was completely open - anyone could bid and anynkit was acceptable

    Anyone, that is, so long as you were French and using any kit, so long as it was a certain German PLC. Yeah, right. European open organisation. Almost as bad as the Yanks.

    Anon 'cos I'm still pissed.

  4. sbt
    Meh

    You had one job...

    It seems like while costs scale with organisational size, competency is inversely proportional.

    Silver lining: another re-assuring data point that the threat of world government is overstated; the resulting bureaucracy would collapse under its own weight.

    1. Steve Kerr

      Re: You had one job...

      Unfortunately, ultimately, it will produce a black hole that will suck everything into it.

    2. Zolko Silver badge

      Re: You had one job...

      sbt : the threat of world government is overstated; the resulting bureaucracy would collapse under its own weight

      good comment, yes, this is comforting to know.

  5. Long John Brass
    FAIL

    So in short...

    Managers after massive organisational clusterfuck recommend more managers and bureaucracy?

    Seems legit. And the SOP for most organisations I've ever dealt with or been a part of.

    *sigh*... Jeezus wept.

    1. Anonymous Coward
      Anonymous Coward

      Re: So in short...

      Closely followed by a second report on how to cut ballooning costs by removing non-management staff.

      And then maybe a third recommending outsourcing to friendly vendors to ensure that the necessary skill sets are available.

      1. phuzz Silver badge

        Re: So in short...

        These friendly vendors, would they be ones in which the managers making decisions, happen to end up with non-executive directorships in, by any chance would they?

        1. Rich 11

          Re: So in short...

          These friendly vendors, would they be ones in which the managers making decisions, happen to end up with non-executive directorships in, by any chance would they?

          Those managers don't make decisions. They call in management consultants to do that for them, consultants who belong to one of the Big Four and who have a healthy rotating-door policy with regard to industry executives looking for a two-year placement position.

      2. vtcodger Silver badge

        Re: So in short...

        Closely followed by a second report on how to cut ballooning costs by removing non-management staff.

        You're suggesting they should outsource the non-management part of the effort? There ARE non-management workers somewhere in the organization structure -- right?

  6. Ben 56
    Facepalm

    It's Phil's fault!

    I told him to make sure he made the DST adjustment, but the berk just left it to go home early that evening, now 24 hours have passed we'll have to wait until end of Summertime next year.

    With the bureaucratic decision to remove DST in the EU altogether from March the whole project is royally f'ed and all the satellites will not be allowed to resync at all.

    We all hate Phil and his croissant stuffing face, he had one job in the time sync department, but all he does is bloody clock watch.

    1. Mage Silver badge
      Thumb Up

      Re: It's Phil's fault!

      Ah, that's why the EU is abolishing the twice a year clock changing!

      1. Rich 11

        Re: It's Phil's fault!

        The end-of-year leap-second added most years (assuming no earthquakes) will be the next to go.

      2. Antron Argaiv Silver badge
        Thumb Up

        Re: It's Phil's fault!

        We USAians are beginning to realise it's stupid as well. For example: Daylight "Saving" Time occurs in the summer. Think about that for a minute or two (I'll wait).

        Why are we "saving" daylight in the summer, when there's more daylight than in any other part of the year? Conversely, Standard Time is used in the winter, when there is the least amount of daylight.

        Nobody seems to be able to justify the silly thing anyway. Safety, energy conservation, farming considerations...none of them actually hold up under critical scrutiny. Time to dump it.

        1. Myvekk

          Re: It's Phil's fault!

          Nobody seems to be able to justify the silly thing anyway. Safety, energy conservation, farming considerations...none of them actually hold up under critical scrutiny. Time to dump it.

          It did when it was introduced, as it meant that people could do the things they needed to after work without needing artificial lighting, thus lowering the energy grid requirements during the war, (when it was introduced, I believe).

          These days, however, you are quite right, and it serves little functional purpose.

  7. Herby

    And we wonder why people want to exit the EU

    Bureaucracy is one of those things that just grows like a virus. Unfortunately there is no cure.

    Then again, planting a BOFH in the right place might have some results. We can only hope.

    1. Warm Braw

      Re: And we wonder why people want to exit the EU

      How does that solve the problem?

      You could argue that a European GPS is an unnecessary vanity project but the UK, for example, wasn't forced to take part: we thought it was a good idea at the time and it was the UK that wanted it kept to EU members.

      If you argue that it is necessary, there is no way that it could be funded either by a single country on its own (despite the UK's aspirations for a local system for local people), nor that the other European countries would accept such a system being under the sole control of another nation's government.

      It's inevitable that a project of this kind will be a multinational operation with the consequent vested interests of multiple governments wanting their piece of the action, regardless of the umbrella under which those governments gather.

      1. Tim Worstal

        Re: And we wonder why people want to exit the EU

        "there is no way that it could be funded either by a single country on its own"

        The alternative system that actually works was built by whom with the aid of which countries?

        1. Persona Silver badge

          Re: And we wonder why people want to exit the EU

          Alternative systems. As well as GPS there is GLONASS which works too as does BeiDou albeit currently restricted to Asia till next year.

        2. Adrian Midgley 1

          Re: And we wonder why people want to exit the EU

          Big ones. Similar in size to the EU.

          The USA one was built by the federal government of a grouping of 50 states.

          1. Persona Silver badge

            Re: And we wonder why people want to exit the EU

            "a federal government" and that is the difference. The EU doesn't have a federal government, yet.

            A friend of mine worked for the European Bank of Reconstruction and Development. Over a beer (well crying over a beer) he once described to me their structure and how they have to work to get things done. It was similar in ways to the structure governing Galileo, and 28 "equal" (in their eyes) bosses each had their own "creative" content to add to the party.

          2. Anonymous Coward
            Anonymous Coward

            Re: And we wonder why people want to exit the EU

            Big ones. Similar in size to the EU.

            While the US and China are of the similar "size" (economically; that's the only size that really matters) to the EU, Russia's is not even of the same order of magnitude:

            From https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal):

            Entity GDP[1012 USD]

            1. USA 21.4

            -. EU 18.7

            2. China 14.1

            3. Japan 5.2

            4. Germany 3.9

            5. India 2.9

            6. UK 2.9

            7. France 2.7

            8. Italy 1.9

            ...

            11. Russia 1.6

            [IMF values; The numbers vary by the agency, but the ranking remains similar enough]

            Note that the EU members Germany, UK, France, and Italy each have the economy substantially larger than Russia's; if Russians can afford to run their own GNSS system, there is obviously no overwhelming economic reason why either one of these three can't. And it's not like the Russians are even spending hand over fist on their space program: it costs about 1/4th of one percent of their GDP; to put that number in perspective, just the annual cost of implementing the Universal Credit programme is about 10% of the cost of the Russian space program; the total amount sunk into UC would be enough to run Russia's space program for the full year.

            1. Mage Silver badge

              Re: if Russians can afford to run their own GNSS system

              Possibly they can't afford it. The reason to have it would be both political and military.

            2. hittitezombie

              Re: And we wonder why people want to exit the EU

              .. And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980 and hardly managed to build any new hardware - remember Kliper shuttle? Got absolutely nowhere but made a lot of noise.

              Same with the Russian military hardware, they're ambitious and make a lot of noise, but their main bulk of aircraft produced right now is just upgraded versions of Soviet designs. Their advanced aircraft are nothing but techonolgy demonstrators with a handful of production samples.

              1. batfink

                Re: And we wonder why people want to exit the EU

                And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980

                You left out the words "mostly successfully" from your sentence...

              2. tip pc Silver badge

                Re: And we wonder why people want to exit the EU

                "And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980 and hardly managed to build any new hardware"

                Like Soyuz, that is used to carry US and other nations astronauts to the ISS

                https://en.wikipedia.org/wiki/Soyuz_(spacecraft)

                1. Augie
                  Thumb Up

                  Re: And we wonder why people want to exit the EU

                  Yeah that bit often gets overlooked.. designed in the 60's.. still going while other designs have come and go.

              3. Peter2 Silver badge

                Re: And we wonder why people want to exit the EU

                .. And that low funding is why Russians are mainly still firing up rockets designed between 1960-1980

                Yes. Their Soyuz rocket is a bit shit in terms of design in comparison to wonderful bits of kit like the space shuttle.

                However, bits of American kit like the space shuttle are so frigging expensive (>$1 billion per launch) that there was a total of 131 shuttle launches before it was grounded the last time. The currently used soyuz rocket costs something like $20-40 million per launch, and as a result has done 727 launches, 708 of were successful at a cost of $15-$30 billion. The shuttle projects 131 launches cost $209 billion.

                Honestly, on this one the Russians win with more launches and more payload delivered to orbit with less cost. There is frankly no point them developing something else; when you have a business model of producing lots of cheap rockets then your R&D goals are to optimise the production to minimise production costs, not design something fancy, new and overly sophisticated that pushes up the production cost.

            3. Ken Hagan Gold badge

              Re: And we wonder why people want to exit the EU

              "the total amount sunk into UC would be enough to run Russia's space program for the full year."

              Perhaps it was? There's eff-all evidence that it went to the intended recipients.

        3. vtcodger Silver badge

          Re: And we wonder why people want to exit the EU

          Actually, there are several alternatives cited in other answers. Or, if you have unique requirements, you can possibly design and launch your own GPS/GLOSNASS,Beidou,IRNSS (pick one) compatible satellites as Japan seems to be proposing with QZSS.

        4. codejunky Silver badge
          Coffee/keyboard

          Re: And we wonder why people want to exit the EU

          @Tim

          You literally made me laugh into my coffee! I was just thinking of that then saw your comment.

      2. Stoneshop
        Trollface

        (despite the UK's aspirations for a local system for local people)

        That would be a single transmitter on top of Ben Nevis? Oh wait, that would be scuppered by ScotExit. Scafell Pike then, or just fire up those DECCA transmitters again.

    2. Jason Bloomberg Silver badge

      Re: And we wonder why people want to exit the EU

      ^ Do people seriously believe it would be any better if we ever do embark on our own post-brexit British Satellite Positioning System? Or any other British project for that matter.

      It will be an omnishamble clusterfuck like every other, no doubt with Crapita and all the usual suspects involved.

      The EU doesn't have a monopoly on those.

      1. Anonymous Coward
        Anonymous Coward

        Re: And we wonder why people want to exit the EU

        "It will be an omnishamble clusterfuck like every other, no doubt with BAE and all the usual suspects involved."

        FTFY - it will be classed as a defence project rather than just a run of the mill omnishambles.

        While I'm not suggesting Crapita could do a better job, they could likely provide a similar end product for significantly less. But it wouldn't be "BAE secure"...

      2. John Brown (no body) Silver badge

        Re: And we wonder why people want to exit the EU

        "British Satellite Positioning System?"

        Untied Kingdom Independent Positioning System. UKIPS.

        1. TRT

          Re: And we wonder why people want to exit the EU

          Federated

          United

          Kingdom &

          Independant

          Nations

          Global

          Position

          Indicating

          Satellite

          System

          Unification

          Project

        2. PerlyKing
          Go

          Backronyms ahoy!

          British Alternative Positioning System :-)

          1. Simon Harris

            Re: Backronyms ahoy!

            That's a bit close to BAAPS.

            Appropriately the British Association of Aesthetic Plastic Surgeons.

      3. Persona Silver badge

        Re: And we wonder why people want to exit the EU

        I don't believe we need our own. With 3 competing superpowers having working systems even if we really piss of any two of them the third would fall over themselves to help us get maximum benefit from their system.

        1. eldakka
          Coat

          Re: And we wonder why people want to exit the EU

          With PMs like Boris Johnson, the chance of pissing all three of them off, at the same time, not to mention the entire rest of the world, is pretty high in my opinion.

          1. Simon Harris

            Re: And we wonder why people want to exit the EU

            Britain's answer to GPS will be sending Phileas Fogg up in a balloon with a compass and a megaphone to tell you where you are.

            (seriously though, I haven't forgotten Decca, which served quite well from the 1940s until satellite GPS took over).

    3. Tomato42

      Re: And we wonder why people want to exit the EU

      yes, no big, technologically complex project started by UK government ever run out of budged and all of them were delivered on time and with complete feature scope

      and we can read about this successes daily on this very website /s

    4. John Jennings

      Re: And we wonder why people want to exit the EU

      https://www.jerrypournelle.com/reports/jerryp/iron.html

      Its a factor in all Bureaucracies - I have seen it in the UK Civil Service - the EU seems more keen on it than most, though

      1. Charlie Clark Silver badge

        Re: And we wonder why people want to exit the EU

        While EU is pretty bureaucratic it is also, by comparison, remarkably efficient. Or perhaps that should be suprisingly less inefficient than many national bureaucracies.

        Setting up complex structures with lots of competing committees in order to piss money against the wall is what bureaucracies do.

        1. codejunky Silver badge
          Thumb Up

          Re: And we wonder why people want to exit the EU

          @Charlie Clark

          That is a damned well reasoned comment for why we should leave.

    5. Charlie Clark Silver badge

      Re: And we wonder why people want to exit the EU

      Prety certain nobody was thinking about this in June 2016. Since then there has been more than sufficient evidence of how shit the UK's own bureaucracy is.

    6. Peter Prof Fox

      Boris will sort it out

      Having had personal experience of this positional persiflage by not being able to find his ditch, Boris-UK will make its own navigational system. Written in Latin, calibrated in fathoms, with an origin of the London Garden Bridge it will be staffed by Etonians and be 'on-time' for fifteen years.

      1. Charlie Clark Silver badge
        Pint

        Re: Boris will sort it out

        Have one of these, sir

      2. Simon Harris

        Re: Boris will sort it out

        Boris Johnson's personal GPS seems to have failed spectacularly in the past.

        When directing it to guide him to Parliament for the Heathrow expansion vote, somehow he ended up in Afghanistan.

    7. TRT

      Re: And we wonder why people want to exit the EU

      Well, we would have left the EU by now... only for some reason we couldn't seem to find the road to the way out.

  8. AlgernonFlowers4

    The Road to Bohemia is paved with good intentions!

    (Galileo) Galileo.

    (Galileo) Galileo,

    Galileo Figaro

    Magnifico-o-o-o-o.

    Oh oh oh oh

    No, no, no, no, no, no, no

    Oh, mama mia, mama mia (Mama mia, let me go.)

    Beelzebub has a devil put aside for me, for me, for me.

    1. a pressbutton

      Re: The Road to Bohemia is paved with good intentions!

      ... but you wont be able to know where you are on it.

  9. Boris the Cockroach Silver badge
    Facepalm

    Sounds a typical

    government project where the poor guys trying to run the damn thing are micromanaged by 6 dozen competing agencies all trying to assert their control of said flagship project.

    But then having the EU ask ESA to build and run it as a seperate agency to ESA, responsible only to ESA is far too simple (and doesn't employ enough middle managers/failed politicians/career minded bureaucrats)

    As for the 'one tech guy fekked up' story.... how many of us have heard the same thing from senior manglement when things have gone wrong ... especially when said senior manglement are the guys who made the decision not to have a backup system ..... (London ambulance computer dispatch system springs to mind as a fine example of this)

    1. Doctor Syntax Silver badge

      Re: Sounds a typical

      "government project where the poor guys trying to run the damn thing are micromanaged by 6 dozen competing agencies all trying to assert their control of said flagship project."

      The first thing to specify on a project like that are the locks. The ones on the doors that keep the micromanagers out.

    2. Sapient Fridge

      Re: Sounds a typical

      I was part of a team brought in to upgrade a large multi-terabyte company database. We did have multiple upgrade rehearsals but these took most of a weekend each so to save time the managers decided (against our advice) to use an old snapshot of their database rather than copying the live system, which would have added 3 hours to each rehearsal. This meant we were testing against the same old static data over and over again. When the real upgrade happened it failed because the live data had something unexpected which wasn't in the test data! Guess who got the blame?

    3. Dazed and Confused

      Re: Sounds a typical

      As for the 'one tech guy fekked up' story.... how many of us have heard the same thing from senior manglement when things have gone wrong ... especially when said senior manglement are the guys who made the decision not to have a backup system ..... (London ambulance computer dispatch system springs to mind as a fine example of this)

      Especially when the "one guy" has probably jumping up and down for the last couple of years scream at how totally broken the system is and that we're one small step away from disaster.

      The senior mangler is now blaming them "coz it would never have died if you hadn't said it was broken"

      1. Gaius

        Re: Sounds a typical

        Sounds familiar.

        In a previous job, the thing I had been warning for months would happen, happened, to within a few days of when I predicted. Had all the email I’d sent about it as evidence.

        The moronic manager who should have acted and didn’t blamed me anyway. Said I hadn’t made the case strongly enough!

        That company no longer exists and said manager went on to have a successful career at one of the outsourcing companies that’s always appearing in El Reg. Successful for him, not any of his clients, naturally.

  10. bert hubert

    if there are any questions

    Hi - I'm the "Bert Hubert" mentioned in the article, if anyone has any questions, I'd be most pleased to answer them!

    1. GrumpenKraut
      Pint

      Re: if there are any questions

      From me just a big thanks for publishing you report linked from the article. I assume that was a LOT of work!

      1. bert hubert

        Re: if there are any questions

        Yes, this took a lot of work - especially to gather all the relevant documents, but I am so happy many people get to read about it now! I hope this contributes to some new transparency. Because the good news is that fundamentally Galileo appears to be a good system, once the kinks get worked out.

        1. Anonymous Coward
          IT Angle

          Re: if there are any questions

          Having a precision, 24/7, multi-billion $, purportedly mission-critical electronic system into which a single person can alledgedly introduce a debilitating flaw, without adequate or accurate release testing and with no backup to restore the system to a working, previous build is one hell of a set of "kinks".

          I certainly hope that all these can be dealt with within the limitations of not having physical access to the existing satellites, but avoiding any one of the kinks I mention above is systems administration 1A stuff for a high-profile, mission-critical system. I'm not even a technologist, and I know that.

          1. Tomato42

            Re: if there are any questions

            > without adequate or accurate release testing and with no backup to restore the system to a working, previous build is one hell of a set of "kinks".

            as if most IT systems couldn't be brought down with one wrong person getting sudo access on a domain controller or such – yes, you can protect against maid pulling the power cable of the root DNS server, can't protect against the authoritative root server advertising empty or wrong root zone – there needs to be a single source of truth, but you need a good process/syste how it is updated, which appears to be lacking in Galileo. So, yes, that's a minor kink compared to overall complexity of the system.

            and it does look like they did have backups to restore, but because of the precision necessary, and given that the satellites are in in non-inertial reference frames, it does mean that getting it all back synchronised isn't a matter of seconds

    2. Anonymous South African Coward Silver badge

      Re: if there are any questions

      Hi Bert

      Why was the system built as a complex system, and not as a simple system?

      Just curious to hear.

      Keep up the good work.

      1. bert hubert

        Re: if there are any questions

        Thanks - well, at a guess, only people that have experienced the pain of complex systems will demand simplicity. Complexity is just like entropy - it happens, without anyone asking for it. You only get someone simple if you have battle hardened cynics in the room that keep fighting against complexity. And I guess those people did not win the day.

        1. Anonymous Coward
          Anonymous Coward

          Re: if there are any questions

          This is true of almost any collaborative project, irrespective of size (I've only been involved with small ones) or industry.

          The cynical voices asking "what if" and "how can we be sure" will be silenced by senior managers from the different quarters,who are committed to get said project completed within a promised time frame that was chosen to please their managers, the council, the board or whoever, rather than a realistic estimate. And within a budget, ditto.

          So that when it's completed there will be doors hanging off, IT missing because the figures were fudged so that, say electronic security was built in, but without enough finding budget to pay for it, the fire exit will open onto the bin areas, so that no one can get out because they promised to sort out the bins later ( this not being an important aspect, it's only bins after all, until, of course reality sets in), the cupboards won't be big enough to hold the kit, there's a water pipe where the toilet is meant to go, the green heating/cooling system will be inadequate, because no one listened when a staff member pointed out that the large doors do actually have to be opened several times a day, and so on.

      2. Roger Greenwood

        Re: if there are any questions

        Q. "Why was the system built as a complex system, and not as a simple system?"

        A. How about "mission creep"? With so many folks likely involved at so many meetings it seems plausible.

    3. maffski

      Re: if there are any questions

      Thanks for your work.

      Do they not have anything like a 'live' and 'beta' data stream so they can test their updates on the system before publishing them?

  11. J4

    Chortling sound coming over the ocean

    US DoD - the reason we don't want to release the hi-res GPS feed is because we've spent a lot of time and money getting it to work and we don't see why everyone else should get it for free

    EU - Pah. It's just clocks and maths. How hard can it be ?

    US - Be my guest.

    1. Anonymous South African Coward Silver badge

      Re: Chortling sound coming over the ocean

      Plus a poorly tested backup method, and no rollback or fallback plan.

    2. Ken Hagan Gold badge

      Re: Chortling sound coming over the ocean

      "because we've spent a lot of time and money getting it to work"

      It is, of course, perfectly possible that this statement is entirely true. Specifically, we don't know how many times the high-precision parts of the US system fell over in their early stages, because we didn't have access to them. But it is only clocks and maths, so I expect Europe will get there eventually, just like the US did.

  12. herman Silver badge

    Hmm, Navstar is not immune to cock-ups either, but it usually resolves quite quickly.

  13. Will Godfrey Silver badge
    FAIL

    Oh yeah?

    That "Lone employee" sure gets around, eh.

    1. a pressbutton

      Re: Oh yeah?

      Last seen on a grassy knoll?

      1. Clunking Fist

        Re: Oh yeah?

        Last seen entering a federal penitentiary in New York in a blue dress.

    2. It's just me

      Re: Oh yeah?

      Probably the same rogue engineer that worked for some auto manufacturers.

      1. Anonymous Coward
        Anonymous Coward

        Re: Oh yeah?

        His great-great grandfather was responsible for buoyancy control and passenger evacuation systems on the Titanic.

      2. A. Coatsworth Silver badge
        Holmes

        Re: Oh yeah?

        Yup, he got this gig after leaving Google. Back in 2012 he worked in the comms systems for street view cars.

        He got massive good recommendations from his PHB at Google, too.

      3. This post has been deleted by its author

    3. Ken Hagan Gold badge

      Re: Oh yeah?

      That's because the lone employee is probably an incompetent manager who has jumped from one project to another whilst always pinning the blame on whoever was closest to the cock-up rather than whoever was responsible for it.

      See also https://en.wikipedia.org/wiki/Proximate_and_ultimate_causation

  14. The Average Joe

    Windows XP! DOH! they should have upgraded to a newer OS!

    so the Windows XP is out of support, better send a helpdesk guy up there to repair the OS. It is always cheaper to use windows for mission critical processes, just look at the factory floor...

  15. Anonymous Coward
    Anonymous Coward

    It's a glorious metaphor.

    In short, the EU is a classic European venture: a great idea with (some) talented people that has turned into a bureaucratic mess in which no one wants to take the blame for problems caused by unnecessary organizational complexity.

    BETTER OFF OUT!

    1. Dan 55 Silver badge

      Re: It's a glorious metaphor.

      When has a simple answer for a complicated question ever worked?

      You have no idea what to do next after leaving and if you think you do it can be shot down in 57 ways in the first five minutes.

      But carry on, I guess learning the hard way will work if nothing else does.

    2. Anonymous Coward
      Anonymous Coward

      Re: It's a glorious metaphor.

      >> in which no one wants to take the blame for problems caused by unnecessary organizational complexity

      Hmm, that smells a lot like how the Brits do things as well... From recent memory..

      Unified Credit Benefits, National Rail and Brexit itself.

      Throwing stones and glasshouses?

    3. Anonymous Coward
      Anonymous Coward

      Re: It's a glorious metaphor.

      In short, the UK is a classic British venture: a great idea with (some) talented people that has turned into a bureaucratic mess in which no one wants to take the blame for problems caused by unnecessary organizational complexity.

      BETTER OFF OUT!

      Sounds like we need UKxit by the same logic..

      Discuss.

    4. batfink

      Re: It's a glorious metaphor.

      ...so the UK can have its own proper British fuck-ups.

      1. GrapeBunch

        Re: It's a glorious metaphor.

        The story is an abject example of how the lessons of this day, November 11th, have only partly been learned. Shouldn't we borrow from the Americans and make it "three strikes and you're out" for political meddling?

  16. CAPS LOCK

    Handily eLoRaN base stations always know...

    ... where they are.

  17. Filippo Silver badge

    Yes, this is typical EU. But, all things considered, the fact that "typical Europe" now means "coming up with awesome shared projects and then fucking them up due to political bickering" is a massive improvement. I mean, "typical Europe" before the EU used to mean "war". Had been since literally forever. If it takes another hundred years to get a EU that really works, the process will still look pretty good in a history book.

    1. Laurent 💔 🐧

      The best comment so far for this article, without contest.

    2. David Beck

      Ask a Bosnian how that's going.

      1. Anonymous Coward
        Anonymous Coward

        QED

        Last I heard Bosnia is not yet a fully fledged member of the EU.

        1. IDoNotThinkSo

          Re: QED

          But it is in Europe. The EU did nothing. It took NATO to sort it out.

  18. Paul Johnson 1
    FAIL

    Where were the risk assessment and reversion procedure?

    I used to work in a related area where high accuracy, uptime and reliability are critical. Anything done on the live system had to be rehearsed on the test system first. It had to be done according to a written procedure which had been reviewed and approved beforehand. Part of the review was a risk assessment (i.e. ask "what could possibly go wrong?"). There also had to be a reversion procedure (i.e. "We screwed up; put it back the way it was").

    We did have occasional outages, including one particularly embarrassing incident where "Do routine thing" was next to "Shut down the system" on a menu. But they were rare, and both management and engineering took justifiable pride in that.

    One thing we would *NEVER* do is blame the engineer holding the mouse (short of actual malice). If they made a mistake, its because the system upstream of them enabled that mistake and set them up to fail. You don't shoot the engineer, you fix the system.

    1. Anonymous Coward
      Anonymous Coward

      Re: Where were the risk assessment and reversion procedure?

      @Paul Johnson 1

      In an ideal world I should be down voting you for stating the bleedin' obvious, but the reality is I have to upvote you for banging the drum for automation and repeatability.

      I hope everyone here is scripting their changes - not just following a written procedure - although that in itself is better than doing it ad-hoc. If not you need to start asking yourself why not. Not working for ESA is not a valid reason.

  19. dnicholas
    Paris Hilton

    We've all been there...

    ...(over)confidently bashing a command in the ...(wrong)terminal window

  20. Anonymous Coward
    Anonymous Coward

    Amazed, trully

    "The EU is planning to create a new European Space Agency, called EUSA, which will largely be a renaming of the existing Global Navigation Satellite System agency. Yet another space entity, GSA, will become the EU Agency for the Space Program, and the EC will soon have a new director general position in charge of the “defense industry and space.” In short, there are a lot of political maneuverings and that is causing all kinds of other problems."

    Really ????? WTF ? Why ? Why creating issues whereas a very good company, called ESA, had already so many successes ? Sure, this mishap is embarrassing, but everyone went there, I'm sure the US had some glitches as well, back in the time, but since it was new and military restricted, no-one got the message.

    I'm baffled and sad. This is all gonna destroy what was done ...

    As for ESA, I'm sure they'll figure out how to setup rollbacks processes for this. It is surely not simple.

    1. Mage Silver badge

      Re: The EU is planning to create a new European Space Agency

      Because, actually the ESA isn't an EU organisation. Not all EU countries are in the ESA, though the the EU as a whole contributes. Also there are three (I think) non-EU countries in the ESA.

      It's explained accurately in this otherwise fictional account of Aliens visiting earth. "The Solar Alliance" by Ray McCarthy. The Aliens quickly figure out that the UN is mostly a talking shop. The ESA and EU take them longer to understand. A new Earth agency is setup by the countries with space launch facilities, or willing to make a major commitment, but who is REALLY running the Solar Alliance, with its HQ in Shannon Ireland, an Israeli Finance director and an ex KGB/SVR Russian in charge of Security?

  21. amanfromMars 1 Silver badge

    Hiding in plain sight and disguised as another Global Navigation Satellite System

    Is Galileo really just a private mesh communication system outside of wacky wild west third party command and control?

    1. David Shaw

      Re: Hiding in plain sight and disguised as another Global Navigation Satellite System

      No, amfM1, I don't think so. though I don't know how random the pseudo codes are, so in stego theory, yes, but that's only a downlink - so no mesh.

      I think it (Galileo) primarily came from the fact that L1 was unreliable due SPOF, serious CIP required a backup, "A-GPS," where A should be as many things as possible. Why not build one's own!

      And there will always remain a serious question mark over your statement outside of ...3P C&C, to which I just answer, really?

  22. beast666

    In short, the EU project is a classic European misadventure: a tyrannical idea with corrupt people that has turned into a bureaucratic mess in which no one wants to take the blame for problems caused by unnecessary organizational complexity.

    1. Intractable Potsherd

      In short, the UK project is a classic English misadventure: a tyrannical idea with corrupt people that has turned into a bureaucratic mess in which no one wants to take the blame for problems caused by unnecessary organizational complexity.

      FTFY

      1. Oh Matron!

        Those who govern are, ironically, those least suitable to do so.

  23. smudge
    Trollface

    A gift for the Brexiteers

    But far too complicated for them to understand :)

    Wasn't the UK going to build and launch its own system as soon as we left? I suppose it's all Parliament's fault that that hasn't happened yet. Still arguing about whether the clocks should be set to 1953, 1940, or 1887.

  24. fuserly

    Why does the author assume that it was a man?

    1. rg287 Silver badge
      Joke

      Why does the author assume that it was a man?

      Because a woman would have read the instruction manual, or stopped and asked for directions when she became unsure - instead of insisting that she knew where she was and "it's just around here somewhere in one of these menus".

  25. Anonymous Coward
    Anonymous Coward

    Who uses it?

    Who uses Galileo GPS? Genuine question - I realise that the original GPS system was USA based and that this was to reduce reliance on the Yanks (don't Russia have their own system?), but I don't I have ever seen a GPS or smart phone that has a "Compatible with Galileo GPS!" type sticker on it.

    The IT Crowd reference -

    So it was fixed by switching it off and on again? Was it running on Vista....?

    1. rg287 Silver badge

      Re: Who uses it?

      It's built into most GNSS chips as standard now. There are over 150 Android smartphones on the market which can receive Galileo (in addition to GPS and GLONASS) and it's been available on the iPhone since the X/8/8Plus were released in 2017.

      There's no sticker, it's just baked into the chips by default, same as your handset being quad-band and being able to connect to networks using 2/3/4G in multiple countries, allowing for different regional spectrum allocations.

  26. -tim
    Boffin

    The lovely problem of a complex problem

    One of the great issues of any GPS system is you don't know exactly where anything is at any given time. You know were it was and you can predict where it is going to be but its only a very good guess. The satellites are being tracked but there is a delay between signal tracking them and getting the info into a computer half a continent away. The weather is going to delay signals in odd ways that usually allows compensation using different frequencies but only some times. The clocks are ticking away with some very high degree of accuracy yet subject to all the oddness that relativity in a gravity well has to offer. The ground stations are busy floating on land that is cruising in different directions at a few cm a year which was considered slow and stable until better GPS systems showed drift rate can vary over the months yet maintain a rock solid annual average. Yet in all that chaos, my phone still can display a map of where it is down to a few meters. I guess this problem demonstrated just how related the chaos of all guesses can be.

  27. Zarno
    Mushroom

    I sure hope they don't use this for military purposes...

    Can you imagine if the official response was "turn it off and on again", in the middle of an actual conflict, 15+ years down the road?

    I'm going to hazard a guess that were this military only, it would cost 4 to 8 times more than it currently does, but at least the org chart would be a bit more vertical, if porky.

    1. Ken Hagan Gold badge

      Re: I sure hope they don't use this for military purposes...

      I think in the middle of an actual conflict 15 years hence, there would be no point in turning it back on again before you'd shot down the several dozen "interference satellites" in low orbit beaming noise on the relevant frequencies at several thousand times the GPS signal strength.

  28. Clunking Fist

    Brexit

    Remain, they say, you'll be part of Galileo, they say. I guess this is one more reason why it is safe to leave.

  29. Anonymous Coward
    Anonymous Coward

    This really isn't helping anyone.

    You know, I think a lot of people wanted to move from GPS to Galileo due to it being a lot less boggled down in politics and better accuracy. However neither Galileo going down or the spat between ESA and EU because of politics is really convincing any manufacturer to actually adopt Galileo rather than sticking with just GPS.

    1. Spikehead

      Re: This really isn't helping anyone.

      There isn't an either-or decision. GNSS receiver chipset manufacturers have all available GNSS built into their chips.

      My phone is able to track GPS, Galileo, GLONASS and BeiDou.

  30. HKmk23

    I know which one I choose...

    Would you trust getting to your destination to the makers of Renaults and Citroens or Jeeps and Hummers?

  31. Turgut Kalfaoglu

    Are they running Windows?

    That's what it looks like..

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like