back to article Woe Canada: Rather than rise from the ashes, IBM-built C$1bn Phoenix payroll system is going down in flames

Canada is about ready to pull the plug on its IBM-built error-plagued Phoenix payroll system that has cost the nation nearly CAN$1bn ($790m). According to the latest version of the Canadian government's spending plans, the Trudeau administration will move to dump the ailing platform in the coming years, but not before blowing …

  1. Anonymous Coward
    Anonymous Coward

    Payroll no go

    IMHO, integrators should no-bid / not get involved in any payroll projects. Especially if you're trying to do this with another organizations product. It rarely goes well....

    1. Anonymous Coward
      Anonymous Coward

      Re: Payroll no go

      IBM are still in business?! god knows how.

      1. Justicesays
        Devil

        Re: Payroll no go

        I Blame Morons

  2. elDog

    Without reading the contract specs or anything, I would think their would be withholds

    for tardy/sloppy work. Or even bonuses if things came in under budget or even on time.

    It's probably fair to put some of the blame on the gov't project management (or lack thereof) but it's probably also true that the vendors should have started calling out warning signals knowing how bad it was progressing.

    Another "lesson learned" and quickly forgotten. Another chapter in the "Worst Practices" best-seller.

    1. Anonymous Coward
      Anonymous Coward

      "...their would be withholds..."

      You may have used the wrong spelling just their.

    2. Yet Another Anonymous coward Silver badge

      Re: Without reading the contract specs or anything, I would think their would be withholds

      the vendors should have started calling out warning signals knowing how bad it was progressing.

      why ? That's the entire business model for this sort of project.

      The vendors job is to deliver something that doesn't work but exactly meets all the specifications in the contract - then get paid 10x as much for "changes"

      That's why the contract is 6000pages -

      1. EarthDog

        Re: Without reading the contract specs or anything, I would think their would be withholds

        Because ethics and professionalism. This is why we will never has Software Engineering, no one cares.

    3. eldakka

      Re: Without reading the contract specs or anything, I would think their would be withholds

      > but it's probably also true that the vendors should have started calling out warning signals knowing how bad it was progressing.

      Why?

      Unless it was specified in their contract that they were to provide such advice why would they do so?

      If IBM weren't hired as consultants to do the full analysis, strategy, planning and project management, why would they provide those services for free?

    4. Anonymous Coward
      Anonymous Coward

      Re: Without reading the contract specs or anything, I would think their would be withholds

      but it's probably also true that the vendors should have started calling out warning signals knowing how bad it was progressing.

      But the people who knew that were the one that got "resourced". ((lucky thing IBM isn't in the business of making Soylent Green).

  3. Ole Juul

    embarrassing

    . . . the software is functioning as intended," a spokesperson told El Reg

    In that case one can only conclude that the relevant civil service management is incompetent and should be replaced by people with a better understanding of the system.

    1. shedied

      Re: embarrassing

      Wouldn't the incoming clueless PHB bump have to share the lift to the office suite with his predecessor carrying a carton of desktop schwag? That would be an awkward moment.

    2. Anonymous Coward
      Anonymous Coward

      Re: embarrassing

      "...software is functioning as intended..."

      As I'm perfectly fluent in Government of Canada Speak, I can translate that into the Queen's English.

      What is means is that some naive but impressive in the flesh wingnut (a.k.a. Project Management Professional) within the government's Project Management Office thought that a major software project like this could be precisely, correctly and fully "Specified" in Specification Document spanning perhaps 1874 pages. As it turns out, they were wrong. Exhibit A is the smoldering ruins of the project.

      So the Contractor gritted their teeth and did precisely what they were asked. As it happens, that Spec led to a system that didn't do what was required. The naivety by all concerned is breathtaking.

      There are ways to manage such projects that would be at least 20x cheaper, and then be brought on-line and made to work incrementally at essentially zero risk. The end result would be better, faster and vastly cheaper.

      Crikey, running the payroll for 120,000 employees is only just out of reach of a spreadsheet.

      1. Anonymous Coward
        Anonymous Coward

        Re: embarrassing

        There are ways to manage such projects that would be at least 20x cheaper, and then be brought on-line and made to work incrementally at essentially zero risk. The end result would be better, faster and vastly cheaper.

        But then they went with IBM.

        1. Anonymous Coward
          Anonymous Coward

          Re: embarrassing

          Maybe you should have bid for the work?

  4. cantankerous swineherd

    "software functioning as intended" well yes, obviously hugely profitable.

  5. thames

    The History Goes Back Further Than That

    El Reg said: "Launched in 2016, Phoenix was an IBM implementation of the Oracle PeopleSoft platform"

    It was actually started well before then, in 2009, under the previous government. The contract was awarded to IBM in 2011. It was part of an overhaul of IT systems which were consolidated as a "cost saving" measure. It went live just as the present government came to power after the election.

    None of the other projects which were initiated as part of this cost saving project were successful either. It became obvious soon after it went live that the project was in trouble. The opposition are of course blaming the government for not having pulled the plug on it immediately after being elected.

    The Auditor General's investigation didn't really address whether IBM did a good job or not. Rather it focused on whether the government's response to things going wrong was adequate.

    From what I can see, the major problem was that the project was rammed through, ready or not, prior to the election in order to claim the cost savings for electoral campaign purposes, and there was no "plan B" if "plan A" didn't come off perfectly.

    It turned out though that payroll for such a broad range of employees was much more complicated than had been envisioned back when the project was started nearly a decade ago.

    The main failing of the present government has been in persisting in trying to salvage something from the mess they inherited instead of pulling the plug on it earlier. Their response to that however is that there was no fall back position available. The Auditor General however noted that the government of Queensland had pulled the plug much sooner when they faced a similar problem.

    The project which was supposed to save $70 million per year has turned into a persistent financial black hole which will continue to cost money for years to come.

    1. as2003

      Re: The History Goes Back Further Than That

      > The Auditor General however noted that the government of Queensland had pulled the plug much sooner

      The project which was originally projected to cost A$6m was canned after the costs had ballooned to A$1.2b (yes, billion), and thousands of health workers had been paid incorrectly, the after-effects of which are still being felt years later.

      It was a monumental disaster in every regard, and if that timeline is what classifies as "much sooner", god have mercy on Canadian souls.

      1. Anonymous Coward
        Anonymous Coward

        Re: The History Goes Back Further Than That

        Oh, you know, we also blew 2B$ on a long gun registry system that never worked. Never mind that handguns are involved in many more crimes, it was unkillable due to the infamous Montreal women's massacre.

        But I've worked a bit w public sector and they seem uniquely competent at effing up software projects. It's really not JUST predatory vendors, it's in their organizational DNA.

        Nowhere more than in payroll that must have covered dozens of union regimes.

        The Feds' previous systemm was largely already PeopleSoft so extra bonus points for screwing up a system they should have known for good or bad.

        1. Robert 22

          Re: The History Goes Back Further Than That

          There would have been pressures from the government of the day to deliver unrealistic results.

          I will offer that it is likely that many of the major decisions would have been made by managers who did not understand the issues at hand - there is a widely held notion in the Cdn government that managers don't really have to understand what they are managing, Apparently the final decision to go online was made at a meeting of 45 deputy minister level officials; it is not difficult to imagine how hard it would be at such a meeting to make controversial arguments, even if one had the knowledge to support them.

          1. grumpy-old-person

            Re: The History Goes Back Further Than That

            The disease that managers only need to know how to manage and have no idea of the area they are managing pops up everywhere!

            In my 70 years I have only come across 3 genuine managers (and hundreds of fakers) themost impressive of whom told the MD of the company publicly "not to f*ck him around, as he had been f*cked around by professionals".

            Surprisingly, he was not fired or "punished" in any way.

            Was it because he was actually an asset to the company who got things done?

    2. Robert 22

      Re: The History Goes Back Further Than That

      There was the further complication of a parallel scheme to relocate the compensation staff to another geographical area (i.e., moving government jobs around for political reasons) AND reduce their overall numbers. The end results were that most of the incumbent personnel quit or were laid off (as expected) and you wound up with a buggy system operated by mostly inexperienced people.

    3. AJames

      Re: The History Goes Back Further Than That

      Not to mention that the government managers in charge of the project were given hefty incentive bonuses to finish the project on time so the government of the day could call it a success. Needless to say, the project was pronounced finished and everyone got their bonuses, only to reveal months later that it was the equivalent of the Irish builder's work on Fawlty Towers. No bonuses were canceled. Nobody took responsibility.

  6. Anonymous Coward
    Anonymous Coward

    It doesn't sound like an IBM problem at all

    It sounds like data entry problems and software issues if the software is performing as intended.

    1. EarthDog

      Re: It doesn't sound like an IBM problem at all

      obviously the users don't know how to use the software and refuse to change the reality of how payroll is done to meet the software's expectations.

    2. Anonymous Coward
      Anonymous Coward

      Re: It doesn't sound like an IBM problem at all

      You don't "data entry" payroll to that volume, you convert.

  7. EarthDog

    What was wrong with the old one?

    And why can't we move back? Duh! There's always a defacto fallback which is often less complicated than building a whole new system.

    1. Anonymous Coward
      Anonymous Coward

      Re: What was wrong with the old one?

      They restructured all the compensation specialists (it's not difficult to find work as a payroll specialist, so they weren't waiting for a call to come back to work) and cut-over to the new system without running parallel. The data base for the old system had point-in-time data that couldn't be rolled-back to because the new system was in production for months before it really became apparent how farked it is. They couldn't re-mediate. It is a text-book case of terrible organizational change and project management, mostly because of political goals defining the triple constraint, rather than good project management.

      I've followed this saga for a couple of years because a) i know Canadian public servants who were done-in by the new system (I used to be one, many years ago), b) I do enterprise project management, including HR/payroll, and c) right from the get-go IBM crushed their client, as usual, by claiming the change management process meant that they had delivered exactly what they were contracted to deliver, numerous change orders included. People really do need to get fired for hiring IBM.

  8. EarthDog

    IBM cop out

    They are the professionals. They could, for a fee, helped define the system with the Gov't. I've been through such exercises and this was the way it was done. We had people compare systems, look at workflows, and then help map data and functions. IBM's "you tell us what to do and then we take your word for it" is a cop out.

    1. Pascal Monett Silver badge

      Re: IBM cop out

      Unless IBM started discussing things and was told to shut up and do as the specs said.

      It is specified that IBM was not the project manager, and a project's success or failure is largely on the shoulders of the project manager.

      So who was the project manager ?

  9. Anonymous Coward
    Anonymous Coward

    The Real Question

    There are tens, if not hundreds of thousands of companies with very large, complex workforces. To a degree a payroll system should be an off-the-shelf item.

    I doubt we'll ever know what really went wrong, but I suspect someone in the Harper government just HAD to stick their fingers in when they should have left well enough alone.

    Hmmm, paybacks? Envelopes of cash? Been known to happen.

  10. Anonymous Coward
    IT Angle

    Remember when...

    "Nobody ever got fired for buying IBM"?

    1. Pascal Monett Silver badge

      Nobody's been fired yet.

    2. Anonymous Coward
      Anonymous Coward

      Re: Remember when...

      "Nobody ever got fired for buying IBM"?

      Now it's starting to sound like "Nobody got PAID by buying IBM".

  11. jerehada

    I can think of a few disasters like this. One where there was a massive gap between what was promised and delivered that led to termination. A friend told me the integrator had written up internally a positive case study on the same project. Big difference between promises and detail design after contract signature. Most companies unlike governments can hide these as sunk costs; CAPEX rich and blurred by EBITDA reporting.

  12. Adam 1
  13. arthoss

    i didn't know peoplesoft could do payroll. if it was such a big project why didn't they hire directly oracle to implement it? since it would have been a prestige project to peoplesoft/oracle.

    1. Blank Reg

      Given the level of sliminess of the previous government, there were probably a few pockets that would be filled by adding in IBM as an intermediary rather than dealing directly with Oracle.

    2. GruntyMcPugh Silver badge

      A salesguy once told us,...

      ... that Peoplesoft did everything we needed. At the time, (and this is about 16 years ago, maybe a little more) I was testing Microsoft MetaDirectory for a large ISP, to link all the disparate systems that managed user IDs together, as they'd grown through acquisition and so had a lot of replication. We got it working pretty well, the contractor lady we had crafting bespoke connectors knew her stuff, and soon we had Bob Jones in one payroll system mapped to the Bob Jones in Active Directory, and his car lease in the assets database, etc etc. We matched 95% of the employees across systems in the end, just had to mop up the John Smiths etc manually.

      The Peoplesoft came along, and sold themselves to HR, 'Peoplesoft can do all that!' he said. HR believed. Oh dear. Our working project got canned for a promise of 'jam tomorrow'.

  14. oneguycoding

    Good riddance

    Conservatives (formerly Tories but in Canada the Cons are now closer to the tea party) again, they're good at fucking up almost everything with their PPP initiatives (giving buckets of cash to their donors.) My wife works for the feds, ostensibly, and her pay is different every pay period. It takes 6 months to be reimbursed for any expenses or overtime, and the idiot managers are refusing to let her take overtime as vacation. Trudeau and the Liberals were between a rock and a hard place with this boondoggle, but it was time to shut the door on the pile of dung. Fingers crossed that we won't replace it with more of the same, but I'm not holding my breath.

    1. EvilGardenGnome
      Megaphone

      Re: Good riddance

      I've said it before, and I'll say it again. "Don't bother blaming the current government for the former's mistakes. Just give them time to make their own."

      Initially used in reference to Nova Scotia, Canada, but pretty universal, I think.

      1. BruceNadeau

        Re: Good riddance

        Yup people THINK this is the Liberal's BUT this WAS the Harper Government who brought this crap in. :(

  15. colinb

    "It turned out though that payroll for such a broad range of employees was much more complicated than had been envisioned back when the project was started nearly a decade ago."

    Really? turned out?

    Its been known for a looong time that large scale Payroll is difficult since there are so many micro rules and variations to apply and can change yearly with legislation.

    See the famous Chrysler C3 1993-1999 for a classic example.

    then you add PeopleSoft hahahahaha

    and then IBM hahahahaha

    Incompetence never goes out of fashion.

  16. Howard Hanek
    Childcatcher

    It's An Old Story

    .....I think Cyrus the Great conquered the Assyrian Empire after his Persians got the contract to 'modernize' the payroll system.

  17. Brian Allan 1

    What the hell, it is only taxpayer money these idiots are spending so no problem! Keep filling the trough... NEVER, NEVER let government manage any sort of project!!

  18. WatAWorld

    What do you think the sales commission on that is?

    "CAN$460m has been spent on support and fixes."

  19. Stevie

    Bah!

    From Ollie White's MRP introduction:

    "The reason people think computers will make everything faster [and better] is that the first thing ever computerized was the company payroll. Payrolls were already well understood and manual systems had evolved that described [their workflow] perfectly."

    He then goes on to illustrate why that is never true for anything else, especially in the manufacturing world.

    It would seem that the new payroll system would never have worked manually either. Spotting that before anyone puts mouse to desk is the trick (and the greater part of Ollie's talk that day).

    1. DJSpuddyLizard

      Re: Bah!

      It would seem that the new payroll system would never have worked manually either. Spotting that before anyone puts mouse to desk is the trick (and the greater part of Ollie's talk that day).

      And it does make me wonder why not write a payroll system mainly from scratch?

      Use an existing database platform, and write an application that mimics your manual payroll system.

      The complexity of the system is usually much lower when rolling your own rather than customizing or munging an "Enterprise!" solution to try to fit your need.

  20. JustWondering
    Unhappy

    Where to start

    One of the problems with this is the government had a few ideas to customize the program. One of these ideas appears to have been that instead of entering an employee's actual negotiated wage, the program would automatically enter the lowest wage possible for the position.

    When it came time to roll out the program, those working with the program said it was not ready, but the executives looking to collect their bonuses said it was.So guess what happened?

    To add to the fun, years before they had removed the pay advisors out of the various offices, reduced their numbers by more than half, and centralized them in a smallish town on the edge of nowhere.

    Fun times. These are just a few of the highlights.

  21. BruceNadeau

    Well we CAN thank the Harper government, this was THEIR project and Harper's PC's who brought us this flawed system.... How shocking lol

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like