back to article Flame Canada, flame Canada ... Botched govt payroll computers spew smoke ahead of probe

The Ottawa data center housing Phoenix – the Canadian government's bungled payroll system for federal workers – was shut down on Wednesday after smoke was detected inside. The Shared Services Canada server warehouse also housed computers handling government email, as well as some government websites, which were switched off, …

  1. Bob Dole (tm)
    Flame

    ¯\_(ツ)_/¯

    Did someone take Milton's Swingline stapler?

  2. Jim Mitchell
    WTF?

    "department managers were allowed to view information, including names"

    I'm not sure that being able to see the names of employees in other departments is a security issue.

    1. Mark 85

      between March and July of 2015: employee names, pay, and personal record identifier (PRI) codes were left encrypted on a test system.

      Same for this one... encrypted on a test system... who had access and the keys?

  3. Sureo

    Our tax money at work.

    Incidentally, when are they going to consolidate the income tax collection department?

    1. Anonymous Coward
      Stop

      Re: Our tax money at work.

      Careful now, once that system is up, you could mistakenly have your Canadian citizenship revoked and be classified as an American. :)

  4. John Geek

    yeah, parliamentary hearings outta fix their IT problems, real good.

    1. thames

      What will be even funnier is that the politicians who are responsible for this fiasco are now sitting on the opposition benches and will be harumphing about it at length when parliament resumes sitting.

      What isn't funny is the situations that the employees face. Some of them haven't been paid in months. The situation is positively third world.

  5. boozy

    payroll systems seem to be IBM's specialty

    or at least, broken payroll systems seem to be.

    1. Anonymous Coward
      Anonymous Coward

      Re: payroll systems seem to be IBM's specialty

      There was a time when hiring IBM meant the job would get done and be of acceptable quality. Sure it cost an arm, a leg, and your first born, but it got done.

      I bet the it's not gotten cheaper, they just have left out the part about getting it done and doing a decent job. Their profit margins are probably higher as well given all the offshoring of consulting work they've been doing.

    2. JeffyPoooh
      Pint

      Re: payroll systems seem to be IBM's specialty

      Payroll is hard.

      Keeping track of all those integers and strings... Crikey.

    3. Anonymous Coward
      Anonymous Coward

      Re: payroll systems seem to be IBM's specialty

      It does stand for It's Better Manually.

  6. Brock Knudsen
    Flame

    Have we tried....

    Maybe some engineer figured that since the system was named Phoenix it would rise renewed from the flames...

  7. Alistair
    Pint

    @Brock

    My immediate assessment was that someone had determined the reason for the payroll issues was that there had been no ashes from which Phoenix could arise.

    Either great minds, or we're both punch drunk from too many hours at the keyboard.

    1. allthecoolshortnamesweretaken

      Re: @Brock

      QWERTYtis?

  8. Anonymous Coward
    Anonymous Coward

    Shit Service Centres

    HRMess

    It is always thus.

  9. Anonymous Coward
    Anonymous Coward

    shared services canada

    Government IT in Canada has been in shambles (well, much bigger shambles than would be normal for any government IT) ever since it was consolidated in a single super-department a few years ago. The idea was of course to save money while improving services and securing govt data.

    As usual, the exact opposite happened - the costs skyrocketed (the new SSC has been given the old IT budgets from tbe departments; since nothing gets done, the departments have to spend more money to create a parallel shadow IT to keep things going); the major infrastructure and datacentres are not been replaced and are gradually falling apart (and the departments are not allowed to replace them, since this is the SSC turf now); the morale is literally awful (well, if I were told that my salary will remain fixed for the next 15 years - not even inflation increases - because my old department was paying me "too much" according to the new IT payscale, I won't be too happy either; this happened to several people I used to know). As the result, nearly all competent IT people and managers who were not close to retirement have jumped the ship, and there is almost nobody left who either knows how things work, or can fix them when they break.

    Phoenix saga is only the beginning; many more IT-related project failures will occur in Ottawa in the next few years as chickens are coming home to roost.

    Obviously anon, since I still know way too many people in that system.

  10. Neoc

    "Launched in February after a joint development project with IBM, the Phoenix system has been beset by a number of technical issues"

    Sounds a lot like the Queensland Health Payroll system debacle... also featuring IBM.

  11. Anonymous Coward
    Anonymous Coward

    Cannot code for stupid

    The issue is not the system, the issue is certain government departments took the transition / cut-over seriously, while others did not. Those that took it seriously, meaning they actually attended the training sessions, have experienced ZERO issues with the system.

  12. Doctor Syntax Silver badge

    Surely payroll has been a solved problem for decades. Anyone proposing a new payroll system should be regarded with the deepest suspicion.

    1. Anonymous Coward
      Anonymous Coward

      Anyone proposing a new payroll system should be regarded with the deepest suspicion.

      The old system was working just fine. However, it required some 3,000 pay advisers (about 1 per 100 employees), most of whom were stationed locally and knew both the people and the department for which they administered the payroll.

      The mail goal of phoenix was to replace all local pay advisers with less than 400 mostly newly-trained staff (many of whom used to administer the long-gun registry, abolished by the conservative government, and had no prior experience with payroll or personnel tasks). Naturally, when the inevitable teething problems arose, they were overwhelmed. Since nobody bothered to add more staff or resources for the first six months after the problems started, by now they are completely snowed under.

      1. Anonymous Coward
        Anonymous Coward

        Re: Anyone proposing a new payroll system should be regarded with the deepest suspicion.

        I know this is a really radical and far out idea, but companies like IBM are supposed to be experts in these fields with many years of experience of what works.

        Do you think maybe they should be issuing contracts which specify what the buyer will be doing? Like ensuring that managers attend training sessions and perhaps pass tests, specifying the number and grade of staff to be managing the transition, and so on?

        The attitude that people are outside the scope of the project should have died about the time Alan Turing was experimenting with state machines.

        1. Bob Dole (tm)

          Re: Anyone proposing a new payroll system should be regarded with the deepest suspicion.

          I think there are a few problems with the idea that companies like IBM are "experts".

          IBM may be treated as a single entity for tax purposes etc, however it's really the people that's the important bit, and people come and go. Someone can work at IBM, as a programmer or project manager, and literally never have worked on a payroll or accounting system before. For all I know the entire team on this never tackled such a system. So, obviously that "expert" status means nothing unless the same people are doing the work.

          The next problem is one of institutional inertia. Even if IBM had used people that know how to make accounting and payroll work - if the people on the Canadian government side insist on doing things like they have before - only in a new system - then there are going to be failures. I've run into this exact situation many many times. We sell a software package that to get the most use out of it requires that the buyer throws away previous attitudes and processes and conforms to a different way of doing things. This causes a lot of issues because people hate change.

          I don't know the details of what actually caused the failure in this situation, but I'd be willing to be that the leading contributor is one of the above. Either IBM has a bunch of jr level (ie: cheap) resources on this that don't know what they are doing OR the Canadians made a large number of illogical special change requests to try and fit a square peg in a round hole.

      2. GrapeBunch

        Re: Anyone proposing a new payroll system should be regarded with the deepest suspicion.

        One payroll advisor per 100 employees? So each advisor spent two full working days per year "advising" each employee, on mean average. In anywhere but Canada that number would be astounding, considering that everything is deterministic, i.e., government employees don't negotiate their wage rates or pensions.

        The long gun registry also did not shine, financially. The government predicted that registration fees would pretty much cover the cost of the registry, but in the first ten years of said registry, the fees were about as expected, and fell $860 million short of breaking even.

        http://www.cbc.ca/news/canada/scrapping-the-long-gun-registry-some-relevant-numbers-1.861912

        My tentative conclusion is that the long gun registry employees were not as good at time management as the payroll advisors whom they replaced. "No, dearie, I know that your former payroll advisor used to give you a full mani-pedi every year, but those days are gone. It's a manicure or a pedicure in alternate years, or a full mani-pedi every third year." That didn't happen because Canadians don't like to say "No".

  13. et tu, brute?
    Joke

    Phoenix...

    ...rises from the ashes, so surely smoke is a good sign, as fire is needed to produce ashes?

    1. allthecoolshortnamesweretaken

      Re: Phoenix...

      Not neccessarily - there also was the other Phoenix.

  14. allthecoolshortnamesweretaken

    Makes you wonder what kind of payroll system IBM uses internally, doesn't it?

    1. Anonymous Coward
      Anonymous Coward

      "Makes you wonder what kind of payroll system IBM uses internally, doesn't it?"

      SAP, presumably.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like