back to article State of Maine orders review of $54.6m Workday project as it alleges delivery failure and threatens cancellation

The US state of Maine is requesting an official review of its $54.6m project to renew its HR system, currently being built by Workday under a contract the state is threatening to cancel, a move which could leave the state government continuing to rely on its 30-year-old mainframe-based system. If the state ends the project, it …

  1. Mike 137 Silver badge

    Been there

    "The state discovered this functionality "could only be achieved if all security roles were removed," the letter alleged.

    Long ago I had to implement an online sensitive records sharing system to deploy an application provided by a government agency. This was in the days of Win NT4, but they'd written the application for Win 95, so it had to run with administrator privilege via the web browser. In vain I pointed out the hazards.

    But BTW, why does an HR system cost $55M to implement? Maybe because they saw you coming?

    1. Version 1.0 Silver badge

      Re: Been there

      It costs $55M because the company is making a profit - $55M would probably more than cover the cost of the state hiring programmerss to not only write, but also maintain, the system. But then some politicians friend will not be making money.

      1. Dave314159ggggdffsdds Silver badge

        Re: Been there

        Amy mention of money, hey presto the Adolph fan club appears.

    2. Plest Silver badge

      Re: Been there

      I can imagine if the gov org has an old mainframe system there's a ton of old archive data to convert and import, various utils to write. No doubt due to budget cuts they probably made all the State of Maine gov employed IT developers redundant a few years ago and that $55m is to have WorkDay hire their own developers at some horrendous hourly rate!

      1. Doctor Syntax Silver badge

        Re: Been there

        It might be worse than that. If the previous system was payroll only there might be manual records to take on-board.

      2. teknopaul

        Re: Been there

        There is clearly a market migrating 30 year old mainframe systems to the raspberry pi.

      3. David 132 Silver badge
        Coat

        Re: Been there

        I can imagine if the gov org has an old mainframe system

        I believe that at present they have a Maineframe.

        Yes, I'll get my coat.

    3. Dave314159ggggdffsdds Silver badge

      Re: Been there

      The price for a job doubles (at least) when the client is suing the first people to take the job.

      When the client is the local government buffoon squad, you don't worry about ever finishing the job, because they won't let you. You set it up so you're in profit at every point.

  2. Version 1.0 Silver badge
    IT Angle

    Replace COBOL with Python?

    $54M would probably cover the maintenance of the old system and COBOL programmers to maintain and possibly update the existing code - which seems to have been working for 30 years? Writing a new system is not a bad idea but then we're dealing to Version 1.0 of the new code (LOL) so it's going to have new bugs and issues which means a few more years of rewrites before it meets the current performance standards.

    1. Mike 16

      Re: Replace COBOL with Python?

      From acquaintance (as a bystander) with such a "legacy rewrite in hot new language" project, pretty much as soon as they had an Minimum Viable Project, they needed to re-deploy some largish fraction of staff just to keep up with shifting language specs and rapid, often undocumented changes to the pile of dependencies.

      It's a Red Queen's Race, where you have to run as fast as you can to stay in place.

      OTOH, I've also been on the other side where the customer made regular "small" (their definition, meaning "Hey, come on, you're not going to make us write a change order") changes to the deliverables. Great way to end up a contractor on a defense-related project netting below minimum wage.

    2. Michael Wojcik Silver badge

      Re: Replace COBOL with Python?

      Typically systems like this aren't just COBOL -- there will likely be some screen definitions (BMS, MFS, whatever), a fair bit of JCL for running reports, quite possibly some 3xx Assembler, maybe some PL/I or REXX, often some COBOL that's generated from a 4GL...

      Migrations are often driven primarily by the expense of leasing mainframe hardware and software, and the potential cost of upgrading to newer hardware.

      But, yes, rip-and-replace projects frequently fail, and often migrating with minimal changes is a more successful strategy. Of course, the latter pay my wages, so take that with a grain of salt.

      1. SecretSonOfHG

        Re: Replace COBOL with Python?

        You're absolutely right, migration with minimal changes is the only viable technical option wherever there is a large amount of legacy data and/or complex processes. However, that always clashes with the higher-ups view of "how is that we cannot add changes if we're spending an awful lot of money?" and usually is not the preferred approach.

        No matter how hard you try to explain that, after the "migration with minimal changes" done properly, they'll be able to introduce changes at a fraction of the cost of the current system. Or that the minimal change approach is really the only viable option because the "let's redesign our current processes to fit an imaginary best practice that no one is able to actually implement in the real world" approach always fails.

  3. Natalie Gritpants Jr

    There must be a better way to build these systems

    It was done 30 years ago probably in COBOL on a mainframe that probably has less compute power than the average PC. I think the problem word in this instance is 'comprehensive' as it tells the vendor that you don't know what you want, but you think you can afford the best.

    1. Plest Silver badge
      Facepalm

      Re: There must be a better way to build these systems

      Think of the best and worst project managers you've worked with and the stupid ideas they've had.

      Now imagine an external company saying they can handle the data conversion for 30 years worth of data held in some obscure IBM mainframe formatted store and not bothering ask the PMs how much archive data to covert and the PM simply not thinking about it, so WorkDay are going ahead with months of transformation work.

    2. Tom 7

      Re: There must be a better way to build these systems

      How much easier would it be to run the Cobol on some modern hardware than convert it. I'm fairly sure Maine hasn't had the 100,000,000 times increase in staff that we have in computing power in the last 30 years.

      1. sgp

        Re: There must be a better way to build these systems

        Agreed, but where to find staff that do cobol?

        1. Gene Cash Silver badge

          Re: There must be a better way to build these systems

          > Agreed, but where to find staff that do cobol?

          Raises hand. But it'll cost...

        2. Doctor Syntax Silver badge

          Re: There must be a better way to build these systems

          "where to find staff that do cobol?"

          Amongst the staff that IBM chucked out.

        3. teknopaul

          Re: There must be a better way to build these systems

          http://cobolcowboys.com/

        4. Michael Wojcik Silver badge

          Re: There must be a better way to build these systems

          There are places in the US which still teach COBOL.

          Any competent programmer can learn COBOL. The IT labor market is tight, but it's not that tight. And there are plenty of people who could use a better job and would be capable of learning how to maintain a big COBOL back-office application. It's not trivial, but it's not rocket science either.

    3. yoganmahew

      Re: There must be a better way to build these systems

      Even 30 years ago a mainframe would handle multiple parallel users with no loss of performance, I worked on a system that in 1991 had 20,000 simultaneous connections processing workloads. Every year they get faster and better where now the mainframe I work on supports millions of simultaneous connections.

      You can read up on some of the characteristics of what people mean by 'mainframe' here - https://www.suse.com/c/mainframe-versus-server-farm-comparison/

      Offloading tightly coupled applications that six people in the company understand well enough to describe the business flows is nightmarish. The lack of people with skills is the reason that mainframe needs to be offloaded, but the same lack of people is what makes it ludicrously hard.

      1. standbythree

        Re: There must be a better way to build these systems

        "Offloading tightly coupled applications that six people in the company understand well enough to describe the business flows is nightmarish."

        ...particularly when four of the six have already retired, and the remaining two don't agree on anything except that they don't want a new system.

  4. Deft

    Big systems IT disasters?

    I always preface my posts by saying I don't work in IT, but am interested observer and it intersects with my job as a scientist a reasonable amount.

    There seems to be a common theme of these mega ERP systems being a nightmare. Presumably the majority do go smoothly, we just never hear those stories??! It must be mature technology, or as a society we sort of know how to do these things? People must have some expectation of these projects when they get started (on both sides).

    1. RM Myers
      Unhappy

      Presumably the majority do go smoothly

      I would presume at least 2/3 rds go anything but smoothly. The number of failed ERP projects is extremely high, and that doesn't include the projects which are declared a success even though they are late, over budget, and don't deliver many of the benefits promised. Companies (and government agencies) often don't want to admit that a project was a failure, since it looks bad for everyone involved. Instead, the project is declared a success, the customers work around the deficiencies, and the attempts to fix the mess are buried in maintenance budgets or called enhancements.

      1. Michael Wojcik Silver badge

        Re: Presumably the majority do go smoothly

        If memory serves, there was a pretty big study done some years back which showed about an 80% failure rate among large IT projects.

    2. Kevin McMurtrie Silver badge

      Re: Big systems IT disasters?

      - First problem is the requirements. Legalese may sound very specific but it's obfuscation for grotesque logical flaws. It does not translate into deterministic software.

      - Second problem is that the software vendor wants to hire thousands of tech workers in a few months. There's no magic pool of thousands of skilled tech workers ready to take a low paying contract and there's definitely no pool of skilled hiring managers to interview them properly. Most of the software produced will come from Stack Overflow and expensive software/hosting vendors.

      - Third problem is integrating with an older system that also has these three problems. The data isn't going to match requirements and it might not even be valid. There's a reason that it's being replaced.

    3. Dave314159ggggdffsdds Silver badge

      Re: Big systems IT disasters?

      While one doesn't like to point out that some of the reg writers are alt-right conspiratorial loons, there is an awful lot of evidence for that notion. It's outright propaganda to push the few failures.

    4. Doctor Syntax Silver badge

      Re: Big systems IT disasters?

      ERP systems try to provide catch-all provision to a lot of individual requirements. As RDBMS products they depend on a database schema that probably started off trying to capture a "typical" business with various add-ons to pick up variations.

      For instance the core might be something that sells finished items such as a garment business. It might have to deal with quantity discounts but it's simple. Stock control is just how many of this, that & the other. A business that sells the fabric from which the garments are made may, provided it just sells whole rolls of fabric may have the same simple model but one that cuts and sells lengths of fabric has somewhat different requirements as there may be a number of rolls of the same material, all with different lengths and there may be a need for algorithms to determine the bast way to choose which of them to pick to fill an order to minimise being left with short ends. The factory which makes the garments has to deal with both types of items as it needs to account for raw materials and stock.

      Other customers might have more complex requirements such as complex bills of materials - a kit comprising several parts might be charged at a different price than the sum of the various parts sold separately. Another requirement might be a need to account for serial-numbered parts. In the early days of the mobile phone industry both were combined in that the product might be a receiver, a car kit and a variety of other bits and pieces giving the bill of material problems together with the need to keep track of the serial numbers.

      Some businesses have quite different approaches - I had a client who had a conventional ERP which was based on stock control and order processing which dealt with stationery logistics and a separate system to deal with print ordering because the print business looks at things in a quite different way and there are specific products for that.

      So an ERP system which provides for the various application fields grows to be a complex beast. That's just the start, however. Businesses will have their own ways of doing things which, rightly or wrongly, they think gives them a competitive edge. If they find themselves using the same vanilla package as their competitors they'll have to give up their quirks, in which case they have the costs of adapting their processes and lose their supposed edge, or have the package modified to suit. Again, the client I mentioned had to get custom additions to their general ERP because they needed to acquire data to be passed on to a digital print operation to produce custom printed product (I was lucky in this, the printers were also one of my clients so I could handle both sides of interface between the businesses).

      It's this last part that's the killer. If the changes needed to the business to fit the package or the changes needed to the package to fit the business are too great then failure beckons.

    5. Anonymous Coward
      Anonymous Coward

      Re: Big systems IT disasters?

      Define success. I've worked on about 12 large implementations of a certain "BIG ERP" with three letters. Not one has been declared anything but an unqualified success. Every single one of them had issues that weren't resolved for months or years, if ever. So if you're in charge of a project where the stakeholders hold you responsible for millions of dollars in fees, billings, licenses and the like, you will tell them everything is going well until the thermonuclear weapons blow up.

      The ones that kept to the original scope and did gradual rollouts did best. The other 10? Not so much. These days I mostly do upgrades, as opposed to implementations. Upgrades tend to illustrate magnify just how much technical debt and bad decisions were generated and made when implemented.

  5. Brewster's Angle Grinder Silver badge

    Fail once and, okay, it's probably the supplier. Fail twice, and you have to start to look at what the customer is doing wrong.

    1. Doctor Syntax Silver badge

      "Fail twice, and you have to start to look at what the customer is doing wrong."

      Choosing the supplier?

  6. Ken Moorhouse Silver badge

    Spec Creep

    Probably Workday have a system that will do the job, and this is what they've put forward.

    But then the bespoking requests came in, one by one.

    Ah yes, we can add this function with a bit of bespoking.

    Ah yes, we can add this..., etc.

    Presumably Workday saw the Dollar signs and kept saying Yes, but if the final requirement is that everything has to be compatible with the old system (stated some considerable time after Workday took the job on), then they would be better off writing it from scratch. But then, you're already a long way into things and it is too late to change.

    Having layers of management filtering messages through on both sides to the troops on the ground is the recipe for disaster "send 3/4d, we're going to a dance" springs to mind.

  7. HammerOn1024

    Workday

    Workday, the nemesis of rational thought. Since I too have to use this nightmare POS, obviously development to keep HR vampires in positions of control and knuckle dragging, moronic twit programs employed, I find it incredible that an organization would speak the truth about this utter pile of bovine excrement.

  8. Anonymous Coward
    Facepalm

    Fall back or Fail

    There are worse things to fall back on than a system that's working, even if it's a 30 year old mainframe running COBOL.

    Mainframes can be upgraded. COBOL can be migrated almost anywhere (there's even a FOSS version). And COBOL is easy to learn, albeit tedious to code.

    My advice would be to first define what your legacy system doesn't do and what it would cost to update it so it could do that.

  9. ecofeco Silver badge

    Workday? More like Workdon't

    IBM strikes again!

    In case anyone forgot, Workday partners closely with IBM. Very closely. See the problem?

  10. kirk_augustin@yahoo.com

    Never hire out for a proprietary system.

    When the code writers are gone, you can't modify it and you got nothing.

    Always hire you own permanent programmers, and use students to help fill in.

    If you want to use a company to write it, then use generic programs many already use, so that you an be sure the software will be maintained.

    Cobol, Fortran, Pascale, BASIC, or even Python, can all easily be rewritten in C or C++.

  11. nxnwest

    Legacy Nightmare People

    I have always found when creating a new system, several manglement types insist every last piece of data from the legacy system migrate into the new one without regard to the fact the legacy data does not conform to the new business rules and has glaring omissions. No one,I repeat NO ONE wants to be held responsible for or perform the task of curating the legacy data. Indeed, the original creators of the legacy data are long since bead, retired, left or pining for the fjords. What there original intent was can never be known. I've watched several projects collapse because of this. "It's there. It must be considered gospel."

    No thanks.

    1. CujoDeSoque

      Re: Legacy Nightmare People

      The amount of incorrect or corrupt data seems to expand at the perceived importance of it.

      It somehow seems to follow the same algorithm as Moore's Law. Funny that.

  12. CujoDeSoque

    BTDT - God help them

    About 15 years ago I worked on much the same project, get a HR/Payroll system for a large Gubmint organization off a 24/7/365 mainframe system to a shiny new ERP based system. I was brought in because the original consultants for the project were dumping more and more verticals from HR into the mix despite having little or no experience with the software. This isn't the end of the world but you should note that while a couple of buckshot pellets may not bring down a duck, it will wound them seriously enough to not withstand the next volley of pellets. Did I mention it took almost 3 years? It did.

    Their real problem was with the architecture and the "experienced" consulting company they brought in. First off, they insisted on a 32-bit architecture in Windows and an older version of the underlying database. Next off, considerable structural modifications were made to "proprietary" internal structures of the vendors software. The consulting company itself had little taste for doing anything other than managing the project and hiring their own contractors to perform actual tasks like coding, training or troubleshooting. (That's why a Big 4 (US) isn't always a great choice)

    At this point I was brought in to keep them honest and provide for a differing viewpoint. As soon as they figured that out I was excluded from meetings. This is when I first suspected that the client was powerless for reasons that extended past the contract negotiations. As the client kept rearranging the deck chairs, their ability to dictate events became even less. Another point, Gubmint IT is hopelessly outclassed when negotiating contracts with large companies. I don't know how many surcharges for work not specifically in the contract were billed but they were considerable.

    <AHEM>

    One of their most egregious system mistakes was a global search system in 32 bit windows. This despite every expert other than the consultants telling them it will be impossible to maintain and keep running. What happens is the indexes used for search have only a 2GB filesize. You have to allocate files before it grows to the limit otherwise you have to rebuild the search DB from scratch which takes a few days. Of course this is exactly what happened over and over again. I believe this was a big reason for the later upgrades.

    So after 68 million for the system they were left with a crippled system that was only online for users for 12 hours a day during the week. A far cry from the 168 hours a week with the mainframe.

    At this point I was deemed too expensive and in truth had already had a job offer to start in a month. They later tried to bring me back but I was engaged and the pay offered was about 30% less than what I'd want to go back. Finally I'd quoted them at 150% of my last rate as a starting per hour rate and hoped that would have them go away. They did.

    Did I mention the project took almost 3 years and the version they were on was no longer supported at the end? They brought in the software vendor to perform the upgrade and it cost another 15 million to upgrade and fix all the modifications done to their proprietary structures. I think they allow users on during the weekends now. So that additional 24 hours was probably worth the 15 million.

  13. Matthew "The Worst Writer on the Internet" Saroff

    Reminds me of a quote from the quote from Taavi Kotka , former CIO of Estonia

    He was asked how they wired their entire government spending only $100 million, and replied, "If you don’t use Accenture or McKinsey, you’d be amazed at what you can get done."

    Privatizing governmental IT infrastructure never saves money over the long term.

    Come to think of it, it never does over the short term either.

    See http://40yrs.blogspot.com/2019/11/tweet-of-year.html

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like