How many screwed up IT projects suffered from similar issues before being put of their misery?
Canada's top auditor has issued a scathing postmortem report on Phoenix: the nation's disastrous attempt to overhaul a key government IT system. In its comprehensive dossier outlining the nine-year-long fiasco to replace the software that handled the country's payroll system for government workers, the Office of the Auditor …
The problem with these sort of "replace something we've been using for decades" projects is that no one really knows what the current system does. It has been tweaked and tweaked over the years as issues were found and worked around, but they weren't properly documented as to why they were done.
When you have people spec'ing the new system, they what it does on a day to day basis and the requirements capture that, but the thousand exceptions that have been built up over the years aren't and that's where they always fall down.
I have no idea how to solve that problem, and I don't see how it can be solved to be honest. All you can do is develop the new system and live with the pain while you build up the same capabilities to handle all those corner cases - and make damn sure you document them well so the next time you rip and replace your requirements capture 99.98% of what needs to be done instead of only 98%.
As someone going through this again (I'm on the 50 year old application side), part of the problem is the unwillingness to actually pay for the expertise in the old system to oversee the functional design of the new system. It's less that the old lags who keep the old system running and up to date aren't willing, more that the minion-driven-development in the outsourcer doesn't want to pay the cost. Meanwhile, the old lags watch on with horror as 50 years of mistakes are repeated...
Good points, but it is not clear that this is the issue in this case.
It seems more like a lot of de-scoping anything that might be "a bit hard" as the budget and delivery deadlines were enforced as if set in stone.
That way you get the delivery party while the rest of the organisation wonders how they can operate and work 24x7 to keep things moving via workarounds.
This is not unusual. It is more likely where projects are primarily run by third party integrators too. They have no incentive other than the contract which will largely define budget and timescale rather than any useful attributes of the deployed solution.
I have experienced the above multiple times now in my career across different scale projects. It usually stems from the translation of guesses into a project plan with little or no proper analysis phase (which can look expensive) to more effectively define and scope what actually is supposed to be delivered.
I was once involved in specing a computer system for a department of about 15 people, all in one large office. We got everyone together to sort out what the new system would do to replace the current method of working.
It emerged from that meeting that nobody in the office actually knew exactly how the current system worked.
is that no one really knows what the current system does
And the supplier knows this.
In spite of 6000 page specifications in this case - there will be things missed.
The suppliers bid to lose money on the original contract and then make the real profit on billing vast amounts for any inevitable changes. For extra bonuses you implement things in the original spec wrongly - but legally in keeping with the spec - and hope for big $$$$$ to fix it
Am I the only one to suspect that a 6,000 page specification is too long to be a useful specification document?
Would anyone read, comprehend, and remember the whole thing? Would any group of people doing 'specification level' analysis be able to discuss it adequately in a week?
A detailed design document, possibly, but even that seems overly complicated for a payroll system, even one that can do retroactive payments.
So what is the procedure for paying time in lieu to a federal employee of union A required to be on standby on a provincial holiday to supervise an employee of Union B completing a task at a site that qualifies for remote living allowance?
There are literally 1000s of these - miss one out and IBM will happily charge you $10,000 a day to document, specify and implement the fix.
How many screwed up IT projects suffered from similar issues before being put of their misery?
There have been numerous books listing quite a few. I know from personal experience of at least 8 in my 20 year career in current, and in almost all the cases I know of, not being utterly clear on the objectives and requirements has killed each and every one of them by running it out to the middle of the span before there was even a dream of a bridge being built.
picture Wiley E in that moment, just after he's passed the cliff edge, before he glances down
Icon is the inevitable result.
Good to know.
And maybe one day the government department of some government will actually apply those lessons before it causes such a massive f**k up.
Basically they chose "On time and budget" over "does the whole job."
If storing and processing the information correctly is not a key requirement, guess what, I can guarantee I can deliver your system.
It won't actually be very good, but it will be on time.
Hmmm, at $850m and counting, "on budget" wasn't exactly a high-priority priority they achieved either ;-)
They should have tested w at least 2 major departments, preferably with 2 different collective agreements/unions underpinning payrolls. This project was supposed to consolidate multiple departments, so even a 1 dept successful pilot might not have unearthed challenges w merging many departments.
Feds already run PeopleSoft payroll, so messing this up was not a given from the start.
Although for political reasons part of the plan was to shut down the previous dept on mass just before the new system was rolled out - so that no long term employees would be carried forward.
Can any 5 year olds see a flaw in this plan ?
I just worked on a payroll implementation and we had exactly the same issues... management emphasising schedule and budget over function. Management trying to get us to bypass testing for the sake of the schedule and really pushing back when issues were raised. The last thing that the project manager wanted to hear was that something didn't work as expected and the response was always "Can we fix it after we go live?" The big issue still is "retroactive actions". The project manager had us go to senior management to request that they completely ban backdated changes (in this organisation that is never going to happen) because the payroll system couldn't handle it. Even now there are issues with some backdated actions that require manual calculations (when we find out that it has happened) to have people paid correctly.
I'll never forget the statement from a colleague on a payroll project.
"It's OK, only six people didn't get paid..."
I didn't really know what to say to that, but looking over these comments it is apparently normal business practise, I must have missed that bit of my training somewhere.
Looking ahead, the auditor general is recommending a set of new requirements for how the Canadian government should handle major IT projects, including the use of internal audits and oversight groups that would help keep a close eye on how projects are being rolled out.
So you have to have greater, more adult adults to oversee the supposed adults in charge of the project?
Why won't anyone read the classics? Or apply lessons learnt during the last 70 years?
Why, I have a 620-page manual on my desk full off good, classic advice: "Software Engineering
in the Systems Context - Addressing Frontiers, Practice and Education" (Ivar Jacobson, Harold “Bud” Lawson, Eds).
It's frankly required reading, but of course when you put this in front of the PM or The Decider, he/she will immediately ask "Do you expect me to read this?". Upon which the correct comeback is "No, I expect you to fail". "We need to have this done by X" shall be answered with "You won't". "Failure is not an option" will lead to "It is now". "I have an MBA" can only be countered with "LOL".
From said book:
1. The ignorant do not know known good software engineering practice.
2. The neglectful may know known good software engineering practice but lack the respect and discipline to apply it rigorously as intended.
3. Those who intentionally defer work or take expedient shortcuts are arrogant practitioners who believe their superior skills will permit them to dodge the bullet of consequences for their action.
I often hear that PeopleSoft is a hazard zone of well-remunerated dark consultancery that delects to remora onto deep-pocketed, possibly taxpayer-funded organizations that are too big to fail.
IBM was chosen as the contractor to deliver a system based on Oracle's PeopleSoft system.
It could be true.
same outsourceries in charge also.
The PHBs at IBM wanted a cookie-cutter business model to cheaply replicate their past "successes", albeit at an unchanged cost to customers, and that's what they now have.
IBM: Guaranteed failure, anywhere, any time, any place.
And that was the key problem: there were big management bonuses tied to on-time delivery. People are people. They were determined to deem the system delivered on time and get their bonuses no matter what, and they did (and they got to keep them!). Somebody forgot to specify that what they delivered should work.
You can't imagine the nightmare this has caused for employees incorrectly paid. Some have gone without pay for months and are in danger of losing their homes due to defaulted mortgage payments. Others have been overpaid, taxed incorrectly as a result, and are now trying to fix that error, which will result in more fixes next year, and fixes to the fixes the year after that etc. etc.
Here in Down-Underland, the Queensland Department of Health had similar problems with, yes, a payroll system developed by IBM.
... and ...
Once again, due diligence by the government agency (or agencies) was not adequately performed (if performed at all).
Of course, let's not mention the Australian Census debacle. But then again, let's do:
"The Department did not completely and properly test Phoenix before its implementation, which is contrary to recognized practices for developing a system," the audit stated. "Phoenix executives cancelled a pilot project with one department that would have assessed whether Phoenix was ready to be used government-wide."
In the automotive assembly plant world, this would be going with production tooling and skipping prototype tooling.
this has gone on for ages and ages. And it will continue for ages and ages...
The problem is two-fold:
1) Management is paid (as already indicated) partially by bonuses upon delivery of something. That leads to rushed schedules. Always has and always will.
2) Even when the project is intended to be agile, there's always somebody fixing the budget. That leads to delayed functionality.
I always ask my clients: Is this an agile project? If they say yes, I present them with a never-ending, all-encompassing contract where they pay me $y a month, and I promise to deliver x hours worth of work doing whatever it is they tell me to do. They always come back with: "But we only have $ amount"... To which I always reply: "So the project is not agile, then, is it?"
If the clients indicate that it is a waterfall project, I immediately pull out a Business Analyst and let that person have a go at finding all the requirements. Then I add a System Architect / Data Architect to translate those requirements in costable work-units. Then I add a Finance Guru / Sales specialist to come up with a cost for the *agreed* scope. When everything is signed, I add a dogmatic scope-and-finance driven project manager to control the client, whilst I let developers get on with the job at hand.
If the client wants to restrict time, or start early, we can always talk about versions of the software-to-be-built. The balance the *client* has to find is: what functionality is needed for the first version. A very difficult balance to strike.
Ultimately, even in properly scoped, documented, managed projects, there will inevitably be delays. What most projects and project managers forget is that you *need* a communications matrix that tells you who your friendly counterpart is to help you solve which type of problem (money, schedule, scope). For *massive* replacement projects, you need to take all the end-users from day one on a journey that tells them about the new system, shows them how it works, and gives key-users the ability to interact with the new system while it is being build... *Never* *ever* develop something in a closed-door environment to reveal it on D-Day to the hordes of end-users... They will inevitably not like what they see, because it is "different"...
There you have it... More than $0.02 worth, but you're very welcome!
"2) Even when the project is intended to be agile, there's always somebody fixing the budget. "
When someone tells you that a project more complicated than a simple web site is 'agile', my observation is that the person telling you this somehow thinks they can avoid doing a proper analysis and design phase with no penalty.
,,, probably due to inexperience, inflated self-regard, and a tendency to believe vendors selling 'magic bullets'.
"1) Management is paid (as already indicated) partially by bonuses upon delivery of something. That leads to rushed schedules. Always has and always will."
Maybe bonuses should be tied to to successful implementation/module stages (50%) and the remaining 50% on meeting the time constraints, the latter being null and void if the former isn't met.
"Phoenix executives prioritized certain aspects, such as schedule and budget...
The Trudeau government estimated the total cost to fix the system could hit CAN$1bn by 2022"
Six years late and well over three times the original budget doesn't really sound as though schedule and budget were prioritised. Cheap, fast, good - pick none.
Not incomprehensible at all, sad to say. Most of the core issues here have been well documented for decades. The problem is that most people in IT (especially upper management) don't read established basic works like _The Mythical Man-Month_ (Brooks) or _Peopleware_ (DeMarco & Lister).
Failed IT projects has been my major area of professional focus for over 20 years; I've advised major corporations, testified before the US Congress, given lectures, published articles, acted as an expert witness in lawsuits involving failed projects. I currently teach a senior-level (US) university course on software engineering, and the two books above are among the required reading. What is utterly depressing is how predictable most failed IT projects are and how they all typically stem from a handful of well-known root causes, most of which involve a tremendous amount of wishful (but erroneous) thinking.
Way back when, I was given a project to port a number of engineering applications mostly in Fortran from VM to (yes) Multics. So I picked an application guinea pig for a pilot project to quantify the effort that would be required to do the whole lot.
The vendor touted its Conversion Support Team; so I brought them in for a meeting. In their eyes, my pilot would be a major project - not a good omen.
The pilot quickly crashed and burned with several mysterious messages from Multics and the compilers. My conclusion was that the technical demands vastly outweighed the available expertise including the vendor Conversion Support Team.
The conversation project was called off. I heard some talk that I had failed while I had in fact spared the company from a fiasco that would have cost many times my salary over my entire tenure.
The Multics system was eventually brought in to replace the vendor's earlier Time Sharing System. It gave no end of trouble and I occasionally stuck my head in my manager's office to remind him of the misery I had spared him from.
The basic functionality is only just out of reach of a Spreadsheet.
Due to widespread reading comprehension issues, I'll restate it again more clearly: The basic functionality *is* clearly out of reach of a Spreadsheet, but not by a wide margin on a Log scale. Perhaps 10:1 ratio, roughly. In the late-1970s, this function might have been coded up in BASIC and run on a TRS-80. The month's data would probably fit onto a 5.25-inch floppy.
Same folks were presumably involved in the aborted 'Long Gun Registry', another billion-dollar Canada government IT boondoggle. Yes, there is a trend.
The history of computing/IT is largely the history of failed 'magic bullets' which were going to make everything easy, solve all the problems, slash costs and delays, and often, eliminate programmers, analysts, IT professionals, etc.
Some highlights from the list include COBOL, subroutine libraries, time sharing, spreadsheets, relational databases (no need for IT pros getting between end users and their data if you have one of those!), remote computing, GUIs, structured programming languages, gotoless programming, object oriented languages, the cloud, microservices, agile, devops....
In many cases these fads are driven by a blind faith that a particular technique will solve all the major problems, or a desire to fix irritants that are the result of a solution to a previous problem... which often returns when the new magic bullet sweeps away the technique or practice that eliminated the older problem.
Often these ideas are 'validated' by anecdotal results achieved by the people who designed and built the new tools or techniques.
Curiously, random user populations with diverse skills and requirements often under-perform when compared with the developers or researchers using the tools they designed to solve the problems they are working on.
Biting the hand that feeds IT © 1998–2021