City council Oracle megaproject got a code red – and they went live anyway

Birmingham City Council, a massive local authority in England, went live with new Oracle Fusion ERP software in April last year knowing the system was likely to fail. Since the introduction of the replacement for an ageing SAP financial management system, the council cannot close its books and file financial reports. The …

[b0llchit]

Oracle declined to comment.

But they will be sending the lawyers because Oracle has not been paid since the system refuses to pay bills sent through the new system. The city being bankrupt is no excuse and Oracle is eyeing replacement payment in other currencies, such as all the land and buildings of the city centre. In the meantime, a statement from Oracle counsel was received to say: "The radius of Oracleham city ownership will extend for every day we are not properly compensated.".

The city of BirmingOracleham declined to comment.

[jay_bea]

Had Oracle decided to comment, I suspect it might have been "Kerching".

[J.G.Harston]

Is Birmingham Council allowed to steal loads of other people's property in order to pay off their own debts?

[b0llchit]

Have you read the quantum dots at the bottom of your contract?

[Anonymous Coward]

A drop in the ocean

Looking at how much Birmingham have frittered on noisy, slow trams and the wider messing up of Brum city centre, I suspect the Oracle overspend is not really significant.

[A Non e-mouse]

Re: A drop in the ocean

This is my beef with El Reg's articles on the Birmingham Bungle: They present the story as the failed Oracle implementation has brought down the council. My reading of El Reg's articles are that the failed ERP project is just one piece of the puzzle and not the catalyst. If the ERP project has suceeded would Brum council still have gone backrupt? Yes. The £760 million equal pay claim is gonna be a big part of the failure to balance the books.

[The Mole]

Re: A drop in the ocean

Without the ERP system it seems like they have no way to even know if the books are balanced let along work towards balancing them. Its not the only factor but is no doubt a big contributory factor and example of how the council was running everything. As they have said on other stories it isn't the whole but it is the part of the story the technical audience of El Reg are most interested in.

[SVD_NL]

Re: A drop in the ocean

Yup, the cost may be a small contributing factor, but i'd wager not having a functioning ERP system doesn't help with identifying and resolving your financial issues...

[Lurko]

Re: A drop in the ocean

"The £760 million equal pay claim is gonna be a big part of the failure to balance the books"

The £760m is an estimated current liability, and is on top of £1.1bn in equal pay settlements already made by BCC in the last ten years. In terms of going bust, BCC's £2.5bn pension deficit doesn't seem to have got much of a mention, but would appear to my causal eye to be the biggest chunk.

[Graham Dawson]

Re: A drop in the ocean

The irony is that Brum council rather screwed themselves over in the equal pay matter. The city's officials and press office made a huge fuss about equal pay for years, loudly decrying the "inequality" of lower wages for certain, female-dominated employment sectors when compared to others that tended to be male dominated, regardless of the differences in work requirements and risks.

All of the council's criticisms of other organisations' practices became evidence of their own malfeasance in the claims. They could hardly argue that they were providing equal pay when their own pronouncements said otherwise.

A few small pay rises and a little more circumspection would have saved them hundreds of millions.

[fnusnu]

Re: A drop in the ocean

^^^^ would have saved the taxpayer hundreds of millions.

[Mooseman]

Re: A drop in the ocean

"loudly decrying the "inequality" of lower wages for certain, female-dominated employment sectors when compared to others that tended to be male dominated, regardless of the differences in work requirements and risks"

Thanks for clarifying that you have no idea what wage equality actually means.

[Graham Dawson]

Re: A drop in the ocean

Tell it to Birmingham council.

[aerogems]

Re: A drop in the ocean

Roughly 80m pounds may not cover everything, but it would certainly go a long ways towards covering any other shortfalls. Just sayin'.

[sketharaman]

Re: A drop in the ocean

+1. My heart bleeds to see Oracle being made the Fall Guy here (Disclosure: Ex-Oracle employee).

[keithpeter]

Re: A drop in the ocean

Trams? Talk to Andy S not BCC.

PS: I'm quite relieved at the moderate speed the Metro runs at in the city centre myself. Try Manchester just outside the art gallery one day - but be prepared to jump quick.

[Anonymous Coward]

Re: A drop in the ocean

"Trams? Talk to Andy S not BCC."

The trams kicked off back around 1999, the West Midlands Mayorality started in what, 2017? There was of course Centro (Wumpty as was) doing the wider West Mids "thinking", but Wumpty was always dominated by BCC.

Seems it's a local government thing - Edinburgh, Sheffield, Manchester, Nottingham, Brum, they're all keen on these ludicrocusly expensive railways down the street, as a form of local government jewellery. Birmingham's latest "in progress" acheivement is the East Side extension, building 1.4km or track for over quarter of a billion quid, and that's to connect the city to the misbegotten HS2 station at Curzon Street. Save ten minutes on the journey to London, but only after fifteen minute tram rattle to Curzon Street, genius. Then zip to The Smoke in not time and find yourself at the urban desert that is Old Oak Common......maybe it's not just a local government thing.

[keithpeter]

Re: A drop in the ocean

Cost per mile is bonkers agreed. Same pattern as HS2 - start building while consultations going on &c and adjust as you go so more cost.

The point I was making is that Metro is a combined authority responsibility now. Not everything that happens in Birmingham is BCC.

Another e.g. Clean air zone = central government mandated air quality standards.

(HS2: I have always been somewhat sceptical of the journey time argument. The capacity argument used to make sense when the NE and NW segments were still in the plan but now its just embarrassing).

[Tim Almond]

Re: A drop in the ocean

The capacity argument hasn't made sense for years. We know what the predicted growth was in 2013 when the planners drew a straight line from the previous 15 years of rail growth.

But business rail barely grew and then a few years before COVID it went into reverse. What growth there has been has been leisure rail which isn't worth building for.

[Ian Johnston]

Re: A drop in the ocean

Passengers like trams. And while the original Edinburgh system was undoubtedly a Category A, chrome=plated clusterfuck, the Nottingham system was a model of cost-effective planning and delivery. What is it about trams which so upsets Daily Mail readers?

[Tim Almond]

Re: A drop in the ocean

Trams: all the cost of a train with none of the flexibility of a bus.

[Ian Johnston]

Re: A drop in the ocean

Much higher capacity, quieter, runs on electricity, more popular with travellers.

[BOFH in Training]

Why did they kill off the existing SAP system before Oracle system could fully take over?

[Derezed]

Presumably someone’s bonus was tied to the retiring of old and implementing new. Job done!

[Doctor Syntax]

End of contract with SAP and/or the H/W it was running on?

Train crashes depend on the momentum of the train.

[aerogems]

I'd guess more that the version of SAP they're using is EOL and the amount of money SAP wanted to continue supporting it just for them was too high. Now why they went with Oracle instead of migrating to a newer version of SAP is anyone's guess. It probably involved a lot of "business dinners" at really nice restaurants and "business meetings" held at luxury boxes at sporting events.

[Lonpfrb]

SAP ECC 6

SAP ECC 6 on NW 7.5 EOL 2027 so that's not a technical blocker. Intention to Fit to Standard is aligned with S/4 Green field implementation so no worse in general to a Oracle green field implementation.

So another reason does appear likely...

[James Anderson]

Because there is no easy migration path from one SAP release to the next. So its often wisest to delay upgrades as long as possible. The trouble is when your current system is about to go out of support you are three or four releases behind.

In many cases all the essential customisation has to be redone from scratch.

Upwards compatibility? "Bitte überweisen Sie die erhöhte Lizenzgebühr auf dasselbe Konto".

While I applaud the decision to escape the SAP forced upgrade nightmare, it was a case of out of the frying pan into the fire.

[Lonpfrb]

SAP Upgrade

"no easy migration path from one SAP release to the next" is generally an untrue allegation. Rather huge effort has been put into the upgrade path for customers.

HR line of business will have mandated legal change upgrades so finance is required to align which means no case for delay to upgrades. Configuration will be fine, but custom code will depend on the continued use of sap enhancement concepts to be upgrade safe, as we built it, and not creating technical debt through poor practices.

However SAP has been very clear that ECC 6 is EOL 2027 and S/4 Hana is the strategic direction but not a succession product with an automated upgrade path. The BCC decision to Fit to Standard is perfect for S/4 green field implementation.

New infrastructure is required for S/4 but by now the infra would be worn out despite several hardware refresh actions e.g: 20 years / 5 years depreciation cycle.

Whichever ERP is chosen the implementation partners experience and understanding is key to enable making good (low TCO) decisions. The leadership and discipline to Fit to Standard is not easy, and that failure is how to over run by 4 times.

[James Anderson]

Re: SAP Upgrade

A UK council could not possibly, practically or legally “fit to standard”.

[tmTM]

Flip-Flop

oh look, another large public body that can't make it's mind up when it comes to large IT investments.

Perhaps if you had a properly researched plan in place before blindly ploughing ahead you might not find yourself in this gigantic mess.

[A Non e-mouse]

Re: Flip-Flop

I'm gonna say that it's probably management (or the councillors) that flip-flopped. The article says the original plan was to change process to match Oracle. I bet someone got cold feet at trying to push through change (or someone "important" complainted). So they then swapped to hack customise Oracle to match their processes. It's that core change mid project that probably did it in.

It's always the mid-project changes that kill you.

[Richard 12]

Re: Flip-Flop

Or even more likely, they discovered that "standard" Oracle doesn't work for anyone at all, ever, and that customisation is absolutely necessary.

I'm not aware of a single Oracle (or SAP) ERP installation that is "standard". They're all customised to a significant extent.

[veti]

Re: Flip-Flop

Indeed, "out of the box" software at that level is a myth, and Oracle should be prosecuted for deceptive sales practises for even allowing the phrase to be used.

[SVD_NL]

Re: Flip-Flop

I know someone who worked as a sales consultant for a certain ERP software vendor.

He described his job as follows:

'I just say "yes the software can do that" and then tell the technical guys the software has to be able to do what i promised"

[ecofeco]

Re: Flip-Flop

More research than the right family vouched for it and the right brown envelopes were exchanged?

How plebeian.

[Missing Semicolon]

Product not suitable

It is entirely possible that the people who signed up for the Oracle product actually did not fully understand what it does, and in what ways it does not implement the (probably legally) necessary financial reporting and integrations that the SAP system has.

So the decision to go customised may well have been because it is legally required.

In fact, the Oracle system should not have been purchased at all.

And, "The council's political and administrative leadership" will experience exactly no hardship and no censure for being so incompetent.

[Anonymous Coward]

Re: Product not suitable

Yes well - therein lies the problem. Being involved in the early stages of prepping for procurement of an ERP trying to get the business to clearly articulate what they do beyond "it's complicated and you wouldn't understand" is an interesting experience.

[Brewster's Angle Grinder]

Re: Product not suitable

"...it's complicated and you wouldn't understand..."

I can do Quantum Field Theory and General Relativity. Try me.

("It's complicated and you wouldn't understand" being code for, "it's complicated and I don't really understand it - certainly not well enough to explain it to you and I know I'll end up looking a fool.")

[gerryg]

Re: Product not suitable

Here's a quantum of advice "If you can't explain to a five yr old you don't understand it" Richard Feynman

[Ian Johnston]

Re: Product not suitable

Over many years working in a academia, it has been my invariable experience that the more eminent the expert, the more lucidly and concisely they can explain their work to a lay person. Waffle and circumlocution always indicate third-rate minds.

[Bebu]

Re: Product not suitable

《"...it's complicated and you wouldn't understand..." I can do Quantum Field Theory and General Relativity.》

I suspect Brumy City Council Accounting Theory isn't renormalizable.

A distant relation to Douglas Adam's Bistromath perhaps.

How is it that Boris Johnson and his defective crew aren't responsible for this fiasco?

[Brewster's Angle Grinder]

Re: Product not suitable

That explains why they went bust. It if was renormalizable, even Oracle would have solved it by now.

[Anonymous Coward]

On the RAG

Ahh highlighted as red in the red/amber/green report? That'll be printed in black and white because "colour is too expensive" so will have been missed for many consecutive months.

[Mark #255]

Re: On the RAG

Having reached an age wherein I have run out of fucks to give about how I'm perceived, I now loudly decry red/green combos as an accessibility failure (because I'm colour-blind), with the back-up argument of "what if it's printed in black and white?".

Red/amber/[yellow]/white are the colours to use, with solid/dashed/[dotted]/no borders

[mikus]

Has there ever been a proper ERP implementation that hasn't run 5x or more in cost than intended? I've never seen or heard of one, pretty much anything Oracle or SAP is simply destined to be a disaster when put toward every weird and wacky business or government use case.

[old_n_grey]

"I've never seen or heard of one"

Do you think that might be because no-one ever bothers to report successful implementations? I used to be an Oracle functional consultant and worked at three large system integrators and I can say without fear of contradiction that none of the projects on which I worked went five times over budget (obviously I do realise that the hyperbole was not meant to be taken literally). Did any large projects go over budget? Yep! Sometimes because we, the integrators, got something wrong and others because the client kept changing the requirements - "I know that is exactly what we asked for but it's not what we wanted"! But I also worked on several that went live on time and on budget. But who wants to read about those?

The other thing about Oracle and SAP projects is that they are likely to be the large projects, with complex requirements and so more likely to be problematic.

Mind you, I am trying not to defend Oracle the corporation, who used to be only too happy to agree that the system could do everything the potential client wanted. On one particular project, that cavalier attitude to real life caused me an enormous amount of grief when I had to tell the client that the system really did not do what Oracle said it would. Of course, not all integrators were squeaky clean either. Nor indeed were clients who sometimes preferred a lower initial cost based on a set of requirements that had had functionality descoped that would need to be added during the project via change control. And, unfortunately, change control never comes cheap!

[ecofeco]

Feature creep is the bane and often death of every project, service and product, from tiny to large.

The Rube Goldberg Effect is the default setting of everything.

[A Non e-mouse]

Vendor, lie???

Shocked! Shocked I tell you!

I was using a vendor's system for about five years and we decided to go to tender for a replacement as the vendor couldn't provide certain features. We made sure to detail these features as mandatory in tthe tender. Our incumbant bid claiming they could do these missing features. I challenged them on this and they said "Ah, we added those features just now." Bollocks they had. I asked legal about this contradiction and legal said there wasn't anything we could do: We had to believe what they wrote in the contract.

Fortunately the vendor lost the tender due to other flaws in their bid.

[cookieMonster]

At this point in time…

You would have thought that after years and years of failure and countless billions wasted, SOMEONE would have figured out that it would be better and cheaper to hire a shed load of developers and build custom solutions

[Lurko]

Re: At this point in time…

If you're not competent to buy and customise software for some very common use cases, what chance of properly managing a self-build?

If you have the right people it is indeed fairly simple to create a new accounting, resource planning, ERP, logisitics, or similarly complicated systems, I have worked with people who built one of the most modern EPOS systems for large companies from scratch, with five nerds in a garden shed*. But if you're a ponderous local government body? No chance. No chance of getting the business case right. No chance of getting the user requirements right. No chance of mapping the adjacent systems and interfaces. No chance of correctly specifying the reporting requirements. No chance of knowing who to turn to and manage the dev team. No chance of being able to make informed choices when the programme manager has to get direction. No chance of leaving them alone to build the thing. No chance of sticking by the spec.

* that system is now (whisper the name) in the Oracle stable.

[James Anderson]

Re: At this point in time…

Most of these ERP “out of the box”, “implement best practices” perhaps work out of the box and implement best practices for a medium sized mid-western manufacturer. But for every body else it involves tons of customisation.

In effect you are writing a bespoke system in a really crap programming language.

[Graham Cobb]

The world has changed

And board priorities have changed. I know a company who's board used to be really engaged with the business efficiencies and competitive advantages IT were bringing.

But in the last couple of years, all they care about is cyber-security. Which is probably right as ransomware is an existence-level threat to many companies. But it means that big suppliers, and big consultancies, are now highly preferred by the board - certainly over innovation. IT has become the new HR - just keep it running and don't bring us any problems.

[gerryg]

Theory of the firm, bear with me

In the private sector eventually the internal transaction costs exceed the external transaction costs, limiting the size of the organisation, (Ronald Coase). That's one reason why there's more than one supermarket.

Enterprise software is a sales pitch designed to convince large organisations they can lower internal transaction costs. In the private sector a firm has to cope with the decision or die. In the public sector they just keep spending and put it on the tax bill.

[James Anderson]

Re: Theory of the firm, bear with me

Er companies have to spend less on their product than the selling price ….. who knew.?

[James Anderson]

Re: Theory of the firm, bear with me

Not sure why the downvote.

The original post was just a jargon filled complicated way of saying firms need to make a profit.

You don’t need to pick obscure wards out of a text book to say this.

[ecofeco]

They used Oracle?

That's a red code nothing can fix.

[aerogems]

I feel their pain

Living through a SAP migration (from another ERP) and it hasn't been quite as bad as this, but it's been plenty bad. Seems like today my inbox for tickets has been blowing up this week, and all for little changes basically because the consultant company they hired decided that, "Oh, you don't need to worry about that, someone else will do it" constituted as training. So, now everyone thinks everyone else is going to do everything for them.

It's virtual, but it's the best I can offer for my fellows in frustration over in Birmingham. If not for a little thing called the Atlantic Ocean in the way, I'd buy a round and we could share war stories. ===============>

[Big Softie]

The fundamental issue with all the public sector disasters we see is that the people appointed to manage them are incompetent. They lack the skills, knowledge, ability, experience, track record, and training to do the job they're doing yet somehow get appointed to the position anyway. So whilst they are the immediate cause of the cock-up, the solution lies in tackling recruitment in the bigger picture. The same happens in the private sector of course but there corrective action has to occur within otherwise the enterprise will go under. In the public sector it's always the taxpayer who takes the hit whilst the mess continues by putting out the message "We're sorry, we made mistakes, we know have to do better, lessons have been learned...." And the band plays on. It's high time that society stopped treating the public purse as a bottomless pit to make the chosen few rich and failing to ensure cost-effective and successful delivery of the projects it bankrolls.

[Grinning Bandicoot]

What learning curve?

Same song, same blues on ITS ONLY A MINOR TWEAK.

And this to Megalodon.