Europe's largest council kept auditors in the dark on Oracle rollout fiasco for 10 months

Birmingham City Council did not tell its official auditors about the disastrous Oracle implementation for ten months after the suite of applications went live, and appeared to obstruct access to the new system needed to complete their work. Since it replaced aging SAP finance software with Oracle's cloud-based Fusion for HR, …

[F. Frederick Skitty]

"Every other council in the country can implement these systems effectively"

No, no they really can't. My local council still does a lot of it's work on what they call "analogue" systems. Paper based in other words. Nor are they alone, with 40% of UK local authorities still mostly paper based, despite many spending vast sums on projects to try and go digital.

[abend0c4]

Also worth pointing out that this very site reported on the dissolution of Birmingham's previous attempt - a joint venture with Capita - back in 2017, on the basis that the expected savings were not being realised.

So it's not as if there isn't longstanding evidence.

[Anonymous Coward]

Aren't East and West Sussex messing this up as well?

Sorry, of course I meant "Haven't El Reg covered similar stories for the Sussexes* recently?"

* The counties, not the (ex?) (part-time (when it suits them)?) Royals loitering in the States at the moment...

[Tron]

'on what they call "analogue" systems'

You should be over the moon about that. Paper will never cost you the money and services that a failed tech system will.

[Brewster's Angle Grinder]

Re: 'on what they call "analogue" systems'

Part of what's racking up costs for Birmingham's current system is paying people to do what the machines ought to be doing. However expensive this becomes, permanently employing staff to do it manually will be even more costly.

[Andrew Scott]

Re: 'on what they call "analogue" systems'

"One day there was a fire" or flood or earthquake or tornado.

[Robert Grant]

Came here to say the same thing. They definitely can't all do it.

[Anonymous Coward]

Since the needs of councils are, more or less similar and really only different in the scale of each bit, surely it's not beyond the wit of man to design a generic council-specific tool whose components can be simply scaled for individual councils?

[Alex 72]

The UK native software industry was killed with neglect. Fujitsu bought ICL, Steria bought FI Group... near the end of a long winding tale where the US gave military contracts and support and the UK government gave warm words sometimes. So now out of the box commercial products assume US standards. Given the blame games, procurement policy, risk aversion, and the fact there is no vendor to blame if Open source and DIY systems are used by councils themselves to do it, Councils are left with at this scale with Microsoft (still tiny market share so seen as a risk to use), Oracle, and SAP. Success or failure then comes down to contract and vendor management, as well as knowing your own processes.

P.S. I have no doubt if a local or international System Integrator or even council employees were given a full correct map of processes independent of existing SAP implementation, along with Oracle, New SAP or even Apache OF Biz and the project actually worked with best practice in mind to agree the new process, with leaders and staff before building anything, as well as did the work in manageable chunks based on available resources, a system that worked as well or better than the old one could have been put in for less money. The point about the death of UK software was just that any failure would likely be less bad if the starting point of an implementation was closer to what a UK council actually needs the delta would be smaller by definition.

[Sgt_Oddball]

Having worked in a UK based software house

I can wholeheartedly agree. We tried to get council contracts from time to time but the sheer effort involved in finding the tender requests and stupid requirements/hoops to jump through to prove that yes, we were capable of providing a simple CRM package for one sports centre got ludicrous to the point I was sure the requirements were written by the tender winners just so only they could take the contract.

Smaller companies who can't afford to take councillors out for steak dinners every other week don't stand a chance for these things (despite being more flexible and able to bend over backwards to accommodate the effective, cost efficient implementation of their own package, safe in the knowledge that other councils will crawl over broken glass to get it once a package is proven good).

[Steve B]

Re: Having worked in a UK based software house

You have hit the nail on the head.

Councils used to work with specific vendors to ensure that only certain products could tender, with most having to admit to a bit of a botch to meet all the requirements.

That was fine when the council officers knew what they were doing, but they all seem to have been replaced by officers prepared to ignore local requirements and enforce OTS products and processes, to meet short term budgets, satisfy inept management teams, but furthering their own career.

Even better if they can move on with a good CV before the SHTF.

That's one of the reasons for busted budgets and missing work targets. Locally, it also led to a significant increase in nervous breakdowns amongst the stressed workforce.

I still think it was short sighted to get rid of all the good council programmers.

Whereas our local council used to spot a problem, report it and get it fixed immediately, I have been told that there can be a two year wait just to get the OTS people to recognise the existence let alone dream of fixing it.

Meanwhile our local council is missing budgets by millions and constantly failing to meet targets.

[Tim99]

Re: Having worked in a UK based software house

We sold a specialized management system to councils, heritage organisations, not-for-profits, and small government departments. A medium-sized council requested a tender. We offered "list price": A stand-alone single seat for <$1000, a 4 user network for <$2,500, and an unlimited version for <$7,500. We knew the officer wanted our system as it was "a standard" on >350 sites. Our nearest competitor (based in another country, with a local agent) had a system that started at probably ~$18,000, often requiring a custom install, adding at least another $5,000. The tendering department wanted $500,000 liability insurance for the network version, which would exceed the software's profit (or cost!). I told them that we weren't interested. The officer was upset and suggested including it in the quote, but found that their rules didn't allow it.

I occasionally saw the officer at conferences and learned they had bought the competitor, as they were the cheapest that had tendered. They paid about $45,000 for a 3 user system they didn't like (about 18 times as much as our system). Much of the extra was for building a data transfer model to upload data to our federal government, which was already standard in our software. I assume that the extra probably covered the liability :-)

[Bongo_xy]

Re: Having worked in a UK based software house

your company should have just increased the price, and would more likely get the contract

[Tim99]

Re: Having worked in a UK based software house

Yes, I probably could. It was shrink-wrap with published list prices and there was an ethical or reputational question?

[Doctor Syntax]

"no vendor to blame if Open source and DIY systems are used by councils themselves"

OK, so they have Oracle to blame instead. How far has thet got them? When you worry about who's going to be there to be blamed you're planning for failure.

[Alex 72]

I agree I was stating that's what happens in public sector not claiming it is good practice

[Anonymous Coward]

Lots of parish councils generally do this for their sites. It's a base template with minor tweaks for each council. The problem with larger councils is they want the bespoke shiny solution rather than the thing that works for other councils.

[Brewster's Angle Grinder]

Like these people? (I'm not in the sector, but they were pointed out to me last time it came up. According to the anonymous coward, "I've used some bad software in my time but Civica Financials really does take the cake." They have a follow up post that outlines the problems.)

I suspect the problem is that when you have a captive audience, you can afford to be shit.

[JimmyPage]

Yet weirdly

it managed to collect everyones council tax.

[AMBxx]

Re: Yet weirdly

Have you checked how many residents are in arrears?

Depending upon source £165, - £195m

[JimmyPage]

Re: Yet weirdly

If they are in arrears then - by definition - the system has done it's job and billed them.

[AMBxx]

Re: Yet weirdly

That's only half the job - has to manage the collections too.

[A Non e-mouse]

Re: Yet weirdly

I suspect an ERP won't manage the council tax: It'll just have an interface to that system to record the monies coming & going out.

[Anonymous Coward]

Re: Yet weirdly

If only we didn't have a system that still bases council tax off house prices in 1991... Alot of things have changed in that time, especially house values in certain areas (Waitrose effect anyone?). That could go a long way to rebalancing things.

(For the record we own a flat and a family home, both have the same council tax rate... The flat is worth about 4.5ish times less than the family home. Go figure).

Anon because scummy landlord etc etc (despite never once increasing the rent on a tennant, and keeping the place regularly serviced/maintained but it was either rent it out or sell it for only 20k more than purchase price. After owning it for 25 years)

[Richard 12]

Re: Yet weirdly

That's not really the main issue.

The first major problem is that many homes have always had the wrong band, because most of the original valuations were done incredibly badly ("second gear valuations") then became the basis for the area forever.

Rebanding an entire wrongly-done street is basically impossible, so those have stuck forever, even when ridiculously high or low.

The other large problem is that the top band is really, really low. Band H covers everything from a luxury Kensington flat to something the size of Chequers. Creating Bands I, J and possibly K would make a big difference.

[96percentchimp]

Re: Yet weirdly

Just this morning I needed to check some details on my Birmingham council tax account. Horrible experience that seems to cobble together at least two systems in an interface where everything is written by, and for, the the operators instead of the end users. Worse, even, than the horrible system employed by Southwark council, which is a significant achievement.

[JimmyPage]

Re: Yet weirdly

Remember they *had* to custom write that app "in house". There was nothing in the FOSS sector that could be developed to do the job.

Proof that plus ca change

https://www.theregister.com/2017/05/23/capita_service_birmingham/

[wolfetone]

This is all starting to whiff a little of criminal acts doesn't it?

Hard to know what smells worse though. This shit or the fact the bins haven't been collected since January.

[Anonymous Coward]

Rather ironic, this unemptied bins business. It was Labour councillors giving bonuses to their Labour trade union mates on the bins, and not the women who cleaned the offices that kicked off Birmingham's massive equal pay settlement costs of around £1.8bn. The costs of the Oracle screw up are chicken feed compared to that. However, as the council struggles belatedly and ineptly to balance its books, it's the boys on the bins going on strike, because they don't think they should contribute to book balancing: They'd rather it was all the other workers who weren't beneficiaries that should be axed.

The best solution for Birmingham council tax payers would be for the council to sack all of their bin workers, and outsource the function to a private firm.

[wyatt]

Yep, outsourcing to reset the pay and get rid of the union seems the only fix atm.

[UnknownUnknown]

https://www.cps.gov.uk/legal-guidance/misconduct-public-office

Misconduct in a Public Office.

https://www.gov.uk/government/publications/procurement-act-2023-guidance-documents-procure-phase/guidance-debarment-html

Debarment of Suppliers for public contracts.

“Tough on crime, Tough on the causes crime …. and proceed of crime”.

Go on Yvette Cooper …. I challenge you.

[Missing Semicolon]

Surely actively conspiring to mislead the Auditor is misconduct?

[UnknownUnknown]

The legislation already exists. Just use it.

In Birmingham CC, In Post Office these are in scope of this legislation. For FJ and Oracle, the new debarment powers are now in scope - the can’t take prior performance as a guide for a different contract is now bypassed.

[wiggers]

To err is human...

To really cock things up takes a computer.

Stick with paper-based systems. Simpler and cheaper, never needs replacing. (Apart from the pencils.)

[Anonymous Coward]

Re: To err is human...

Remind me why you’re reading a tech website?

[Anonymous Coward]

Re: To err is human...

Schadenfreude?

[IGotOut]

Re: To err is human...

"Simpler and cheaper, never needs replacing. (Apart from the pencils.)"

Those affected by the Windrush scandal would like a quiet word about how great paper based systems are.

[Richard 12]

Re: To err is human...

The major reason for that was that the Home Office deliberately destroyed the paperwork, so only a small number of people still had their copy.

If it had been computer-based, deleting it would have been easier and left fewer traces.

I e-signed a contract a few years ago, and the e-sign company had "lost" their copy within a year - despite it being their entire business.

How many of you still have letters from two decades ago? How about emails?

[Andy The Hat]

Re: To err is human...

"How many of you still have letters from two decades ago? How about emails?"

I haven't deleted a non-scam email since about 1996 ... and it absolutely saved my bacon in 2000 ...

The issue for me will be with withdrawl of support for .pst files in outlook (that's what we get so no alternatives) as I don't want multi-megabytes of live email database that's mainly cover-my-arse historic.

[UnknownUnknown]

Re: To err is human...

Loss of statutory documents is enough to summarily gain judgement against an organisation. Lack of due diligence

… say if a Finance company lost the documentation around a Car Finance contract mis-selling investigation for example.

[Nematode]

Re: To err is human...

"To err is human...

To really cock things up takes a computer."

And to make an absolute mess of everything takes a Council.

Councils: every day finding ingenious new ways to not do things.

Pet gripe: our local county council will (eventually) mend a GBFO pothole but having mobilised the road mending team to a distance of possibly 20 or 30 miles, the team leave another obvious pothole, a mere 3 metres away, totally alone so they have to re-mobilise some weeks later, having meantime wrecked Many Suspension Components.

[User McUser]

Why does Oracle always seem to get a pass?

Granted, I'm not privy to every single Oracle fuck-up or success and it might just be straight-up confirmation bias, but in all of the articles I've read about these disastrous deployments, the blame seems centered on the customers and not on Oracle's apparent inability to get these systems working.

Is there no clause in the contract that Oracle has to deliver a working product? They seem to collect (a lot of) money, deliver "a" system, and then they get to just sit back and watch the fallout. ¯\_(ツ)_/¯

[Excused Boots]

Re: Why does Oracle always seem to get a pass?

As I understand it, I believe that many of these issues are down to the customer failing to absolutely nail down a specification for exactly what they want the system to do, how they want it to work, etc before putting out the tender. This might be down to lack of skill, failure to listen to the people who actually use and/or understand what the system needs to do, reliance on 'what their mate at the golf club told them’ etc.

In this way the specifications are so poorly defined that Oracle could hand over a bag of spanners, claim that this will technically do what the specifications required and have a fair chance of defending it in Court should the customer kick off or refuse to pay!

Of course Oracle are not entirely without sin, they must know what will work and what won’t, and I suspect, could head some of these issues off at the pass - but then Larry won’t be able to afford another yacht - which would be a shame.

[StewartWhite]

Re: Why does Oracle always seem to get a pass?

It's all about plausible deniability. Oracle themselves avoid getting involved as the actual project is usually handled by a 3rd party consultancy who can then be blamed when it all goes south.

Much as it sticks in the craw me to say it, Oracle software isn't one of the primary causes of the screwup in this case (if El Reg at al are to be believed). Buying a system on the basis that you're going to use it in a vanilla fashion but with the council then proceeding to want it customised to match existing processes was always going to result in an expensive omnishambles. The consultants involved should have told the council this (but wouldn't due to the extra £££ that result from customisation) and councillors are only going to want to hear that everything is going swimmingly (until it's too late).

[wolfetone]

Re: Why does Oracle always seem to get a pass?

"It's all about plausible deniability. Oracle themselves avoid getting involved as the actual project is usually handled by a 3rd party consultancy who can then be blamed when it all goes south."

I think this is it. There are plenty of Oracle "experts" involved here before Oracle themselves get involved, and I think a lot of the problems start there. Of course the client needs to spec out what they want, but we all know the customer wants the moon on a stick. It's up to that 3rd party to translate the abstract needs of the client then give it back to them in a way that they understand and know makes sense.

[Tron]

Re: Why does Oracle always seem to get a pass?

They have lots of really excellent lawyers. Lovely people, all of them.

I would point out that if customisation is offered as a feature, then it should work when implemented. You should not be able to customise a product to the point where it does not work. At least not without a dialogue box popping up warning you that it will end badly. That said, local and national government can break anything (cf. Brexit, Trump & Musk).

Everyone in the industry knows that to make money selling tech to a local authority or Govt. Dept., you quote low to bag the contract, wait for them to screw it up and then offer expensive additional services and support.

Pretty much everything in the UK is run badly due to incompetence, corruption or both. You need to read 'Private Eye' to find out what the mix is in any individual case. The CQC was covered there but I haven't seen it on El Reg. Only a few cases make the mainstream news, dumbed down for the chavs.

If your local council is still using paper, you can sleep peacefully. Once they sign the software contract, you are merely a revenue stream for GAFA.

[Sam Shore]

Re: Why does Oracle always seem to get a pass?

At least 50% of lawyers, are by any definition compulsive liars. In every court case, at 1 lawyer is going into court knowing he is going to be presenting lies. Truth doesn't come into the legal profession.

[ecofeco]

Re: Why does Oracle always seem to get a pass?

Why do they always get a pass?

Why do you think? Yes, that.

[Eclectic Man]

Lessons ...

... will probably not be learned, either by Birmingham council or any other council.

I need a drink.

[IGotOut]

Re: Lessons ...

Well forget the pub, the landlords will sell them of and BCC will grant permission for a badly built faceless block of flats to go in it's place.

[Anonymous Coward]

The real truth ?

All the competent people at BCC that could have properly specc'd the system *and* properly made sure what was delivered worked were let go years before this.

So you had a lot of very highly paid "managers" whose glaring lack of ability in managing meant they were shafted from the off.

The last 5 years have been a round of moving people around as much as possible to ensure there is nobody to be held accountable. Except maybe a few refuse collectors.

The same way the 2008 financial crash was caused by public libraries being open more than a day a week

[veti]

Re: The real truth ?

When I was in the database business, I heard a story once of a customer who had written a proper spec. One that described every feature they needed, with use cases and test scenarios, all the functional and non functional requirements, the text and behaviour of every message box...

Then he woke up.

The "properly specc'd" system is like an "out-of-the-box" install. It's a myth. There never has been such a thing and probably never will. Even AI won't be able to write one, because you'd need examples to train it on.

The spec is a weapon for the contractor to use against the customer. Simple as that. It tells them when they can demand their money. The last thing anyone on the supply side is motivated to do is to question it.

[dinsdale54]

Re: The real truth ?

The spec is the spec.

It's what has been agreed to be delivered. If you change the spec then you are changing the contract - that doesn't usually come for free.

In the early part of my career I worked for a company that supplied systems to local government written in Oracle. The system was decently complex and did what it said on the tin. Most councils took it as is, or with a few customizations which were generally done on a T&M basis.

The couple of councils who asked for extensive customization were back using the standard system within a few years. They discovered that they weren't actually as special as they thought and the costs of maintaining a customized version outweighed any predicted benefits.

The quality of IT departments at the councils was a mixed bag. Some were pretty good while some didn't really have anybody who knew what they were doing. There were very few Oracle skills there. Somebody who actually got good at Oracle would likely be off for a better paying job pretty soon.

[John Smith 19]

Hmmm. *Obstructing* auditor access to the financial system

stopping them conducting the audit to begin with?

Yes that starts to sound actually criminal.

Hard to say which is the bigger bunch of c**ts. The Councillors, the "Officers" of the Council, the SI's (I strongly doubt all the boots-on-ground were direct Oracle employees).

[Anonymous Coward]

Re: Hmmm. *Obstructing* auditor access to the financial system

Keeping things secret is the lifeblood of the people with the dividers and secret handshakes.

Does that have any influence in the likelihood of successful redress, via Plod or whoever?

[Orwellian83]

Fusion still WIP

A few years ago I was the Snr Business Analyst at a mid sized charity looking for an ERP.

Oracle was not even in my list of potential vendors invited, however the “wiser” Finance Board wanted them to be included in the RFP.

They demo’d Fusion and couldn’t match even basic requirements Out of the box such as bank rec’s without 3rd party apps.

I cut the meeting short, they’d wasted enough of my time.

[Anonymous Coward]

Re: Fusion still WIP

Out of the box both Oracle and SAP do almost nothing at all.

They have to be customised to work for any business, no matter how simple.

Weirdly, Jira is the same.

It repeatedly shocks me, because in my industry we get complaints if we didn't include a built-in connection two years ago for a brand-new third party device that doesn't even have a datasheet.

[LateAgain]

Not defending anyone but...

If the "Oracle Project sub-committee" only meets yearly then 10 months is really quite good :-)

[Anonymous Coward]

Americas Cup

Quote: ".....adopt Oracle's standard processes....."

Larry Ellison needs a lot of money for projects like the Americas Cup.

So..........the standard process looks like this:

(1) Mushroom Model - Keep everyone in the dark......and s**t on them once a week.

(2) Given item #1, the ORACLE billing will be hard to challenge.....even when the bills balloon to ten times the original estimate!

....and of course it helps that the ORACLE sales team have assessed the "clueless coefficient" of the client "management".

(Note: the ORACLE sales team only needs to be LESS clueless than the client!!!!)

I'm looking forward to the next Americas Cup!!

[GrizzlyCoder]

On a slightly different tack, I have seen a similar US-based OTB solution for HE administration that has gone from strength to strength as the Universities that initially heavily customised it in the early days realised the pain caused by baseline upgrades that undid them all and then needed to have the customisation re-applied and then re-tested to make sure none of it got broken. The difference was that the supplier was (and still is) very active in eliciting the requirements that were missing from the baseline (and thus causing the customisations) and incorporated the best ones (not all, as some were based on weird local business processes) into the baseline. Over time more and more Uni's realised that, as the baseline approached 80%+ of their requirements, it might be better to reanalyse some of their business processes to see how many were a result of "because we have always done it this way" since the days before computing existed. BUT most universities have a strong in-house support element for their IT as well, which sadly seems to have disappeared down the PFI/outsourcing drain for our local authorities.

[keithpeter]

Enquiry please

I'm very grateful to The Reg for giving time for Lindsay Clark's detailed and thorough reporting of these issues. I'm getting more information from here than from any of the Birmingham press sites. Just some questions (and an opinion)...

"Birmingham City Council did not tell its official auditors [...]"

Should that not be 'Birmingham City Council Officers did not'? There was a change of political administration part way through this saga and previous articles have suggested that a group of council officers decided to start asking for customisations of the Oracle system.

"During the meeting, Councillor Lee Marsham, a Labour member of the audit committee, called for an inquiry into the Oracle ERP implementation."

YES PLEASE. And could that enquiry's scope include questioning around the suitability of the original Oracle system for running the audit function of a large and complex corporation in the UK without customisation? I have doubts about that. Lindsay Clark's previous reporting has already pointed out that school accounts in Birmingham are now 'out of scope' for the system.

"Stocks said the council had to be ready for the required business change to adopt Oracle's standard processes. "You do have to drive change because the things have to change within how departments work and how you implement the next ERP system because it can't go wrong again. I don't think any of us can survive if it goes wrong again," he told the committee."

Are these Oracle 'standard processes' documented in public anywhere? Are they fit for the tasks required? Who decides that now? We know that many Oracle implementations both in public sector organisations and in commercial concerns are having problems.