City council audit trail is an audit fail after disastrous Oracle ERP rollout

Birmingham City Council, Europe's biggest local authority, has no way of knowing if financial fraud has been committed after it failed to run security and audit features in a new Oracle Fusion ERP system. The English council, which was responsible for £3.7 billion revenue [PDF] in 2021, also continues to struggle with " …

[wolfetone]

"Birmingham City Council, Europe's biggest local authority, has no way of knowing if financial fraud has been committed after it failed to run security and audit features in a new Oracle Fusion ERP system."

I'm no expert, but I would start with the clown who suggested the use of Oracle and follow that particular trail.

[Pascal Monett]

That is almost certainly just incompetence, not necessarily fraud.

[M.V. Lipvig]

A better way would be to audit the finances of all involved, and find out who got rich during the time of no audits.

[The Organ Grinder's Monkey]

So, less an audit trail, more an Audi trail?

[Fruit and Nutcase]

Larry Ellison?

[Anonymous Coward]

Same with us. The fool that convinced management to pick Civica. He was a consultant, they always listen to consultants over their own staff. We're sure he was getting brown envelopes from Civica as it wasn't the best system and has none been proven to not be the best system now live. I suspect same has happened here, an consultant has recommended Oracle due to brown envelopes.

[Anonymous Coward]

OMG yes, Cívica are the pits!

I have to deal with one of their aged products which provides 'controlled' and audited access to a major national database and it is the biggest heap of excrement I've ever encountered.

All bug reports get rejected as being either features or not important.

It's tightly and pretty much irrevocably bound to a much-loathed database for user controls and auditing

User passwords are stored as symmetrically encrypted stings without salts (i.e. easy to spot who has the default password!! And all can be decrypted using a rather unimaginative fixed key anyway).

I wish I could say more here because they are taking six figures levels of public money annually for horrendous software and it wouldn't surprise me if some of their more public software is just as terrible (the council websites I've seen bearing their name do not give me any confidence!)

TL;DR For the love of your favourite choice supreme universal power: do no but anything from Civica !

[Anonymous Coward]

I have seen similar issues a few times in the past. Performance was one reason, having logging on at the required level killed performance or the service completely fell over.

The other reason was incompetence. Someone forgot to switch on the logging service and rather than fix it the issue was avoided or forgotten as the responsible person moved to another contract.

I understand that neither of these reasons are excuses, simply what I've seen.

[tfewster]

Mistakes happen. But that situation would come to light when the Finance Director or Audit committee demanded to see the audit logs prior to signing the system off to go live, during parallel running and at least monthly thereafter.

What, the beancounters didn't do the basics on that either? What a surprise.

[katrinab]

Possibly they knew the audit trail report wasn’t working, but didn’t know why, or how to fix it?

[ITMA]

Ah, the "Angela van den Bogerd" defence.

[Eclectic Man]

Re: Performance

Performance was one reason, having logging on at the required level killed performance or the service completely fell over.

In that case the system was incorrectly specified. A financial system which is legally required to provide audit reports that cannot perform when auditing is enabled is clearly not compliant with user requirements. And the sales team that proposed it, and must have known that auditing was a legal requirement are as culpable as the purchasing team that did not specify auditing along with performance levels.

[mtategcps]

Re: Performance

If I had to guess, a system was proposed that would have performed properly, but the council pushed back on the price, so the sales people revised the specs to make the sale.

[Decimal5446]

Requirements tracability

I doubt it was missed on the requirements. I also doubt it was due to technical performance constraints. Suspect it just got lost somewhere along the project. Requirements traceability could have caught this or maybe a pen test / assurance review which certainly would have caught this one. Probably!

[hmas]

Occam's Razor

The simplest explanation is that it was turned off during implementation and someone forgot to re-enable before production go live.

[Ian Johnston]

Re: Occam's Razor

Perhaps the simplest explanation is that someone in the council really, really didn't want auditing enabled.

[Vince]

The flaw is that you can even configure an on/off for any sort of auditing for a financial system.

[Anonymous Coward]

A cynic may wonder whether Oracle's growing interest in the healthcare market might in any way be affected by a growing cohort of former officials who may at any moment become too sick to be held responsible for the IT fiascos they have left in their wake.

[David 132]

There's certainly good money to be made investing in Alzheimers' support facilities for all those senior Post Office executives who have no memory of anything whatsoever that happened on their watch while Horizon was running!

[Anonymous Coward]

Murdoch Syndrome.

As first seen from James and Rupert at the Levinson (phone hacking enquiry) Enquiry. Harvard Business School must have done a Grad Research Project on it and added it the MBA Curriculum.

Didn’t see anything, didn’t hear anything, don’t remember anything, don’t know anything…. Nothing to do with me… … sorry I’m just the CEO/Chairman. I’m really sorry about it though. Is it lunch yet. Garrick??

More recently seen in the wild at the UK Covid Enquiry, Post Office Enquiry, MoD Arms Corruption to Saudi, Investigations into Water Industry Institutional Sewage dumping …….

[Cliffwilliams44]

It's called the Hillary Clinton defense. "I'm sorry, I don't recall!"

[Anonymous Coward]

I'm not defending Hillary, but that defense was used long before she did.

It might even be older than her.

[Charlie Clark]

Donald Rumsfeld was happy to boast about what he couldn't remember about illegal weapons shipments to Afghanistan.

It's a great strategy: it's not perjury if you say you can't remember.

[Rich 11]

Ronald Reagan, in the Iran-Contra inquiry by the Tower Commission, back in 1986.

"Er, um, I cannot recall." Said later in the inquiry, after he initially said he had authorised the deal. Thankfully lots of documents had been destroyed which might have brought his reputation and competence into dispute.

[Mike Friedman]

Reminds memos the Florida US Senator Rick Scott who was CEO of HCA Corp (a large hospital company) that was fined $1BN for defrauding US Medicare ( health insurance for older folks) and claimed to know nothing about it.

So he was either lying or incompetent. Or both. But voters mysteriously believed him

[Charlie Clark]

Sounds like a prerequisite for holding elected office in the US. Basically, voters seem to want salesmen so they shouldn't be surprised when they get them.

[Ian Johnston]

The fundamental truth about the Post Office is that as an institution it still believes that all those sub postmasters were guilty and are being let off on a technicality and therefore that nobody in the organisation did anything wrong or has anything to be apologetic, ashamed or embarrassed about. Which is why they keep apologising to the enquiry with all the forced insincerity of a child saying sorry to get pudding.

[Tron]

Shocking. Absolutely shocking.

I really must contact them about my £2.4m unpaid invoice that seems to have vanished in their system.

Would it have been an idea to have run the new system in parallel with the old one until the new one worked, or is that just crazy talk.

If they'd have gone back to paper, a hundred A4 pads, a lot of 4-colour biros and several calculators later, they would have sorted their finances and saved themselves £90m. A moral there for everyone.

[Lee D]

Re: Shocking. Absolutely shocking.

I would also pitch for parallel systems. I don't understand how large projects EVER decide to play switcharoo in the middle of active use of the system and think that's acceptable.

At this scale, I'd want parallel systems with the same data put into them and see what happens.

[neilo]

Re: Shocking. Absolutely shocking.

Have you ever been involved with an ERP cut-over? I have, multiple times. I'm yet to see a company have sufficient staff and resources to double-entry processes into the new and old systems; in fact it's probably a really, really bad idea. With a cutover, you pick a time - say, after all the end-of-month processes are completed - and block access to the system. You migrate the live data over the nearest weekend, set the old system to read-only, and start transacting in the new system on the Monday. The old system is now there for reference, and will be used repeatedly for about three months, then it's use will fall away dramatically.

Entering data into a test instance of the old and new systems and watching what happens is robust testing. Entering live data in the old and new systems is a recipe for disaster.

[Plest]

Re: Shocking. Absolutely shocking.

"Would it have been an idea to have run the new system in parallel with the old one until the new one worked"

What and have people find out that the new system is a pile of poop and there never really anything fundamentally wrong with the old system that a little TLC couldn't have fixed?

[neilo]

Re: Shocking. Absolutely shocking.

Once the old system is out of vendor support, no amount of TLC will keep you regulations compliant, unless you commit to developing the system in-house from now on. Which, as you can guess, is a very expensive and very risky thing to do: what happens if you in-house developers leave?

[Vince]

Re: Shocking. Absolutely shocking.

Yeah because going to the 'not out of support' system has been an epic success of regulation compliance.

[Michael Wojcik]

Re: Shocking. Absolutely shocking.

They moved from SAP to Oracle. I expect both the old and new systems can reasonably be described as "a pile of poop".

[Anonymous Coward]

"audit trail", "segregation of duties"......

Yup.. let's make guesses about exactly who might think "audit trail" and "segregation of duties" might be an impediment to doing business........

Someone INSIDE the council?? Someone OUTSIDE the council?? Larry Ellison??

Ah yes.....How about Larry Eliison and that ONE BILLION DOLLAR INVOICE???????

[Anonymous Coward]

Who does their technical support :o

“decision was made .. to switch off .. the control environment of the ERP system”

Shomit’ wrong here ..

[I am David Jones]

What’s the opposite of “the gift that keeps giving”?

[Korev]

Gift is German for poison, it fits!

[Charlie Clark]

And Swedish for marriage.

[Anonymous Coward]

The public sector IT project that keeps on taking?

[Anonymous Coward]

What’s the opposite of “the gift that keeps giving”?

The gift that keeps taking the piss...

[mtategcps]

>What’s the opposite of “the gift that keeps giving”?

The recurring invoice?

[spacecadet66]

Oracle.

[Ian Johnston]

In other news, Lamborghini dealers in Birmingham are expecting a busy year.

[Anonymous Coward]

RE: Lamborghini dealers in Birmingham

That's no joke. The ring roads are frequently buzzed by racers in all sorts of souped up cars. The racket they make in the tunnels is unbelievable.

[markrand]

Re: RE: Lamborghini dealers in Birmingham

To be fair, the worst offenders are VW Golf drivers who spent all their money on baffle free exhaust systems and can't afford to get the valve guide oil seals and piston rings sorted...

[David 132]

Re: RE: Lamborghini dealers in Birmingham

>To be fair, the worst offenders are VW Golf drivers who spent all their money on baffle free exhaust systems and can't afford to get the valve guide oil seals and piston rings sorted...

Because Halfords don't offer valve-guides or piston ring kits slathered in plastichrome and green LEDs, obviously.

[Anonymous Coward]

Re: RE: Lamborghini dealers in Birmingham

Mainly Daddy’s V8 Mercedes AMG’s.

Must be doing a delivery from his Curry House.

[Anonymous Coward]

So if I stop paying my council tax

they have no way of knowing ?

[heyrick]

Re: So if I stop paying my council tax

On the other hand, if you do pay your council tax they equally have no way of knowing...

[Cynical Pie]

Re: So if I stop paying my council tax

Unlikely as Council Tax will almost certainly be on Northgate which isn't an Oracle system (breathes sigh of relief) but is I believe part of Capita....

[Anonymous Coward]

Re: So if I stop paying my council tax

Northgate is NEC, not part of Capita as far as I'm aware.

[Throatwarbler Mangrove]

Re: So if I stop paying my council tax

More like, if you called your mate who works for the council and asked him to update the database to show that you had paid your taxes, they'd have no way of knowing.

[TM™]

The Bureaucrat's Paradox

Ironically, these expensive, late delivered, 'm'uck ups are usually caused by a culture of extreme risk avoidance, time estimate fixation and penny pinching.

[Anonymous Coward]

Re: The Bureaucrat's Paradox

They certainly avoided any risk of an Audit.

[Fred Daggy]

Re: The Bureaucrat's Paradox

Alternate translation, "Avoid. Don't risk your Audi."

[Charlie Clark]

Re: The Bureaucrat's Paradox

And well-meaning government initiatives, the most recent of which have encouraged councils to take large financial risks. The shysters on the other end are happy to go along knowing that the taypayer will end up making them whole, very whole.

However, in one respect this was the chronicle of a disaster foretold. They were warned about the very clear risks regarding equal pay and decided to ignore the warnings. It's possible this may lead to charges of negligence against those involved. Well, it's nice to hope.

[aerogems]

It's like peeling an onion with this story, and each layer brings some fresh new case of, "What the bloody fuck where they thinking!?"

[David 132]

Indeed, but "leaving the auditing and security systems disabled" rather renders the first four words of your question moot. "Were they thinking", full stop?

[aerogems]

I'm pretty rarely the sort who says, "Someone should lose their job for that," but in this case it seems absolutely justified. Anyone who signed off on that particular decision* should consider themselves lucky if they only lose their job as opposed to also finding themselves up on charges for something like misappropriation of public funds.

* Unless they have one hell of a good reason. No idea what that reason could possibly be, and I'm doubtful one exists, but I leave open the possibility, however faint, that there may be one.

[Ian Johnston]

Let's just hope that there is a clear trail showing precisely who made the decision. Oh, wait.

[Brewster's Angle Grinder]

Also, it keeps making you cry (with laughter, despair, or both - your choice.)

[Terry 6]

Biggest in Europe

On one hand it's hard not to think that maybe it's just too big to manage.

On the other, presumably lots of other very large authorities do manage. Ultimately it's an issue of scale rather than complexity.

Which brings us back to the question of why there isn't an off-the-shelf system that any local authority can use, maybe with bits that they can just not use if they don't need them.

[Yorick Hunt]

Re: Biggest in Europe

Because then nobody would get the kickbacks so lavishly sprinkled by vendors when signing contracts for bespoke solutions.

[Pascal Monett]

Re: it's just too big to manage

It's only big because there are no subdivisions. Somebody is obviously a control freak and cannot stand the idea that his local empire be divided and his "power" be diminished.

So there's just one big blob of incompetence at the top, instead of spreading the risk and maybe finding someone competent somewhere else.

[ITMA]

Re: it's just too big to manage

You are talking about the disease which has always infected local government - empire building.

[Ian Johnston]

Re: it's just too big to manage

Not just local government, but you are right about always. It's almost seventy years since C. Northcote Parkinson wrote "the number of workers within public administration, bureaucracy or officialdom tends to grow, regardless of the amount of work to be done. This was attributed mainly to two factors: that officials want subordinates, not rivals, and that officials make work for each other."

[ITMA]

Re: it's just too big to manage

You are Sir Humphrey Appleby under an assumed name, aren't you!

Go on, admit it!

LOL

[TVU]

Re: Biggest in Europe

That is an excellent point and perhaps the size of unitary local authorities ought to be capped at the 300,000 population size to stop management becoming so complex and top heavy.

[ITMA]

Re: Biggest in Europe

But... But.... But...

How would their CEOs be able to justify their telephone number salaries?

[Cliffwilliams44]

Re: Biggest in Europe

NetSuite, OpenGOV, Priority Software, RDA Systems

That's just a quick Google search.

But I have to say, and please understand this is just a personal observation having worked for a UK based company for 23 years, that, IT managers over there on that little island tend to only consider the biggest and shiniest options. Like replacing a functioning ERP, developed and supported by a local UK based company that was specifically targeted to our industry with a bloated, expensive, monstrosity from Oracle that didn't have a presence in our industry. Like migrating to Google Cloud and then to Azure when Google got too expensive, all because some idiot believes AWS is a "consumer grade" cloud provider. Like farming out our entire UK support services (Help Desk) to Fujitsu and buying their crap hardware which resulted in a 50% failure rate on hardware and an average 45-day resolution rate of service calls.

These are just 2 of the decisions that turned out to be absolute cluster F's because someone just HAD to hang their name on something big and shiny thinking it will make their career! It didn't, it destroyed their careers!

[katrinab]

Re: Biggest in Europe

Largest Lower Tier Local Authority in Europe. Not even the largest local authority in Birmingham.

Largest in Birmingham in the West Midlands Combined Authority, largest in Europe is Île de France (Paris Region).

But Birmingham is probably responsible for more things than either of those. Île de France has 2 layers of local government below them.

[claimed]

Probably because you never asked for it. Iron clad RFI, RFP, timeline, stage gates… but no one who actually gives a shit about getting the thing on, working, and useful. 1000 jobs to move from tick box one to tick box two , with nobody autonomous enough to say: what the fuck are we trying to do again? Oh, let’s just do this thing.

Nope, if it’s not in scope it’s not getting done, because there is hell to pay if you’re 30 seconds late with the deliverable for a modern, flexible, process tracking user input/output system with validation and data input control (a tick box)

[Anonymous Coward]

no way of knowing if financial fraud has been committed?

Getting involved with Oracle should have been a clue.

[Derezed]

Sounds like it was too hard to configure / specify so got put off until go live…

[ColinPa]

And the quality award goes to

40 years ago I worked in Germany for a couple of years and companies had to have some software installed to monitor access to data. This was a government requirement, and was audited.

This software kept getting awards for software quality. It has been installed at x thousand sites - and only a small number of defects were found - have an award!

They then changed the audit question from "Is it installed? " to "Is it installed and running", and the answer went from yes to no.

If you enabled it, and ran with it, it killed performance.

When customer's turned it on, it had so many bugs, the government eventually removed the question from the audit ( because their systems stopped working)

10 years ago I was doing an audit on a customer system, and asked if a security feature was turned on. They said yes. I then asked to see the reports, and they did not know how to do this. No one had been given the job to process records and produce reports. It was always another department's job.

Hanlon's razor: Never attribute to malice that which is adequately explained by stupidity.

[IanRS]

Re: And the quality award goes to

Back in the days when I was a real techie, rather than an architect, I occasionally had clients who wanted an IPS installed. If I already had a good working relationship with them I would ask whether it had to be functional or just auditor compliant. A worrying number, mainly in the financial sector, just needed to be auditor compliant. "Yes, we have an IPS." "Yes it is running" Just don't ask if anybody looks at the reporting dashboard.

[sarusa]

Anything involving Oracle is going to be disastrous ^_^;

But I guess such a simple and accurate summary would negate the deliciousness of all the specific examples.

[Coastal cutie]

Just when you think this whole thing can't get even more bizarrely incompetent, along comes another jaw dropper

[FirstTangoInParis]

What’s going on, El Reg?

So we have the rolling popcorn-fest that is Birmingham, and elsewhere City of London is entering “Here be dragons” territory with its SAP upgrade. I’m sure commentards can regale endlessly with similar stories.

Question is, what is the in-depth take on this? What are the typical bear traps? What needs time and effort and experience to get it right from the get-go? What checkpoints should be mandatory to stop things going off-piste? What does best practice look like? In short, how can organisations stop themselves becoming ransomware victims, not from the usual scum but from consultants from the likes of Oracle and SAP?

How about it, El Reg?

[Cliffwilliams44]

Re: What’s going on, El Reg?

If you want to distil this down to the most important piece of this puzzle, I'll give you my opinion.

Having watched from afar our UK business convert to Oracle and having participated in merging our 2 largest US businesses into one and migrating business 2 into business 1's ERP. The answer is the Project Manager! Both of these efforts did not go well. The Project Managers on both of these projects were completely incompetent for the task.

1. The did not have the technical and subject matter knowledge.

2. They did not have the strong personality to force past the resistance from the business and the foot dragging from the vendor. (Both females)

3. Poor communication, not just with management but with the staff who had to slog through these projects while also desperately trying to their work done.

The decision to move to Oracle was a bad decision out of the PMs hand, but it was made, that was not going to change, At that point, the PM has to realize this is going to be a challenge and attack it diligently.

[Martin M]

Re: What’s going on, El Reg?

> (Both females)

How is that relevant?

[Dunstan Vavasour]

Public Money, Public Code

Quite simply, anything developed for the public sector should be public code. Councils do about the same thing as each other. Hospitals have the same IT requirements.

Basic public service software: general ledger for a council; appointments booking system for hospitals --- should be developed by central government and made freely available, and UK public sector bodies should have *&^% good reasons for not using it. Councils are not that complicated, the legislation is the same for everyone, and to fuck up a basic ledger system is so inadequate.

https://publiccode.eu/en/

[Ken Hagan]

Re: Public Money, Public Code

With the added caveat that central government is not allowed to move the legal goalposts until central government has endured and verified that the public software can handle both the new requirements and the switchover.

Updated: This requirement has the added advantage of ensuring that the legal requirements are self-consistent. Not that there is any risk that they wouldn't be, of course ...

[Shaun Winfield]

Re: What’s going on, El Reg?

I'm a SAP Payroll consultant, so not normally involved in all the stuff that causes a total shit show, but adjacent to. I've also worked directly employed in local government.

Don't want to get ripped off? Know what you want is key, probably hire your own tame consultants so they can check what is in the contract is what you think it is. Not because the vendor is necessarily trying to screw you. But converting technical detail into real life requirements is rife with honest misunderstanding.

But it's shocking how many clients have incredibly vague requirements, so assumptions get made for the sake of contracts / pricing which they never read. When it turns out they meant something else entirely, that's really on them.

I see, why don't all councils just have a standard ERP a lot in the comments, well, none of the different departments in a single council would ever knowingly agree on a process, let alone separate authorities! Efficiency is the enemy and must be stamped out seems to be the normal driving principle.

[Mike Friedman]

Oracle rips off everyone. This is not a new type of issue. In the us they are notorious for burning through wads of money and the implementations never work properly. Oracle is evil.

[Terry 6]

But they still get the contracts. See also G4S , Crapita, and the rest of them. Ever failing upwards to the next mutli-million quid fuckfest.

[xyz123]

Deliberately accepted "faulty" system with no security because Birmingham councillors etc have been on the take for many MANY years.

They basically raped the local area of 100s of MILLIONS in funding, straight to their own bank accounts.

[The man with a spanner]

This is quite a strong claim. Is there any evidence?

[Ian Johnston]

OK, so given that the council changed the specifications, their consultants ripped them off, the supplier is avaricious and the IT industry generally couldn't find its arse with both hands - all entertaining subjects for discussion - where on earth does the council go from here?

A quick google suggests that Birmingham City Council has an annual budget of £3.2bn, so presumably they are spending around £8.8m per day (that's close to £100 per second) with no effective accounting, records, audit trail or oversight. How can they recover from that? They already have several years of incomplete and/or inconsistent records, and the effort of reconciling all that while keeping an eye on current expenditure, all manually, boggles the mind. I honestly can't see any solution. Can anyone?

[Ashto5]

Legal Liability

That would focus minds.

From the most basic grunt to the CEO they are all held financially responsible.

Snatch back pensions / bonus etc and if that does not cover it then you go after their assets homes / car / clothes off their backs.

Pretty sure they would be reading the small print then.