City council audit trail is an audit fail after disastrous Oracle ERP rollout Birmingham City Council, Europe's biggest local authority, has no way of knowing if financial fraud has been committed after it failed to run security and audit features in a new Oracle Fusion ERP system. The English council, which was responsible for £3.7 billion revenue [PDF] in 2021, also continues to struggle with " …
"Birmingham City Council, Europe's biggest local authority, has no way of knowing if financial fraud has been committed after it failed to run security and audit features in a new Oracle Fusion ERP system."
I'm no expert, but I would start with the clown who suggested the use of Oracle and follow that particular trail.
Same with us. The fool that convinced management to pick Civica. He was a consultant, they always listen to consultants over their own staff. We're sure he was getting brown envelopes from Civica as it wasn't the best system and has none been proven to not be the best system now live. I suspect same has happened here, an consultant has recommended Oracle due to brown envelopes.
OMG yes, Cívica are the pits!
I have to deal with one of their aged products which provides 'controlled' and audited access to a major national database and it is the biggest heap of excrement I've ever encountered.
All bug reports get rejected as being either features or not important.
It's tightly and pretty much irrevocably bound to a much-loathed database for user controls and auditing
User passwords are stored as symmetrically encrypted stings without salts (i.e. easy to spot who has the default password!! And all can be decrypted using a rather unimaginative fixed key anyway).
I wish I could say more here because they are taking six figures levels of public money annually for horrendous software and it wouldn't surprise me if some of their more public software is just as terrible (the council websites I've seen bearing their name do not give me any confidence!)
TL;DR For the love of your favourite choice supreme universal power: do no but anything from Civica !
I have seen similar issues a few times in the past. Performance was one reason, having logging on at the required level killed performance or the service completely fell over.
The other reason was incompetence. Someone forgot to switch on the logging service and rather than fix it the issue was avoided or forgotten as the responsible person moved to another contract.
I understand that neither of these reasons are excuses, simply what I've seen.
Mistakes happen. But that situation would come to light when the Finance Director or Audit committee demanded to see the audit logs prior to signing the system off to go live, during parallel running and at least monthly thereafter.
What, the beancounters didn't do the basics on that either? What a surprise.
Performance was one reason, having logging on at the required level killed performance or the service completely fell over.
In that case the system was incorrectly specified. A financial system which is legally required to provide audit reports that cannot perform when auditing is enabled is clearly not compliant with user requirements. And the sales team that proposed it, and must have known that auditing was a legal requirement are as culpable as the purchasing team that did not specify auditing along with performance levels.
I doubt it was missed on the requirements. I also doubt it was due to technical performance constraints. Suspect it just got lost somewhere along the project. Requirements traceability could have caught this or maybe a pen test / assurance review which certainly would have caught this one. Probably!
Murdoch Syndrome.
As first seen from James and Rupert at the Levinson (phone hacking enquiry) Enquiry. Harvard Business School must have done a Grad Research Project on it and added it the MBA Curriculum.
Didn’t see anything, didn’t hear anything, don’t remember anything, don’t know anything…. Nothing to do with me… … sorry I’m just the CEO/Chairman. I’m really sorry about it though. Is it lunch yet. Garrick??
More recently seen in the wild at the UK Covid Enquiry, Post Office Enquiry, MoD Arms Corruption to Saudi, Investigations into Water Industry Institutional Sewage dumping …….
Ronald Reagan, in the Iran-Contra inquiry by the Tower Commission, back in 1986.
"Er, um, I cannot recall." Said later in the inquiry, after he initially said he had authorised the deal. Thankfully lots of documents had been destroyed which might have brought his reputation and competence into dispute.
Reminds memos the Florida US Senator Rick Scott who was CEO of HCA Corp (a large hospital company) that was fined $1BN for defrauding US Medicare ( health insurance for older folks) and claimed to know nothing about it.
So he was either lying or incompetent. Or both. But voters mysteriously believed him
The fundamental truth about the Post Office is that as an institution it still believes that all those sub postmasters were guilty and are being let off on a technicality and therefore that nobody in the organisation did anything wrong or has anything to be apologetic, ashamed or embarrassed about. Which is why they keep apologising to the enquiry with all the forced insincerity of a child saying sorry to get pudding.
I really must contact them about my £2.4m unpaid invoice that seems to have vanished in their system.
Would it have been an idea to have run the new system in parallel with the old one until the new one worked, or is that just crazy talk.
If they'd have gone back to paper, a hundred A4 pads, a lot of 4-colour biros and several calculators later, they would have sorted their finances and saved themselves £90m. A moral there for everyone.
I would also pitch for parallel systems. I don't understand how large projects EVER decide to play switcharoo in the middle of active use of the system and think that's acceptable.
At this scale, I'd want parallel systems with the same data put into them and see what happens.
Have you ever been involved with an ERP cut-over? I have, multiple times. I'm yet to see a company have sufficient staff and resources to double-entry processes into the new and old systems; in fact it's probably a really, really bad idea. With a cutover, you pick a time - say, after all the end-of-month processes are completed - and block access to the system. You migrate the live data over the nearest weekend, set the old system to read-only, and start transacting in the new system on the Monday. The old system is now there for reference, and will be used repeatedly for about three months, then it's use will fall away dramatically.
Entering data into a test instance of the old and new systems and watching what happens is robust testing. Entering live data in the old and new systems is a recipe for disaster.
"Would it have been an idea to have run the new system in parallel with the old one until the new one worked"
What and have people find out that the new system is a pile of poop and there never really anything fundamentally wrong with the old system that a little TLC couldn't have fixed?
Once the old system is out of vendor support, no amount of TLC will keep you regulations compliant, unless you commit to developing the system in-house from now on. Which, as you can guess, is a very expensive and very risky thing to do: what happens if you in-house developers leave?
Yup.. let's make guesses about exactly who might think "audit trail" and "segregation of duties" might be an impediment to doing business........
Someone INSIDE the council?? Someone OUTSIDE the council?? Larry Ellison??
Ah yes.....How about Larry Eliison and that ONE BILLION DOLLAR INVOICE???????
>To be fair, the worst offenders are VW Golf drivers who spent all their money on baffle free exhaust systems and can't afford to get the valve guide oil seals and piston rings sorted...
Because Halfords don't offer valve-guides or piston ring kits slathered in plastichrome and green LEDs, obviously.
And well-meaning government initiatives, the most recent of which have encouraged councils to take large financial risks. The shysters on the other end are happy to go along knowing that the taypayer will end up making them whole, very whole.
However, in one respect this was the chronicle of a disaster foretold. They were warned about the very clear risks regarding equal pay and decided to ignore the warnings. It's possible this may lead to charges of negligence against those involved. Well, it's nice to hope.
I'm pretty rarely the sort who says, "Someone should lose their job for that," but in this case it seems absolutely justified. Anyone who signed off on that particular decision* should consider themselves lucky if they only lose their job as opposed to also finding themselves up on charges for something like misappropriation of public funds.
* Unless they have one hell of a good reason. No idea what that reason could possibly be, and I'm doubtful one exists, but I leave open the possibility, however faint, that there may be one.
On one hand it's hard not to think that maybe it's just too big to manage.
On the other, presumably lots of other very large authorities do manage. Ultimately it's an issue of scale rather than complexity.
Which brings us back to the question of why there isn't an off-the-shelf system that any local authority can use, maybe with bits that they can just not use if they don't need them.
It's only big because there are no subdivisions. Somebody is obviously a control freak and cannot stand the idea that his local empire be divided and his "power" be diminished.
So there's just one big blob of incompetence at the top, instead of spreading the risk and maybe finding someone competent somewhere else.
Not just local government, but you are right about always. It's almost seventy years since C. Northcote Parkinson wrote "the number of workers within public administration, bureaucracy or officialdom tends to grow, regardless of the amount of work to be done. This was attributed mainly to two factors: that officials want subordinates, not rivals, and that officials make work for each other."
NetSuite, OpenGOV, Priority Software, RDA Systems
That's just a quick Google search.
But I have to say, and please understand this is just a personal observation having worked for a UK based company for 23 years, that, IT managers over there on that little island tend to only consider the biggest and shiniest options. Like replacing a functioning ERP, developed and supported by a local UK based company that was specifically targeted to our industry with a bloated, expensive, monstrosity from Oracle that didn't have a presence in our industry. Like migrating to Google Cloud and then to Azure when Google got too expensive, all because some idiot believes AWS is a "consumer grade" cloud provider. Like farming out our entire UK support services (Help Desk) to Fujitsu and buying their crap hardware which resulted in a 50% failure rate on hardware and an average 45-day resolution rate of service calls.
These are just 2 of the decisions that turned out to be absolute cluster F's because someone just HAD to hang their name on something big and shiny thinking it will make their career! It didn't, it destroyed their careers!
Largest Lower Tier Local Authority in Europe. Not even the largest local authority in Birmingham.
Largest in Birmingham in the West Midlands Combined Authority, largest in Europe is Île de France (Paris Region).
But Birmingham is probably responsible for more things than either of those. Île de France has 2 layers of local government below them.
Probably because you never asked for it. Iron clad RFI, RFP, timeline, stage gates… but no one who actually gives a shit about getting the thing on, working, and useful. 1000 jobs to move from tick box one to tick box two , with nobody autonomous enough to say: what the fuck are we trying to do again? Oh, let’s just do this thing.
Nope, if it’s not in scope it’s not getting done, because there is hell to pay if you’re 30 seconds late with the deliverable for a modern, flexible, process tracking user input/output system with validation and data input control (a tick box)
40 years ago I worked in Germany for a couple of years and companies had to have some software installed to monitor access to data. This was a government requirement, and was audited.
This software kept getting awards for software quality. It has been installed at x thousand sites - and only a small number of defects were found - have an award!
They then changed the audit question from "Is it installed? " to "Is it installed and running", and the answer went from yes to no.
If you enabled it, and ran with it, it killed performance.
When customer's turned it on, it had so many bugs, the government eventually removed the question from the audit ( because their systems stopped working)
10 years ago I was doing an audit on a customer system, and asked if a security feature was turned on. They said yes. I then asked to see the reports, and they did not know how to do this. No one had been given the job to process records and produce reports. It was always another department's job.
Hanlon's razor: Never attribute to malice that which is adequately explained by stupidity.
Back in the days when I was a real techie, rather than an architect, I occasionally had clients who wanted an IPS installed. If I already had a good working relationship with them I would ask whether it had to be functional or just auditor compliant. A worrying number, mainly in the financial sector, just needed to be auditor compliant. "Yes, we have an IPS." "Yes it is running" Just don't ask if anybody looks at the reporting dashboard.
So we have the rolling popcorn-fest that is Birmingham, and elsewhere City of London is entering “Here be dragons” territory with its SAP upgrade. I’m sure commentards can regale endlessly with similar stories.
Question is, what is the in-depth take on this? What are the typical bear traps? What needs time and effort and experience to get it right from the get-go? What checkpoints should be mandatory to stop things going off-piste? What does best practice look like? In short, how can organisations stop themselves becoming ransomware victims, not from the usual scum but from consultants from the likes of Oracle and SAP?
How about it, El Reg?
If you want to distil this down to the most important piece of this puzzle, I'll give you my opinion.
Having watched from afar our UK business convert to Oracle and having participated in merging our 2 largest US businesses into one and migrating business 2 into business 1's ERP. The answer is the Project Manager! Both of these efforts did not go well. The Project Managers on both of these projects were completely incompetent for the task.
1. The did not have the technical and subject matter knowledge.
2. They did not have the strong personality to force past the resistance from the business and the foot dragging from the vendor. (Both females)
3. Poor communication, not just with management but with the staff who had to slog through these projects while also desperately trying to their work done.
The decision to move to Oracle was a bad decision out of the PMs hand, but it was made, that was not going to change, At that point, the PM has to realize this is going to be a challenge and attack it diligently.
Quite simply, anything developed for the public sector should be public code. Councils do about the same thing as each other. Hospitals have the same IT requirements.
Basic public service software: general ledger for a council; appointments booking system for hospitals --- should be developed by central government and made freely available, and UK public sector bodies should have *&^% good reasons for not using it. Councils are not that complicated, the legislation is the same for everyone, and to fuck up a basic ledger system is so inadequate.
https://publiccode.eu/en/
With the added caveat that central government is not allowed to move the legal goalposts until central government has endured and verified that the public software can handle both the new requirements and the switchover.
Updated: This requirement has the added advantage of ensuring that the legal requirements are self-consistent. Not that there is any risk that they wouldn't be, of course ...
I'm a SAP Payroll consultant, so not normally involved in all the stuff that causes a total shit show, but adjacent to. I've also worked directly employed in local government.
Don't want to get ripped off? Know what you want is key, probably hire your own tame consultants so they can check what is in the contract is what you think it is. Not because the vendor is necessarily trying to screw you. But converting technical detail into real life requirements is rife with honest misunderstanding.
But it's shocking how many clients have incredibly vague requirements, so assumptions get made for the sake of contracts / pricing which they never read. When it turns out they meant something else entirely, that's really on them.
I see, why don't all councils just have a standard ERP a lot in the comments, well, none of the different departments in a single council would ever knowingly agree on a process, let alone separate authorities! Efficiency is the enemy and must be stamped out seems to be the normal driving principle.
OK, so given that the council changed the specifications, their consultants ripped them off, the supplier is avaricious and the IT industry generally couldn't find its arse with both hands - all entertaining subjects for discussion - where on earth does the council go from here?
A quick google suggests that Birmingham City Council has an annual budget of £3.2bn, so presumably they are spending around £8.8m per day (that's close to £100 per second) with no effective accounting, records, audit trail or oversight. How can they recover from that? They already have several years of incomplete and/or inconsistent records, and the effort of reconciling all that while keeping an eye on current expenditure, all manually, boggles the mind. I honestly can't see any solution. Can anyone?
That would focus minds.
From the most basic grunt to the CEO they are all held financially responsible.
Snatch back pensions / bonus etc and if that does not cover it then you go after their assets homes / car / clothes off their backs.
Pretty sure they would be reading the small print then.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
