How the collapse of local cloud provider caused biz continuity issues in UK government The collapse of a relatively small "local" cloud hosting service caused "real business continuity issues" in the UK's central government, according to one commercial lead. Speaking to MPs last week at a Public Accounts Committee hearing to investigate the government's relationship with digital technology suppliers, Andrew …
Exactly this.
outsourcing between business entities has its advantages (IMHO far outweighed by the risks, but as I said, that's just my opinion) but when government - ANY government - tries it, it always ends up with crappy service and the business entity sucking so hard on the teats that the poor cash-cow practically turns inside out.
Depends it gets Fujitsu, Oracle, Cr@pita, Motorola, Sopra Steria, IBM etc to do any work in them.
Some suppliers perhaps fall into the new Supplier Debarment from Public Contracts legislation that just came until effect Procurement Act 2023. See Grenfell Tower/Angela Rayner.
Fujitsu and the Post Office debacle & Oracle and Birmingham Council seem like ideal test cases for debarment.
Arguably, the cloud business is sufficiently mature for there to be some open standards for service provision that would facilitate migration between different providers and possibly provide an opening for alternative operators.
I'd have thought this would be an opportune moment to start funding such an effort, leaving the question of whether the standards might be advisory or mandatory hanging in the political air.
> the cloud business is sufficiently mature for there to be some open standards for service provision that would facilitate migration between different providers
You're cutting the providers too much slack here. This has nothing to do with the maturity of the market- it was an obviously-desirable option even when The Cloud was still relatively new and little more than a concept. (*)
While I'm sure the various providers will make excuses and rationalisations, it's quite clear that they're in no rush to get rid of proprietary standards *because* they're an obstacle to migration, i.e. an effective means of lock-in.
(*) Indeed, I seem to remember that such potential flexibility was one of its selling points, along with the idea that cloud services would be both resilient and location-independent because the setup of the underlying infrastructure- and transparent allocation of compute power- was to be the provider's problem.
"why is it outsourced?"
Because successive governments have chosen to outsource everything that possibly could be outsourced. That's why air sea rescue is outsourced to the private sector, military aircraft maintenance is outsourced to the private sector, ownership of air refuelling tankers is a PFI, FFS. Armed services recruitment outsourced to serial failure Crapita, armed services accommodation outsourced to some shitbag outfit who failed. Private sector welfare and disability assessment, another outsourcing failure. Likewise £80bn of staggeringly poor value PFI dumped on the NHS. It's why private companies run prisons, and transport convicts between courts and prisons, operate failed tagging and probation services. DfE outsourced a national pupil tutoring programme for post-Covid catch up, and the crappy provider was encouraging schools to report tutoring sessions delivered even if there were no pupils in attendance.
It's worth knowing that other governments (eg Germany, Sweden, Holland) outsource similar or even higher proportions of government spend, but there's an unfortunate belief in the UK that outsourcing is the answer to everything. Link below is somewhat dated, but still relevant:
https://blogs.lse.ac.uk/europpblog/2018/06/13/why-public-sector-outsourcing-is-less-efficient-than-soviet-central-planning/
Outsourcing never works well in public sector because you pay for service and someone else's profit. Depending on a markup, tax payer might be paying even 10 times over what it should cost.
In case of outsourcing to foreign companies this is even worse, because money doesn't stay in the UK and doesn't benefit our economy.
Outsourcing works well for services that are consumed by both public and private sector. (e.g. Office cleaning) It doesn't work well for services that only the public sector operate. (e.g. Prisons)
Even then, it depends on the type and level of outsourcing.
A department or building manager contracting in a cleaning agency with clearly defined responsibilities and standards, and that staff are actual employees of the agency - not temp/zero-hour labour? Okay, that's one thing.
A department outsourcing building management to a property management agency, who then outsource cleaning to a cleaning agency, who don't employ any staff and just get ad-hoc zero-hour workers in from a temp agency, fully embracing evil maid opportunities?
Alas, the latter is too common. Private Eye spent a while reporting on a hospital cleaner trying to get £45 of unpaid wages out of 3 levels of outsourcing who all insisted they'd paid their bit down the chain. The coffee and biscuits at the meetings to sort it out probably cost more than the sum the poor cleaner was owed. At some point you have to ask "well why can't the NHS just employ their own cleaners?"
"The only dentists I have seen in the last two decades have been in adverts on TV."
There's few dentists taking on new NHS patients for the simple reason that the government/NHS have set standard costs for procedures such that unless a dentist stuff their books with more NHS patients then they can feasibly treat, they lose money treating them. NHS dentistry was widely available, but essentially all governments since Thatcher have been pushing "semi-health" services like dentistry, opticians, podiatry etc towards the private sector.
" Private sector welfare and disability assessment, another outsourcing failure".
My daughter who has severe fibromyalgia along with major arthritis issues had absolutely no issues what togever to get PIP. Admittedly I had to go to the original hearing as:
* her representative in her initial hearing (appeal rejected)
* her representative in her appeal hearing (appeal rejected)
* Write her request for permission to appeal to a Tier 1 tribunal (unpaid for all of this)
* Read her response from the request to appeal to that appeal (this bit is just unbelievable) Appeal denied as it wasn't necessary; 5 major flaws in law in a three page judgement.
* Attend the new phone heard case along with my wife, which was cancelled 15mins before the appeal as apparently no one had noticed that the three judges had been recused from th original case. (Well I did but apparently that was insufficient until the court had already been assembled - when someone finally bothered to read the papers. We had no prior notice of the panel, and so could not challenge it in advance) My wife took an unpaid day off work, leading to about 20-30 patients being cancelled at relatively short notice, with no compensation for their fuck up, but they sent my daughter an email which said they couldn't send it to me as I wasn't her representative [ Want to take a bet on that ] but no offer of compensation nor any sugestion that they would do anything to ensure this didn't happen again (like, read the paperwork, you know, it can't be that hard. Perhaps between important things like making a cuppa , or sucking the arse of the boss).
* Getting a letter which said they had decided there were no problems, ands thus removing all support. The Dr involved assumed that my daughter had walked to the shops, refused to listen to me as "they were now getting somewhere". After she had demonstrated that she had the forensic skills of a dead wasp I could finally ask a simple question; "how did you get to the shops, {name}", to which she replied "You drove me there". And how often have you done this ("not sure but 2 or three times in the last year"). This was followed on by "I see from your medical reccords provided that you have an almost normal range of motion of your arms, how do you explain that?" Again she would not let me help her with her desire to read that was which not there. Eventually I got a chance to make the point that this was passive movement (i.e. someone moving her arms, did not describe the pain involved as it was an assessment for osseus blocks), and got the response: "Oh I didn't notice that." Being the noce friendly chap that I am and bloody furious I asked the tribunal to adjurn until a competant medical advisor could be found. And then asked her where she qualified and if that was there normal standard.)
* eventually getting a letter from DWP who said that they had agreed my daughter needed the highest level of attendance allowance buts the tribunal had decided she didn't need mobility allowance. Which also stated "In view of previous decisions I have decided to overide that action
So no, I think you are as deluded as the system if you think there are any issues at all.
(Oh, and BTW, before I entered IT I was a clinician, and if I had made most of the mistakes they did then I would have been suspended, if not struck off.
So, it may or not be private, but fuck them all.
Perhaps they should have taken it k to public ownership on that basis - see banks, rail franchise, Carilllion … and used it this way as local Sovereign Cloud Provider and now AWA/Azure/Google - as Tangoman led USA is proving an unreliable partner to the world now..
I would imagine it was probably actively pushed into administration by HMRC as some delicious irony.
Quite. Bear in mind that the numbers quoted here are insignificant in Government Spending terms (annual budget of £1Tn). £17m? Rounding error.
The MoD bought Sheffield Forgemasters in 2021-22 in order to secure a domestic specialist castings capability - SFIL lurched around from one acquisition to another as "lumpy" defence contracts led them to have a year with £20m profits and then another year with £20m losses. Not appealing to a private investor but of literally no consequence to a nation state who have a fiscal time horizon of a decade to deliver new aircraft carriers/nuclear submarines/whatever. They're quite happy if it breaks even over the course of a decade (or even runs at a loss, since it's a national security issue).
There's no particular reason why GDS shouldn't run their own data centers/on-prem, or at the very least manage their own infrastructure in suitable colo facilities (depending on departments, what counts as List X, etc), some of which may constitute a private cloud running OpenStack or whatever package they deem most suitable/useful.
There's absolutely no reason to leave Continuity of Government at the behest of market forces or the stability of a private operator.
Cost lots of money.
Company worth little, but you could by it cheap as a going concern. It would keep things running for you and potentially save money in the long term.
Why have the Government let it fold if it's that important?
Something strange going on here unless it's a demonstration of how we should feed the MS/AWS pigs at the trough and not doing things ourselves ...
And not the first time it's happened recently.
"Forgemasters*" is now a wholly-owned subsidiary of the MoD.
It's not a stretch to think they make the pressure vessels for British nuclear powered submarines.
Somewhat surprising that BAE didn't snap them up as part of their every-bit-of-the-defense-supply-chain-we-own strategy.
*And IIRC suppliers of bespoke "Gaspipe" to Saddam Hussin back in the day.
Government should have mandated the use of UKCloud only for UK data, why put billions in the pockets of USA corps who nobody really trusts with the data or the US Govt to keep off, especially with the current fascist Trump in the Whitehouse determined to ignore any and all rules and agreements if it helps his MAGAland theme park.
>Government should have mandated the use of UKCloud only for UK data
They did.
The Government Security Classification Scheme specifically mandated that any data formerly above BIL 2-2-x must be resident in the UK on an approved, assured and accredited platform, right up till it was re-written in June of last year (to remove that requirement).
As a result - formal Ministerially signed HMG policy from 2014 to 2024 was specifically to use UK providers.
had that policy been applied UK Cloud (and probably other UK Sovereign prooviders) would have a solid footprint in the UK today and we wouldn't be exporting our critical data assets around the world.
However.... no-one adhered to that policy, and GDS/CCS/Cabinet Office/etc. all also pushed the 'use Public Cloud' model in favour of the Hyperscalers which was (perversely) NEVER part of the Cloud First policy when it was Ministerially created in 2013 - that was just policy by "blog and innuendo".
The reason it caused continuity issues is doubtless the same reason people were using them at all - they were a provider of cloud services suitable for certain official levels of security and that isn't exactly a big pool of players as it's a lot of effort.
If you needed 'cloud' that couldn't go on the usual public hosts and you weren't (or couldn't) running something suitable privately then they were where you went, especially if interacting with something already on there.
It's been a while but I seem to remember that they were involved in stuff like secure Azure hosting; if you wanted Azure at a high enough level of security it was actually them running a private instance.
I used them too for a couple of things that fell between what suited the normal public clouds and what we would allow on our various private ones.
Ultimately they filled a useful niche but didn't have the scale, and even if they had managed to last longer they'd still have got squeezed out by the hyperscale players on one side and in-house stuff on the other.
... & run various services for both, other taxpayer funded entities, & private businesses - worldwide.
I worked with some amazing software/ hardware/ network people, gurus, really.
I was a 'lowly' data centre “operator”, but the role entailed much more.
The entire site is within the purview of the business/ scientific research arm of the administrative state.
I won't name it.
There are dozens of similar sites across the UK.
What possible reason is there for not using this capacity, & expertise? Beyond corruption, I mean.
This could've been handled 'in-house', but, it appears, 'friends, & family' need some lovely taxpayer money.
Signing a renewal contract just before declaring insolvency smacks of corruption - they (the company leaders) must have known the true state of said company.
Sadly, this is simply one more instance of wasting taxpayers money, add it to the tome.
Read “Parliament Ltd” by Martin Williams (2016), & weep!
Is Mr Forzani getting muddled up? UKCloud, Datacentred and others were providing cloud services to HMG long before the hyper-scalers became dominant. It was the not so subtle change from a "Cloud First" to a "Public Cloud First" policy that did the damage in terms of over-concentration and lack of leverage with AWS and MSFT (with the consequent resilience and dependency issues). It simply isn't true to say there was a policy to move away from hyperscale and introduce local providers. The reality was the polar opposite and there is a wealth of evidence to demonstrate it.
WHAT:
I'd put it all on AWS, try to move it to a managed EKS with managed Services, centralise around a sane supportable set of Technology choices.
Push everything into K8S operators which we can interchange, and try to kick start a UK market in those services around common use cases for those technology choices.
Recruit and retain expertise in these choices, and slowly start to migrate to an on-prem Kubernetes for compute, using the AWS purely for managed services interconnected with a DirectConnect (MPLS tunnels)
Bring managed datastores back on-prem one at time, e.g from RDS to in-house supported DB platform tuned by our DBAs.
WHY:
Terraform/IAC tooling exists to standup / teardown infra and scaling tooling with price awareness can deliver quite good savings. It's easier to turn stuff off, we need to move in house, but lets build the mess in their sandbox, and lift and shift the good stuff in house.
We should be able to get vast discounts from AWS, so it could be quite competitive to use AWS in addition to on premise footprint.
We should be able to run on K8s Clusters to ease migrations to on-prem Linux Server, we can replace the various components as needed - but if S3 works - use S3 and move on.
Put components we'd like to replace from AWS out to commercial tender with the deployment deliverable being K8S operator/CRD pairing.
Provided a deployment meets our standards, we don't care about implementation technology, and we get an escrow artefact, with a reproducible path.
Staying fairly close to widely used tech means we can swim with the tide to some extent picking up security improvements.
So along that path, we'd then get to increase the UK pool of skilled talent, for a globally marketable skillset.
We should be able to get vast discounts from AWS, so it could be quite competitive to use AWS in addition to on premise footprint.
Oh you sweet summer child. That isn't how it works. Anyone who's tracked AWS spend over the last decade knows it's the basic drug dealer model. Get customers hooked, jack up the prices and lock them in.
I think a lot stems from purse keepers not really understanding either what the cloud really is, what their requirements are, or what the risks will be. So years ago I went through this exercise with some government types and showed them that the cloud is really just the same kinds of HPE servers, switches and abstraction stuff that they could buy themselves. So for organisations the size of the MoD, NHS etc we/they could buy a pile of tin, plonk it in a datacentre and have their own 'cloud'. Then if we were running it instead of them, and they ever wanted to bring it back on-prem, there would be an option to yoink the rack(s) out of the datacentre and let them have it.. Which isn't exactly something you can do with AWS or MS. Or many other 'clouds'.
Oh I agree it's a huge cost, but the billing model is visible so challenge UK market to beat them in competitive tender.
Either we can do it so we save money, or the cost of AWS is acceptable.
The Gateway drug for AWS is IAM once you're high on that ACL supply, it's very hard to migrate the permissions but we can plan for this by keeping permissions tied to the Clusters themselves.
We need some IAM type - thing - is it so terrible if we rent from AWS.
The tender is the saving grace of this plane, we have to prove you can't do it for better value, and if we do it for better value. Happy days.
The K8s data model is essentially yaml interchange formats, so no reason that can't be farmed out across the UK to meet a specific need. I think as a GOV platform for open / fluid mixing between AWS and On-Prem with a goal to fund a OS replacement component, it could work.
Deliverables have fixed scope, are easily specified, integrated compared and delivered.
We can't run it all in house currently because we've not got a coherent platform.
Exactly as you say, we could run our own kit, in our own DC racks, and employee local people to carry the pagers.
I think we need to force a fait-acompli and the greed of Bezos combined with the lure of saving money to Public Acclaim.
See we give a shiny platform with "open-standards" we "deliver taxpayer value" supporting "British Business" by "Encouraging competition".
The more Bezos gouges us, the more compelling the replacements are.
Problem us always CapEx. You can't expect government to stump up money for hardware (or building a sovereign DC), even if the lifetime TCO is better. Much less the increase in headcount to run it (even though you're saving a king's ransom in Cloud payments).
"You want to spend how much on servers? Can't you just run with AWS for a while and we'll review next year?"
Rinse and repeat for every aspect of the economy - transport infrastructure, housing, energy security/infra, hospitals, schools.
The perpetual bleat of "Oh, but it's cheaper to rent/outsource". And 20 years down the line you're bleeding money to PFI schemes who have abandoned the property they're supposed to be maintaining.
UK Cloud whilst partly accountable for their fate were pushed over the edge by government changing from supporting UK service providers to throwing all their cash at the American hyperscalers with no control. The government and civil servants are 100% to blame and 70% of government cloud use is nothing more than IaaS purchased in the most expensive way possible. Even worse, those public cloud providers contribute nearly nothing back in terms of tax contributions.
In local gov its always a bullshit tender process where the cheapest bidder wins. So the big companies can undercut everyone to win the contracts. Smaller companies never have a chance. They may have gone in low to try and get the contract then realised it was a mistake after.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
