UK tax dept's IT savings created 'significant risk', technical debt as it faces difficult conversation with Chancellor Cost-cutting at UK tax ministry HMRC has resulted in IT systems that “constitute a significant risk to the department”, according to national spending watchdog the National Audit Office (NAO). As the holder of the keys to the UK's treasury, chancellor Rishi Sunak, prepares his Spending Review for 25 November, the NAO said the …
@Hollerithevo
"Free meals during school breaks during COVID? Stuff like that?"
I read something about that. How parents would let their children starve or something. If I read the solution right it had something to do with opening up schools for the kids to go eat, with all the associated heating/lighting/cleaning costs to keep such buildings habitable but without the teachers and any learning. Assuming I read that right it would be a huge waste of tax payer money unless the teachers showed up and classes were on.
I assume that is the right plan that was proposed, let me know if not.
Please, we spend billions on so called "poor people" in this country and still we hear "children are starving, give us more money". The problem is that the WOKE crowd in this country won't do anything about the problem of parents neglecting their children and trading their welfare benefits for drugs. Their only solution is to confiscate more of the countries wealth but not fix the problem so they can continue to use it as a campaign issue.
Unfortunately government departments only administer the laws they are given. Bad laws = Bad administration.
As has been seen in Universal credit it was intended to be a way back to many when we had near full employment, it turned out it turned out to be a bully's charter.
Combine this with a reluctance to pursue sensible taxpayer targets HMRC are bound to fail.
Sack the idiots at the top until they start sacking the idiots in the middle.
Technical debt in IT? Deferred maintenance and upgrades? For hardware maintenance, the cases need to be popped open for cleaning out the dust and what have you, hard drives need periodic defragging. This is not including replacing parts that fail. Software maintenance includes running backups, applying patches for security and bug fixes. The SDLC for government systems can be measured in decades, and is usually part of waterfall methodology. That's government thinking for you. Mission critical systems, such as the systems that handles the collection of taxes and then doles it out to the different agencies as part of their capital budget, you cannot accrue technical debt in the maintenance of those systems. You might be able to do it for awhile and get away with it, but when something fails, you got the PHB running around with his two tuffs of hair on fire.
At some point, you have to upgrade the hardware. The two most replaced items in the box due to failure (in my experience) are hard drives and power supplies. Other components such as memory do fail, but it's rare. Video cards are another item that fails, but those are seldom. Backup tapes are another thing that needs regular replacement. The budget for failed components should be factored in to the total cost of ownership, but it rarely is. How many of you work or have worked in shops where they maintain an inventory of spare parts? I've worked in a few, but when budget cuts roll down the hill, that's the first victim. When the inventory is depleted, then chaos ensues because a system is dead for several days because the parts are on order, if they are available at all. There was one IT department where the budget was so tight, the management asked the IT folks to buy replacement parts out of their own paychecks and file for reimbursement because it was faster than waiting for official channels.
In corporate America, management types get promoted to their level of incompetence. This is especially true in government. It's good to know that this seems to be the standard everywhere.
Most of this kind of BAU maintenance is priced into their contracts - almost everything is run "as a Service" in this respect. The tech debt accrual is in areas where you need a change project to perform some form of technology refresh or implement a nice-to-have change. Think things like running older versions of RHEL, or not upgrading your SAS estate, or having a domain running on some prehistoric compatibility mode because you haven't got the capital investment to perform the upgrade.
These are all things that are too-easily cancelled when you're asked to deliver 10 years worth of new customs infrastructure in 18 months while your departmental budgets are slashed 40%.
To Quote "For hardware maintenance, the cases need to be popped open for cleaning out the dust and what have you, hard drives need periodic defragging"
Ahahahaha. There speaks the Windows home desktop support guy!
Try, These systems run in a clean datacentre and don't use Windows, for a closer to the truth view.
Anonymous due to circumstances (I might just have an inkling how Govt. IT actually runs
I think you're focusing too much on Hardware. No doubt you can keep hardware running more or less indefinitely, we've seen NASA picking up parts from eBay. But this is in a closed system where you don't need to give a stuff about security.
The problem comes when you're not running in a closed system, and have oodles of an end of life middleware and appliances. For example, you can get away with an out of date weblogic server for a while, but sooner or later you're going to end up with security vulnerabilities. You can get away with running an end of life firewall for a while, but it's not going to protect you against vulnerabilities as well as the shiny new one might, and sooner or later the firewall will itself have vulnerabilities.
An enterprise environment will have hundreds of different appliances and middleware from tens or hundreds of different vendors, most not the least bit interested in supporting end of life software, especially if the entire product range has been killed off. Sooner or later one of those will have a vulnerability for which you don't have adequate mitigations, and that's exactly the sort of thing auditors are going to flag as a "significant risk"
Yes, and then there's bespoke application code, requiring updates to business logic to keep in line with current legislation, written in languages that only those whose IT career precedes Java can support and how much it costs to tempt them out of retirement, or compiled on processor families that are no longer manufactured, with a file format that only a previous generation of tape drive can read and a user base that would take months of retraining per person to migrate off of it.
You know that quote about when you owe the bank £500 you have a problem, when you owe the bank £500 million, the bank has a problem? Extend that to technical debt. Past a certain point it's not even an IT problem anymore.
written in languages that only those whose IT career precedes Java can support
They may as well be in Sanskrit given the grasp most current devs have of anything but their native programming language.
(compared to them I'm a polyglot: Algol60, Pascal, C, x86 ASM, Python, some Perl)
But this is in a closed system where you don't need to give a stuff about security.
O rly?
For the past seven months our entire team (a dozen bods) has been WFH. We do second-line application support in what's supposed to be a walled-off environment, inaccessible from the Greater Internet. Other teams: networking, OS and hardware support, storage and middleware, work from home as well. This basically is no different from being on call outside office hours: you have to have access to that walled-off environment, using a VPN, Citrix, whatever. That gets you into the office environment, and from there via a jump host you can get into that walled-off area.
That's going to need a few security hurdles before you're in. And once inside all the standard authentication to get on to specific hosts and run privileged commands should still be in place too, if only for accountability.
For the avoidance of doubt, I was talking about a truly closed system which has been kept running forever, such as that on a Space Shuttle, or the famous 8" floppies on a missile launch control system. If you're on a space shuttle it's unlikely that your going to run nmap during re-entry...
I never suggested that HMRC was running in a closed system, and it's rare that anything is actually in a closed system.
I had to call the HMRC last year (pre pandemic) on something unrelated to my tax affairs. I asked about Making Tax Difficult at the same time which was 'popular'. The lady I spoke to said I wasn't the first to ask about it. I'm no longer self employed but I rent out a flat and have some small investments so still have to fill in a tax return. I asked why they needed the info my accountants normally gave them at the end of the year four times through the year. That's on top of including that info in my usual annual tax return. Didn't this just cost me more beacuse my accountants would be doing four times the work.
The reply was that I was going to have to do MTD and nothing between now and the start date was going to change that. I could do it myself if I was worried about paying my accountants and the costs involved. I said I might investigate whether letting my tenants pay less as I'm not much above the 10k threshold. That would mean that I don't have to do the extra reporting and avoid the extra hassle and cost from my accountants. She wasn't keen on that at all and I said this created a massive opportunity for people to pay their rent part cash in hand. Asked me if I was going to do that and I said most certainly was not. I was just considering reducing the rent to which she said that MTD wasn't going to be that much extra work.
She asked how I normally sent my accountants my info and how often. I said I send them the PDF statements my lettings agent sent me. I do this shortly after the new tax year starts along with the two paper statements from my investments. She said okay so in my case it will mean I need to do some extra work. She again pushed the idea that I could use an app on my phone to upload the data to the cloud. I stated I was wary of using cloud apps/the cloud given the inability of companies and people to properly secure the data they store on things like AWS etc.
Then I asked about the whole MTD project being flagged Red by the Infrastructure Projects Authority (which I read about on this very site). She said she couldn't comment on that and I needed to wait for the start of MTD before I could complain about it. She thanked me for the call and then hung up at that point.
Articles like this don't fill me with confidence about the HMRC and cost savings.
Beancounters cut into IT budget and it went wrong... how it is possible? At least they learned their lesson!
“take better advantage of technical innovations and keep pace with technology trends in order to support HMRC’s digital transformation and move to lower cost and highly resilient cloud services
Maybe not.
You get what you pay for. As we say in French, you can't have at the same time the butter, the money for the butter and the creamer lady's arse.
you can't have at the same time the butter, the money for the butter and the creamer lady's arse
I haven't heard that one, but then again, I'm not French. Anyways, In that situation, if you are a good charmer, you just might be able to have all three if you bring the first two.
Lower Cost Cloud might refer to the fact that there are now more than just one (expensive) cloud provider ratified for Govt, workloads here in the UK now.
Initially only SkyScape (now rebranded UKCliud) was available and you'd better believe they weren't as cheap as Amazon.
> "HMRC has recognised that, due to the need in the past to forgo operational maintenance and upgrades to its systems to secure cost savings, its IT systems now constitute a significant risk to the Department".
It never ceases to amaze me how any company with a significant investment in existing IT infrastructure thinks they can save money (cut costs) by not keeping it up to date. All you do by not regularly upgrading (ideally before you near the end of the hardware and/or software support lifecycles) is punt the problem down the road. You have to do it sooner or later, and the longer you leave it, the more difficulty, risk and expense you have to deal with when you finally have no other option than to finally get on with it.
Not to mention the security risks associated with not keeping up with regular maintenance schedules.
I suppose this is what happens when the company is run by accountants who don't get the importance of IT thats fit for purpose.
I agree where you're coming from but like to or loath it you have to consider things from the other side.
For example, the suggestion of replacing kit / software / services before nearing the end of life is clearly going to get culled by someone needing to balance the books. I understand why you'd want to do that but I also understand why someone else (who's job it is to find unnecessary spending today) would think otherwise.
The problem is when the mantra that 'as long as it works it's fine' is taken, and we all know problems are inevitable in this instance. And that's where I am. That's where I work. A significant IT spend occurred during a two year project which concluded in 2010(!) and apart from some upgrades, scaled up provision and the rollout of 365, we're pretty much stuck where we were. Madness.
The problem is the bean counting way does not accurately measure costs and risks. Regardless of whether the risk materialises if you are exposing the business to a risk that will cost £100m at 20% likelihood to save £10m then you are an idiot, especially if you do not recognise that the risk is not really 20% but a rising curve. Often you get lucky, but eventually in the words of Susan Ivanova: "No boom today. Boom tomorrow. There's always a boom tomorrow."
Business needs to get better at weeding managers that are gamblers out.
Another way to look at it is this: IT investments are just that, investments. Those investments are usually capital expenditures. When the investment stops bringing value to the business, then it's time to upgrade. One metric is to keep track of failures and cost of replacement (parts, systems, etc...). The moment that starts curving upwards could be a trigger for a new round of investment in IT infrastructure. That's assuming that everyone does their job properly, which we all know doesn't always happen.
One metric is to keep track of failures and cost of replacement (parts, systems, etc...).
Ah, the small-business Windows view again.
Larger businesses and government departments don't work from that metric, as maintenance contracts already cover the replacements. What they do track is downtime, although more often than not that downtime tends to be caused by software or operational errors, not hardware. And when the time comes that the hardware is closing in on the far end of its bathtub curve those businesses and departments will find their maintenance contracts getting more expensive. Still not pure cost of replacement, but a result of simple statistical arithmetic by the vendor.
There's also the cause of operational errors increasing as not only the kit gets older but the staff does too, experienced bods leaving (the company, the workforce or this earthly plane) and being replaced with less experienced folk, often outsourced.
But replacing kit can also incur downtimes due to the new software not being quite ready, integration problems, users being insufficiently trained and several other reasons. And as long as this is seen by the beancounters as the greater bunch of risks they will put off the necessary changes, increasing the technical debt.
Interesting book (well for certain values of interesting):
How to Measure Anything in Cybersecurity Risk
The basic point is to come up with a measure of the true cost of failing to address your risks, in a language that the bean-counters can understand. He's targeting security risk, but you could equally apply the same principles to obsolescence/reliability risk.
You know that in bean-counting country there is something called "depreciation"?
That is: the money to put aside in order to replace anything that has a finite lifespan.
It may even be deductible from the profits.
Just wondering where the money went...
Currently on year 6 of our "four year replacement program". Year 5, which was agreed would have 1/4 of the original purchase cost for everything allocated to replace 1/4 of everything. Never happened. Year 6, same. Never happened. Should have replaced/upgraded/supplemented literally 50% of the equipment, etc. by now. Nothing. And if they'd done it right, they'd have had a fixed, annual expense every single year that they could put into budgets and plan for, and a complete replacement of everything every four years.
Had catastrophic failures. Suddenly we're spending three times what we normally would to put in new stuff. Then we find that we're having to do extraordinary budgeting to cope with that, and tell other departments that they have no money this year because of that.
So IT are the enemy, it's all our fault, nothing works, all our kit is ancient and we've stolen everyone else's money. Amazing that.
Oh, and obviously you have to hire a very expensive outside consultancy to come in, audit everything and then tell your IT guy that and try to upsell all those replacements to things that nobody else understands what they are.
Schools by any chance Lee? Used to do a similar thing with ours - Every year, budget to replace 25% of your kit with new stuff that has a three year warranty on it, and plan on doing that every year so you always have decent kit. Worked well until budgets tightened, and then the false economy of not replacing stuff kicked in and looked like a genius money saving plan until the fifth year of the cycle when stuff failed and you had to start charging for replacement parts/hardware on an ad-hoc basis and maintenance costs started going up... at which point, they start complaining that the kit you sold them 5+ years ago is no good and you're charging too much for maintenance compared to previous years.
@Maelstrom,
You are indeed correct that spare parts inventories are the first victim of a budget cut - assuming there was a spares budget in the first place. But over the last few years I've seen a trend where PHB read about "Just In Time" manufacturing methodologies and misinterpret the manifesto to mean that one actually saves money in an IT or other operation by eschewing minimum essential spares.
"We can get what we need through the channel! Just in time!" As the mission critical system lies silent.
The problem is that "Just In Time" usually isn't. Especially when there are supply chain disruptions due to unforeseen issues like COVID-19. Then if your kit comes from China, you might be waiting a month for it. A solution would be to migrate to the cloud, but that has a variety of it's own problems.
The "channel" in these instances often turns out to be a company who are going to charge you an eye-wateringly high price for the item.
This is because:
a) They have one on the shelf ready to be couriered.
b) They know you must be really desperate to call them.
And sometimes: c) Your BOFH is their CEO
@Loyal Commenter
Two points you might want to consider-
1. If you consider trolling a dyslexic by being a grammar nazi you must have a pretty boring life.
2. You replied to one of my comments and quoted a completely different one! Muppet
Thanks for the laugh, I probably deserve it
If you consider trolling a dyslexic by being a grammar nazi you must have a pretty boring life.
For the long term observation of your posts over several years, I can see no evidence of dyslexia. Not knowing the difference between "its" and "it's", between "their", "there" and "they're", or in this case, between "who's" and "whose", on its own, with no other indications of dyslexia isn't dyslexia, it's a lack of understanding of the rules of grammar. If you were dyslexic, there would be other instances of you mis-spelling or confusing words. So I'm calling bullshit on that one.
You replied to one of my comments and quoted a completely different one! Muppet
You do know the same universe continues to exist in the space between two separate comments on an internet message board, don't you? In colloquial terms, such a logical construct is known as a callback, and is often used in comedy. I'm sure it has in fact been used on the Muppet Show. In this instance, it is used to illustrate the inherent contradiction between your two posts, by employing your own words against you for comedic effect. Glad you laughed.
But then I'm probably too much of a muppet, right?
@Loyal Commenter
"For the long term observation of your posts over several years, I can see no evidence of dyslexia"
I take that as a complement. Thanks. Especially as I am diagnosed as such by 2 professionals.
"If you were dyslexic, there would be other instances of you mis-spelling or confusing words"
I must admit to cheating a little. Computers are very handy with their spell check capabilities. That you complain about my poor command of English but then complain my English is too good for my condition I am wondering what really is your problem?
"In this instance, it is used to illustrate the inherent contradiction between your two posts, by employing your own words against you for comedic effect. Glad you laughed."
So not only did you look like an idiot for being picky about my grammar and posting it as reply to the wrong comment, you claim it wasnt the wrong comment but an attempt at comedy by parroting my comment where it makes no sense.
"But then I'm probably too much of a muppet, right?"
I think we both have that understanding. And thanks for making me laugh again. Never before have I met someone too stupid to accept they are dealing with a dyslexic.
codejunky> To be fair we had a previous government who's stupid mistake of thinking increasing any costs would always improve the outcome. In the business world they would have been bankrupt in no time.
Are you referring to Theresa May's Government or David Cameron's two governments?
There's a lot of valid reasons you might object to a more socialist government in the UK but borrowing more and repaying less doesn't seem to be one of them, if the actual Treasury figures are to be believed.
Not only can failure to invest have a hidden cost, failure to maintain is similar. No point vastly reducing your (or local authorities') road budgets and complaining about the potholes.
But, in the end, this stuff isn't about 'left' or 'right' - it's about ancient truths like 'a stitch in time' combined with the modern practice of rewarding the people who say "oh that doesn't need a stitch yet" without (and this is the crucial bit) ever holding them responsible when they are wrong.
"Are you referring to Theresa May's Government or David Cameron's two governments?"
I was referring to the Labour government who spaffed money up the wall. However you are not wrong about the Cameron/May govs who had more public spending than the Brown years even while claiming 'austerity'.
Not sure if your trying to troll me with that comment but it is very valid and I agree.
Ahh.
A decade on and it's still NuLabour's fault.
It's always someone else's fault with you guys, isn't it.
In another 10 years it will still be Jeremy Corbyn's fault. Or the Leave campaign. Or the hordes of Turks. Or the Communist students. Or poor people. Or ...
(Also, I'm surprised there is no link posted to an appropriate Tim Worstall article. Maybe the one about there being no such thing as Tory austerity and you've never had it so good?)
@Dr_N
"A decade on and it's still NuLabour's fault."
For what? Go back and read the thread.
"It's always someone else's fault with you guys, isn't it."
Who is you guys? What? I know you troll me generally but what are you on about?
"In another 10 years it will still be Jeremy Corbyn's fault. Or the Leave campaign. Or the hordes of Turks. Or the Communist students. Or poor people. Or ..."
For the dodgy managing of public systems?
The 'Technical debt' appears to be inability to deliver what was promised earlier. (Oh and now that's gone tits-up we have to spend more on fixes and more on work-arounds.)
Then 'technical debt' seems to mean they can't do what they've been doing. What's the reason for that? Really? Oh [shuffles papers after three months of continual prodding] here's the answer: We want to go all cloudy. That'll fix everything.
I kind of have to agree with you since "technical debt" is a software development term. But what I'm thinking of is that they are applying the concept of it to IT in general, which does make sense when you think about it. Not exactly equivalent, but it does get the point across.
That's unusual. My sister is an accountant, and she takes her job *very* seriously. That job being to shrink the distance between the top line and the bottom line by as much as possible using any means necessary. I have come to discover that is the common mindset of most bean counters. Although there are exceptions as your case demonstrates.
It's worth stressing that it's not just cost cutting that has hit HMRC. They have - like every department - been subjected to enormous budget cuts across the last decade, HMRC has been subjected to three successive, significant stresses that are unique to the department.
Since around 2015 the department has been in the process of dismantling the old Aspire contract, first via novation, then bringing some functions back in-house and ultimately rebuilding their procurement capabilities for SOTF. This made dealing with certain of their embedded suppliers _exceptionally_ difficult, as those companies unlikely to win future work (rhymes with Jujitsu) went into full bleed-them-dry mode. Then, of course, since 2016 the department has been dealing with Brexit. This has meant crash-investment programs to rebuild things like European-facing customs declaration systems that hadn't been seriously considered since the 1970s. That amounted to a near-total stop on all other investments. And of course now the department has been grappling with Covid, in particular being responsible for the bulk of the delivery around the Coronavirus Job Retention Scheme - which is a superb example of public sector delivery done well - all while reconfiguring the department for remote/flexible working. Again, this amounts to a stop-the-world on almost all other investment. That's four or five solid years of budgets being slashed and all available cash being redirected to things that aren't collecting tax or stopping tax avoidance. That will begin to bite, soon.
The department isn't entirely screwed yet. The advantage to having such a deeply embedded set of suppliers is that they do - for better or worse - know how to keep the lights on without too much fuss, and HMRC as a result have one of the better track records in government. The risk is that the Cummings-driven technical exceptionalism is forced onto the department, wherein the solution to all the department's problems is just deemed to be "Cloud". Never mind that the department's key/core systems are mostly mainframes and SAP. Never mind that the cost-per-hour of cloud instances at scale is off-the-charts. Never mind that the department hasn't been given the budget to rearchitect any of this to work well on AWS or Azure. Between Deloitte's bullshit reporting and No. 10's fantasy-technologism, the department is going to be forced to target all new investment towards Amazon and _that_ is when things are going to go to total shit.
I'm a contract PM and make a lot of my living going in to organisations who have deferred upgrade to the point where they cannot any longer. Acritical business application can no longer receive annual updated for things like tax tables as its going out of vendor support. The App cant be upgraded because the new version isn't supported on the existing database tier. The database cant be upgraded because the next version isn't supported on the version of the O/S in use and the O/S Can't be upgraded because the later version isn't supported on the server in use.
In general I'm called in to reimplement the application on a new hardware platform. This then requires a mammoth data migration project, a full re-engineering of all the business processes, retraining of the staff who use the application, rewriting of most of the business intelligence reports and all to an absolutely fixed timescale (usually about a year). These projects are always incredibly painful and go live is normally very close to the drop dead date. I've got a great record of delivering these kinds of projects but the experience is so painful that I'm not invited back. After the huge cost of the hardware replacement, the lost opportunities caused by diverting key staff members to deliver the project and the disruption to operational departments what do you think they do next.... That's right defer the next application upgrade 'for a couple of years to let the system stabilize. If they don't move to a SAAS solution they then repeat the whole process 5 years ;ater (as the speed of technology change has increased in the past decade)
> A review by consultants Deloitte recommended a 12-month “Rapid Remediation Programme” to focus on the most critical priorities.
Wow, focus on the priorities - I would never have thought of that!
Less sarcastically, it's also wrong because a rapid remediation programme will inevitably focus on 'low hanging fruit' (because the projects selected need to have a vague chance of completing within 12 months) instead of the tough-nuts-to-crack systems that are holding everything else back because they are so complex and entrenched.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
