
Making changes
On a Friday - what could possibly go wrong?
The UK is full of unhappy workers that are unable to manage their payday cash amid online service outages at a host of major banks. Downdetector indicates trouble at Lloyds Bank, Halifax, TSB, Nationwide, First Direct, Bank of Scotland, and Barclays, although the latter's woes appear to have been resolved since the surge of …
Actually the rules came in 31 March 2023 but as you say the last day before they are enforced is 31 March 2025.
(If you think U.K. Banks are bad I know one Bank in the EU where ALL it’s mainframe software is out of support, in some cases by 7 years and their mid-tier vulnerability/exposure index has over 5 million entries!
Murphy works at competition.
But if there is failure, embrace it and learn from it. If it was down to the team, not circumstances outside of control, study it, talk it through, how it could have been mitigated.
If it was due to gaps in the knowledge? Replenish training fund and make sure team members who contributed get to learn more.
If it was due to lack of care? Investigate if there is burn out. Maybe the workload is too large or pay is too low (team members worry about the bills and not the product).
If there was miscommunication? Was specification as correct as it could be? Was the failure and edge case that was not considered? Did we listen to input of everyone? Are people afraid to raise concerns?
etc. etc.
> how it could have been mitigated
One popular mitigation is to not perform risky manoeuvers immediately before the entire workforce of your employer and all of your suppliers goes home for 2 days, so that the people who can fix a problem are available to do so when one occurs.
Rather than the negative rule of... dont make changes at the end of the week and the end of the month.... there should be a more positive approach of - we will plan majour changes in the first week of the month, idealy on Tuesday.
Urgent changes - any week except month end, and idealy mid week.
Screamingly urgent changes are permisable at end of month or end of week if and only if the A team is on alert.
This really should be a hard and fast rule.
"One popular mitigation is to not perform risky manoeuvers immediately before the entire workforce of your employer and all of your suppliers goes home for 2 days"
The timing is important and so is having people on hand when something is being updated, just in case. A plan to immediately roll back an update is a good move as well and to try it out first for one entity rather than all customers at one time.
No effective rollback, at a bank? What if they want problems? Ever considered that?
What was that Sherlock homes adage; "when you have eliminated the impossible, whatever you are left with no matter how improbable ...".
How often is commercial banking and inter-banking effected?
That only happens when management is not constantly chasing next quarters results (and hence share price) and allow the IT budget to be split favourably for resilience as opposed to "digital" and new apps. Also, the crazy rush to remove all privacy and responsibility of choice. Try getting a decent amount of cash out or paying a crypto related business.
If your engineering function is correctly set up for ci/cd then the next minute is the same as the last and you should be safely deploying on demand.
There's no excuses for outages anymore. Uptime is a solved problem.
The answer here is to fine banks per account per hour or part thereof of downtime. Then they'll stop using cheap engineers the way you'd stop using a cheap parachute.
All major UK banks testing at the same time on payday? Not unless their managers have all suddenly been replaced with idiots.
Common payment problems would mean a problem with SWIFT, but that wouldn't explain the frontend problems.
Also this is the second month running, Barclays was a month ago. There's something we're not being told here?
IR35 prevents worker-owned businesses from operating as real businesses.
Consider this: You work at a consultancy and realise they’re making massive profits from your work while adding minimal value themselves. Naturally, you might think - "Why not cut out the middleman?" You could leave, set up your own business, find clients directly, and do the same work while earning more.
In a fair system, you’d be free to quit on Friday, start your business on Monday (or after a reasonable waiting period), and continue working - only now with the added benefits of setting your own direction, training, handling sales, and growing your business. IR35 makes this impossible by taxing all your company’s revenue as if it were a salary under PAYE - before business expenses are even considered - making it impossible to turn a profit. Meanwhile, big consultancies face no such restrictions, allowing them to continue dominating the market while small businesses are effectively shut out.
Furthermore, British employers are notoriously reluctant to invest in training. If you wanted to stay at the top of your field, going independent was the best way to do it - you could upskill yourself in areas that interested you and had market demand, rather than whatever knowledge gap your employer thought needed filling (if they didn’t just hire someone who already had the skills instead). This created a highly skilled, mobile workforce. Since IR35, that incentive is gone - you can't expense training, and wage compression (Boriswave, etc.) has destroyed the market. The result? A deskilled workforce, stagnation, and fewer opportunities, all while "bums on seats" cheered about cracking down on so-called tax dodgers. Big corporations won again.
Yes, you got treated as any other company until somebody at HMRC noticed that individuals were making money out of it, and got religious about it. HMRC invented IR35 and have been hitting contractors with it ever since.
Even though HMRC's interpretation of the rules has been shown to be false in a number of high-profile cases, the dogmatic nutters at HMRC.insist on applying those rules all the time everywhere. Mpst people do not have the resource to fight it, so cave in.
As others have said, no.
The more nuanced answer is that some people were taking the urine. Imagine a "contractor" who only has one client, has worked for that client at the client's site for several years, doing what the client want, when the client wants it, and using the client's equipment. You might look and think - that's an employee in reality, but using a tax dodge to make more money.
Where the same person works for a large(r) firm, and in theory can be moved around and substituted, then it looks more like a genuine contractor.
The problem is, HMRC looked at the first example and thought it bad - how dare someone (to coin the old adage) arrange his finances within the rules to minimise the size of the shovel HMRC uses to take it's share ! And HMRC introduced the IR35 rule - which essentially says that if HMRC decide you look like a disguised employee, then you'll be taxed as if you were an employee - but you won't get any of the benefits (job security, holidays, sick pay, etc.) that an employee gets. And IIRC they can do this retrospectively, so having paid the tax due, HMRC can rock up and decide you should have paid income tax - so hand over a big wedge, plus penalties and interest.
But the biggest problem is that there isn't a simple rule you can apply and come up with a yes/no answer to whether IR35 applies. There are "woolly" rules related to whether the contractor can substitute another person (in a one man band, not possible), whether they work for multiple clients (big consultancies have many clients, one man bands tend to have one at a time for the duration of a contract), whether the contractor or the client dictates things like working times (often the contractor has to work with the other staff so have to work something resembling the clients standard working hours), whether the contractor uses their own or the client's equipment (often, the client requires the contractor to use the client's systems for security). All of these stack up against a sole contractor, but have zero effect on the big consultancies. And as mentioned, HMRC lose "many" cases. Oh yes, and for good measure, HMRC has an online tool to check these questions, but apparently it is "majorly incorrect".
And because of the risks, many businesses simply stopped using contractors, or made them all go (temporarily) on the books as an employee, or just switched to using "safer" big firms.
All this together has caused massive disruption. True, some were taking the urine, but of the contractors I've dealt with during my career, they've mostly been specialists where the client wants the flexibility to "buy in" some knowledge for a short term specific project - without all the overheads that come with taking someone on as an employee (even if only temporarily). And here, "clients" can include government departments - c.f. all the "government IT failures" which often stem from not having the right people in place at the right time. As mentioned, you could go and work for a bigger consultancy firm - only to see your skills squandered based on manglement whims, while manglement creams off a very significant slice of what the client is paying them.
And for good measure, I've read reports that suggest it's cost HMRC more to introduce and enforce IR35 than they've actually taken in though higher taxes. I.e. it's possibly resulted in a negative net benefit to the tax coffers.
Lest there be any doubt, I'm a permy, always been a permy - so this isn't a disgruntled contractor having a moan. I've not considered contracting as I don't have the non-technical skills to manage it. As mentioned, as a contractor you have to manage the whole business, allow for reserves to cover sick, allow unpaid time for holidays, allow unpaid time for self funded training, and accept that your client can inform you on Friday afternoon that they don't need you back on Monday (oh yes, you need to find unpaid time to cultivate your next clients).
Lloyds, Bank of Scotland and Halifax - all part of the same group. Llyods and BoS apps are the same platform / layout etc, just a different skin overlaid it seems.
My BoS app seems to work, the Lloyds one has the account showing still but not data!
TSB was part of Lloyds, perhaps the same platform in the mix still ~10 years they left the party?
Or is there a 3rd party supplier of app based banking they all share?
TSB migrated out of Lloyds in 2018, although that didn't go well...
You're right about the other three being the same banking group, they probably run different OS instances for the brands, but there will be crunch points somewhere in the chain where they use the same bit of infrastructure. Doesn't explain how Barclays got caught up in it, so could well be some third party service which threw a wobbly and some banks didn't cope as well as others. Or it just some random coincidence, we may find out from the banks' post mortem comms.
I have accounts with both and being a sad IT git, have my own domain. Email address was halifax@aaaaa.com. I got a Lloyds one, set that with lloyds@aaaaa.com and then halifax emails went to the same address.
Tried to complain and was told "we are the same group", but pointed out that according to FCS you are independant (hence £75k protection) and so everything should be retreated so, including email addresses. They didn't get it and nothing done
Now they are doing a new project where you can go into any Halifax / Lloyds / BOS and do ALL banking for any of those 3 banks regardless of the branch. This is a new feature - only becuse they are trying to close more branchines IMHO.
My mum moved banks due to bad policies and customer (dis-) service at Bank A. Things were good for her at her new bank, Bank B. Bank B was taken over by Bank A, so she escaped to Bank C. Bank C also was taken over by Bank A. She fled to Bank D.
As she put it, "The turkeys gobble up everything."
"Tried to complain and was told "we are the same group","
At that point, it would be prudent to move one or all of the accounts someplace else. If all of those brands are under one umbrella, they might be managing them all with the same team, using the same software and management scheme. Any whoops is going to lock you out of all your accounts at the same time.
I can concur of the same experience, it seems the data matching/transfer seems to work as well with actual account data being overwritten from either side.
I know this because as a stupid and naive 18 year old I didn't use my full legal name to apply for my first current account with the horsey bank. Think like putting down Alex as opposed to Alexander.
Many years later, I applied for a credit card with the bank that gives you extra, giving my complete legal name and for a time everything stayed separate.
Then, as if by magic, and without warning the shortened name on the Lloyds account became the one on the Halifax one, and I started getting Lloyds emails to the Halifax one.
I haven't bothered to correct them on the issue, however, all the emails are now blocked from both. It seems that because a bank gets your email it gives them carte blanche to send emails all the time, and there was also a time where they kept sending me emails about something in my digital inbox in the internet banking, I looked, and there was nothing there. Obviously their email send script got jammed at some point, and I got about 10 copies of the same email - very annoying.
So you can now just write to me, and keep the postman in a job!
My TSB branch was in Kirkwall, to reach it involved a 2 hour ferry crossing and a 10 minute taxi ride (and the same for the return journey). Now the Kirkwall branch has closed and my nearest branches are in Wick & Thurso which would involve a further 2 hour ferry crossing and 10 minute taxi ride from Kirkwall. And, because of the ferry timings, it would probably involve at least one overnight stay in Kirkwall.
Fortunately I can check my TSB accounts online and use my local Post Office to withdraw cash.
So who's the systematic point of failure here?
I recruit mobile developers for my team - in finance, but thankfully not in retail banking. Judging from the content of CVs coming across my desk, the consultancies are pretty active in building mobile and web banking platforms and white-label apps. Tata and Deloitte in particularly. Not a good omen. I've also learnt a lot about which retail banks outsource... R...cough...B... cough... Nat...cough, and even their unreleased products. We've not hired a single candidate who's previously worked on these products.
Given this is the primary point of customer interaction, you'd think that retail banks would see building out high quality apps (and back-end infrastructure for them) as a business critical activity, and not throw it over the wall.
"Given this is the primary point of customer interaction, you'd think that retail banks would see building out high quality apps (and back-end infrastructure for them) as a business critical activity, and not throw it over the wall."
I wouldn't think that. Logic would say that companies would care, but experience says no, they really don't give a flying anything. And rather than employ a good UK or European development team they're largely happy to outsource to any third world hell hole or corporate IT charlatans.
My experience of mobile apps is that most of them are ropey at some level - either the core functionality is inadequate or illogical, or they're flakey and unreliable, or the interface and UX is poor, including basic crapness like failure to work well with scaled displays, or the company can't be bothered to update to address basic failings around taking payments. Sometimes they "update" their app and if keeps asking the same stupid questions everytime you open it, showing that there was zero testing before release. It's pretty obvious that when they can't get those basics right that most apps will have very poor security. Not to mention shitty privacy policies and shittier licence agreements.
And sadly, banks generally do have better than average apps. The rats nest of awfulness that is on display in car parking payment apps shows that some sectors slam together a poorly thought through spec, outsource to anybody who says they can do it, and the resultant bodge is released without any proper QA. And there's plenty of big companies that could afford to and should know better - look at the world class awfulness of Sony camera apps (the reviews for Sony Imaging Edge are worth reading purely for the comedy value of some of them).
"Not to mention shitty privacy policies and shittier licence agreements."
I recently installed Blokada 5 on my phone following a recommendation here.
My bank's app immediately fell over because it wasn't able to access Firebase, the Google app analytics stuff. I had to whitelist it. I'm thinking of contacting the bank and asking them to explain to me how sending usage information to Google (in the US) is in any way GDPR compliant, and why they are basing their "is the internet active" test upon the presence of an unnecessary (and often blocked for privacy reasons - it's on Blokada's block list) site instead of, you know, their own site (that wasn't blocked)?
Given that it's a French bank (I live in France), I don't expect to hear any response whatsoever.
My experience (of blocking various tracker sites - both at DNS & ASN level) is HSBC just refuses to work. Barclays is fine. Kent Reliance blocks access full stop (although I suspect this is more down to my use of VPN’s) Nationwide is okay.
The worst offender is Google Captcha (I block Google, so frequently don't even see the challenge and wonder why page has stalled) I hate Google …
As an aside found sometime ago an interesting article written by a developer analysing his Barclays account logon process. Whilst most of it goes over my head, some of the more technically accomplished amongst us may enjoy a read (note it was circa June 2021, so a lot could have changed since)
https://www.bentasker.co.uk/posts/blog/security/732-uk-banking-protecting-you-by-exposing-you-to-risk.html
It's advisable to keep some cash around for this sort of problem, at least you "might" be able to buy food. But sooner or later the whole banking system will get closed while they devalue your currency and introduce your full tracked and controlled digital currency. It will be Putin wot dun it though ...
All the existing transactions, card usage and such are said to be working normally, it's the customer facing stuff that's sick. Pretty much matches the TSB meltdown a few years ago when they were transferring from their former parent company.
It reinforces my belief that banking IT has a hard split between the money handling teams, probably still maintaining ancient COBOL, and the webby types who may be less rigorous. OTOH everyone being on fire at once suggests a single point of failure outside of the banks themselves. What centralised service could they all be using just for the internet UIs?
Anyone have relevant experience to comment?
I have no experience but I'm going to put a pound coin on it being a balls up due to the web back end (which is the exact same thing for each bank, just with different graphics and styles) pulled in a dozen unvetted and unverified third party resources (sort of like npm), and one of those has changed in some manner, like maybe an unexpected 301 redirect, and everything fell over in a heap because of it.
That's my guess.
Icon because it is a pleasant afternoon and I've had a shitty day at work so I'll get my coat and go for a long walk.
I have definitely never worked for a bank, even posting as AC.
I had I would say that what the public (including business and most internal staff) see is completely separate from the core banking system. I would also suggest that the core banking system is probably on a mainframe installed in the early 2000s or even older, and most things people do will be in external apps the have a very strictly defined interface to the core system.
Mainframe hard ware (basically IBM Z series as it's the only one still standing ) is usually refreshed every five years or so. The operating and related software systems although descended from 1970s MVS are actively supported and constantly updated. As are the probably COBOL based core banking systems (still some CICS assembly code out there though).
For the most part these systems just work and almost every large bank has at some point made a failed attempt to replace them with a more "modern" technology.
The current trend is to preserve the COBOL code and port it to a fake mainframe environment in the cloud.
Everyone's cash handling runs on mainframes, as far as I could tell from my stint in the industry. The web stuff talks to servers that translate between the mainframe's batch-mode, database-driven design, and GET requests. In effect there are three main systems—not always from the same provider.
Here in the US, Jack Henry & Assoc. provide the frontend stuff for many regional banks. No clue who's in that space in the UK.
There have been a few big ISP outages as well, notably virgin but also others. Maybe just a coincidence and they are different issues, or maybe some systemic issue with multiple companies trying to cut costs, making the people who know how it all works redundant, outsourcing to the cheapest provider etc. I guess it could also be probing cyber attacks as geopolitical ‘games’ start heating up…
I managed to log into Lloyds bank this morning. The secret was to be patient. Like most websites, the first order of business is to contact a third party analytics/advert providing website to customise and improve your experience. The problem was that my firewall blocked the connection to the analytics website, which had changed from the one I had previously allowed, and my browser sat waiting for the site tags.tiqcdn.com to load. Eventually, after waiting a few minutes, and getting no response, the bank’s website loaded.
Also this morning, I had the same problem with the Royal Mail website, when trying to track a parcel. The page was also stuck trying to contact tags.tiqcdn.com
Maybe it was just coincidental?
"If you've sent money already or are waiting for money to arrive you don't need to do anything, it's in a queue and will arrive ASAP," its website reads.
"You can still send money, but this won't go through straight away. Direct Debits and standing orders are working normally."
So if direct debits will still go out but money won't come it, when you go overdrawn are they going to waive the £35 (or whatever it costs these days) charge for each failed payment? Because I doubt it.
"when you go overdrawn are they going to waive the £35 (or whatever it costs these days) charge for each failed payment? Because I doubt it."
yes, they will. But the quality and morality of the bank will show depending on whether they cancel/refund charges automatically for everyone or manually only for those who complain.
Found these old Bank of England token things in a spare pocket this morning. Had the KIng's head on one side (or his mother's). Weird! Anyway, my local shop was kind enough to accept a few in exchange for the morning's fresh bread and milk. Wow! So simple and easy! The way of the future?
... I had zero problems this morning, but I can commiserate with those who were trying to get into their accounts.
That said... if many banks have the same issue, but they are all using their own backends, it's something they do in their websites or their mobile UIs (which no doubt are all web-based). The web/mobile teams at all those banks will need to have a think about not relying on external services maybe?
Maybe it was as "simple" as a broken/missing library
(Although why large orgs can't pay to self host critical code is beyond me)
... but not HSBC. I was involved in integrating their systems back in the day when they merged.
Back then First Direct just became a trading-name for the senior bank.
Have they separated again in some way?
Retired now so haven't been watching things banking closely.
I adamantly don't have any financial stuff on my phone. I do keep cash on hand as a reserve and try to do the bulk of my day to day from my pocket rather than with a piece of plastic. If I have a limited amount of cash on me, I am limiting impulse purchases. I do still have a debit card and will use it for things like filling up with petrol or buying groceries, but not a quick coffee or mid-day take away. I've seen people with bonk-to-pay or receipts/tickets on their phone when the phone runs flat. They aren't getting on the train, but I am with old fashioned paper. One video I watched, the person managed to talk themselves through as they left their phone on the train (long distance) in their compartment and didn't realize that the station had a closed platform since it isn't that common in the US.
My advice is to always make plans for when electronic access to your funds is not available. It's going to happen and if you have plans for the weekend, it will suck to cancel due to the bank being stupid.
You'd think - and I know this is a wild one, so try and go with me on this - you'd think that any huge enterprise that makes £billions and moves £billions, would actually care about their computing structures and spend money on it rather than engage in smoke and mirrors so as to eel out a few more Shekels from punters.
Wouldn't you?
Just me, then.
Banks that email out helpful communications that advise how to guard against online scams etc. One such tip: never follow links in unexpected emails - no matter who they claim to be from.
"Click here to be taken to our webpage outlining further ways to secure your money."
Genuine emails from real banks - obviously run by a management that listens to IT security experts before dismissing them with the Marketing Department's claim that such communications will show customers (victims, saps, etc.) that they really are a caring establishment.
Frankly, given how the banks skimp on computing systems and express mock horror and surprise when things go wrong, I'm surprised anyone trusts them with their money.
Oh, there's no competition? Oh, I see. But what about cryptocurrencies?
Ah, but no real existence there. No branches. No one to talk to if you need to sort anything.
Er, isn't that the exact situation we are facing now with the banks?
- El Reg Reader, living in a town where the last bank standing is shutting its branch this summer...
I fairly regularly get emails from a bank who issued me with a credit card.
At the bottom of the email on SOME of them, there's the following bordered (for emphasis) message:-
We want you to recognise a fraudulent email if you receive one. *** will greet you personally using your name and the last four digits of your account number: 0123
*** obfuscated
Two problems:-
One is that my credit card does not end in 0123.
Second is that I receive communications from them on 4 (FOUR) different DOMAINS. (Note: not email addresses, of which there are more, but four different domains). One problem arising out of that is if I want to search for one of their communications I have to do so using different search strings.
At the moment they are hassling me with PGP secured messages purporting to be KYC type messages which look really spammy. TBH if they want to revoke my card, fine, they don't follow the rules, I don't want their card.
How the #### is the average Joe supposed to know what is genuine and what isn't?
Still no real detail from the Barclays outage a few weeks back and now this.
Let’s not forget that Halifax and Lloyds also had issues when Barclays went down.
Parliament also issued a demand to 9 banks to explain what happened and find out how resilient they are.
The lack of transparency doesn’t smell right?