Lloyds app glitch turned transactions into shared experience for 447k users

A botched overnight software update at Lloyds Banking Group left up to 447,000 customers briefly seeing other people's transactions in its mobile apps, with the bank now acknowledging the scale of the incident and compensating affected users. Details of the incident emerged in a letter from Jasjyot Singh, the bank's CEO of …

[Chloe Cresswell]

Strange interactions

For once the inability for me to use Lloyds banking app was a plus?

Who'd have thought it...

[Mishak]

"a flaw in how the updated API handled simultaneous requests"

So, a race condition.

A race condition in a banking app was one of the first examples we were shown of "bad things" in multi-treaded systems when I was at uni.

Doesn't give me a lot of confidence in the current banking platforms, especially as this is something that needs to be right-by-design rather than something you try to detect/eliminate in testing.

[Anonymous Coward]

Re: "a flaw in how the updated API handled simultaneous requests"

And not using a distinct context for each session's database accesses

[elsergiovolador]

Re: "a flaw in how the updated API handled simultaneous requests"

Didn't they move their IT to India?

https://www.ft.com/content/a304cf5a-5d91-4d4d-a41f-16651b59e758

[frankyunderwood123]

Re: "a flaw in how the updated API handled simultaneous requests"

I wasn’t aware of that, but it doesn’t surprise me.

The huge corporate I work for is doing similar, in a fairer way though.

New software engineering hires are now 80% from India, mandatory.

The skill level once limited to more junior roles is now up to architect level.

The software output quality?

Not good.

Clearly a lot of vibe coding.

Comms is awful, most the engineers will not speak until spoken too, waiting for architects to tell them what to do.

They will agree with everything, not daring to challenge.

Test coverage is either poor or non existent.

I’m clinging on for as long as I can, but I expect it’ll just be a matter of years before I’m considered too expensive.

Four Indian engineers can be hired on my salary.

[Anonymous Coward]

Re: "a flaw in how the updated API handled simultaneous requests"

Fancy that - almost the same points as told to me by a friend of mine who has worked for Lloyds!!!

[Valeyard]

Re: "a flaw in how the updated API handled simultaneous requests"

They will agree with everything, not daring to challenge.

The dangerous thing is that this is exactly how Indian QAs act, which makes them worse than pointless since you think something has been checked

[Lee D]

Row-level

All this tells me is that their central database has zero row-level security.

I don't really care about "yeah, that's the norm in finance" - that's ATROCIOUS.

Even off-by-one's, API errors, timing issues, etc. should not allow an app only authenticated as me to see other people's financial transactions.

[lordminty]

Re: Row-level

Lloyd's are still using a very secure database on backend mainframe, with the online transactions almost certainly provided by CICS. They've been secure for absolute decades.

This is very clearly a fault in the 'modernisation' they've done to bolt on a sexy online banking and app web interface as a front end.

[Anonymous Coward]

Re: Row-level

'modernisation' - literally, head in the "cloud". And let's not forget the drive on AI

[Dan 55]

Goodwill payout

But if you check in the app, how can you be sure it's your goodwill payout? Perhaps just one customer received it.

[Anonymous Coward]

Re: Goodwill payout

Sorry

I think I got yours.

[IGotOut]

Re: Goodwill payout

If you need to claim, let me know your phone number and I'll get that nice person in India to help guide you through the whole process. They'll even log onto your pc to get the payment through even quicker.

[Fruit and Nutcase]

Re: Goodwill payout

Perhaps just one customer received it.

Goodwill hunting!

[dippy1]

And I was just told 2 days ago by Lloyds security that I really should use the app as it is the most secure way to access my account!

[Mishak]

Unfortunately...

That may be true.

[Lee D]

I fell out with NatWest early in their online banking era because they told me to use their online banking because it was "secure".

What it did was load an unsecured page with a Java applet that popped up and showed a padlock, and you banked inside the applet.

I don't expect the poor branch manager to understand but... that's not secure.

There was no reason they couldn't at least secure the page with the Java applet, even, but they didn't bother. And plenty of their rivals were using proper SSL and plain web-based banking systems at the time.

They even tried to tell me that it was somehow "more" secure... so I closed my account with them and went with a rival that understood SSL instead.

[Anonymous Coward]

Cookies

There are at least two UK banks that use cookies to 'secure your browser' or 'secure your computer'!

I changed to a new PC with a new OS recently and moved everything from the old PCs browser to the new one, including the browser cookies.

Neither bank noticed the change, and treated the new PC just like the old one, even though the OS had changed.

No "we see your using a new device", "we noticed a new login on Windows 11" or additional MFA checks. Nothing.

And don't start me on all the sites online banking polls out to when you log in. Why is it even calling Google Analytics, never mind some online chat outfit entirely external to the banks domain?

Security? What security?

[Eric 9001]

Re: Cookies

Being able to login to an account with the username and the secure password, without braindead additional login steps, like an additional weak password, is not a indication of a security problem.

It should be irrelevant what OS you're using to access a website.

The indication of security problems is rather ??? 3rd party JavaScript and also 1st party JavaScript that implements functionality, that is worse than what HTML5+CSS natively offers.

[Anonymous Coward]

To be fair, it probably is.

Ironically, it's also the most secure way other people can also access your account.

[Amblyopius]

There was a Reddit thread the Saturday before of a user experiencing the exact issue already. So they did a Canary release days in advance and then totally ignored customer feedback (even worse, they just blocked the customer from using their account for days). So the answer to "The bank is now reviewing how that defect slipped past its design, testing, and quality assurance processes." is them not actually doing any proper testing/QA given they had all the needed feedback, sat on it for days and then just released it to all their customers. I'd be surprised if they reported the earlier breach in time to the ICO given they keep pretending it wasn't an issue until the one morning they exposed all their customers to the problem.

[StewartWhite]

The bank is now reviewing how that defect slipped past its ... testing ... processes

Didn't you know, testing is for wimps! We need to go at AI speed to do more stupid things even faster as AWS seem to have now mandated.

[Anonymous Coward]

Re: The bank is now reviewing how that defect slipped past its ... testing ... processes

Yup, move fast and break things.

Might be fine for video streaming platform, banking not so much.

[Anonymous Coward]

Do you have a link to that Reddit thread?

Should make interesting reading.

[elsergiovolador]

Lloyds groups should have their banking license revoked over this. This is banking not a fking dating app. If they lose their license maybe other banks will start treating IT a bit more seriously, not something that you can outsource to Asia like your trainers or t-shirts.

[Anonymous Coward]

As a Lloyds Commercial customer...

... who is unable to see any transactions in their online portal, to be honest this actually sounds like progress.

Luckily the "download everything as a CSV" function is stil working.

[Mishak]

Downloads

I download my statements at the end of the tax year to pass them to my accountant.

I moved on from HSBC as it only allowed me to download statements for a month at a time, with each download needing three clicks though a number of tabs (select month, open in tab, right click to "save as", or something). The downloads were called 'Statement.pdf", "Statement(1).pdf", ...

I used to then (manually) pull them all into a single pdf, removing the ads and notes they added at the end of each statement (which were basically what they mailed out if you got paper copies). They then added "security" to the pdf that meant I had to "print to pdf" before I could do that (so the "security" was not effective).

Worse still, I had more than one account.

New bank (Starling) - select account, select start date, select end date, click download - gives a single pdf for the whole year.

[elsergiovolador]

Bold

It says it will consider further claims if any financial harm emerges.

Bank should have been closed down for this.

[Anonymous Coward]

private static List<Transaction> transactions

Is enough.

[Blue Screen of Bleurgh]

This has to be a breach in GDPR. One would hope ICO will deep-dive into this upgrade bodge because it may well happen again either with Lloyds or some other bank.

Data breaches are happening far too often as of late, which doesn't offer much confidence when age verification requirements happen across the board and your PI is at the mercy of IT managers who don't fully understand what is going on under the bonnet

[MatthewSt]

Not the trade off they think it is

> What this incident brings into focus is the fact that there is a trade-off. By moving more interactions with our bank online, we place our faith in technology which can suffer unpredictable errors.

Unless their staff are using a disconnected paper system, this has very little to do with customers interacting with the bank in an online manner. A race condition could still happen on staff terminals.

[Anonymous Coward]

Offshored & no testing

The change in question was to reduce daily API calls needed to show in-app transactions from ~40m to ~10m. Implemented, "tested", and approved to ship to prod entirely off-shore in India, where Lloyds have been scaling their presence over recent years.

The test report for lower environments was no more than "the healthchecks work" on the API to be changed, and some, limited, load testing. Classed as "low risk" & "low impact", hence no need for wider oversight before the change went into prod (The risk-based system exists to prevent moderate/high risk/impact changes making their way to prod without additional review from expert teams - typically UK based)

This was not properly tested. However, an issue like this should not be possible - row-level security can be expected as standard, and why this was omitted, either in the DB or Cache, is unknown at this time.

Their back-out plan didn't work, hence the near 2-hr recovery window once issues were detected.

Heads really ought to roll on this one, and accountability taken by those at the top who've been behind the drive to offshore and knowledge-drain their UK staff over recent years. After all, Mr Nunn's mission is to "Help Britain Prosper!"

[CrazyOldCatMan]

Re: Offshored & no testing

Of course it was tested - it worked fine on the Dev PC with the testing script that tested one of each sort of transaction (mostly)

[Cav]

"The crossover works both ways: some customers saw other people's transactions, while others had their own details briefly shown to strangers." Isn't that stating the obvious?