Egg meet face.
UK 'meltdown' bank TSB's owner: Our IT migration was a 'success'
TSB's Spanish owner Sabadell has proudly boasted about the success of its IT integration - amid widespread chaos for customers in Britain still grappling with the fallout out of a botched systems upgrade. Last weekend TSB was supposed to complete the move of its customers' data from Lloyds' IT systems. The bank split from …
-
Tuesday 24th April 2018 12:59 GMT katrinab
"With this operation, Sabadell demonstrates its capacity of technological management not only in national but also international integrations."
Well yes, I suppose they have given a demonstration of their capabilities in technological management. Only one problem, they are not very good.
No doubt their supplier will get the contract to do the Brexit customs system.
-
Tuesday 24th April 2018 14:00 GMT Rusty 1
Perhaps part of the problem was that the CEO was submerged with his teams in the first place. No place for such a person on the ground during an obviously supremely stressful time for the techs, other than taking orders for all the pizza they can possibly consume.
He was probably asking "is it done yet?", or "what does that red flashing light mean?". Likely followed by "why can't you go faster?" and "why isn't it done yet? - I have to play golf!"
-
Tuesday 24th April 2018 14:01 GMT michael cadoux
Top down or bottom up?
I would like to think that the people tasked with actually coding and implementing the changes had been asked whether that was feasible in the timescale. However, if so, it would sadly be a rarity; usually the managers emerge from a meeting where it's all been sewn up, and the luckless techies are told to be positive and just have it ready it by a prescribed date.
-
Tuesday 24th April 2018 14:38 GMT Anonymous Coward
Re: Top down or bottom up?
That's exactly what happened. The original deadline was October 2017 to avoid the rise in Lloyds IT fees from £100 million to £200 million in 2018. However, that simply couldn't be met and so an excuse was mooted around interest rates or Brexit or something and a new date of April 2018 was chosen.
Had anyone bothered to ask the testers on the ground what they thought, the answer would have been unanimous. It ain't ready and won't be for another six months minimum. But no-one asked and anyone with any sense got out of the project as soon as they could.
Migrating an entire bank to a new platform in less than 2 years from start of requirement gathering was never going to happen and Sabadell was warned.
-
-
Tuesday 24th April 2018 14:37 GMT Jemma
Paul Pester... Really?
Do any of these organisations ever consider people's names when they choose a CEO? off the top of my head only Nickolas Steal would be more apt for a bank CEO.
It's almost as bad as calling a girl after a 70s Toyota (Cressida) after fate has stuck its finger in and given her the surname Dick.. And *then* she joined the police - Monkey Dust couldn't make *that* one up and they gave us Ivan "I only confessed so they'd take my nuts out of the magimix" Dobsky and Mr Hoppy...
"People on the toilet" was far far far too prescient - although a 2G1C Kardashian cross over would possibly be the "end of the beginning" of a suitable revenge. Them and every soap - but I'm thinking pig farm slurry pits, head first, for anyone involved in Hollyoaks.
-
Tuesday 24th April 2018 14:41 GMT TRT
Re: Paul Pester... Really?
Mr Hoppy!
Or more like...
"So what is it this time, Clive. Why are you late back from the TSB?"
"Well dear, you'll never believe it... I was just about to fill in some details on a change of account form when a highly skilled computer hacker who served time for infecting the FBI's Carnivore program with a computer virus, was forced by someone called Gabriel to use his hacking skills to siphon $9.5 billion from several government slush funds.
Anyway, Stanley secretly coded a back door in his hydra hacking program that reversed all the money transfers after a short period. So Gabriel and his men stormed the bank branch, and strapped a ball-bearing-based Claymore to me and used threatened innocent people, forcing Stanley to drain all the bank accounts. And they kidnapped his daughter too. So that's what's happened to all the money from our accounts dear, and it really happened."
"That's the plot of the film Swordfish. What really happened?"
"I was being pegged by a 12" monster BBC across the console in the computer room at the bank, and I came all over the keyboards."
-
-
Tuesday 24th April 2018 15:19 GMT Joseph Haig
Lost in translation?
Their logo adequately sums up the contents of that press release.
Really, companies need to consider their branding when they start operating in foreign countries.
-
-
Tuesday 24th April 2018 15:44 GMT Allonymous Coward
Re: Where did it go?
This isn’t the level of service that we pride ourselves on providing, and isn’t what our customers have come to expect from TSB, and for that I’m truly sorry.
We’re still seeing issues with access to our press releases. One of the steps we need to take to resolve this is to take our press releases down for a few hours.
We’ll be taking this offline and we hope to be back up later this afternoon. We’ll let you know as soon as it’s available again.
-
-
This post has been deleted by its author
-
-
Tuesday 24th April 2018 15:29 GMT John Brown (no body)
402 customers?
TSB said 402 customers had access to some data they would not normally be shown on Sunday during a "20-minute window".
My wifes friend told her that she went to access her TSB account yesterday and got her husbands account details. So her husband trid his account and got his wifes details. I suspect as time wears on number will increase significantly.
-
Tuesday 24th April 2018 17:03 GMT Killing Time
Re: 402 customers?
' she went to access her TSB account yesterday and got her husbands account details. So her husband trid his account and got his wifes details'
Sounds like a joint account or potential grounds for divorce.
Me, I keep my personal account in a completely different bank for multiple reasons including that they can't rob 'Peter' to pay 'Paul'.
-
-
-
Tuesday 24th April 2018 21:28 GMT Phil Endecott
Re: 402 customers?
> Should have been picked up at UAT and probably pentest.
To me it sounds more like it’s time to sack the entire team and throw away all the code they wrote.
If you’re even anywhere close to one user accessing a different user’s bank account, it means several layers of security are borked or maybe just not there.
Almost OK for some stupid PHP webshite, but absolutely not for a f***ing bank.
-
-
-
Wednesday 25th April 2018 07:19 GMT TonyJ
Re: 402 customers?
"...Me, I keep my personal account in a completely different bank for multiple reasons including that they can't rob 'Peter' to pay 'Paul'..."
This is called "setting-off". I must admit, I'd thought they'd been stopped from doing this, but apparently not: https://www.moneysavingexpert.com/banking/setting-off
It allows banks to take money from savings or current accounts to pay debts such as credit card or loan arrears
-
-
-
Tuesday 24th April 2018 16:07 GMT Anonymous Coward
On a scale from 1 to 10...
Funnily a market research company rang on their behalf last week, spent what felt like twenty minutes asking me on a scale from 1 to 10 how happy I was with the service TSB provide on my business account. For quite a few questions, I said 10.
I've got this little card reader gizmo for transactions - works a treat, really simple. But that was going to be discontinued by the 20th and I was supposed to obtain one of these "apps" I've heard so much about.
I told them I don't have an "Android" (whatever that is) or a "smart phone" and they sent me an identical card reader. I checked the part number on the back, it's the same thing.
Then this afternoon DHL delivered a "smart phone" and the instructions say I have to create an account with Google and go to the "Google Play Store" and download this "app". Yeah, fuck it. I hate, despise Google as much as I do Facebook, so unless they can suggest an alternative, I'll close my business account. I'm not selling my soul to the devil name Google for their convenience.
-
-
Tuesday 24th April 2018 17:06 GMT Boris the Cockroach
Quote
From The Guardian "It’s worth noting that millions of pounds of bonuses to senior TSB staff were dependent on this IT migration."
And they'll still get their mega bonuses for being super agile at deflecting the blame and working hard to restore the company's image by 'entertaining' the ICO and the banking watchdog.
Meanwhile the low level techs who are currently putting in 24 hrs of overtime a day to sort the problem will be dismissed and outsourced as soon as they've repaired the system as we cant afford to pay them that much.
Trebles all round
-
Tuesday 24th April 2018 19:42 GMT Electric Panda
It'll be like the household name megacorp I used to provide my services to.
Under normal circumstances, you could barely move for suffocating red tape and doing the simplest of things was like staging a military operation to be viewed by the Queen.
Ridiculous acronyms (often used in wildly different contexts - no, not that "IFA" used by the backup team, the other "IFA" used by the firewall guys etc.), impenetrable processes with "beware of the leopard" approach to documentation and how this process actually worked, hand-offs between five different teams when it could and should be done by just one etc. This was the norm until something was considered "high priority" and would be exposed to millions of paying customers, or seniors set an arbitrary deadline.
At that point things became suspiciously efficient. No lead times, the red tape was swept aside, no silly interpretive dancing through utterly byzantine and farcical processes - and everything was relentlessly escalated. Any jobsworth who stood their ground was absolutely trundled over.
To be honest it was very refreshing and in retrospect I do wish it had happened more often.
-
Tuesday 24th April 2018 17:06 GMT clanger9
18:00, day 4, still broke
So much for "we hope to be back up later this afternoon"! It looks like it's getting worse, not better.
While it was (briefly) working yesterday I managed to complete a payment from the app (not the website), but both have been down all day today.
I wonder if there's a point of no return? As more and more backlogged transactions pile up, the more the the pent-up demand is likely to flatten their systems when they are eventually fixed...
-
Tuesday 24th April 2018 18:16 GMT Anonymous Coward
19:00, day 4, still broke
The login button now just takes you to the service status rather than even letting you attempt to log in. Even the TSB home page is throwing the occasional internal server error now, but that's probably down to it getting swamped.
A point of no return? I guess in the worst case there has to be one sometime, where they just say sorry we have no realistic hope of getting things up to date again, here's the address of the FSCS.
-
Tuesday 24th April 2018 19:11 GMT Electric Panda
Re: 19:00, day 4, still broke
You know, it's totally terrifying.
RBS nearly packed up in 2012 because of their computer issues. Apparently, all that saved them was the fact that the botched upgrade had only been applied to Natwest and Ulster Bank leaving the main RBS retail arm untouched.
Had that also gone wrong, that could have sunk the entire RBS Group as they would have no transaction history. You don't recover from something like that.
-
-
-
Tuesday 24th April 2018 19:08 GMT Electric Panda
Does anyone want to place a bet on what the root cause of this actually is? What part of the migration seems to have gone wrong given people are seeing the wrong data entirely?
Some people on another site say they are seeing mortgage arrears worth millions of dollars. Yes, dollars, and millions thereof.
Millions of dollars of mortgage arrears shouldn't even be on the Lloyds or TSB accounts database - so where the hell did that data come from? An overflow perhaps?
-
Tuesday 24th April 2018 21:04 GMT Anonymous Coward
One Million Dollars!!
My mortgage was showing arrears of -$1.7 million during the brief window when I could log in.
Yep, negative arrears...
More worrying was that the balance was also wrong by at least £10k. I only get annual statements, so it's hard for me to prove the right numbers.
Not managed to log in since.
Credit card bill is due on Friday. Fortunately it's a TSB card so...
-
Tuesday 24th April 2018 19:13 GMT Special_Brew
Ever read a lessons learned document?
They've thrown a migration and upgrade together (never a good idea) and also considering timelines, they no doubt have dumped a load of new GDPR code to support next months change into the mix.
More than likely run by one of TSB's first attempts at a psuedo agile project.
At some point over the weekend a decision was made to go live and not restore to the backups and now fixing forward a steaming pile of shite is the only option hence the site being up and down like a (insert similie here).
-
Tuesday 24th April 2018 19:31 GMT StuntMisanthrope
You don't qualify for a technology license.
“Thank you. Since we decided a few weeks ago to adopt the leaf as legal tender, we have, of course, all become immensely rich.”
Ford stared in disbelief at the crowd who were murmuring appreciatively at this and greedily fingering the wads of leaves with which their track suits were stuffed.
“But we have also,” continued the management consultant, “run into a small inflation problem on account of the high level of leaf availability, which means that, I gather, the current going rate has something like three deciduous forests buying one ship’s peanut." Thank you to the late great Douglas Adams. #hesawitcoming #cryptotulips
-
Tuesday 24th April 2018 19:40 GMT Anonymous Coward
Bad before it started.
I was already pissed off before Sunday, when I logged in to make a fairly important transfer to find my bank account effectively closed. No telephone banking backup, no service in branches either, so a total inability to carryout basic banking functions for 3 days and that was if it all went well!
We're now getting close to a week and this is turning into a real problem. Surely as it gets close to a week, they've got to consider rolling it back, or maybe they can't?
-
Tuesday 24th April 2018 19:43 GMT Anonymous Coward
Re: Bad before it started.
I'm hearing rumours of a a similar story to what someone suggested down below.
They have apparently tried to do a simultaneous migration and upgrade. Rumour is the migration went okay (which is why Sabadell are publicly praising it) but it's the upgrade that's gone Pete Tong. It would apparently be quicker to forge ahead and fix it rather than rolling back.
Given that people are seeing mortgage balances that are millions of dollars in arrears, this suggests default configuration and one badly mangled dataset. TSB doesn't handle dollars and nobody sane has a mortgage that large, let alone that far in arrears. That is spectacularly broken.
-
-
Tuesday 24th April 2018 19:58 GMT Anonymous Coward
That apology in full...
1. We followed best practises.
2. Lessons have been learned.
3. All account balances are probably correct for certain values of correct.
4. We now have the IT platform of the future; we just didn't have it for quite a bit of the present. And we've lost the past altogether.
5. Everyone* will get compensated appropriately.
6. This was a long and complex project so we decided to shorten that time and go all dev-ops and agile. Well you'd do the same wouldn't you?
7. We followed best agile practises.
8. Lessons have been learned in an agile way.
9. There will be a full risk assessment and post mortem. We won't publish this. It'll blame all the people we've sacked who actually knew what went wrong.
10. But we will get that Scottish woman with the lovely sing-song voice from Trainspotting to do some more ads about added 'plusness' (which is what happened some of our customer's balances!)
11. C'mon! We're not as bad as the co-op! He snorted crystal meth off martian hookers or something.
(* except customers).
-
Tuesday 24th April 2018 20:23 GMT Anonymous Coward
From the opposite point of view
I've known several people that worked in IT at the other financial institution acquired by the Spanish.
Apparently they like to drive the IT policy from there.
As it's 'their' system and apparently has worked for them, they have no doubt jumped to the conclusion that it would just be a plug and play job, until of course they got the call from the UK to tell them it wasn't going according to plan (assuming there was one).
Let's hope they don't let the techies go as it does sound like it's proving to be somewhat of a Herculian effort to put right.
-
Tuesday 24th April 2018 20:24 GMT Anonymous Coward
Tory Stolen Bank
The TSB was dedicated to serving the community and effectively competed with the big clearing banks by successfully attracting deposits.
TSB served its customers well but retained some of its profits instead of passing them on to its customers by more competitive interest rates. So its reserves accumulated, had become very substantial, were in effect serving the community.
Ownership of the bank and its reserves became important when a Conservative government decided to 'privatise' the bank and convert it into a shareholder-owned profit-maximising bank just like the commercial banks.
A 1973 report had stated that if the banks are to be considered as mutual organisations without shareholders the depositors are entitled to the full value of the bank. (TSB 03}
The TSBs were freed from competitive restrictions in 1976 {TSB 05}. The 1981 TSB Act in turn states that the TSB can accept deposits, accumulate the produce and
return the deposits and produce to the depositors after deducting any necessary expenses of management, but without deriving any benefit from the deposits or produce. {TSB 01}
The TSBs serve the community. Trustees are meant to ensure the TSBs are run for the benefit of their depositors and of the community, their assets being looked after by Trustees on behalf of the depositors.
But then the UK government decided to 'privatise' the TSBs, to sell them off, stating in the 1984 White Paper:
The TSBs do not belong to the Government. Nor do they belong to the trustees, to their employees or to their depositors. {TSB 05}
So the government is proceeding to sell the TSBs although it considers that they do not belong to the government, on the basis that the TSBs and their assets belong to nobody, that no one owns the bank.
The TSB's assets were to be transferred to a public limited company before flotation {TSB 01}. This is how Andrew Murray commented in the Financial Times:
As it considers that the TSB belongs to no one, the government apparently intends to appropriate the assets by Act of Parliament, fix the price, make the sale and return the payment with the goods. In other words, the existing assets would be a free gift to the purchasers of the shares. {TSB 06}
The 1985 Act then took away the powers of the depositors. The bank is to be run first and foremost for its shareholders. {TSB 01}
It did not seem to make sense to depositors and the matter was fought out in the courts in Scotland, then in England and Wales, and finally decided in the highest UK court, the House of Lords.
Their reasoning was interesting, their arguments appeared almost unreal, the conclusion was unexpected.
See http://www.solhaam.org/articles/tsb.html for full document
Once you understand where they came from and what was done to them then everything makes sense
-
Wednesday 25th April 2018 02:21 GMT James Anderson
For the curious
Protea is a hacked version of Accenture's alnova banking package. Which according to this
https://www.microfocus.com/media/success-story/alnova-financial-solutions-wit_tcm6-202106.pdf
Is a classic COBOL mainframe system hacked to run on UNIX using the .NET framework.
For the UK they made further hacks to run in an Amazon cloud.
What could possibly go wrong?
-
-
-
Wednesday 25th April 2018 09:43 GMT Anonymous Coward
Re: For the curious
I'm too old to have done much cloudy stuff, but isn't the whole point of using a scalable cloud like Amazon's that you can just sling as much resource as you need at the task in hand? Though if the whole thing really is based on mainframe-era back end software, maybe that's what is bottlenecking the whole thing.
-
-
-
Wednesday 25th April 2018 10:48 GMT colinb
Re: For the curious
nice find, you had me at Accenture.
So a converted 11 million line Cobol system into Dotnet, add in a message brokers,new databases, loads of services for mobile, document storage etc...
then the frontend using JAVA (people are getting beancontroller errors on the site)
Sweet Jesus, i'd heard of that Dotnet Cobol product but didn't think anyone would be crazy enough to use it.
-
-
Wednesday 25th April 2018 04:20 GMT Anonymous Coward
Are they running TSB in the 'cloud'?
Welcome to Business Internet Banking
$host dpm.demdex.net
dpm.demdex.net is an alias for gslb.demdex.net.
gslb.demdex.net is an alias for edge-usw2.demdex.net.
edge-usw2.demdex.net is an alias for dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com.
dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com has address 54.70.206.135
dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com has address 52.33.54.46
dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com has address 52.26.195.87
dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com has address 35.167.226.133
dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com has address 54.244.32.108
dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com has address 52.38.50.31
dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com has address 34.214.245.56
dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com has address 52.32.186.42
-
Wednesday 25th April 2018 07:10 GMT Tim Kemp
Saw Pester on the news last nigjht, when asked when the systems would be up he replied "I've been told 4 PM but we can't always believe what IT people say".
That's disgraceful. He is CEO, buck stops there. If he can't trust his IT department and leadership then why did he allow the transfer to proceed?
-
Wednesday 25th April 2018 11:01 GMT dervheid
Crock of shit
The new web interface *may look OK on a PC, but it's a complete POS on a mobile device.
In Portrait orientation it's basically using a narrow central column, with around 40% (in my estimation) of the screen width unused white space. If you want to see transaction details in full, you're forced to use landscape orientation, as the rows don't resize fully (if only it weren't for all that wasted space either side...). Frankly, it's not much better in landscape, as there's still those wide, unused borders either side, leaving you with a whole load of scrolling.
Your coders need there arses soundly kicking TSB.
* not logged in on a PC yet.
-
Wednesday 25th April 2018 11:36 GMT Stephen Booth
URL change does not help
Part of the migration is a change to a new server.
online.tsb.co.uk -> internetbanking.tsb.co.uk
If you go to the main TSB site and follow the links you get there and It appears to be UP
If you use an old bookmarked you go to the old server which shows as still DOWN
Also the site could remember your (unmemorable) username for you via cookie but (inevitably) you need to actually type this in the first time you go to the new site.
These changes by themselves would cause lots of people problems so even If everything is working now I'm not surprised people are still seeing it as down.
-
Wednesday 25th April 2018 12:29 GMT d3vy
Re: URL change does not help
"Also the site could remember your (unmemorable) username for you via cookie but (inevitably) you need to actually type this in the first time you go to the new site."
If your luck it will let you in. I have re-typed my credentials so many times and been told that they are wrong that the account HAS TO be locked now despite being 100% sure that they are correct.
-
-
Wednesday 25th April 2018 15:58 GMT Ivan Headache
3 days
In an hour's time it will be 3 days since the service was supposed to be back on line.
Except that it's still not. I get directed to maintenanceweb.tsb.co.uk. It's been like that since about 0700.
I went into my branch this afternoon just to make sure that my debits due next week would be covered and the girls behind the desks were able to get in and transfer funds - but very, very slowly.
I'm quite chummy with them and was told that there had been a lot of tears as irate customers were taking it out on them.
It also got me thinking about the folly of shutting down branches 'as everyone does internet banking nowadays'.
Apparently my local branch of Lloyds will be closing later this year.
-
Wednesday 13th June 2018 19:52 GMT PhilCollinsUK
Currency reset
All these upgrade outages seem very suspect to me, three in the UK within weeks of each other!
I think there is more to this than meets the eye. I suspect because the west is completely bankrupt they will do a currency reset rather than let it drag on and watch one company, bank, state fall over like dominos.
Biting the hand that feeds IT
