https://en.wikipedia.org/wiki/Brooks%27s_law
Who will fix our Internal Banking Mess? TSB hires IBM amid online banking woes
In the midst of its six-day online banking meltdown, which is showing no sign of letting up, TSB has hauled in the systems integration big gun or as it is otherwise known, IBM. The British bank confirmed the signing of Big Blue this morning as part of its rather uncomfortably timed first quarter financial results for 2018, in …
COMMENTS
-
-
-
-
Tuesday 22nd May 2018 12:21 GMT Tydfil
Re: Out of the Paella pan
IBM to the rescue!!!!! ...... now there's an 'oxymoron'. What with no more TSS for support, IBM could fix it on site with some highly skilled project staff and developers. But they won't be around to support it afterwards - and that is something IBM will want from all of this once fixed, and sufficient time has passed for TSB to draw breath.
-
Thursday 26th April 2018 10:31 GMT HmmmYes
'But the damage has been done – vast numbers of Twitter users and numerous Reg readers have been reporting plans to switch banks (as soon as they can access their money again).'
Surely thats:
'But the damage has been done – vast numbers of Twitter users and numerous Reg readers have been reporting plans to switch banks (*IF* they can access their money again).'
-
Thursday 26th April 2018 10:56 GMT Peter2
'But the damage has been done – vast numbers of Twitter users and numerous Reg readers have been reporting plans to switch banks (as soon as they can access their money again).'
Why wait?
I signed up for a new account a couple of years ago. One of the questions they ask in the setup process was "would you like us to get your old account with the other bank closed down and the balance transferred to your new account with us?"
It turns out that most banks these days have teams that are dedicated to doing transitions between banks. In this situation, personally i'd be availing myself of their services so they can deal with the hassle.
-
-
Thursday 26th April 2018 11:50 GMT Ktsecful
So instead of filling in a simple form (and with many banks being paid to switch using the current account switching service), you'd rather:
Go in to a bank, withdraw all of your cash, close your account
Go to a new bank, open an account, pay your money in
Contact all businesses you have outstanding DDs with, and set up new ones.
Recreate all of your payees on your new banks online banking
Set up any standing orders again.
-
Thursday 26th April 2018 12:35 GMT Dan 55
Yes, I would rather do that. I don't know if you've noticed but TSB's IT currently resembles a smoking crater and they've got no more chance transferring your money out of your account than you have getting the money or a bankers draft at a branch, plus by now TSB will have a huge backlog to work through.
DDs, SOs, and payees can be done online after getting the money in the new account, given an evening and a cup of tea.
-
Thursday 26th April 2018 15:26 GMT Yet Another Anonymous coward
TSB's switchover team use the same back office systems as branch personnel and will probably be inundated.
So if you do it you get nowhere.
If another giant bank gets no response to a milllion transfer requests it phones up the banking regulator and suggests TSB get fined a few million/day until they comply.
I don't think - we would love to abide by the rules for the time it takes to transfer a customer to a competitor, but the computer is acting up - works as an excuse
-
Friday 27th April 2018 12:05 GMT Aodhhan
Yes, I'd rather do all of this. It's known as:
- Responsibility
- Being forward thinking
- Someone who isn't going to support poor business practices
- Taking care of my financial future and my family
- Oh, and this will strike at the heart of many snowflakes... NOT BEING LAZY.
Taking several hours out of my life to deal with switching banks will likely save me many days of headaches and late payment fees in the future.
Anyone who isn't willing to do this, only perpetuates poor business practices. Someday, this is likely going to bite you in the backside.
Finally... when all this is over, TSB is going to look for ways to cut costs to cover the large expense this muck-up is costing them. It doesn't take a world class seer to figure out how the effects will eventually fall back onto the customer.
-
-
Thursday 26th April 2018 15:13 GMT Anonymous Coward
Current Account Switching Service
I don't understand why the Current Account Switching Service which the UK Government exhorted the banks to implement was made literally only that, a switching service, and one which closes down your old bank account once the transfer has been fully completed.
Certainly, being able to transfer things such as salary payments in, and standing orders and direct debits out, from the old account to the new account as smoothly as possible is very welcome, but I think that all of the problems that various banks have had, coupled with the risk of card theft, all go to show the importance of having at least two current accounts (with completely separate banking groups) at all times, so that you always have a backup available ...even if one of the banks doesn't!
-
Thursday 26th April 2018 17:12 GMT Anonymous Coward
Re: Current Account Switching Service
> I don't understand why the Current Account Switching Service which the UK Government exhorted the banks to implement ...
Sounds like an absolutely fantastic Social Engineering opportunity.
"Hi this is <foo> calling from <some bank>. Just calling to switch another client over. Name: [ie: Jeremy Clarkson], Details: [found on social media]. Target bank sort code: [..]. Target bank account: [...]."
Wonder how employees from each bank are supposed to auth to employees from different banks, who they probably don't always know? Hopefully it's not something stupidly easy to fake.
-
-
-
-
Thursday 26th April 2018 10:32 GMT djstardust
A disaster
As a TSB customer of 33 years (It was Aberdeen Savings Bank when I signed up) I have finally decided to move on. Shame really as the local branch have good long term staff .... but the real issues are higher up the chain.
This is yet another example of outsourcing not delivering (Accenture) and another company using outsourced IT support rather than a professional, committed in-house team.
Good luck hiring IBM, because most of their decent people either got the Ruud Gullit or moved on somewhere else.
I have a real issue with outsourcing, especially when seing what's recently happened with Carillion and Crapita, and any company who I find willingly using it to save a quick buck (in the short term of course) will be losing my business.
-
Thursday 26th April 2018 10:37 GMT Doctor Syntax
Re: A disaster
"Shame really as the local branch have good long term staff"
It was one of the staff of a local branch when it was part of Lloyds that prompted me to leave Lloyds although the damage had been done when Lloyds closed my preferred local branch. Just as well I did - maybe I'd have been transferred over to TSB when they split.
-
-
-
Friday 27th April 2018 08:11 GMT Dan 55
Re: A disaster
Sabadell bought the system off Accenture, but Accenture is involved in this migration as well as in-house staff. I guess Accenture sold that snake oil by claiming they still had the expertise but anyone who saw that system more than a decade ago has either left or is doing something else by now.
So what happened is Accenture got paid money to get rid of a piece of crap and still raking in the consultancy fees now for a failed migration.
-
-
-
Thursday 26th April 2018 10:38 GMT JimmyPage
promising that no one would be "left out of pocket".
but to what extent ?
When HSBC went titsup there were tales of house purchases which failed (HTF can you recompense that ????) plus business contracts which failed, and (presumably) bargains via eBay etc which were gone - forever.
Admittedly there haven't been any stories from that fuckup about custmers being left out of pocket. But I remain deeply cynical.
-
-
Thursday 26th April 2018 10:44 GMT Anonymous Coward
Re: not a customer
"sounds like a complex undertaking done for good reasons. Customers should just be patient and not get agitated."
Sure, they're doing it for the right reasons, but it seems to have been a monumental cockup.
Nationwide did the same thing a few years back with a much higher level of success. I wonder how much outsourcing, offshoring and asset stripping they've had over the same period compared to the for profit big boys?
If you couldn't get access to your money, pay bills, etc, you'd quite rightly be "agitated". Not even the branch staff can help as their system is completely fluffed too (the BBC report quotes a manager, who says their system even helpfully has all the error messages in Spanish!)
-
Thursday 26th April 2018 10:47 GMT gv
Re: not a customer
On the contrary, customers should get very agitated. This is a core banking platform migration, so there should have been extensive contingency planning, testing and a fallback system in place.
"to help identify and resolve performance issues in the platform"
A good rule of thumb is to do this BEFORE going live.
-
Thursday 26th April 2018 10:49 GMT Doctor Syntax
Re: not a customer
sounds like a complex undertaking done badly for good reasons.
FTFY
I'm not a customer either - I escaped that fate. If I were I'd certainly be agitated.
There's an old saying, "if you fail to plan you plan to fail". What does the present situation tell you about their plans?
-
Thursday 26th April 2018 11:01 GMT colinb
Re: not a customer
They are not a ffing corner shop.
If a Bank can't be trusted to handle your money responsibly it can cause things like bank runs. Bank runs can be contagious.
RBS was fined £56 million for a botched IT upgrade, so no the regulator doesn't like to chill either when avoidable crap happens.
-
Thursday 26th April 2018 13:12 GMT defiler
Re: not a customer
Customers should just be patient and not get agitated.
Day 1, I agree completely. Day 6, we're running headlong into the end of the month.
Missed direct debits is one thing (I'll have a few between now and Monday, and I'd be miffed if they failed), but the bank can generally explain the problem to the recipient, handle the penalties etc. But what if I (not banking with TSB) were unable to get paid by my employer (hypothetically banking with TSB), and I am the one stung with penalties? Will TSB reimburse me?
Yes, it's a complex task. It's a huge task. And it needs huge planning, and regular checkpoints with back-out plans already in place. It seems that somebody skimped on the planning, and they committed to a course of action that didn't have a back-out plan ready to go.
I'm not saying I'm perfect. I've committed to system changes without a fast back-out that ended up biting me before, but we had a workaround already on the back burner before it happened, and were able to keep users running on that for the couple of weeks it took the devs to sort the latency issues with us. But this is beyond amateur hour. It's a bank. Not a small bank. They claim to have over 5 million customers. That means that a small disruption is a big issue.
-
-
Thursday 26th April 2018 16:34 GMT d3vy
Re: not a customer
"I'm a TSB customer and as far as I can see, my direct debits and standing orders have worked just fine since Monday morning."
They have been "Core services" were up and running at the weekend... but no one can access their accounts.
I have a few accounts with TSB one which I use day to day and one for the bills. At the end of every month I work out the next months outgoings and transfer them into the bill account which I then dont touch - this means that no matter what happens during the month the bills are all covered.
The problem now is that as its the end of the month that bill account is empty, I cant put any money in it because the account with the money in is a TSB business account (and the internet banking for the business accounts is in a worse state than the normal accounts).
So shortly my mortgage and a host of other payments will come out of my bill account - this will do one of two things (Maybe both)
1. Put me WELL over my overdraft
2. DD will get declined and go unpaid.
-
-
Thursday 26th April 2018 13:29 GMT d3vy
Re: not a customer
"sounds like a complex undertaking done for good reasons."
Well, You're right to an extent.
It IS a complex problem and they do need to do it.
"Customers should just be patient and not get agitated."
I wasn't agitated until at 7pm on Sunday (AFTER the switch over was meant to be complete) when my debit card stopped working (Its still not working now).
ATMs also seem sporadic - Though to be fair I've only tried once.
I can tell you now that 6 days on I am more than agitated I have two personal accounts one which is used just for bills and by this time of the month will be empty. The other has money in it but I cant get to it.
I also have three business accounts which I cant get to to pay wages out.. or check that Ive been paid by clients..
So my only option now is to go into a branch and HOPE that they can transfer the funds for me - from what I have read online this is a bit hit or miss at the moment.
-
-
Thursday 26th April 2018 10:49 GMT Gordon 10
I pity the poor schmoes working on this.
Now not only do they have a badly planned, poorly managed cut over to repair they are also going to have a bunch of senior muppets from IBM looking over their shoulder second guessing them with the benefit of hindsight.
I bet somewhere in the depths of TSB's tech dept there are a number of people muttering "told ya so" under their breath who were ignored when they raised issues.
I don't see what value IBM can add at this point other than making things worse. I get a very strong whiff of "something must be done"
-
-
-
Thursday 26th April 2018 17:56 GMT Vincent Ballard
Re: I pity the poor schmoes working on this.
Anonymous Coward is spot on. A former colleague of mine is caught up in the middle of this, having joined Sabadell recently in a QA role, and reports that people are pulling seriously illegal amounts of overtime and staying awake with the aid of pills and Bolivian marching powder.
-
Sunday 29th April 2018 05:47 GMT HelpfulJohn
Re: I pity the poor schmoes working on this.
"Even if the problem was caused by management ignoring advice from techies, it'll still be the techies fault ... somehow."
Space Shuttle. O-rings.
You'd think those who don't know how things work would have seen that one as a Really Big Wake-Up and would now be paying attention to those who do Know Things.
Or am I expecting too much from Admin and Management Plonks?
-
-
-
-
Thursday 26th April 2018 10:49 GMT BoldMan
What are IBM going to do? Wave a fucking magic wand?
So how do they expect a bunch of outsiders to come along and fix everything when their own internal people who built the fucking thing can't get it working quickly?
Bringing in outsiders always looks good to the mouth-breathing morons up the org chart, but doesn't fix the problem just means that the people who are supposed to be fixing the problems spend pointless time explaining things to the newbies.
This Peter Pester twat sounds like a complete waste of space.
-
-
-
Thursday 26th April 2018 13:50 GMT Anonymous Coward
Re: What are IBM going to do? Wave a fucking magic wand?
There are a lot of people in this thread who are jumping to some remarkable conclusions.
A couple of disclaimers - I have no inside info here, and I once worked for IBM (left a few years back).
Everyone here's pointing the finger at TSB and their new owners here but the reality is most likely that this is all a great example of the evils of lock-in, which can be entirely attributed to Lloyds. There is a lot of IBM presence in that bank and they are very dependent on overpaid contractors to keep it running, along with oodles of offshore bollocks.
My guess (and it is admittedly a guess, although I hope an educated one), is that TSB are trying to change as little as possible, so are splitting out IBM systems, or at least duplicating them, and someone somewhere has not done the numbers, or more likely (in my vast experience) has been told those numbers (performance metrics, technical necessity etc.) do not match the other numbers (£££) and some serious corners have been cut. I hope the relevant technical people kept their emails on the subject.
IBM being brought in now is unlikely to be a free market choice; IBM are most likely the best people to fix it because it's IBM technology that's involved. If it's an architectural issue or a sizing issue, as is most likely, and this was not IBM's fault, then IBM aren't going to fix it without getting paid. Support doesn't cover that. The fact that TSB have little choice other than calling IBM is entirely down to lock-in. Of course, they may have had the option to move off IBM here, but probably decided to change as few things as possible to improve chances of success.
I'll reiterate that this is conjecture on my part, but I have a lot of experience here - being sent halfway round the world to fix customer issues. Of course, I no longer work there (took a package) but there are still people who can probably fix this. It won't be quick though and I can guarantee that the people who do fix it will get zero credit. Which is one of many reasons I no longer work there.
I am hoping the details do get leaked, but I for one am not pinning this entirely on TSB.
-
Thursday 26th April 2018 14:30 GMT Doctor Syntax
Re: What are IBM going to do? Wave a fucking magic wand?
"I am hoping the details do get leaked, but I for one am not pinning this entirely on TSB."
What has to be pinned on TSB is that they took stock of the situation and decided they were in a position to go ahead in a single operation. And then found they weren't.
-
Thursday 26th April 2018 15:25 GMT I ain't Spartacus
Re: What are IBM going to do? Wave a fucking magic wand?
It's a bit of an EU cock-up in my book too. RBS have apparently
wastedspent a couple of billion trying to sell several hundred branches to comply with competition rules - but nobody in the end wanted to buy them. So they're now spending billions more trying to spin them off as Williams and Glyn. That means setting up a head office, management, IT, and a cash pile/reserves.This TSB thing is a similar exercise in regulatory box-ticking. That's doing nobody much good. How much have we really gained as customers attracting another Spanish bank into the retail market, as a minor player with just a few hundred branches?
We had state aid and shotgun mergers during the financial crisis and that broke competition rules. But they really should have ignored them. At least until a later date. There's a reason that when banks buy other banks they tend to leave them on the old systems for years, and only slowly transition across (if ever).
Rules are important, but given this was a once-in-a-century level of global banking crisis, I think a bit of pragmatism would have served better. Worrying about moral hazard was what caused the US to let Lehman Brothers collapse and the Bank of England stuff Northern Rock. Nothing good came of those 2 decisions. Similarly the rigidiy in stopping Italy from trying to solve its banking crisis has stored up potentially horrible trouble for later.
Presumably the main blame still goes to TSB and Sabadell management for doing the migration on the cheap or too fast.
-
Thursday 26th April 2018 17:22 GMT Anonymous Coward
Re: What are IBM going to do? Wave a fucking magic wand?
> IBM are most likely the best people to fix it because it's IBM technology that's involved.
Just because IBM made something doesn't mean they're competent with it past the sales point.
Remember the many waves of outsourcing support, and deliberately removing their more experienced technical staff? (still ongoing)
Bringing in IBM is unlikely to achieve an meaningful win, apart from giving the CEO someone else to blame.
-
Thursday 26th April 2018 18:04 GMT Vincent Ballard
Re: What are IBM going to do? Wave a fucking magic wand?
My inside information from a former colleague is that Sabadell tried to port their existing code for their Spanish bank, which was cowboy quality. Hard-coded values including IP addresses for servers and telephone area code prefixes. Copyright-violating ripoff of a Netflix library from GitHub where the copyright header was changed but not the references to Netflix in error messages. Compounded by the insistence of the suits that 500 simultaneous users was a sufficient load test.
-
Friday 27th April 2018 07:10 GMT Anonymous Coward
Re: What are IBM going to do? Wave a fucking magic wand?
"port their existing code for their Spanish bank"
That would seem to be confirmed by the BBC report of one TSB branch manager's bad week - they said that the branch systems have been spitting out Spanish-language error messages when things inevitably fail or time out
-
-
-
-
Thursday 26th April 2018 20:48 GMT d3vy
Re: What are IBM going to do? Wave a fucking magic wand?
" just means that the people who are supposed to be fixing the problems spend pointless time explaining things to the newbies."
That in itself might be a useful excercise. It's even got a name. Rubber duck debugging.
Sounds stupid but it works.
-
-
Thursday 26th April 2018 10:54 GMT Steve Graham
I was once manager of a team which was migrating more than 25 million customers to a new infrastructure overnight. I was so confident that we had covered all contingencies, including total roll-back, that I went home and went to bed.
It went perfectly.
It wasn't easy, and it wasn't cheap, but failure is always avoidable.
-
-
Thursday 26th April 2018 13:40 GMT WallMeerkat
Seriously?
If I was up all night on a stressful cut over migration of 25 million accounts while my manager skipped off home to bed it would be anything BUT an "excellent attitude".
There is trust in the team (do not micromanage) but it's nice to know the general is beside you when the troops are at the front line.
-
-
-
Thursday 26th April 2018 11:09 GMT tip pc
1 way migration
Looks like this was a last minute 1 way migration with extremely poor testing and over site.
T minus 7 days: heads we do the change tails we don’t.
T minus 5 days: any plans this weekend, weathers meant to be good. what about this change? The change that’s always delayed? Apparently the ceo is adamant this time!! I’ll beleive it when I see it.
T minus 4 days: notify the customers it’s happening, cancel your weekend plans!!
T minus 2 days: we’re down 10 senior tech lead heads for this weekend, 1’s getting married, another 2 on maternity, 5 on leave and the other 2 are “busy”. Everyone is saying they are not ready but the pm’s think the remainder can get the job done.
The rest is history.
I couldn’t imagine doing something like this in 1 go over 1 weekend without a credible blackout plan.
You’d pause all internal transactions and record all externals, do your migration, test the hell out of it, then roll it back and replay transactions And unpause stuff, then during the following week analyse the tests to check for problems, repeat until all the kinks are ironed out then roll out to a small number of customers per weekend, looking and checking for issues including fraud.
All a very simplistic high level glossing over huge amounts of detail but my fag packet looks a lot more detailed than their packet of Rizla with roaches torn off.
-
Thursday 26th April 2018 11:18 GMT HmmmYes
This seems to be evolving - or going to shit.
http://www.bbc.co.uk/news/business-43904267
'TSB chaos: 'We are on our knees,' says boss'
A few days ago they are getting their bonuses for a completing the work.
Yesterday it was just a couple of niggles affecting a handful.
Today, Godzilla is running rampant thru the server room.
-
Thursday 26th April 2018 14:33 GMT Anonymous Coward
I was thinking that what's s the betting some of the trouble is being a Spanish bank bits of the software will be in Spanish, adding to the levels of bullshit. then I read this
http://www.bbc.co.uk/news/business-43892840#
"He reported that branch staff were trying to cope with a completely new system after the botched migration but "nothing is working".
"My concern is we have customer bills to pay such as credit cards, but we physically cannot process them, as the system either freezes or completely crashes, like it did yesterday for the whole afternoon."
He said that staff had been assured for more than a year how amazing the system would be, but "it's an absolute joke. Most of the error messages are in Spanish, which is great if you can speak Spanish!"
-
-
This post has been deleted by its author
-
Thursday 26th April 2018 18:53 GMT Adam 52
There's a difference between being able to operate a new system day-to-day, which you may have been trained for, and being able to work around a system you've been using for years and know all the back routes in.
We have a mission critical but old, command line system at work. It's massively reliable (a few minutes downtime in three decades), fast and really efficient for those trained in it but management can't cope without drop-down menus so is being replaced with a much less reliable, much less capable Windows app. Thank's for nothing Steria.
-
-
-
Thursday 26th April 2018 11:28 GMT adam payne
But the bank claimed that "each of these instances were 'connected accounts' where third party access had been granted, typically to family members" and that no personal customer information was shared.
Bold statement but are you really sure? Wouldn't want that to come back to bite you are anything.
They hired IBM! Oh dear, they really do have no idea how to fix it.
-
-
Friday 27th April 2018 14:21 GMT d3vy
"I think I must be the only customer TSB has that's finding this all rather funny"
Now that I have been into a branch and transferred enough to cover next months bills into my billing account Ill join you. It will be a bit of a pain using a credit card for the next few weeks while the sort the issues out but 5% interest is pretty decent for my kids savings!
-
Thursday 26th April 2018 11:51 GMT Aitor 1
No account
On monday, for a short span of time, I could see my accounts. Seemed my personal and shared accounts were kinda mixed.
On tuesday, no access.
On wednesday night , I did not have an account!!
Now I have an account again... the system is sloooow and my salary is not there.. colleages with other banks already have the money. It is no problem for me.. but WTF.
-
Thursday 26th April 2018 12:00 GMT ForthIsNotDead
So TSB's answer is to throw more people at at it.
Outsiders, that had nothing to do with building the system, no less. Clearly, nobody in TSB manglement has read The Mythical Man Month. Ironic, since it was written by a former (and highly respected) IBMer.
-
Thursday 26th April 2018 12:01 GMT Stephen Booth
Not your normal outsourcing
Everybody is talking about this being outsourcing and they should have stuck with their in-house tech people.
My reading of this is that they are switching between two outsourcing suppliers. They lost any in-house system when TSB was spun out and were being held over a barrel by lloyds so they hired somebody to create a replacement platform and got shafted again. Probably because the bank does not have its own technical staff to evaluate the solution properly.
The lloyds in-house techies who were running things till last week are probably still at their desks business as usual.
-
-
Thursday 26th April 2018 12:28 GMT dullpointerexception
We never will, but it would be really interesting to find out the root cause for all of this.
What I don't get is that common sense would dictate that you'd already have moved the vast majority of the account and transactional data beforehand, and then just migrate the delta over the big bang weekend. You'd also move all of your staff's accounts over to the new system (fun for them) to see how it handles that volume of business for an extended pilot period in the run up to go live.
If they've done all of that, and it's still a simple case of the new system just can't cope with the volume of transactions per second, well, best of British / Spanish fixing that quickly!
-
Thursday 26th April 2018 12:29 GMT Anonymous Coward
Using IBM - aka I'm By Myself - to fix this isn't going to help. This smells like a failure of architecture/design & having the wrong developer expertise. Below is their cloudy stack and I bet that this project lacked Engineers with Spring Microservices expertise & true SDETs, rather than traditional QA'ers. I can't help but think they used QTP & Selenium playback QA monkeys when for something of this scale they needed Amazon/Netflix/Google quality SDETs.
... Component of the BancSabadell Architecture team in the TSB project (TSB Bank in UK acquired by BancSabadell group) for the definition and implementation of a new banking platform based on the latest technologies and methodologies and oriented to a hybrid infrastructure between on-premises and public cloud
Technologies:
-PaaS (TIBCO SilverFabric)
-Micro services (Spring Cloud Netflix)
-SOA (TIBCO AMX Service Grid, TIBCO BusinessWorks, TIBCO API Exchange Gateway)
-Single Page Application (AngularJS)
-Asynchronous Messaging (TIBCO EMS)
-APM (Application Performance Monitoring)
-Distributed Search & Analytics (ElasticSearch)
-Containerization (Docker)...
-
Thursday 26th April 2018 13:33 GMT Anonymous Coward
Say what you will about IBM...
And most of it is richly deserved, too.
However, in this case, being familiar with how IT in the banking industry works (think toilet bowl of piranhas), bringing IBM will likely fix the issue to some extent....especially since their own in-house IT made a complete cock-up of things.
Anon because you know, employed in the banking industry in IT
-
Thursday 26th April 2018 13:40 GMT d3vy
ARRRRRRRRGGGG
Ive just tried to log into business banking and it wouldnt let me in. It says my user name or password is wrong so I selected forgot my password.
It said that it couldnt find my account - assuming a mistake in my user name I hit the "forgot my user name" option and filled in the form. Remember this is on the BUSINESS banking page. It gave me my personal account user name.
In short they have properly fucked up - Years ago they tried to link my personal and business accounts together into one online login.... and failed (I got £150 compo for that!) Now it looks like they have lost the business login altogether!
Fan-Fucking-Tastic.
Now to sit on the phone for 2 hours.
Edit :
Just noticed the error message (Verbatim) :
"1100086: Sorry, we can't identify you from the information that you 've given. Please check it and try again."
At first glance that seems ok. Then "you 've" I know its only a typeo but HOW THE FUCK WAS THAT NOT PICKED UP IN TESTING? If they have missed something that basic what hope is there of anything else having been properly looked at?
Oh and the "Help and support" link goes to a blank page - quite appropriate.
-
Thursday 26th April 2018 16:37 GMT clanger9
HOW THE FUCK WAS THAT NOT PICKED UP IN TESTING?
Indeed. More examples:
- The account list page is titled "holding list" (yep, in lower case).
- The pages are trying (and failing) to load resources from internal test domains.
This is pretty basic stuff that even cursory testing would pick up.
I'm sure there'll be plenty of blame to go around, but it does look like they went live with software that wasn't sufficiently tested. On the up side, the website is more-or-less working again now, so hopefully they're over the worst...
-
-
Thursday 26th April 2018 13:44 GMT Anonymous Coward
Not even the biggest baknking failure of 2018...
There are some major changes coming to all UK banks this year, this TSB issue will be small fry compared to some of them.
My prediction for big failure is Cheque Clearing from August bringing various banks major headaches and public disruption of service. Probably a good idea to stash some cash under the mattress.
-
Thursday 26th April 2018 16:49 GMT jonfr
Always have two banks
I read in a article few years ago that the smart idea is to have two or more banks. In the case one of them (your main) messes something up, goes bankrupt or just burns to the ground for some reason. At least that way you have access to active bank service. I think anyone that is having issues at the moment needs to move to a new bank already and forget TSB for the moment.
It is also clear that IT at TSB did not use the process of duplicating the data first and then switch on the new system. Having the old system as a backup just in case something didn't work as planned or not work at all.
-
Thursday 26th April 2018 17:14 GMT TrumpSlurp the Troll
Half and half?
Looks like the back end is churning away doing all the right things, and the front end is totally fucked.
Probably soon after the problems started someone decided that the bank was up and running and handling the usual business fine, and all that was needed was to fix the web front end and all would peachy. So forward, no rollback, we have migrated successfully, break out the champers and put up a notice apologizing to those few users who were having online problems (temporarily).
Just a bit of Agile fail early fail often with those weird webby types. Couple of sprints and all fixed.
By the time it was obvious that online access was borked long term and wasn't going to be sorted any time soon a couple of days had passed and reverting to the old system was nigh on impossible and getting harder every second.
The only option appeared to be to plough on and pray for a miracle.
If there is not enough hardware to handle the front end load then it's going to take more than a quick trip to PC World to stand up more kit and get it integrated. This may be where IBM come in.
-
Thursday 26th April 2018 17:28 GMT Anonymous Coward
Re: Half and half?
From reading various comments over the past few days, though, it looked like people were saying the whole thing was in the Amazon cloud. So if that's really the case, surely then it would be trivially easy (albeit not cheap) to throw more hardware at the problem. Which suggests a deeper systemic issue.
-
Thursday 26th April 2018 20:21 GMT RancidRodent
Re: Half and half?
"Looks like the back end is churning away doing all the right things, and the front end is totally fucked."
Yes, the decades old (ex mainframe) COBOL bit (that has been sunset and "containerised" under MicroFocus (yee gods)) is working - it's the new fancy-dan front end script kiddie stuff that's borked.
-
Friday 27th April 2018 05:41 GMT Anonymous Coward
Re: Half and half?
They did actually break out the champers...
The Guardian has uncovered images on LinkedIn of IT workers in Spain who handled the botched transfer celebrating with sparkling wine and exclaiming “TSB transfer done and dusted!” and “Hell of a team!”. The pictures were posted just as the meltdown was beginning, and up to 1.9 million customers were subsequently locked out of their accounts.
-
-
This post has been deleted by its author
-
Thursday 26th April 2018 18:42 GMT Anonymous South African Coward
Sisyphus is not happy, as getting IBM on the team now means that he now will have to carry the object* upwards whilst farting** a merry tune and also have to take a short course on the OSHA act on a regular basis lest he should hurt himself.
*greased with Face Man's special grease
**past the inserted bluetooth-controlled buttplug which got inserted to stop him from farting
IMHO this may (or may not be) the beginning of the end.
-
-
Thursday 26th April 2018 20:28 GMT RancidRodent
Re: IBM
'Cept they're not running on the old system nor any part of it, nor is there a migration path back to it. That said, IBM probably don't understand the old system anyway - the people who do would have been sacked by now - "replaced" by those who can do the needful - until the needful needs doing...
The reality is nobody ever got the sack for hiring IBM, but there is a first time for everything - IBM is an empty husk, Ginni Rometty has thrown the baby out with the bathwater - there's no in-depth knowledge left. She bet on cloud forgetting that z and the old men who smell of wee who understand it actually pay the bills. IBM is doomed, Ginni strangled the golden goose.
-
-
-
Thursday 26th April 2018 19:18 GMT Anonymous Coward
Re: Drowning Man
More likely, the customers are going to pay for the bank being fined. OK, so this is not quite as insane as fining NHS trusts who screw up because are chronically short of money, but when a company has a single shareholder, they really aren't going to let themselves suffer.
-
-
Thursday 26th April 2018 22:13 GMT Anonymous Coward
What is the new infrastructure like I wonder?
Heavily snipped from the article: "In 2015, Sabadell said ... £450m .... would be "more than sufficient" for the switch ... from Lloyds' to a newly developed version of its Proteo system. That was designed in 2000 specifically for mergers, and is based on Accenture's Alnova system. But the transition of the 5 million-plus customers ... to the new platform ... Proteo4UK ... CEO Paul Pester admitted today .... an "incredibly bumpy" move. Proteo4UK is a new core banking system with TSB-specific customer-facing components."
Gartner article, 2002: "Alnova Financial Solutions, formerly Altamira, is an integrated core banking solution supported on IBM S/390, Unix and Microsoft platforms with Java or .NET technology, backed by Accenture's experience."
So Alnova is at least 16 years old, but formerly something else. Based on that, Sabadell created Proteo. Based on that, they then created Proteo4UK.
I bet it's the sort of infrastructure setup you'd try to move off on as quick as you could, not move 5 million customers on to!
I think Unisys have something to do with it as well.
I'd just get my money out of TSB. A bank is only as good as its IT.
-
Saturday 28th April 2018 16:21 GMT James Anderson
Re: What is the new infrastructure like I wonder?
Not so fast Tenemos T24 is the only successful banking package which was written recently. (Started sometime in the late 1990s).
Part of its success was due to it modest initial ambitions. A fully functioning system aimed at small overseas operations with few transactions and customers. They have gradually increased performance, capacity and scalability to handle large sized banking operations.
Nearly every large bank is running COBOL code written in the 1980s or before on an IBM mainfram.
Some are still running on UNISYS. For smaller banks RPG on AS/400 is the default choice.
These systems are difficult to write, especially when you analise the use cases and find that the "user" in 60 per cent of the cases turns out to be some government department. Use cases involving actual customers probably account for about 10 per cent.
But never mind all good OO text books show an example banking program in 10 lines or less.
-
-
Friday 27th April 2018 02:42 GMT Cynicalmark
I can imagine.....
Maybe something along these lines?
Director - we need to transfer these accounts to our new system
IT - what new system?
Director- oh this new wonderful package we just bought. We had a meeting about it.
IT - Nope. We didn’t have a meeting.
Director- oh we did, you weren’t there but the nice sales chap told us you would say yes so we signed for it.
IT - ffs. OK, we will have to learn about this and transfer say a thousand dummy accounts in and out of it and test all aspects....
Director- ahem no. We have been told by the board that it must happen this way on Friday....
IT - ffs when? We will lose data potentially, and well fuck it - I quit.
Director- fine I will use Colin the Intern - he reckons he was better than you anyway and knows this outsourcing type jobbie who can help him.
IT - Good luck you prat
1 week later
Director - Colin what is wrong? You said you could do it..
Colin - ......door swinging.....tumbleweed rolls across empty IT dept.
Director- did we pay him any money? No? Oh bugger. Hello IBM? Do you know how to do this?
IBM - please wait caller, we are asking on global forums for answers as all our best and brightest were told they were not required a while back. While you are on the line could you tell us if we have answered the question to your satisfaction.....?
Director - Fabulous, yes, i see you know what you’re doing. I’ll break the good news to the board.
Something like that maybe?
Total incompetence comes to mind. It seems obvious that zero test data transfer runs were carried out, the fools just thought lets just transfer the lot in one go.
Just total effing incompetence, I mean clusterfucking moronic twattery of the highest level.
-
Friday 27th April 2018 07:13 GMT Anonymous Coward
I'm not surprised at any of this. I spoke to a former colleague who was on the testing team in the UK and he said it was madness. Supposedly Agile but really waterfall shoved into sprints. Huge numbers of defects being raised which when fixed, would produce more defects. Myriad of minor problems such as Spanish error messages and poorly worded text were ignored as there was no time to fix them. Getting bank hardware to connect successfully was a lottery, I imagine the poor sods in the branches are in as much pain as the customers. The test cases were written by the Spanish developers so you can imagine the level of detail they contained.
Senior management would breeze in from time to time, be fed happy path demos and be on their way none the wiser. There was definitely some spin going on in the project, no bad news was ever reported internally or otherwise. They use to offer crappy prizes for tester and team of the week and even had a test case lottery FFS.
Nobody seriously thought it could be ready by 2019 never mind the original implementation date of October 2017. But dates had to be met to avoid Lloyds increase in their IT fees so it all ploughed ahead regardless.
-
Friday 27th April 2018 07:14 GMT Peter 6
A long time coming.
Visa were chasing Lloyds Banking Group as far back as 2007 to improve their capacity to process transactions. Their IT systems have always been a bit of a joke. I believe at one point Visa were processing about 30-40% of LBG's transactions during one Christmas period because LBG's systems couldn't cope and they were too lazy/scared to upset the carefully built equilibrium made up of decades of mergers and acquisitions so found it easier to pay Visa in yachts to process their traffic for them.
TSB are a casualty of this awful approach to IT policy.
-
Friday 27th April 2018 07:45 GMT Anonymous Tribble
I've been banking with TSB for nearly 40 years. In that time I've only seen a small number of minor cock-ups (two staff issues, one computer), but this one is just ridiculous.
I've been trying for days to make a transfer to another account (not TSB) before that one goes overdrawn, but it says my password is the wrong length. I've also been trying to pay off my TSB credit card, but get the same problem.
Also, the site is really slow. I get templates showing, then when I select an option it just shows as a blank screen for ages before the details appear. Someone has made a bad technology decision here.
Today when I try to log in (normal login screen, all details correctly entered) I get "you have successfully logged OUT".
I don't want to switch, because the staff in my local branch are really nice, and usually very competent and I know this isn't their fault at all. *sigh*
-
-
-
Friday 27th April 2018 13:51 GMT Anonymous Coward
Re: So Which Bank to Flee to?
"I'd go with Nationwide though, as I don't think their systems lend themselves to this scale of cockup."
I believe they've already been through the pain of modernising their banking platform - though it didn't result in a week or more of outages.
Either way, probably the safest bet. Plus you get to vote for the CEO and board every year if you like.
-
-
-
Friday 27th April 2018 10:23 GMT Anonymous Coward
Trustee Savings Bank
On the one hand TSB save £200m by moving off Lloyds systems and we've just announced a profit of £225m.
On the other hand,
No overdraft fees for a month has just cost us £10m.
Extra interest on savings will cost £20m. ( Thankyou as I have an account with them )
That's before we pay
- IBM a vast fee.
- Customer ex-gratia payments
- FCA fines
- ICO fines for data breaches
- Loss of customer goodwill
- Severance packages for directors
I'm sure the next bank to migrate will plan better.