TSB's middleware nightmare: Execs grilled on Total Sh*tshow at Bank

The boss of British bank TSB has insisted it carried out rigorous testing ahead of a systems migration that saw thousands without access to their accounts for over a week – but revealed he won't be getting a chunk of his bonus. In a heated evidence session in front of Parliament's influential Treasury Committee, …

[Voland's right hand]

along with bringing in solicitors Slaughter and May

Nominative determinism?

[Anonymous Coward]

Re: along with bringing in solicitors Slaughter and May

Whilst Slaughters are far from the biggest law firm even in the UK, if you put aside pure scale, they are by a long chalk THE most prestigious law firm in the UK, and certainly top five in the world. So that's going to be VERY expensive.

Unfortunately the top City law firms all know which side their bread is buttered, and will rarely act against the banks. I would wager that they are not going to come in and point any fingers at TSB. They might do against any suppliers involved, but the primary purpose is presumably to ensure that TSB are defended against any high value claims, and they can say "look, we had a proper investigation".

[fruitoftheloon]

@Ledswinger: Re: along with bringing in solicitors Slaughter and May

Ledswinger,

indeed, I was responsible for figuring out how to automate getting their many documents to Office 97 (and back if it went wrong) when the big green button was pressed to migrate to NT & Office 97.

Slaughter & May were VERY switched on.

Obvs this was 'nowt on the scale of a banking migration, but it worked properly...

Jay

[Uberior]

Re: along with bringing in solicitors Slaughter and May

The word "prestigious" never needs to be used to desribe anything that's truly prestigious.

[Justin Clift]

DR?

Wonder how their new DR site is going.

You know, the one they're legally obliged to have at all times in case of disaster.

If they haven't scoped gotten their primary site to a decent state, there's practically zero chance of them having DR coverage at the moment.

[Anonymous Coward]

Re: DR?

"Wonder how their new DR site is going."

It's faithfully replicating the broken applications running in production.

[Tippex]

Re: DR?

"It's faithfully replicating the broken applications running in production." If that is the case then its not just the data migration and the log on that is the issue. The transaction processing should be OK in DR and that is if what TSB are saying is true. Otherwise its a much bigger problem than is being reported.

[Anonymous Coward]

Re: DR?

As one who knows - any paragraph in any document that doesn't contain the assurance of increased profit is not read or digested at the board table.

It does not register. So any regulatory bumf or common-sense belt-and-braces stuff is left for others to worry over - just as the last-minute confidential tag is affixed to ensure that no-one else gets to even read it.

Ooops!

[Blockchain commentard]

Just like at a company I used to work for. The IT boss would decide to do a migration and even when it was obvious it didn't work, he'd push on in the vain hope everything would just click into place.It never did. HIs replacement had the same mentality and didn't last long when he stuffed up the company payroll replacement system.

[Doctor Syntax]

the main "challenger"

Close. Just a letter astray. "Challenged".

[Alister]

The problem, Pester said, is that the middleware systems were unable to deal with the number of customers that wanted to access the banks systems

And he's trying to use this as an excuse? "Please sir, it wasn't me, the computer did it"

Can he not see that determining the expected load and planning for adequate resources to deal with that load are what his job is supposed to be?

It's like he's claiming that suddenly there were twice the number of customers from what they expected.

It's not like setting up a normal publicly accessible website, where planning for expected visitor numbers is always a bit of a gamble.

In this case, there are a finite number of account holders, so working out the expected load should be easy, even if, because of the downtime, more of their customers were trying to log in to see what had happened to their money...

/rant

[Anonymous Coward]

Probably lying too. I have a bottle what says that the backend is effed-up as well.

[Anonymous Coward]

yep, lying

GPDR looming, and the host bank going to IBM cloud.. so they HAD to migrate.

I see several problems here.

The first problem is testing, including volume testing.

It is quite obvious that you can't test the normal volume. After several days of outage, you are going to have several times the requests of a normal day.

Second: quite obviously they did not design a system that refused to allow more users that those it can acommodate. That is horrible, and something I only do for small companies. I have designed big systems and the underlying auth system should also control the number of active users.

On a big system as a bank this is non trivial to do, and many people don't even understand WHY you have to do this.

Third: that all explains the outages. But the auth problems? People had wrongly assigned accounts. At some point they deleted and recreated at least part of my data.. my common account was my personal one --> then no accounts --> no login (deleted) --> correct account assignments, BUT it is quite suspicious that my accounts appear on a different order. for performance reasons I pretty much doubt they are doing an "order by" in the DDBB, therefore probably (speculation) I see the natural order in the DDBB.. so as my personal account is older than the shared one.. to my trained eye this suggests that they run a script to correct account associations, instead of reloading all data. this is pure speculation and I might be wrong, of course.

Anon, as I am looking for a job right now.. and you never know..

[Doctor Syntax]

Re: yep, lying

"I have designed big systems and the underlying auth system should also control the number of active users."

Don't you then have customers complaining that they can't get access and then overwhelming the auth system with repeated retries?

[Anonymous Coward]

Re: yep, lying

On a well designed system? No, not at all, and the compute and memory cost of rejecting them is very small.

the system would say "Currently our systems are whateveryouwanttotellthem", for a period of time, and other users will be able to log in if the system is available.

The cost of doing this (in compute terms) is very small if done properly.

The real issue is how do you determine capacity.

I can demonstrate and do the math for it, but hey, I charge for that!

I have proven it with a live system that has many million time sensitive calls, and other smaller systems.

[Keith Oborn]

Re: yep, lying

I'm a long term Lloyds customer, and their online stuff does exactly that: sometimes it throws a polite error when you try and log in, "please try later" and then works fine a second later.

[fruitoftheloon]

@Keith oborn Re: yep, lying

keith,

likewise for first direct (very rarely)...

[Doctor Syntax]

Re: yep, lying

"No, not at all, and the compute and memory cost of rejecting them is very small."

But what about the customer reaction?

[sabroni]

Re: But what about the customer reaction?

You have to judge that in comparison to the reaction TSB customers are having at the moment. Being told you can't log in, then getting in a minute later and everything being fine is no big deal for most people most of the time. Compared to this balls-up it's clearly a better system.

I prefer to be mildly irritated rather than unable to access any of my money.

[d3vy]

Re: yep, lying

"But the auth problems? People had wrongly assigned accounts"

Ive been into a branch today because my business accounts are still inaccessible - when they went into the system to view it my personal account user name is somehow assigned to the business accounts.. BUT it wont let me log in with those details..

The girl in the branch was really helpful but said that they couldn't rectify it in branch - its been passed over to the IT team. This isnt a capacity issue this is a wholesale data fuck up.

[gv]

PHB

"Can he not see that determining the expected load and planning for adequate resources to deal with that load are what his job is supposed to be?"

I'm guessing they employ the "brightest and best" IT people for this, just like everybody else.

[hplasm]

Re: PHB

"I'm guessing they employ the "brightest and best" IT people for this, just like everybody else."

Top men. Top. Men.

[Duffaboy]

Re: PHB

They got them all from Linkedin

[Dan 55]

Of course the back end's screwed as well. Direct debits don't get collected, standing orders don't get paid, cards don't work, bank branch access don't work. It's a clusterfuck.

[katrinab]

You have to plan for peak load + a margin over and above that.

Peak load times would be end / beginning of month when people receive their salaries and pay their bills, the end / beginning of the tax year when lots of people put money in pensions and ISAs, and around Christmas, when people spend lots of money on things.

[Anonymous Coward]

The middleware...

So, the storage is fine, and the UX is decent. Its only *all the business logic* thats broken? Phew, I thought it was something serious!

[Anonymous Coward]

More TSB incompetence amusement - they sent me an apologetic email yesterday (obviously this has gone out to all their customers) which near the very top says "We want you to recognise a fraudulent email if you receive one. We will always greet you personally with your name and quote the last four digits of your account number..."

The account number digits were correct, but my name was nowhere to be found...

(I'm going to Nationwide today to open my new main current account, but leaving the TSB one as a backup - I suppose that means I'll be a "retained" customer in whatever face-saving report they concoct.)

[d3vy]

"The account number digits were correct, but my name was nowhere to be found..."

I got that email too, what are they thinking?

[DuchessofDukeStreet]

I also got that email. trouble is, I've never had an account with Lloyds or TSB in my life. And the email address that received the letter has never had an email from TSB in the three years that it's been active.

[0laf]

Banking exec - Is my bonus affected to a point where I'll notice any dip in my income?

Yes - This terrible crisis must be resolved to the satisfaction of all affected I reeeealy mean it

No - I do not give a fuck, I will make a show of caring but I really really do not give two shits and will leverage this as an opportunity to cut more within the bank and leverage my share options just prior to jumping ship at the star of the next clusterfuck.

[trevorde]

He sounds like he's already banked his bonus for a 'successful migration'

[Radio Wales]

Harrrumph!

... and NOT with TSB I'll wager.

[Anonymous Coward]

Sound like a perfect Project Management Answer.

Yes the migration for the back end went 100% as planned. The fact that no one can access the new system doesn't matter...the migration itself went perfectly.

[Anonymous South African Coward]

Slaughter and May... lovely.

May I start the slaughter?

[Bill M]

Sad

This whole TSB affair makes me very sad.

[wolfetone]

"The boss of TSB has insisted it carried out rigorous testing ahead of a systems migration that saw thousands without access to their accounts for over a week - but revealed he won't be getting a chunk of his bonus."

I'd say this is correct. They did rigorous testing on the note counting machine to count all the £50 notes that would make up their bonus payment.

[Anonymous Coward]

'The issues we’re seeing in the system are - middleware'

Middleware - Is that another term for humans with long-time IT experience, that HP shit-canned over time... In favor of cheap outsourcers full of hollow promises. All so, those at-the-top at HP could depart with mega-bonuses?!

[iana]

Re: 'The issues we’re seeing in the system are - middleware'

How are HPE involved in this?

[Anonymous Coward]

Re: 'The issues we’re seeing in the system are - middleware'

@ iana... My bad - Good catch - It should have read Ulster / TSB etc. I was reading today's HP article on the other screen at the same time. Wires crossed! Oh well, too late to edit now - Oops!

[Peter27x]

Re: 'The issues we’re seeing in the system are - middleware'

And what does Ulster have to do with this?

If you meant "Ulster Bank", well that is part of the Natwest/RBS group - which has it's own and separate rich history of IT failures - unrelated to TSB or Lloyds.

[Bronek Kozicki]

Re: 'The issues we’re seeing in the system are - middleware'

"Middleware" to banks is what trucks are to a shipping company. Sadly banks execs do not know it.

[grumpy-old-person]

Re: 'The issues we’re seeing in the system are - middleware'

The bank I worked for was moved to TCP/IP while all the others in the country at the time were captive IBM accounts using SNA.

When Burroughs became a problem and a move to IBM s/390 and onwards was done I refused to succumb and used Cisco channel-attached routers - tunneling the SNA between sites and avoiding the FEP and associated software costs.

Imagine my astonishment when the person in charge of IT enquired about our SNA network!

It took 2 days to craft a suitably snotty reply that pointed out that while he was being a big-shot and contemplating his navel we had saved a bundle of money and had only a single IP network,

Execs usually cannot discriminate between execute as in kill and execute as in carry out operations!

[Anonymous Coward]

Continuity

No DR/BC in place.

Failover/Backup plans.

Migration/testing pants.

Project mgt crap.

Senior staff overpaid and under educated.

Dotting the 'i's and crossing the 't's staff under paid and over worked.

Should be fined under DPA/GDPR...

I could go on..

[DontFeedTheTrolls]

Re: Continuity

Are you just trolling?

Yes, there have been issues, but in what universe are any of the above proven today to be the cause. The root cause analysis is still under way, the result of which will probably never be made public. Everything we're being told is rumour and conjecture, including probably what has been fed to MPs.

I reiterate, you're just trolling and that's why you're AC

[Chris Miller]

Obligatory Dilbert. (From more than 20 years ago, nothing changes!)

[Doctor Syntax]

"Obligatory Dilbert. (From more than 20 years ago, nothing changes!)"

Quite so. Look at today's.

[Anonymous Coward]

The buck stops with him

I've been working in a bank when they've applied an across the board 10% pay cut to all contractors to reduce costs. Similarly, Natwest previously seemed to claim outsourcing wasn't to blame for one of their previous outages, because the off-shore team didn't do the failed migration, despite the fact they'd made redundant some of those most knowledgeable about the migration process so the on-shore team that was left had insufficient skills to ensure its successful completion, as a direct result of the off-shoring effort. It's easy to point the finger at one of the IT teams as having done something wrong, here's the point.

Things can and do go wrong in migrations. A process fails, some unexpected data blocks the process. While you obviously try to minimise the chance of things going wrong, you also need to ensure your plans make absolutely sure that you can back out and get a reasonable level of service provision back promptly when things do go wrong. The blame for a lack of adequate planning lies with management, not any of the lower layers.

Customers aren't too fussed about a few minutes extra downtime in the night while you roll back after a failure. They don't care about a new faster system being rolled out a bit later (they probably don't want to have to learn to use a new system anyway). What they care about is when they have days or longer where they can't access their money, or where their data is exposed and leave them vulnerable to fraud or theft. I don't think they'll be finding they've made those savings, because the reputational damage has got to have outweighed many years of cost-cutting - which, unfortunately, just seems to be all too common a result of short-sighted cost-saving measures.

[TheVogon]

Re: The buck stops with him

"I've been working in a bank when they've applied an across the board 10% pay cut to all contractors to reduce costs."

To which the appropriate response is either find a new role to coincide with your notice period and don't sign up, or if that's tricky, each renewal you look for a new role and when you find one, leave at the end of your contract, effectively giving them no notice.

It is generally the really good people that are in demand that walk from these type of demands, so in my experience it down-skills a contract workforce and often ends up costing money.

[Peter Danckwerts]

Re: The buck stops with him

Thast also sounds a bit like the British Airways disaster.

[Herring`]

Middleware eh. Maybe Biztalk wasn't the ideal choice.

[Anonymous Coward]

"Maybe Biztalk wasn't the ideal choice."

BizTalk can scale both up and out - I have used it with hundreds of connections and millions of transactions. It has a benchmark wizard to allow you to check performance under predicted transaction types and volumes. For greenfield sites BizTalk is often a no brainer because it usually has a much lower TCO and is generally easier to use then the common alternatives.

The problem is probably not the choice of middleware, but the sizing and design of the supporting infrastructure and databases. So it likely boils down to lack of adequate testing.

[Anonymous Coward]

Hmm .. IBM called in ... Middleware ... hmm ... WebSphere? ... hmm ... Java and scaling?

[Bronek Kozicki]

@AC you might be onto something. Banks love presentations and UML graphs, and Java is best suited for the kind of software that can be presented as UML. It impossible to find an established bank without large Java presence in the critical systems. When Oracle finally starts charging royalty fees, banks are going to be captive users for decades to come.

[James Anderson]

My thoughts exactly. Would be nice to get some confirmation.

Once worked on a newly migrated system where we did some complex transfers on a DB2 database with COBOL/CICS application -- then uppdates a backend UNISYS/IDMS database and sent a whole bunch of data to various reporting systems.

Of course we had a performance problem. When analised it was found that the Websphere/Java front end was spending more time routing the transaction and formatting the reply the the whole complex backend took to do the actual work.

I remember my WTF reaction when I went through the Java beans/Webspere "HelloWorld" tutorial and ended up with circa 3000 lines of boiler plate coding -- plus a couple of dozen lines of XML config!

[JWLong]

"Maybe Biztalk wasn't the ideal choice."

Did someone forget to tell Pester that you can't run a backend of that scale on MS Office Access.

[Valeyard]

QA

everytime the word "testing" is used another QA CV goes out, everyone can see what that particular buck is winging its way

Whether or not it's deserved something must be done, and blaming QA is something, therefore we must blame QA

[Primus Secundus Tertius]

Talk to the techies

If Parliament really wanted to know what went wrong, they should talk to the techies, not to management.

Except if they go looking for the techies they have either been "let go" or are a continent away.

[Nano nano]

Re: Talk to the techies

Or decamped from Brexit Britain ...

[Anonymous Coward]

Re: Talk to the techies

If banks were characterised by their biggest single internal function or cost base, they would be called IT companies. Problem is, the level of IT knowledge in the average boardroom in the UK is woeful. So, yes, Parliament should ask the techies if they want a straight answer.

It would have helped if the board had asked the techies a few more questions themselves before committing to puny budgets and fantasy timelines. Maybe they did ask but didn't like the answers.

[Andrew Findlay]

Re: Talk to the techies

Disaster management 101:

The last thing we need right now is to expose the techies to public scrutiny. They are the only people who can fix the problem and they will already be stressed out and short of sleep.

At this stage it is the CEO's job to take the flak while making sure that the people doing the real work are looked after and protected. It helps a lot if he can say useful, true things about what went wrong and what is being done to fix it, but that is very much secondary.

Disasters are fascinating and we can learn a lot from them, so I hope to see a few good technical papers come out of this in the next year or two. Parliament won't get the level of detail that Reg readers appreciate, so leave them to talk to the suits...

[PhilBuk]

Re: Talk to the techies

@Andrew,

All well and good but Pester comes over as an arrogant twat who seems to think that he can bluster on about everything being fine now when even the branch systems aren't working correctly. This is not a good example of damage control.

Phil.

[MooseMonkey]

Re: Talk to the techies

Disaster management 101 - BEFORE taking flak, apologise sincerely and properly, not say "it's fixed over here"

[Disgruntled of TW]

Re: Talk to the techies

@Primus - doubtful that they'd understand the techies. Amber Rudd didn't understand encryption and privacy, then criticised the experts that attempted to advise her, and for that reason alone I am pleased that she is no longer Home Secretary. Don't see a realistic hope anyone in government would understand the term "middleware" let alone the absence of a failback/recovery plan for this migration.

[Doctor Syntax]

Re: Talk to the techies

"doubtful that they'd understand the techies."

That's true but having the answers out in public would mean that they'd be accessible to a lot of people who would understand. Nevertheless this is not the time to be asking them; they've got better things to do.

In any case the generic answer to "What's wrong?" is "If I knew it wouldn't be wrong and when I find out it won't be.".

[Rabster]

Re: Talk to the techies

To paraphrase Yes Minister "On the rare occasions when the techies understood the questions, the MPs didn't understand the answers"

[Anonymous Coward]

Global expertise in Middleware ... Ahem

Global expertise in selling middleware software and infrastructure perhaps. Also experts in hyping up execs with delusions of grandeur of always on, agile developed niceness.

You are on your own if you a foolish enough to integrate it with your own systems. Whatever the problem is it will involve an upgrade or additional licenced feature (and I really mean whatever the problem)

[}{amis}{]

TSB..

Totally Shit Bosses....

[The Specialist]

Slaughter and May

or S&M?

[fruitoftheloon]

Re: Slaughter and May

the specialist,

I was in the room when a v senior manager had an interesting discussion with a colleague who was 'wordly wise' about acronyms thereof and urls..

most enlightening for a young techie!

[herman]

"Slaughter and Pester" - Seriously? What is this, Monty Python's Flying Circus or a Butcher Shop?

[groMMitt]

...a Different answer?...

Two points you've all forgotten...first (though surely coincidentally) the date of the disaster, when it kicked off, was Hitler's Birthday and the second, knowing those hipsters at the banks, the date was 20/4...or to Americanise it, 4/20...National Marijuana day...or am I just being ultra-paranoid?

[Mr Dogshit]

Challenger bank

... like the space shuttle

[tip pc]

the emperor's new clothes

He's just regurgitating what he's been fed from people that report to him that either want to keep their jobs or keep their contracts.

he personally has no clue what the issue is and shouldn't, his job is to manage the people that should. The answer is to not outsource but to retain the skills in house and on shore so that loyal expertise is on hand to fix these issues that can occur with systems that have been kludged together over decades.

loose the people and they lose the expertise, experience and combined history telling the story of their now bespoke technology stack all to save a few quid which they likely have now lost.

the upshot is that now this has been done their platform has potential to be one of the most reliable out there as all the old guff should now be gone. I state potential as we have no idea what kludges had to be whacked in to get this working again.

[HmmmYes]

Re: the emperor's new clothes

No, not for this.

For a bank, the comptuers/softare are the entire operation.

This is not the some obscure role, performed by a minion in the arse end of Swindon.

This is the the entire life blood and skeleton of the company.

Sure, he doesn;t need to role up his sleeves and programme but he *does* need to know what they are running and why its failed.

[The Oncoming Scorn]

Re: the emperor's new clothes

Ohh I lived in the Cheney Manor area of Swindon back in the 80's.

[yoganmahew]

Re: the emperor's new clothes

And I lived in Old Town in the '90s - for another industry that skimps on technology, but lives and dies by it - the airline industry. To be honest, we're normally talking about some airline gone TU, so watching the defence of the old TSB mainframe has provided a welcome relief for me from defending old airline mainframes! :)

[Anonymous Coward]

Root cause...

"Technology is dominated by two types of people: Those who understand what they do not manage. Those who manage what they do not understand".

- Putt's Law

[HmmmYes]

I did have a google.

Im curious what 'middleware' TSB uses that can end up with ~50 customers seeing other customers details.

I found this:

https://tsb-applications.advorto.com/Candidate/Candidate/VacancyInformation.aspx?VacancyId=4932

'Hello, we're TSB, and we're different from other banks.

We don't do investment banking, overseas speculation or big corporate finance. We just serve individuals and local business customers – we call it "local banking".

The continued growth and development of our CIO function means that we are now focussing on building our Infrastructure and Architecture teams. This newly created role Manager, Middleware Design Authority has the key purpose to deliver great service to our customers, both internal and external, by providing architectural understanding , leadership and in depth subject matter expertise (SME) in Middleware Technologies.

This person has the responsibility to:

Provide Architectural and SME knowledge to the Senior Manager Platforms & Midrange

Ensure Middleware services meet TSB’s requirements

Provide roadmaps and lifecycle management for the Middleware services

Work with TSB Stakeholders to ensure their applications are hosted on the most appropriate platforms

Includes End-to-end architecture, engineering, deployment and maintenance. Will require close collaboration with Sabadell Information Systems the supplier of IT services to TSB.

Key Accountabilities include:

To be the TSB expert in, and trusted advisor for Middleware technologies and their integration to the larger enterprise platform.

To keep abreast of technical developments, including vendor product strategies and industry advancements and to participate in developing or setting standards as required

To provide Technical design expertise across TSB

To establish and maintain effective working relationships with all stakeholders

To act as TSB subject matter experts, architectural governance and design authority for Middleware within the Platforms & Middleware team.

To assist with improving the standard of technical delivery and competencies within CIO and across the business relating to Cloud and Midrange.

Work closely with SABIS (Group IT) to deliver the full lifecycle technical roadmap

To ensure that the technical authority role is performed including interpreting business requirements, approving technical proposals, and reviewing progress to ensure they meet business needs

Ensure projects meet business & technical requirements across TSB and are within the agreed budget and timescale

To ensure that the infrastructure is developed in a manner that supports appropriate levels of resilience and performance but supports scalability and is agile enough to support the future needs of the business

To work with technical teams in SABIS, product owners and 3rd party suppliers to ensure that technical solutions are aligned with the technical strategy

To establish, deliver and refine a road map for Middleware

Ensures TSB remain current, deliver value to the business and support the needs of stakeholders in terms of scalability, resilience, security, capacity and operational maintenance

Identify areas of development that require engineering activity and establish an appropriate programme of work and execute it '

All for the princely salary of ...... £36,050 - £51,980

Im still no wiser on the Middleware. The HR post might as well as put 'Magic!!' instead of 'Middleware'

[Anonymous Coward]

Let me translate this bit for you:

Provide Architectural and SME knowledge to the Senior Manager Platforms & Midrange

Ensure Middleware services meet TSB’s requirements

Provide roadmaps and lifecycle management for the Middleware services

Work with TSB Stakeholders to ensure their applications are hosted on the most appropriate platforms

Includes End-to-end architecture, engineering, deployment and maintenance. Will require close collaboration with Sabadell Information Systems the supplier of IT services to TSB.

It means Sabis (Sabadell's IT dept that they've spun off into a 100%-owned consultancy for some reason) hasn't got a clue what they're doing and your job will be to design it for them, write the specs for them, do code review for them, and keep banging on at them until they finally deliver something usable.

[PhilBuk]

That's a lot for a pretty crap salary.

Phil.

[eamonn_gaffey]

...and would need double the high end of that to walk into the shitstorm, and work for a management team headed by the likes of Pester. On second thoughts, take the job as a 12 month contract, undercut IBM by 15%, and still walk away with enough to buy a retirement home in France...........

[Anonymous Coward]

@PhilBuk

That's my thought as well. It is possibly a decent salary for a country you'd outsource to, but in the developed economies that seems very low considering the responsibility. I suspect the point is to low ball it so that they can say we offered the job didn't get any qualified candidates so had to outsource the role overseas.

[therealvicz]

‘Middleware’

It’ll be a few thousand lines of spaghetti JavaScript and half a dozen obscure packages with stupid names that no-one remembers I expect.

[The Oncoming Scorn]

Re: ‘Middleware’

"It’ll be a few thousand lines of spaghetti JavaScript and half a dozen obscure packages with stupid names that no-one is available who remembers who wrote it or why they wrote it that way, because the closest\last point of contact was shown the door with extreme prejudice I expect."

FTFY.

The Graybeard engineer retired and a few weeks later the Big Machine broke down, which was essential to the company’s revenue. The Manager couldn’t get the machine to work again so the company called in Graybeard as an independent consultant.

Graybeard agrees. He walks into the factory, takes a look at the Big Machine, grabs a sledge hammer, and whacks the machine once whereupon the machine starts right up. Graybeard leaves and the company is making money again.

The next day Manager receives a bill from Graybeard for $5,000. Manager is furious at the price and refuses to pay. Graybeard assures him that it’s a fair price. Manager retorts that if it’s a fair price Graybeard won’t mind itemizing the bill. Graybeard agrees that this is a fair request and complies.

The new, itemized bill reads….

Hammer: $5

Knowing where to hit the machine with hammer: $4995

[HmmmYes]

Re: ‘Middleware’

The old magic (physical) engineer tales no longer apply to the world of software.

Things are much more complex.

Besides, Ive never got into a place without getting the money signed off.

[Alister]

Re: ‘Middleware’

The old magic (physical) engineer tales no longer apply to the world of software.

Things are much more complex.

My aren't we prideful.

Knowing which bit to hit is as relevant today as it's ever been - except you don't use a hammer, you use a keyboard.

[LucreLout]

ROFL...

.... bet all that outsourced and offshored IT isn't looking such a bargain now is it?

Any company of scale is now an IT company - you might be in banking, or insurance, or car production, or shipping; it doesn't matter. You are an IT company. If the CEO doesn't turn up nobody bar his PA will notice for a few days or weeks. Everyone notices within seconds when the technology doesn't work.

I trust that clarifies the "paying for talent" myth for any budding MBAs reading this.

[Mark 85]

Re: ROFL...

I trust that clarifies the "paying for talent" myth for any budding MBAs reading this.

It should but the reality is that the MBA's won't read El Reg as it's below their pay grade. Now if a financial paper was running these articles it would be a different thing.

[Red Bren]

Which bit of his bonus will he lose?

The bit before or the bit after the decimal point?

[Anonymous Coward]

It's TSB. This time around...

But all the banks are the same, cost cutting, offshoring, ever shifting "location strategies", an executive bonus culture that engenders short term motivations, blah, blah...

I believe TSB should lose their banking licence - moving banks is easy nowadays, so customer impact isn't huge (certainly nothing compared to the recent debacle). Maybe a proper reprimand would actually have an affect for all the banks, and stem the ridiculous cost-cutting-race-to-the-bottom by the big banks.

[Anonymous Coward]

Re: It's TSB. This time around...

"I believe TSB should lose their banking licence "

You can't just go around closing down great lumps of the banking industry and hoping there'll be no effect on the rest. Do you not remember the Northern Rock debacle? And that was only a tiny ex-building society.

[Dan 55]

Re: It's TSB. This time around...

The effect on the rest would be more customers. What's the problem?

If a bank's been incompetent, it loses its licence. How many more weeks would you like it to carry on like this? And if the government does nothing, every other bank will know they can get away with it.

[Anonymous Coward]

Re: It's TSB. This time around...

"You can't just go around closing down great lumps of the banking industry and hoping there'll be no effect on the rest. Do you not remember the Northern Rock debacle? And that was only a tiny ex-building society."

Personally, I'm not seeing the link between the TSB migration screw up and a [globally] screwed up approach to how retail finance organisations were capitalising themselves.

I'm guessing you haven't worked in IT for a big bank, at least not recently? If nothing is done to affect change, then at some point hence, there will be another El Reg forum thread for us armchair experts to dissect and cogitate over. And if that thread relates (say) to a black horse, cyan eagle or a peculiar shaped red triangle, the potential impact is far larger to the banking public.

[Doctor Syntax]

Re: It's TSB. This time around...

"I believe TSB should lose their banking licence"

TSB was hived off from the Lloyds group to provide customer choice. You can't achieve that by closing it down, at least not until the customers have exercised their choice and gone elsewhere.

[fajensen]

Re: It's TSB. This time around...

TSB was hived off from the Lloyds group to provide customer choice.

I somehow imagine that getting your bills paid and your money kept safe kinda beats "customer choice" by a generous margin?

And, It's not there there is actual choice either - presently, TSB victims cannot get their money out and move their accounts to a competent bank!

[Tomato Krill]

Re: It's TSB. This time around...

Effect

[Anonymous Coward]

With so many branch closures - the idea of a national bank can't be a bad one. A bit of competition would be a good thing. Why not ? We've paid hundreds of billions in bailouts and all we've got to show for it is cock up after cock up, branch closures, and top execs awarding themselves huge bonuses. Of course the brainwashing continues that the public sector is bad and the private sector is good - but come on people - private banks have wrecked the economy, RBS were caught asset stripping small businesses, numerous banks are laundering dodgy money.... how bad does it have to get ? Do we run the banks or do the banks run us ?

[John G Imrie]

the idea of a national bank can't be a bad one.

We used to have one of those, National Savings, you could use your post office to access it. Now with Post offices shutting faster than bank branches ...

[Anonymous Coward]

Re: the idea of a national bank can't be a bad one.

Goes around comes around.

"We used to have one of those, National Savings, you could use your post office to access it."

See also, more specifically, Girobank aka National Giro.

At the time National Giro was launched, it was a radical innovation. So innovative for its time (in the UK) that it was perceived (for good reason) as a threat to the then high street banks. Many other countries still have a GiroBank-like system. The UK invented the concept but our bosses at the UK's high street banks didn't like it and were effectively allowed to close GiroBank down (via privatising it) because it looked like fair competition from Girobank would lead to the legacy high street banks being outcompeted for day to day transactions, savings, etc.

Girobank offered cost effective routine day to day stuff, no huge opportunites for Our Friends in the City to rip people off, er sorry reward financial innovation, with endowment mortgages, liar loans, PPI, etc.

The Girobank concept spread way beyond the UK; lots of countries still have them. UK plc still has the City, for another few years anyway.

Those who don't remember Girobank can read about it in Wikipedia and elsewhere.

Some years later, Our Friends in the City and their high street representatives have largely closed their branches anyway (same goes for many Post Offices)., but there's not actually much real competition in the UK high street retail banking 'market' these days is there.

"Do we run the banks or do the banks run us ?"

The fact that the question even needs to be asked is indicative of the sad state of politics in Westminster.

[AndyMulhearn]

Re: the idea of a national bank can't be a bad one.

We used to have one of those, National Savings, you could use your post office to access it. Now with Post offices shutting faster than bank branches ...

Is that the one that became Girobank, got sold to Alliance and Leicester and then finally rests with Santander? If so then I was with them from the point they became Girobank. More years ago than I care to recall...

[cantankerous swineherd]

we're looking at creeping privatisation of the payments system, something that's starting to bother the Swedes. a cashless society as presently conceived hands over control to a few sociopathic institutions, who will go on to screw the country mercilessly, either by malice or incompetence.

[fajensen]

hands over control to a few sociopathic institutions, who will go on to screw the country mercilessly, either by malice or incompetence.

Why not both? Then add "For Their Own Good", with cashless and only a bit of mass surveillance and computing power, we could stop the fatties from buying unhealthy foods, muslims from buying pork alcohol and entrance to bars, jews from buying un-kosher and the vegans could stop anyone buying anything at all except grass.

The Swedes KNOWS that this will be exactly the outcome in Sweden, there nothing the swedish mind likes more than to be the Big Sister to everyone else and they will simply not be able to stop themselves with only "Systembolaget" when there is so much good to be done for other people!.

[fruitoftheloon]

@ac

what have you got to hide?

[d3vy]

"The boss of TSB has insisted it carried out rigorous testing ahead of a systems migration that saw thousands without access to their accounts for over a week"

I still dont have access to to half of my accounts...

[cantankerous swineherd]

as a matter of interest, are they bouncing your cheques? will they let you withdraw cash from a branch? I understand this may not be your usual MO, but I'm wondering how deep the rot is?

[d3vy]

I honestly can't remember the last time I used a cheque. So can't help with that.

I'm pretty much cashless so have been into branch to transfer money but not actually withdraw it, others were lifting cash though so I guess that's a yes you can.

[Anonymous Coward]

To be honest, I'm rather fond of them

They seem quite humble and don't take me for granted. They don't try and sell you stuff.

[Doctor Syntax]

Re: To be honest, I'm rather fond of them

"They seem quite humble and don't take me for granted. They don't try and sell you stuff."

If "they" refers to TSB this is directly contrary to my experience of the local formerly TSB branch of Lloyds. It left me wondering what size of account is worth less to a bank than the £20 the arrogant so-and-so behind the counter tried to extract from me.

[Tom Melly]

Best Bits!

The two bits that cracked me up were when he said it was disappointing that customers were hanging up before anyone could speak to them and 'acknowledge' their complaint. This was with an average wait time of 30 mins.

The next best bit? That that 30 mins doesn't include anyone who hung up after, say, an hour - it only includes people who actually spoke to someone.

[Anonymous Coward]

‘The operation was a success but the patient died...”

[Anonymous Coward]

oo cute template engine tsb

even now....

[Anonymous Coward]

MIA

What's the evil WIndows 10 angle?

[Duffaboy]

Rigorous testing

1. Can I logon Yes Check

2. Does Solitaire still work Yes Check

3. Collect Bonus

[Anonymous Coward]

Re: Rigorous testing [Fixed it for You]

1. Can I logon Yes Check

2. Does Solitaire still work Yes Check

3. Collect Bonus Check ...... No thanks .... 'Cash Only' .......

I saw the 'Fag Packet' System Design !!!

:) ;)

[Anonymous Coward]

do they use computers

That's advanced. I get the impression my bank uses stone tablets and coal fired steam for their system which shuts down in the weekend. Every year they increase the costs and every year you get less service for the privilege of banking with them. They made sure that you can not switch to another bank and keep the same account number.

[RB_]

Why everything is always a "success"

The culture of banking IT management is it's always a "success" even though it patently isn't. This has everything to do with protecting their own backsides and managing their career - nothing to do with executing the "right thing" that is in the best interests of the bank. This permeates down to the lowest levels and even team leads are not immune.

If the FCA is serious about banking stability they need to some rules to codify how IT management receive bonuses - measurable things, such as financial impact caused by change you are responsible, downtime & outages, number of impacted customers, number of trades that failed to book or settle etc.

OTOH that would require meaningful and measurable objetives tied to your compensation.. which doesn't happen in banking IT and that's why we get treated like dirt (better paid dirt that some but banks are no longer premier players on the pay front). Can you imagine a salesperson that didn't have this ? The day a bank wants to admit that you've performed better than "effective" is when you throw your resignation in and they beg you to stay.

[Doctor Syntax]

Re: Why everything is always a "success"

"some rules to codify how IT management receive bonuses"

And that's all the management except those in the investment casino side of the business.

[Anonymous Coward]

Re: Why everything is always a "success"

When I worked in banking, this was my favourite song

[Anonymous Coward]

Migrations...unsung heroes and consequences

Every migration I have ever worked on has been seen as just get it done. It comes as no surprise to me that TSB failed so spectacularly. Leadership thinks they are in charge, surround themselves with middle-tier Managers who performance review the shit out of the technical operators, who are then culled down to a bare minimum (whilst being TUPEd in and out multiple times). This is the operations layer’s way of saying fuck you to the leadership and is in fact a direct consequence from a failure of leadership.

I tip my hat for all the migration engineers out here that have migrated all data over as planned and nobody noticed. Thank you - you deserve a medal but alas Leadership will see that it was easy and probably try to replace you anyways.

Anon obviously but if NT4E means anything to you then you might know me!

[ecofeco]

What went wrong?

IBM. Bank on it.

[Tomato Krill]

Re: What went wrong?

Could be the first example of someone getting fired for buying IBM?

[ricardian]

I got an email from my bank (Santander) at 9pm on Friday 27 April to say that the new debit card that I had ordered was on its way. I was suspicious as I'd not ordered a new debit card. The email was addressed to me personally but the 4 digit number that was supposed to be the last 4 digits of my debit card was incorrect. I phoned Santander and after a lengthy wait spoke to a fairly helpful chap in the UK. He explained that my old debit card had been compromised and they were sending a new one; as to the last 4 digits quoted on the email - they were the last 4 digits of the new debit card which was in the post. He said (in so many words) that the email was a bit of boilerplate that they could not change.

It was only the next day that I realised that I'd been in Kirkwall on Tuesday 24 April and, needing some cash, had used the nearest ATM. Alas, it was the TSB and my card was rejected so I went to the next ATM and was successful. Presumably just using the TSB's ATM and getting my card rejected was enough to get my debit card considered as compromised.

The new card arrived on Tuesday 1 May and has worked OK

[FuzzyWuzzys]

Testing? Ha ha!

"The boss of British bank TSB has insisted it carried out rigorous testing ahead of a systems migration that saw thousands without access to their accounts for over a week..."

I think from bitter experience we all have, we know exactly how that went....

CIO to QA testing manager:

"Sorry we brought you into the migration project so late, we just ran out of time. If you can get some testing done, you have a couple of days left at the end before the drop-date for the migration."

QA and Testing manager to developer leads:

"F**king typical, I'm always last to know! You devs, when you get this thing up and running in UAT, get some users to give it the once over, eh?"

Devs:

"F**king typical, we're always the last to know! UAT testing users, when you get given this thing, can you log into the test system and check it works. Don't go nuts, we haven't got time."

Users get told 3pm Friday afternoon before the drop-date:

"Yeah we logged in, John ran a report and Fred said he could login OK. Yeah, we'll sign it off. Pub anyone for a quick jar before the weekend?"

[Anonymous Coward]

On the bright side...

They've announced that they're going to waive all interest charges for March and April. As someone who has a very sizeable mortgage with them, the news that they're going to charge no interest for two months is very welcome indeed!

[Anonymous Coward]

Middleware / testing

I've have seen (in passing) a particular vendor's name mentioned here - the one who supposedly provided middleware for the TSB project. I have no way of knowing whether or not this information is correct. However, I do know that ten or so years ago I worked on a project with said vendor, in which they turned over supposedly "near production ready" code, with just a need for final testing and approval. But no sooner had I begun my own tests with that code than it fell over hard, and further investigation showed that the vendor had almost completely fabricated the results of their own internal testing of the code! When confronted on the matter the vendor then claimed that the software was actually nowhere near production ready, and that it in fact it was just a throwaway, proof-of-concept. (This is not what our agreement with them stated that they would provide.) Which even if true means that it also failed miserably as a P.O.C.!

After much work on my part I finally got the code working pretty much as intended. However, by this time management had lost faith in the software and was also rapidly losing faith in the vendor, so they didn't let even the newly-working code anywhere near production. This was probably a wise move on their part.

[The Oncoming Scorn]

Re: Middleware / testing

So it finally got put into use & deployed after all this time?

[Anonymous Coward]

Re: Middleware / testing

"So it finally got put into use & deployed after all this time?"

No, that particular piece of software got permanently shelved. Its functionality was eventually built out and implemented during a different project, though, using a different vendor.

The main reason for failure here was that one of the guilty vendor's core middleware software products (which my project was set to utilize) was itself full of bugs, so they did the first pass at project implementation themselves and had to rig the test results. If they pulled a similar stunt at TSB then no wonder there was a meltdown.

[Anonymous Coward]

Re: Middleware / testing

Wow, that means you had management that knew or at least understood stuff. That is an increasingly rare commodity.

[Anonymous Coward]

Re: Middleware / testing

"Wow, that means you had management that knew or at least understood stuff."

Well, even though my manager traveled a lot I kept him informed of my progress, such as it was. As far as understanding, he kept saying "I just don't understand this; these folks used to do really good work!" He had used them before in another life, apparently, back when they were still tops in their field. But after realizing that they had since undergone a change in ownership/management, plus a move from onshore to offshore development and support, all became clear to him.

[greenwood-IT]

Customers at fault?

Yep, ALL of my projects work perfectly and as intended... if you remove the middleware and stop those damn pesky customers from accessing it :-o

[TrumpSlurp the Troll]

Tested under fire?

Does this mean that when it is finally working that it has undergone far more rigorous testing than the latest iterations of the legacy systems?

Kind of like watching a depressing NCAP test when all that is left is broken pieces, then realising that at least you know the passenger compartment survives intact. Unlike your classic vehicle which was too old to have ever undergone a similar test.

[Anonymous Coward]

Not even a large migration....

Pester even claims it was the biggest migration in UK banking. It represents a fraction of the volume of customers & transactions involved when HBOS was migrated to the LTSB platform (seemlessly), I don't think that migration even made it into the mainstream press as it was not considered a newsworthy event.

[Tippex]

I think they should "roll back". I cannot see how they are going to ever prove the data integrity and access security of these systems?

[fajensen]

I cannot see how they are going to ever prove the data integrity and access security of these systems?

And ... Why would Lloyds want to be restoring and auditing TSB-transaction data for a year or two?

[Tippex]

I don't see why Lloyds would be doing this surely the data and access was already owned by TSB. Anyway its just a sugg.. there must be a point at which the data can be trusted then a few weeks of transactions to run into the assured data and accounts.

[Tippex]

Advice

To TSB: Take my advice and roll back.

[toffer99]

Bonus

Bonus? BONUS???? I assume he's paying back last year's bonus.

[maret77]

I've led massive integration tests for major banks and NOT ONE of them have paid proper attention to building a proper TEST infrastructure prior to implementation. Take a look at all of the IT execs at a bank (or pretty much anywhere) and I dare you to find one that has a clue about what's required to invest in proper testing (including, uh-hem, a reasonable test environment).

[Roger Ramjet]

Pot, meet manure....

What an utter utter crock of shite. I logged in tonight (slowly) and still can do bugger all.

I was then kicked up the arse with a 502 error...

Any recommendations for a stable banking system?

[Wobbly World]

Disappointing vapourware integration...

It was rumoured that Paul Flowers (banker & "crystal methodist" ) dealer had similar problems after the adverse publicity and was unable to deal with the number of customers that wanted to access his services but he was only selling vapour ware ;~/

Not that I'm saying the IT team are on drugs though it may seen at times like they are. It's more simple than that, the problems at TSB are down to the Severn "P's" ie. Piss Poor Planing Produces Piss Poor Performance...

[Jack_Rainbow]

Would the threat of punishment work as a means of making pester and fester get their sums right? If so, what form should such punishment take?