IT outages in the financial sector: Legacy banks playing tech catch-up risk more outages, UK MPs told Banks with mountains of legacy tech risk causing more outages as they race to catch up with their "agile" competitors, the Treasury Select Committee was told. Speaking at the hearing on IT outages in the financial sector yesterday, deputy chief executive of the Prudential Regulation Authority, Lyndon Nelson said innovations in …
The correct answer to the original question is : zero. During my bank IT days, new code took weeks, if not months to go into production, after at least 3 levels of testing (the last one before release being a complete mirror of the production environment). Strangely enough, there were never any disasters or outages when the code went live.
Are they wondering why that's started happening since they moved to the very frequent release "agile" model?
I do not really think there is a shortage of COBOL programmers. Rather a shortage of COBOL jobs jobserve list 4 COBOL contract vacancies and 6 permanent positions (only 1 contract offered decent money!).
Compare this with another legacy language like 'C#' with over a thousand job offers most reasonably well paid. Or BASIC with 550 vacancies.
I thinkthe banks are just making up excuses to fob of the regulator.
Indeed. Claims like this one:
He noted not many programmers are left who can use COBOL.
are both wildly erroneous and completely irrelevant. Any competent programmer can learn to maintain COBOL source, and - as anyone who's dealt with real-world applications knows - so can a great many incompetent ones.
The only labor issue with COBOL source code is employers' reluctance to hire the people who already know the language (generally because they're older and want a decent wage), or to pay callow youths enough to learn it.
Stop closing branches!
I'm not going to do phone or computer banking as zero faith that if something goes wrong bank will make it my fault so financial loss.
e.g. zeroday pwns computer, just about any exploit pwns android device with slow / non existent patch role out of many manufacturers (and with grim irony, rooting phone lets you do things to make it safer, but bank apps if detect rooted won't run on rooted phones)
More relevant if I had banking app & phone was .stolen (all sensible knife point thieves make sure you unlock it) its a PITA jumping through hoops to get any dodgy transactions made on phone resolved by the bank.
If I go into a bank branch, get a nice paper receipt for the transaction, then if anything goes wrong its definitely banks fault and so no financial loss.
Most of those old mainframe systems fairly rock solid after all those years, hassles come with making non essential changes (as article implied, not that many people with the skills / knowledge, especially as a lot of the banks decided to start sacking their experienced IT folk)
"If I go into a bank branch, get a nice paper receipt for the transaction, then if anything goes wrong its definitely banks fault and so no financial loss."
I suspect the bank can afford more lawyers who will argue that your little bit of paper is not legally valid than you can afford lawyers to argue the opposite.
"Members of the public would probably be alarmed to learn that some of their financial institutions are running on systems that are possibly 50 years old.. and often are not well understood by the people working with them. "
Only one of these 2 things is alarming. I rarely find myself thinking "Oh no, this thing has been working without causing problems for decades"
and how do people gain experience and expertise in solving these already solved problems?
Compare, if you will the Saturn V and the SLS.
Not having all those corner cases and regulatory requirements documented is a failure of a) the first team b) manglement for not insisting on it. and c) subsequent teams for not keeping the docs up to date.
Not *following* said docs, now *that* is the fault of the newer team. If the docs don't exist, then we just have to learn the hard way.
@Glen 1
In my experience (over more than a decade and a half in finance industry) it was records management failures that gave rise to documentation voids. It was documented, management insisted on it, subsequent teams kept it up to date. It reaches a point where no further change is required. Then, over time, just like the Saturn V, the documentation gets lost. This is generally tied to internal structural reorganizations.
"If the docs don't exist, then we just have to learn the hard way." If you think having the working COBOL source as a starting point is 'the hard way' you have a bit to learn yet. If the source code has also been lost you are about to learn why those who came before you resisted the urge to change this code every other day in order to 'surprise and delight' the customer.
Yes, I remember moving from a bank to a telco account many years ago and having explained to me that the payments system didn't have documentation because IT management a) insisted that the 3rd party development team itemise all costs in their statement of work b) struck out the expensive bits that didn't add customer value, like documentation.
Regulators have (rightly in some cases) been trying to one-up each other continuously since the millennium, and the consequences for global banks have been enormous. The ideal from an operational perspective would be some sort of global standard for tech that mandates a minimum base level of security, but that's never going to happen - it would be out of date as soon as it was written and impossible to keep up to date.
Regulations by their nature are reactive and typically only change in response to an issue, consequently the organisations affected by them are always playing catch-up. All of this has significant impact even before any commercial demands are placed on the bank - incorporate the challenge of trying to keep up with startups not bogged down by legacy technology and less burdened by the regulatory regime, and you end up with a workforce whose job is a constant never-ending race to balance regulatory compliance, customer satisfaction and an effective return to shareholders, and rarely managing to achieve all three to high standards.
The legacy issue is nonsense. Yes, banks do run their core systems on old mainframe hardware with the systems written in COBOL. Those systems successfully shuffle your dollars around flawlessly and have been doing so for the last 30 or more years. There is no need to change those basic functions. All the rapid change nonsense is happening at the front-end: From ATMs, to telephone banking, to internet banking to mobile banking. Rapid but minor-tweaks to payment services and mobile banking all of which call those back-end services to actually move your money around. Somehow, it has become fashionable to keep changing these front-end systems as fast as possible and quality has been sacrificed (as if that should be a surprise). It is not a legacy, back-end systems problem. Finally, you want to replace those systems? Are you raving nuts? You are "betting the bank" by touching these. What exactly is your business case? Is it: please give me $200 million dollars and in 4 years you will have exactly the same functionality you have now, but on a newer system? There will be huge bumps in the road and you will end up ridiculed in various papers and investigated by your regulator for all the client impact, but hey, it must be worth it to have exactly the same capability you had before on sexy new hardware and software. Did I mention these sorts of projects can often go 300% over budget and arrive years late? Only a systems outsourcing vendor would think such a project was a great idea. Raving nuts indeed!
"The Parliamentary inquiry into IT failures in the financial services sector was launched last year after the meltdown at TSB that lasted almost a week in April 2018."
Exactly, ironically the migration at TSB from a proven platform to a new platform was the cause of the meltdown. And now they seem to be suggesting that other banks should do the same.
Proven technology may not sound cool if someone chooses to call it legacy to discredit it. But it's generally rock solid, well maintained and often provides more features than the latest/upcoming technology.
They run it on current mainframe hardware such as the z14: https://www.theregister.co.uk/2017/07/17/ibm_latest_mainframe_z14/
However as stated the old code continues to run. Not having to reprogram when hardware is upgraded was a foundational element of System/360.
"David Bailey, executive director of Financial Market Infrastructure at the Bank of England, said the body has suggested banks provide a full list of their critical services and which specific IT systems are required to support them."
No shit Sherlock?
“Lyndon Nelson said innovations in the sector are driving change.”
Just how difficult can it be to add-up columns of figures?
'how many times in a week can we change an app without it falling over?'
The answer is never update a live system. Test your ‘innovations’ on a second system before going live.
“Alison Barker, director of specialist supervision at the Financial Conduct Authority, said 65 per cent of outages are in retail banks”
The problem is in the nature of the ‘innovation’, as in it consists of a mainframe database connected to a web server using a scripting engine to glue it all together. All this so as you can do your banking on a browser.
How many of the shiny new fintech darling are running customer transactions processing at anything like the total volume and duty cycle that the major legacy banks have been doing for years? Not many, if any...
Give the millennial proles a bright app, some coffee, mashed avocado on toast and a social media platform, and they won't care what is running the back end.
Members of the public would probably be alarmed to learn that some of their financial institutions are running on systems that are possibly 50 years old
Perhaps they'd be even more alarmed to learn that some of the buildings that house their financial institutions are even older!
Honestly, what an astoundingly stupid thing to say.
Certainly there are problems with legacy IT systems, at financial institutions and elsewhere. Often organizations have exercised poor software hygiene and housekeeping over the decades, and have only a vague understanding of what binaries they're running, what sources correspond to those binaries, how those sources implement business rules, and so forth. But that's a solvable problem. There are software packages to assist in source and binary analysis and inventory, and firms that offer consulting services to help clean up the mess; and an organization that wants to make the effort can do it in-house, too.
The use of COBOL, PL/I, assembler, and other out-of-favor languages isn't the problem. Neither is the use of z and other mainframe architectures and their OSes. The problem is financial institutions unwilling to invest areas that need investment, and grasping at the new shiny in the hope of avoiding paying the price. And ignorant ninnies like some of the people quoted in the article are contributing to that problem.
because if they screw up, it doesn't cause billions in loss, and barely anyone hears about it. They can push out change after change, fix on top of fix constantly.
If you are big bank, billions can be lost in no time due to the smallest error, and everyone will know. Stability, accuracy, and robustness, over quick change any time.
I'm reminded of a piece i worked on many years ago, an actuary came in with a hugely complex, recursive, formula for working out a critical (legally important) value. When testing the output, it was compared to an Excel spreadsheet someone in the test team knocked up in an afternoon, and the differences were flagged up as failures of the software. We had to then school the test team on how the whole point of doing this work, was that a spreadsheet couldn't work it out accurately. floating point approximation threw off the result enough to get us fined huge sums.
I don't think you could code it in anything but COBOL, with some serious computing power behind it, to make it work at all. Even C would probably have you delving into assembler to do it.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
