Old, complex code could cause another UK banking TITSUP – study

Another major banking outage similar to the RBS disaster back in 2012 is likely to happen again in the UK, given the amount of legacy code in the sector, according to research. The average mission critical banking application has around 600,000 lines of code, according to a study by software firm CAST which analysed systems …

[Anonymous Coward]

Deja vu

I could have sworn I read about some remarkably similar "research" by the Institute for Studies several weeks ago - oh look, I did.

[Warm Braw]

Re: Deja vu

Interestingly, the CAST Press Release that led to the earlier El Reg article was entitled:

Revealed: Why UK Banks Crash All The Time [dateline March 23, 2016, London]

It was followed by another press release citing the same report [dateline March 24, 2016, New York] and entitled:

Study Reveals U.S. IT Systems Unfit for Digital Future

So it could all perhaps be summed up as:

Vendor alleges crisis in $MARKET can be solved by giving them $CURRENCY

[BebopWeBop]

Well spotted - and the comments on their 'methodology' will all roll back in...

[Anonymous Coward]

Statistics ...

For some reason I remember the story of an army officer (pre 1918, possibly) who did some number crunching and "predicted" that so many men would end up being castrated by being kicked by horses in the coming year.

It wasn't magic - just a realisation that the event happened with a frequency greater than 1/year.

It *may* have been the Belgian army.

[kmac499]

Re: Statistics ...

There was a study of Prussian cavalry, this led to the Poisson distribution. Which I believe is used quite a bit in retail palnning apps. A fairly predictable rate of low occurrence events with occasional high outliers..

[Anonymous Coward]

Re: Statistics ...

"There was a study of Prussian cavalry, this led to the Poisson distribution"

What?! You mean being kicked in the nads by a fish!?!? - instructions unclear.

[Anonymous Coward]

Only devops can save us now!

[Steve Davies 3]

Oh no not the 'devops' inquisition

Please no.

[Dan 55]

"Even if something has been written in Java in 90s that is still 20 years ago."

So what if it's documented, tested, and maintained. Java is a current language. Are they supposed to throw it out and go with Rust, Go, or Ruby on Rails?

[Brewster's Angle Grinder]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

As any fule kno: software documentation wasn't invented until 2003 so the code will be undocumented.

[Anonymous Coward]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

I am struggling to understand why Old Software is Bad Software. Software doesn't perish or rust like hardware, nor do the component parts wear out with use.

What this actually means that it has 20 years of being used in a real environment, 20 years of bugs being identified and fixed. Not like the new, shiny, barely tested stuff.

[Electron Shepherd]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

True, but in a lot of industries, especially finance, the regulatory environment changes, and that makes the software out of date. Effectively, the software gains bugs by not changing, and so failing to keep up with the world in which it operates.

[Skoorb]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

Yes, Things Change.

A brilliant example of this is RBS/NatWest Telephone, Online and Mobile banking.

A telephone banking system was developed to hook into the main banking system. This telephone banking system is separate to the main branch system, logging all transactions against a four digit call number, before passing them over to the main banking system.

Becauase parts of the main branch banking system dates back to the 60s, it was not possible to actually integrate the telephone banking system fully; it has to sit separately and just pass transaction messages back and forth as if it were a special type of branch terminal.

The online banking system was developed to hang off the end of the telephone banking system, so all transactions are passed to the telephone banking system and treated as coming from a telephone "call", before being transferred to the main branch system with a call number.

The mobile banking system hangs off the end of both the online and telephone banking systems, but cannot generate proper call numbers, so simply logs everything against telephone call 0, then leaves the telephone banking system to pass the transaction over to the branch system.

So, a right mess there then, with lots of potential to go wrong. No decent software engineer would even consider designing a system that way if implemented today - the potential for failures and security flaws is just too high.

You will also notice how everything is predicated on having a telephone banking login, and how your telephone banking details also double as your online banking log in details, with no easy way to separate the two.

[Doctor Syntax]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

"that makes the software out of date"

No, it makes it require maintenance. Development is the process by which software is launched into maintenance. It usually spends most of its life there so it's no excuse for assigning the least competent staff to the job. Neither is it an excuse for relying on maintenance to do all the bug-fixing that should have been done during development (did someone say continuous release?) so documentation and testing are equally important in both phases.

[Anonymous Coward]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

Out of date numbers cause software bugs? You should write for Dilbert.

[glen waverley]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

Smooth Newt "Software doesn't perish or rust like hardware, nor do the component parts wear out with use."

I used to work in a place where we used to get a phenomenon called "bit rot". Stuff would stop working for no apparent reason.

[Anonymous Coward]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

The Tao of programming states:

5.1

A well-used door needs no oil on its hinges.

A swift-flowing stream does not grow stagnant.

A deer blends perfectly into the forest colors.

Software rots if not used.

These are great mysteries.

[Anonymous Coward]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

I used to work in a place where we used to get a phenomenon called "bit rot". Stuff would stop working for no apparent reason.

Bit rot is the phenomenon in which code which is written with certain assumptions about the environment in which it is used, breaks, because the environment changes in subtle ways that cause those assumptions to no longer hold true.

e.g. you might have a LSB (systemv) init script that takes arguments after the "start"/"stop" argument to allow you to start/stop subservices. Then systemd comes along and wraps your LSB init script up in a wrapper that ignores and strips these arguments: causing bit rot that breaks the feature.

I've spent the last few days battling bit rot in the Xaraya CMS. Ideally, I'd rip out Xaraya and replace it with something else, but that would require locating the original design docs (most of which were in the head of a now deceased colleague).

[allthecoolshortnamesweretaken]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

"I am struggling to understand why Old Software is Bad Software."

Software a consultant can't make money from is bad software.

[Daniel von Asmuth]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

Indeed. What would happen if the banks start replacing old complex code with fashionable ultra-complex code? The one thing you can be sure of is that the performance will decline (espciallty with Java code).

[PNGuinn]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

"As any fule kno: software documentation wasn't invented until 2003 so the code will be undocumented."

That's news to me.

I naively thought that that patent was years away yet.....

All together now:

"Does - your - sourcecode - loose it's comments on the bedpost overnightttt?"

[Version 1.0]

Re: "undocumented code"

That would be COBOL I assume?

[Doctor Syntax]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

"Does - your - sourcecode - loose it's comments on the bedpost overnightttt?"

Brilliant.

[P. Lee]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

>"Does - your - sourcecode - loose it's comments on the bedpost overnightttt?"

Yes, but the bedpost is strong enough to withstand them.

[a_yank_lurker]

Re: "Even if something has been written in Java in 90s that is still 20 years ago."

If they it was Cobol or something as ancient they I would say they definitely have something. Java is still active but I could see some PHB refusing the money to update some ~2000 Java code which is using some deprecated features that runs erratically at best on Java 8.

[Doctor Syntax]

"However, in the UK mission-critical banking apps have between 800,000-900,000 lines of code."

So that's the apps. What about the application programs?

[Anonymous Coward]

Pah

"However, in the UK mission-critical banking apps have between 800,000-900,000 lines of code."

If that is a problem, then just remove the line breaks. As a Perl programmer, I do that all the time.

[Doctor Syntax]

Re: Pah

As a Perl programmer did you actually put them in in the first place.

[Anonymous Coward]

Re: Pah

As a Perl programmer did you actually put them in in the first place.

Actually, you are right, I didn't. If it can't be done in one line, no matter how long, then it isn't worth doing.

[JeffyPoooh]

LoC: "...just remove the line breaks..."

Back in the Tandy Colour Computer days (circa 1981), there were 'One Line (of BASIC code) Contests'. About 240-some characters to work with.

Many examples were outstanding and/or hilarious. Tiny video games. Utilities. An Adventure game engine (additional DATA statements holding the content).

[P. Lee]

Re: Pah

>>"However, in the UK mission-critical banking apps have between 800,000-900,000 lines of code."

>If that is a problem, then just remove the line breaks. As a Perl programmer, I do that all the time.

Perl programmers are the reason the code has a variable number of lines.

[yoganmahew]

When did measuring in lines of code become a thing again? Are we back in the 1970s all of a sudden?

Anyway, more lines of code=better structured code=easier to maintain.

Well done UK banking ;)

[Anonymous Coward]

And then think of all the crap in the underlying OS too. For instance Microsoft have a rule that no line of code in Windows can ever be deleted in case some 1980s game still relies on it.

[Skoorb]

With most banking systems that would be an IBM Mainframe or two running z/OS and z/TPF etc OSs, with something like Computer Associates' (remember them? ) CA-7 actually triggering all the batch jobs.

Mainframes and their OSs are a little bit different from pretty much any other system you will ever touch.

[Rich 11]

no line of code in Windows can ever be deleted

I would be deeply unhappy if my copy of Alpha Centauri ever stopped working. I might have to ask for my money back.

[WibbleMe]

I am a coder, its the quality of the code not the length, if its there then it has purpose

[Warm Braw]

if its there then it has purpose

In the case of very-long-lived commercial applications, it's often the case that "if it's there, it may have had a purpose at some indeterminate point in the lifetime of the application". Exactly what purpose that was, and whether the code path might be required for some random future transaction is not always something that is clear.

Given banks' reliance on contractors and their enthusiasm for merging, splitting and outsourcing, it's quite surprising that any of it works at all.

[Doctor Syntax]

"Exactly what purpose that was, and whether the code path might be required for some random future transaction is not always something that is clear."

It should be if the code was properly documented and the best place to do that is in the code itself, not in some separate document that's either been lost or never updated since day one.

[JeffyPoooh]

"I am a coder..."

Of course you are.

You've made nine errors in one line.

"I am a coder, its the quality of the code not the length, if its there then it has purpose"

I am a coder. It's the quality of the code, not the length. If it's there, then it has purpose.

[BurnT'offering]

Re: "I am a coder..."

A coder might well have written that sentence. A programmer, on the other hand ...

[Anonymous Coward]

Re: "I am a coder..."

It's not his fault English doesn't have a compiler to catch errors. And it's such a weakly typed language.

[cloth]

Re: "I am a coder..."

I did once argue that in order to be a good coder you had to have a good command of grammatical structures and ability to communicate well. I don't know many story telling 'coders' or 'programmers' - but I do know an awful lot of introverted techies who are always looking for something more interesting than 'making sure that thing runs well and is maintainable'.....Sigh.

[Doctor Syntax]

"I am a coder, its the quality of the code not the length, if its there then it has purpose"

And very often that purpose is to deal with Stuff That Should Never Happen when it happens. The more you've got of that the better.

[Velv]

"... its the quality of the code not the length..."

Quality is a subjective measure. Heavily documented code that's clear and easy to maintain could be quality to one situation where tight efficient complex code is quality in a different situation.

[David Roberts]

Distant memories

Once in my youth I was a COBOL programmer and I still remember the heady excitement of a week writing all the active code.

Followed by a month or so writing all the exception routines which made up over 90% of the average programme and were probably never used in production.

So sheer volume is no real measure of how good or reliable the code is. Although the more lines of code there are the higher the odds of an undetected flaw.

[Doctor Syntax]

Re: Distant memories

"all the exception routines which made up over 90% of the average programme and were probably never used in production."

Like all safety devices, you hope they're never needed but when they are they're really needed.

"So sheer volume is no real measure of how good or reliable the code is. Although the more lines of code there are the higher the odds of an undetected flaw."

You're arguing against yourself here. All those exception handlers are there so that you can detect flaws. It's lot's of lines that don't do checking that are the problem.

[Doctor Syntax]

Re: Distant memories

" It's lot's of lines that don't do checking that are the problem."

Damned apostrophes, breeding when you're not watching. Must be bit rot.

[Anonymous Coward]

Re: Distant memories

Damned apostrophes, breeding when you're not watching. Must be bit rot.

You're going to blame the machine for that apostrophe atrocity when the problem likely exists between the keyboard and chair?

[AndrewDu]

Software Engineering company bewails lack of employment opportunities for Software Engineers in the banking industry.

Surprise!

[Alister]

Has anyone considered that the extra 300Klocs in UK code might be more comprehensive error checking and handling? No?

It seems a real stretch to me to be able to state that more klocs = more errors without any terms of reference.

[JeffyPoooh]

"...more comprehensive error checking and handling?"

Auditing.

Banking loves auditing.

[BurnT'offering]

Re: "...more comprehensive error checking and handling?"

Also, FCA Compliance

[kmac499]

Re: "...more comprehensive error checking and handling?"

Yeah banks love audits, Never happier than when they found that 10p that fell down the back of the cash drawer in branch.

But when it comes to auditing code this usually means a tick box exercise. Did we have a spec (yes)

Did we do some testing (yes) Did we sit around risk assesing the changes (Yes). Ok go live

What I have neve seen is some form of single code vault akin to the banks money vault, where software is received, stamped, filed and held. This should be the one and only place that code can be retrieved from and put into production or formally signed out for future mods.

[JeffyPoooh]

"...a 10-15 per cent charge on that project..."

If that's true, then the amazing conclusion is that some software projects are actually getting finished on budget, thus protecting this "10-15 per cent" reserve. Traditionally, the "last 10%" (suc) of the code requires the other 200% of the budget.

This improvement is amazing news.

Humanity, perhaps via the UN, should celebrate. Maybe a cake.

[Doctor Syntax]

Re: "...a 10-15 per cent charge on that project..."

'Traditionally, the "last 10%" (suc) (sic) of the code requires the other 200% of the budget.'

[Mage]

Outsourcing

Losing core competence due to outsourcing, so called "cloud" or simply off shore developers is stupidity.

These business need to be just as expert at software development as at trading and marketing or else ultimately they are not actually really banks or other financial institutions, but just resellers.

[jake]

"Lev Lesokhin, CAST's senior vice president of strategy and analytics, said: "In consumer banks, there are core components been there for a long time. Even if something has been written in Java in 90s that is still 20 years ago."

Earth to Lev Lesokhin, Java was never secure. Ever. Really. And never used in truly secure systems.

Thankfully, the core systems in the world's banking systems are Fortran & COBOL, neither of which the skiddies are capable of breaking into.

Well, to be truthful, COBOL & Fortran aren't exactly secure as a stand-alone product, but the skiddies have absolutely zero clue as to how they work ...

[Jimbo 6]

one-third above world average

Personally, I believe that 900K is *one-half* above 600k.

[allthecoolshortnamesweretaken]

Re: one-third above world average

Either way, big stack of punch cards...

[justanothermainframer]

The shock of the new

And of course 800,000 lines of NEW code will be INSTANTLY so much more reliable.

[allthecoolshortnamesweretaken]

Re: The shock of the new

Of course, they will be flawless! They will be written using DevOps!

*ducks*

[Captain DaFt]

Re: The shock of the new

"Of course, they will be flawless! They will be written using DevOps!

*ducks*"

Aaaand, after reading your comment, several corporations started outsourcing their code development to aquatic avians as the latest DevOps strategy.

[Caff]

RBS

Bit misleading to mention the RBS outage expect possibly to highlight scale.

The outage would have happened regardless of the age or complexity of the code. It was an operational issue caused by poor lines of reporting and badly designed escalation proceedures in the outsourcing contract.

[drunk.smile]

Well it seems the cure wanted is to replace all the Java code.

I presume the report authors want Python to be the replacement, after all the lack of static typing will present no potential for security problems as the code grows to thousands of lines.

[Anonymous Coward]

The real problem is Britain invaded India...

... and made people there speak English also. In turn it made for English-speaking countries easier to offshore software development to India today.

That's the real reason why English banks code is crappier than continental ones, where, for pure language reasons, offshoring to India or the like is less appealing.

There are good chances code has been developed locally, also software developers salaries may be lower in many European countries, making offshoring somewhat less advantageous.

[Anonymous Coward]

RBS banking outage caused by legacy code?

Except the RBS outage wasn't caused by legacy code but by a failed upgrade to their overnight batch processing system CA-7 and RBS firing most of their experienced UK staff and outsourcing the job to an intern in India.

[JulieM]

Old code that doesn't get run

There's probably a whole bunch of code in there that doesn't do anything, in the form of functions that are declared but never called. Especially if anyone wrote routines to deal with Imperial (240 pence) pounds -- which they might well have done, just in case someone was ever crazy enough to try to revert back from the decimal (100 pence) pound that was already well-established before computerisation.

[Zippy_UK]

DON'T WORRY - RBS HAS IT COVERED

"Because of this problem, RBS executive management has acted quickly and efficiently to solve this problem by 1. Offshoring to cheap, inexperienced Indian IT operatives with a quality industry experience of less than 2 years 2. Cut back on all infrastructure spending and maintenance to bare minimum to re-invest in critical areas 3. Invested a whooping 50% of the money saved in point 2 in quality providers such as TCS and Gap Gemini so we can get the future delivered in the future where its needed. 4. Invested key priority areas such as restarting dividends, so share buy back programs can be re-started. 5. Paid extra money (including share options schemes, see point 4) to invest in the talent at the top (of our organisation)

to ensure it is retained"

DON'T WORRY CONSUMER, YOU MONEY IS SAFE WITH US.