RBS Mainframe Meltdown: A year on, the fallout is still coming

A year ago, RBS experienced its Chernobyl moment – an incident when a case of simple human error by those running critical systems resulted in a crisis. IT staff badly botched routine maintenance of the IBM mainframe handling millions of customers' accounts – a system processing 20 million transactions a day. The mistake was …

[Annihilator]

Outsourcing <> offshoring

Straight away the article insinuates it's "outsourcing". Yet again, the need to point out that outsourcing doesn't mean offshoring.

As I recall, RBS employed their batch support staff directly. Regardless of location, this isn't outsourcing.

[Velv]

Re: Outsourcing <> offshoring

Whatever you call it, getting rid of 7,500 man years of experience is never going to end well.

[Mad Mike]

Re: Outsourcing <> offshoring

From my contacts at RBS, the cause of this issue has nothing to do with the hardware or software. It has everything to do with money saying management decisions. They simply don't seem to understand the following:-

1. Loosing tens of thousands of man years worth of experience on the job is not a good idea.

2. Distance within teams causes problems proportionate to the distance. So, Edinburgh to India is a big problem.

3. When you have different cultures involved, it causes additional difficulties, especially when one is using a second language and (for understandable reasons) don't necessarily understand the nuances etc. if you think of how often misunderstandings occur between people working in the same office, with the same language etc.etc. and then translate this to different cultures, you start to understand the issue.

4. RBS IT had (especially at the higher levels) a grossly overinflated opinion of itself.

5. When RBS took over NatWest, it seems to have kept the worst of NatWest culture.

6. Don't underestimate how much the ex-NatWest staff hated RBS and their staff, partially due to point 4.

All of the above is never going to result in a situation that ends well. RBS is effectively wasting £450million as they simply aren't dealing with the real issue.

[Anonymous Coward]

Re: Outsourcing <> offshoring

Well I for one as a UK tax payer and by implication an indirect share owner applaud RBS senior managements cost saving initiatives be they either outsourcing or offshorig.

Mind I would also add that this statement is a lot more comfortable as an 'ex customer' of RBS............

[Mad Mike]

Re: Outsourcing <> offshoring

"Well I for one as a UK tax payer and by implication an indirect share owner applaud RBS senior managements cost saving initiatives be they either outsourcing or offshorig.

Mind I would also add that this statement is a lot more comfortable as an 'ex customer' of RBS............"

Don't worry, all the major banks and organisations are doing very similar things. Your time will come.........

[Anonymous Coward]

Re: Outsourcing <> offshoring

@Titus: So as a UK tax payer, are you happy for RBS to pay the redundancy to those highly paid techies, the first to go always being the most skilled, longest serving and most highly paid. Are you happy for them to give up work for a few months while they work out what to do? Are you happy for all those ees and ers NICs to go unpaid and for all that higher rate tax to go uncollected?

[Anonymous Coward]

This wasn't a hardware flaw. This wasn't a mainframe flaw.

This was a botched CA7 upgrade that caused chaos during the back-out of said upgrade.

The fact that it was on a Mainframe means that the system was more important, but the mainframe itself performed the actions it was required to flawlessly.

The actions it was required to perform were wrong, and that was the fault of the people operating the system.

[Gordon 10]

Re: This wasn't a hardware flaw. This wasn't a mainframe flaw.

Correct me if Im wrong - but not even IBM mainframes cost £450m its entirely possible that this budget includes new teams to run the stuff.

(Its also possible that they have just spunked £450m just to IBM - but Im a glass half full type person).

[Anonymous Coward]

Re: This wasn't a hardware flaw. This wasn't a mainframe flaw.

"but not even IBM mainframes cost £450m"

Agree, there is no way the mainframes themselves cost anywhere near that amount of money. The cost of the hardware was probably trivial. The cost of rewriting/modernizing millions of lines of COBOL probably is the bulk of these costs.

[tom dial]

Re: This wasn't a hardware flaw. This wasn't a mainframe flaw.

It appears it may not have been a programming flaw, but more a series of operations flaws: first, in committing an error upgrading CA-7, then in compounding it during the attempted rollback, with the finishing touches applied when application recovery procedures were not up to snuff, or perhaps there was not enough available capacity to handle the recovery and normal workloads concurrently. All speak to operations and capacity management, none to the applications or the geriatric implementation language. RBS management may wish to shift blame away and empower themselves to spend great bundles of money, but the facts are as they are.

[Matt Bryant]

Re: AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

"....This was a botched CA7 upgrade....." Which neatly ignores the question did the platform's complexity, inherent inflexibility and convoluted design have anything to do with the inability fo the staff to complete what would probably have been a far simpler task on UNIX, Linux or Windows.....?

I would suggest RBS's money would be better spent spending a million on a redesign using commodity servers and Linux, and it probably come in at a fraction of the cost of the ludicrous mainframe upgrade.

[GreyWolf]

Re: AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

" what would probably have been a far simpler task on UNIX, Linux or Windows.....?"

You have just revealed how little you understand of what the night batch at a bank actually entails...

[Matt Bryant]

Re: GreyWolf Re: AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

"You have just revealed how little you understand of what the night batch at a bank actually entails..." Yeah, 'cos bank systems aren't about shuffling numbers round or anything that other industries already do very well, and in some cases on a far larger scale. It's all magic and unicorn farts, right? Seriously, if you want to pretend that even the largest banks do anything unique then you're probably the muppet that shelled out for the £450m mainframe upgrade.

[Intractable Potsherd]

Re: GreyWolf AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

For the record, I'm the one agreeing with Matt. I therefore want to insert that disclaimer that I don't work for RBS, I am merely a customer that can't see that any of the banks are any different, and so stay where I am and take the free insurances offered through my account.

[Skoorb]

Re: GreyWolf AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

There is a fair point about whether a full rip and replace would have been warranted.

Whilst there does come a point at which a system becomes so unwieldy the only option is to bin it, RBS do not feel that the time has come to put themselves through that pain.

As for moving to a more 'standardized' system, Nationwide are having a shot at moving to SAP, and it seems to be sort of on track. Mostly.

Anybody think that moving to SAP would be a better bet for RBS? It seems more like simply swapping to a different kind of masochism to me...

[Anonymous Coward]

Re: GreyWolf AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

Germans and masochism? Sounds like a great party! Will there be beer?

[tom dial]

Re: AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

Mainframes have, as has Linux (or Windows) complexities. However, nothing about a mainframe environment, from an application development or management viewpoint, is inherently more complex than a single Linux/Unix/Windows server. The hardware has a MTTF measured in decades; the hardware architecture has been gradually extended and polished for about half a century to remove bottlenecks and single failure points. As has the operating system. While great herds of commodity servers surely have a place, it is unlikely that they have a complexity, managability, or reliability advantage over mainframes for large workloads. It is quite possible that they also have no overall cost advantage.

All that I have read about the RBS failure leads to the conclusion that the cause was management and operator error. As another commenter noted, 450 million GBP probably covers a great deal more than a hardware in-place replacement which would not address the underlying cause anyhow.

[Kebabbert]

Re: AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

"...While great herds of commodity servers surely have a place, it is unlikely that they have a complexity, managability, or reliability advantage over mainframes for large workloads. It is quite possible that they also have no overall cost advantage...."

Google thinks it is more cost effective and reliable to have herds of commodity servers for their services, than a few large servers. Google has designed their system so it doesnt matter if a server breaks, they just insert a new cheap server. If you have 100 cheap servers that can switch roles anytime, you will have a better uptime than a single mainframe. Distributed beats centralized both in terms of costs and reliability and performance.

But sometimes, you can not distribute. Some workloads are inherently centralized and can not be parallellized. Google has embarassingly parallell workloads, so they can just use lot of COTS servers.

For banking, Mainframes are the norm. They can be replaced by a fleet of COTS servers, but no one has done it yet, as I know of. There will be years of R&D. Maybe it is easier to just use a Mainframe.

Another note. In banking, Mainframes are the norm. They are slow, have extremely slow cpus, but good I/O. Banking is doing lot of account updates like calculate interest rate, etc. Nothing sexy. Old, boring, dusty stuff. More, accountant stuff. It doesnt matter if your account gets updated 0.1 s later or so. Latency is not important. Much work is done in batches, and can wait another hour. Mainframes have good throughput, bad latency.

In finance, you never use Mainframes. In finance, you typically use Linux/Unix. You are doing real time high performance calculations. HFT. Quant. Math. Algorithmic trading. Linux has low latency, so that is important in some fields. Mainframes are too slow to do finance as they have very weak cpus, much weaker than x86 cpus. For instance, large stock exchanges typically run Linux/Unix. No stock exchange runs on Mainframes, they have too bad latency for that.

Banking: accountant stuff, boring. Needs to be reliable and just work. No requirements for bleeding edge performance. Simple calculations.

Finance: math, high performance stuff, complex calculations. Needs to be reliable and high performance. If you are faster than any one else, you get your order filled and you can earn money.

[Anonymous Coward]

Re: AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

Slow CPU, I suggest some research, google zEC12 processor speed - that's a mainframe BTW.

[David Beck]

Re: AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

Google also doesn't care if you get the "complete" answer to your query. If a section of the data is "down" then you just don't see that in the results. How'd that work with your bank account?

[MikeC711]

Re: AC: This wasn't a hardware flaw. This wasn't a mainframe flaw.

Agreed, put them on a cloud of Windows machines, then the 57 Windows security releases in 1Q2013 could be rolled out to each one individually. Mind you, they could not be rolled out all at once as there was an order and a need to do it in at least 3 separate waves. And also, keep in mind that as you handle the workloads on these 32 and 64 core machines, that you have to pay licensing for your DB and Middleware on all those cores. Still, there are times that this makes sense, but there are countless migration nightmares we can talk about. And the migration consultants (who are eager to assist you for a price) will forget to work out costs for extra hardware for firewalls, dev/test environments, DR environments, ... etc. So as long as RBC can handle a far less reliable environment and a migration that will likely cost at least 3x what the consultants quote you ... it's all good.

[Dominion]

It ain't broke??

450 million to replace a system that is still at the mercy of incompetent staff managed by stupid managers? Do the clowns in charge not realise that throwing shiney tin at a problem will not make a blind bit of difference? Outsourcing / offshoring, it's all the same. Getting rid of competent staff and replacing them with incompetent replacements.

[Brewster's Angle Grinder]

Re: It ain't broke??

Q: "Do the clowns in charge not realise that throwing shiney tin at a problem will not make a blind bit of difference?"

A: They're bankers.

[Anonymous Coward]

Re: It ain't broke??

The Governemnt needs to bring in someone qualified from outside. A familiar face, someone who can be trusted. Terry Wogan.

[Richard Wharram]

Re: It ain't broke??

This wasn't the mainframe's fault certainly. Doesn't quite add up.

[Ken Hagan]

Re: It ain't broke??

It adds up perfectly. It's just doing a different sum from the one you wanted.

By throwing a big pot of cash at the problem, the management appear to be taking "tough decisions" to sort out the problem, thereby justifying their "compensation packages". They don't know how to actually fix it, but this should get the regulators off their backs for a while.

I expect Bruce Schneier would call it "management theater".

[Anonymous Coward]

If an IT focused business like RBS can fuck up your banking, what chances have other companies got who aren't IT focused. Like a Supermarket?

[Dominion]

All large corporations are run by bean counting bureaucrats. IT is just an expensive irritation to them. All they want to do is spend as little as possible, regardless of impact. I include all of the large IT outsourcing companies in that sweeping generalisation as well.

[Anonymous Coward 101]

"IT is just an expensive irritation to them. All they want to do is spend as little as possible, regardless of impact."

That explains why they are spending £450m on a new computer.

[Mad Mike]

"That explains why they are spending £450m on a new computer."

Both points are true. Most large organisations these days are run by accountants of one form or another. A finer example of blinkered sight is hard to find. They will spend as little as possible on everything. An accountant knows that to improve profits (all they care about), one simple answer is to cut costs. There are always loads of people who claim to be able to do that. So, they go to lowest common denominator at all times.

However, once the shit has hit the fan, an accountant (or general senior management type) cares about only one thing......saving their own necks. At this point, it's all about blaming someone or something else. Hence, blaming 'the mainframe'. It was actually their stupid decisions, but can't let that get out as it would result in them having to take responsibility for being idiots. So, you blame something else. Then, you come in with the solution which obviously has to be fixing whatever you've blamed. At this point, you look like you weren't responsible and can gain advancement through providing the fix. Cost becomes irrelevant as it's about making yourself look good, which is no longer about saving money. So, £450million for a new mainframe system........chicken feed. Where do I sign!!

Never underestimate the stupidity of senior executives and other rampantly self-centred people in the desire to advance their own careers.

P.S.

My father was an accountant!!

[vagabondo]

Re: @Mad Mike

"Most large organisations these days are run by accountants"

In the case of banks perhaps a few more accountants, and not so many salesmen/gamblers at the top would have saved the rest of us a deal of grief.

[Richard Wharram]

Re: @Mad Mike

Accountants would be no use. I'm sure their figures all added up. What they didn't have was adequate assessment of risk.

[Mad Mike]

Re: @Mad Mike

"In the case of banks perhaps a few more accountants, and not so many salesmen/gamblers at the top would have saved the rest of us a deal of grief."

The salesmen/gamblers weren't at the top. They were further down, but were employed by the accountants at the top. Problem is, the current problems aren't really the fault of the dealers etc. themselves. Governments have this idea that continuous growth is all that's important. Things simply have to keep going up...GDP for instance. This is why it's all they talk about and recession is the enemy. Problem is, continuous growth isn't actually possible or even desirable in the long run. It encourages people to borrow money that isn't fully backed by real money (cause of the crisis) and then encourages them to do so in circles amongst many banks and government coffers. This leads to the illusion of growth, but in reality, it hasn't. Effectively, you're just inventing money. Loans that are not 100% covered by real money aren't real themselves. Run this cycle for long enough and eventually someone stops paying. Then, it all comes crashing down.

It is, in effect, a giant pyramid scheme. Problem is, it's generally promoted by governments, not banks. Yes, they jump on board and make money whilst they can, but eventually it has to come down. It pretty much started in Margaret Thatchers era, but the larger part of bank deregulation (and certainly the worst parts) actually occurred during Brown/Darlings tenure. They needed more and more money to try and prove to people life was good and they should stay in power. So, they deregulated more and more, leading to lower and lower levels of money backing loans. Then, the inevitable happened. When they couldn't deregulate any more, they then went down the PFI route. Again, it's effectively inventing money that doesn't exist.

This whole issue is much wider than the bankers. Yes, they played a significant part, but politicians especially are, if anything, far more culpable.

[Anonymous Coward]

Re: @Mad Mike

" Governments have this idea that continuous growth is all that's important".

Yes, our industries (especially financial) must grow faster and make bigger profits than the unscrupulous foreigners. Unfortunately, that involves becoming even more unscrupulous than the foreigners. And before we know it, everyone is engaged in a Gadarene race to the cliff edge.

It's going to be very hard, because all our government policies and corporate strategies - even our whole view of economics - is predicated on continuous growth. It's a testament to the sheer innumeracy of our leaders... We badly need a scheme for running the world in equilibrium from year to year and century to century, instead of continually pursuing the growth that is bound to annihilate us.

[Dominion]

Is 450m enough? I genuinely have no idea how much a replacement would / should cost. If a robust replacement should cost 500m, then cutting corners and spending 450m means they are building down to a price, not up to a spec.

[Mad Mike]

The spec and building down to a price are not really relevant here. The problem is they aren't dealing with the actual problem. Therefore, it will reoccur. Doesn't matter how much you spend, it will happen again, because the core issue hasn't been resolved.

[Anonymous Coward]

Well, assuming the Parliament cost about 400M, and the RBS computer prob wouldn't fit, then maybe some kind of giganto-monolith making muffled but deep bleeping noises and lights fashing slowly on and off ought to do it? Perhaps Renzo Piano can do them one, I would suggest on Arthur's Seat or Salisbury Crags? About 1200M ought to do it, for that price you could maybe get titanium skin and some Swarovski, and get bolts for the roof beams that dont require immediate fixing? And maybe train a new set of "priests" for it :P

[Anonymous Coward]

Outsourcing

It's not about the type of organisation, it's about outsourcing, and especially about outsourcing to what might be called "warm body consultancies" where people are moved around projects (I know some very good consultancies that have domain specialization).

I worked as an on-site contractor for a company, managing their intranet and after about 6 months, making changes was 2nd nature. You learn where things are and it sticks. Something breaks, you shorten the process of fixing it by hours because you know the stack - the name of the web page, the service it calls, the table that it writes to. You aren't having to find it out each time.

When I left, it got outsourced because it was cheaper. But... what was happening is that they got new staff every couple of months. So, every change took longer because people had to work out how to make the change. Fixes took longer to resolve because they weren't like "oh, that'll be the overnight update job failing", double-check it, make a fix and kick-off the update.

That's what none of these pinheads doing outsourcing understand. They see development staff as being like checkout staff or plumbers, where you can pretty much hire one working in one place and move them to another.

Simple question: if outsourcing is such a great idea, how come Google and Amazon hardly ever do it?

[David Neil]

Re: Outsourcing

Bang on, they also assume that all of those little nuances of the systems can be captured in documentation - they can't be as things will change slight;y from change to change so you end up with a monster list of exceptions which say do x, unless it's y in which case do z unless b is....

[Ken Hagan]

Re: "monster list of exceptions which..."

...is exactly the list that elicited the comment "you actually understand all that stuff?" when a previous commenter waved it at the people who (through their management decisions) actually wrote it.

It's a bit like politicians and tax law. They've made it so complicated that they don't understand it anymore and when it starts delivering the wrong answers they blame whoever or whatever is following the rules.

[Anonymous Coward]

Re: Outsourcing

You are absolutely correct about consultancies. And that's why it's relevant that there was no outsourcing in this story (and El Reg should really bother to understand that). They were RBS staff working in a different RBS office.

They were inexperienced but this issue happens to all companies... no-one stays at a job forever. RBS should have moved some experienced staff to that office to cross-train, maybe for the first year, say. But they (apparently) didn't.

[Anonymous Coward]

Supermarkets not IT focused?

Well yes and no.

Tesco has multiple z/os mainframes in a quite nice DR configuration and quite a few big Unix boxes, 590's, 595's, I even think they kept the old 690's. A smaller number of Sun boxes and of course a gaziilion windows machines. Shed loads of cross site replicated SAB storage and at one time managed their own national network (now outsourced to BT) The IT team there was a good one, long service, good and deep business knowledge, solid applications knowledge. It needed that IT function to grow as it did and in the late 80's through to the mid 90's IT was respected. From then until around early 2000's the companies respect for IT slowly drained away.

Then the main board and bean counters decided that it would be a wonderful thing for the company ($$$$$) to offshore and all that systems and business knowledge was shown the door. Even as the quality of service provided by IT started to dive and the off shoring costs kept rising the main board still kept insisting that it was a "success". One IT director joined, looked over his domain, challenged the main board on their assumptions and was shown the door but at least he had the balls to do so.

The only reason that the Tesco CA-7 roles were not off-shored was that the compensation they were offering at the time was too cheap even for Bangalore and they couldn't get enough takers.

The bottom line is unless the IT director is on the main board as an equal then the IT department is always going to end up a ripe target for cost cutting. Sadly many directors seem to be deaf to the implications of ravaging their IT departments, could be a genetic thing....

[Anonymous Custard]

Apps?

Since when did mainframes run apps rather than programs? Makes it sound like a big smartphone running the show...

[Jemma]

Re: Apps?

iOS/360 perhaps?

This article has me a little lost.. why, if the problem was some khat stunned PFY who didn't know what he was doing is the cause of a problem do you rip out the hardware? To replace it with more of the same no less.

I could understand doing a burn and rebuild of the software as it sounds like its needed.. but hardware doesn't make sense..

More to the immediate point; how stupid are RBS going to look if they start the shiney and another Nasreen the Nerk frags it within the week?

Methinks a point has been missed here on a truly monumental scale. If the problem is user error you fix the user (now where's that quicklime..), you don't put in a faster computer so they can trash the accounts that much faster.

[John Smith 19]

Re: Apps?

" ...khat stunned PFY..." "...Nasreen the Nerk..."

You aren't by any chance based somewhere around Edinburgh?

"khat stunned PFY"

I'll be remembering that one.

[Corinne]

Re: Apps?

Apps is short for application software, i.e. the software that does the work of the business as opposed to the system software such as the OS. So they aren't running apps rather than programs, they are just different terms for the same thing.

Apps has become the trendy term for software that does stuff the user wants, and tends to be used more often for modern small computing devices because that's what most non-IT people see and use.

[Anonymous Coward]

Would they have if still private?

I wonder, if they were still a private company would they have spent this kind of money?

As theyre currently backed by the public, and theres zero expectation of profit theres really no risk in splashing money around on big ticket projects.

Once the dust has settled and its 'back' to running correctly they can then go ahead with returing the bank to private hands.. After they've used the tax payer to fund all the improvements.

Even 'better' if they leave the old mainframe in the proposed "bad bank" that stays with public while spinning off the new and improved profitable bank back to the private sector.

Or am I just getting too cynical in my middle age?

[Anonymous Coward]

Re: Would they have if still private?

You misunderstand how this works. If you're in government, you make sure you pump public money into something BEFORE you hand it over to your mates in the city. It would be unthinkable for those buying the bank at some heavily discounted price have to buy their own replacement mainframe. What'll be in it for them apart from a knighthood and a seat in the Lords?

[Anonymous Coward]

Re: Would they have if still private?

You misunderstand how this works. If you're in government, you make sure you pump public money into something BEFORE you hand it over to your mates in the city. It would be unthinkable for those buying the bank at some heavily discounted price have to buy their own replacement mainframe.

Not to mention that spending megabucks with a supplier might just end up with a nice little retirement number as a board director.

[Anonymous Coward]

"The exact reason for the problem have been found and poured over in a tedious level of detail, then process will have been put in place to prevent the set of circumstances repeating."

There was story many years ago about a new CEO of M&S. He examined their thick manuals of "process" - and after a few weeks replaced them with a much slimmer one.

He said that the previous manuals had so many processes - added with specific hindsight - that no one would be able to absorb them. Therefore they were only useful for apportioning blame after the event. His new manual dealt with principles that could easily be understood.

[Terry 6]

A bit like education.

There's a policy document or manual for every little thing, and every time something's gone pear shaped they bung in a new one. But no one can actually read all of it, let alone absorb it. And that's assuming the things are actually workable in the first place.

(BTW, "Apps?" Please don't say my bank account is being run on software from Google Play Store. Still that would explain...........)

[Brewster's Angle Grinder]

"Unlike, say, the nuclear or airline industries, where accidents have led to investigations that have produced operation and safety standards..."

And yet you're more likely to be affected by a banking crash than a plane crash or a nuclear meltdown. Admittedly, it's not quite as fatal; but several days without money is tough.

[Ken Hagan]

"several days without money is tough"

If you're counting since 2008, it's quite a bit longer than that.

[Anonymous Custard]

The Edinburgh mainframe was so old that parts of its code had been written in Assembler, a language rooted in the immediate post-war years, with dates going back to 1970.

Umm, didn't the war end in the mid 1940's?

And we haven't fought with Scotland itself since the mid 1500's, which is a little early even for Assember.

[Anonymous Coward]

Assembler/assembly

Of course, the suggestion that assembler is old-fashioned reveals immense ignorance. All computers intended to be programmed have assembly languages, which are simply symbolic forms of machine code made easy for trained human beings to program in. True, assembler was used more frequently in the early days of computing - simply because there weren't any compilers or 4GLs yet.

[Anonymous Coward]

Re: Assembler/assembly

Indeed assembler still have it's uses. I wrote some small modules to perform functions that are awkward and/or inefficent to do in Cobol.

Some examples:

1) Making 32-bit hashes of records and/or files.

2) ASCII to EBCDIC conversion where the record has binary numbers (can't convert those parts)

3) pah and other stuff I forget now.

Also did some simple VSAM access modules for TSO/ISPF panels (CLIST and REXX can't read VSAM files unless you got REXXTOOLS). Plus it makes the panels more responsive with the efficiency of the asm programs.

[Rabster]

so old that parts of its code had been written in Assembler

Wow - spectacular ignorance. A lot of user exits can only be written in Assembler. I'm primarily a java coder now developing web-based tools on WAS/unix but I spent May rewriting the TWS logon submission exit and a Rexx functional extension package to allow rexx execs to talk to MQ.

Tied in with the "apps" quote well........

[Brewster's Angle Grinder]

Re: so old that parts of its code had been written in Assembler @Rabster

"A lot of user exits can only be written in Assembler."

I think we've found the problem. I don't know what a "user exit" is (perhaps mov ax,4c00h; int 21h?) , but a system that requires you to write bits in assembler is broken. And I speak as someone who was once paid to write apps in assembly. That's just no way to run a bank.

[James O'Shea]

"And we haven't fought with Scotland itself since the mid 1500's, which is a little early even for Assember."

<cough> Culloden Moor </cough>

<https://en.wikipedia.org/wiki/Battle_of_Culloden> Note the date.

Still a trifle old for assembler, though I may have once tried to use an old GA-18 which may have been designed close to that time.

<exit, stage right, to the tune of 'Charlie, he's my darling'. and 'Scotland the brave but stupid for following a Stuart'.>

[Anonymous Coward]

Regardez-ci, c'est moi, votre vieil ami, Charly! Aidez-moi, Scottie-garcons! Aieee !

Qu'est-ce que c'est, "Lu-Ther-Ann", c'est un vrai mot?

[Anonymous Coward]

We should check with the Moderator of the General Assembler of The Church of Cobol :P

[Anonymous Coward]

Oh dear...

...oh dear. How many times have I witnessed this panic response in my life as a service management consultant. RBS - think, please think. What happened was either a process failure or the staff the were too lazy, or too stupid to follow the process correctly. Throwing money at the problem by buying new tin is likely to make the problem worse not better. Standby for another disaster before too long.

[JayBizzle]

The articale talks about a LTSB outage based on a change to old hardware... well they moved to a brand spanking new core banking system as part of integration of HBOS, this apperas to have gone very smoothly (from a customer perspective).

Im with the others who are making comments in that user and process error are main root cause faults not the kit itself... but what ever keeps the top brass happy, no one in IT is really going to say no to more investment and new shiny stuff.

[sugerbear]

Porting Apps? Downtime.. Eh

[quote]RBS faces a Herculean job in bringing online a new mainframe operating in a core part of its day-to-day business. It must plan and execute the job without interrupting the existing service by taking the old mainframe offline during the transition.

RBS did not say when it plans to bring the new mainframe online.

But hardware is only one thing: RBS must also determine what do with the existing apps running on the system. Either it must port existing apps to the new system - which is likely - or write or buy new apps. If the former, RBS must design, write, test and then shift. If the latter, RBS must make sure the new apps work on the new mainframe and interoperate with other RBS’s other, connected systems.[/quote]

Spoken by someone that knows nothing about mainframes. Are you a consultant by any chance ?

RBS are just buying a bigger mainframe and then plugging it into their existing parallel sysplex system. Whoop de doo. All that will get them is the chance to use the newest version of the bits of mainframe software and things will run a bit faster.

No porting of apps, no downtime required. That is what makes the mainframe such a great environement to develop and run. Something that lesser mortals dont get.

[Richard Wharram]

Re: Porting Apps? Downtime.. Eh

Indeed. Updating zSeries and z/OS was not a minor task but a lot less effort than desktop and server upgrade programmes in my experience :)

[John Smith 19]

Re: Porting Apps? Downtime.. Eh

"No porting of apps, no downtime required. That is what makes the mainframe such a great environement to develop and run. Something that lesser mortals dont get."

True.

But is not £450m somewhat expensive for a hardware upgrade alone?

[tom dial]

Re: Porting Apps? Downtime.. Eh

It is more than a bit expensive and I, for one, do not believe it without serious proofs offered.

[Geoff Lamb]

Re: Porting Apps? Downtime.. Eh

YES +1

A mainframe is one of the EASIEST systems to upgrade. Most banks (if they pay the IBM tax) run on pretty much the latest kit while still running decades old code. All completely virtualised - decades before VMware came along.

Oh, and the idiot who thinks you can swap a banking system with a google cloud - no one cares if your google search returns the answer that is not quite right. See how many customers you have if you do that with a couple of accounts !

[Anonymous Coward]

Surprise meme!

The press of middle England weighed in, too, pillorying the bank’s already unpopular chairman as he a grovelling apology to MPs for the whole episode.

Surely you mean "as he accidentally a grovelling apology"? ;-)

[Oliver 7]

Is the editor off this week?

[James O'Shea]

Oh, the editor's been off for a while.

It was the smell which first alerted the rest of the staff.

[ukFletch]

...and the IT problems still haven't been fixed...

FCA are investigating... blah blah blah... Replacement Mainframe... refinaciate IBM'icus... DR Plan... of course there is one... somewhere, try looking under that lads mag from 2005, the one with Jordan on the front, a classic that is...

The only thing I really care about, despite working in the IT industry for getting close to 20 years, is why since 1 year after the SHTF at RBS, they still can't update my balance properly? I'm assuming that the roll back they must have done, recovered them to a restore point from 1983, say around April the 1st?

How the hell they can on the 1 hand roll out a new "helpful" system to give you cash when you've lost your card while on the slash, yet can't tell you if you have any money left to spend on the monthly shop when your stone cold sober, beggars belief. Or am I living in cuckoo land?

[Anonymous Coward]

Re: ...and the IT problems still haven't been fixed...

Err... I think you're living in cuckoo land. I never had any problem being told how much money is in my account for immediate access. Unless, perhaps you've got lots of non-faster payments going through the account or a very high volume of card transactions?

[ukFletch]

Re: ...and the IT problems still haven't been fixed...

Then I and a number of others I know, suffering the same problems are all proud residents of CuckooLand. Which I am assuming that Jamie and His Magic Torch (along with Wordsworth the dog) will be visiting soon then.

[Richard Wharram]

Re: ...and the IT problems still haven't been fixed...

No two nights are the same.

[Anonymous Coward]

Lessons learned from the meltdown?

"it's quite possible that what happened at RBS might be replicated elsewhere as old and overloaded mainframes like the one at RBS hold millions of accounts at other banks who’ve also sent their IT jobs overseas"

What happened at RBS wasn't caused by 'old and overloaded mainframes` but by errors introduced into the overnight CA-7 batch processing job made by a single operator. Subsequent measures to roll back the error made matters worse as they rolled it back to the beginning of the financial quarter and not just the previous nights transactions. Took ages to manually restore all the accounts.

"The exact reason for the problem have been found and poured over in a tedious level of detail, then process will have been put in place to prevent the set of circumstances repeating."

So, who was it that fired the most experienced staff, then handed responsibility for overnight batch processing over to a single individual with only four months experience with the system.

[Rabster]

Re: Lessons learned from the meltdown?

Exactly. Most mainframe shops have large teams doing the ITIL Capacity Management process.

[Anonymous Coward]

Looks like the CIO decided to exploit the crisis and pitch a bid for new kit (like we all did for Y2K!)

Maybe the clapped out mainframe can be given to the RBS 'bad bank' along with all the other toxic assets?

[Dominion]

Jesus wept! The mainframe isn't clapped out - the staff and management running it are!

[Mad Mike]

The mainframes aren't clapped out. You wouldn't believe how often they're replaced!!

[Caff]

new mainframe solution??

Switching out one IBM mainframe for a newer one is pretty par for course and relatively painless.

From what I heard about the RBS meltdown caused by the CA7 upgrade most if not all of the ensueing madness was down to the way that the various mergers over the years affected their ability to roll forward daily batches to catch up with what they had missed over the days while still allowing some day-to-day banking.

The hardware upgrade will do nothing to fix these issues, rather they need to go back and properly intergrate the various disparate batches and clearing processes of the banks that make up RBS ( not straightforward by any means )

[Anonymous Coward]

Re: new mainframe solution??

RBS don't have one mainframe. They had something like seventeen spread over multiple sites when I worked there (IIRC), they were fairly proud to say that they had the largest footprint of IBM mainframe in Europe.

[Caff]

Re: new mainframe solution??

That would complicate things more than a little if they inherited all the smaller mainframe from the member banks and tied the batch of each ones together with file transfers and attempting to schedule and trigger jobs between them.

This could tie in with the "new mainframe" plan mentioned, which would go someway to explain how this could be a solution to their problems.

[Anonymous Coward]

Not a good article

Quite apart from already-pointed-to fact that this was not a hardware problem at all, a lot of what the author is saying amounts to mainframes are big and complicated, and people have to know a lot to use them.

Banks are big and complicated, and are supposed to be managed and staffed by people who do know a lot about them, their systems, and how those systems are computerised. And if you are going to computerise something like that, you're going to need something big. Now, lets see, what are those systems called that are up to the job? Ahh, yes, mainframes.

It's not the sort of task one does with a PC and a bit of cloud.

[Lord Elpuss]

"25 of the world’s top banks use mainframes from IBM, according to Gartner"

25 out of how many? Without context this statement is meaningless...

[Jemma]

Context...

The ones that got bailed out and are still trading?

God knows there can't be many left now... scary to think there are more MG6 customers than there are competent banks.

Even scarier that these people are CHARGING us to take our own money and lose it.. and people seem OK with this..?

Is it me, or is there something wrong with this woodcut?

[Dominic Connor, Quant Headhunter]

True

Actually, I'd expect the 25 to be 25 out of 26 or 27, ie nearly all of them.

As the article says, it is insanely difficult to replace these things and even where they aren't the spine of the firm they do run some critical bits.

Also the biggest banks grew by sticking bits of smaller banks on, so even if the "main" bit of the bank didn't use mainframes, some other part will often have it.

[Anonymous Coward]

There will never be a need in the world for more than 26 banks :P

[Captain Scarlet]

Heh yeah right, everywhere I have ever worked has treated IT as a thing that can just be upgraded and replaced. Its odd to see peoples reactions when they realise something isnt as simple as plugging someone in

"financial IT where the technology is treated as black box – meaning it can be installed and operated, without much thought to who runs it"

[Arbee]

To the people asking what £450m will buy a bank...

For a good reference of what £450m will buy you, Nationwide recently completed their 5-year Voyager project, which replaced their Unisys mainframes running whatever banking application they did, with SAP Banking Platform. Essentially, this was a complete replacement of the back-end banking platform, a rewrite of most of the middleware, and a rewrite or replacement of most of their front-end applications. We commonly called it "rewriting the bank."

That cost £400m.

(Note they still have the mainframes, but they just have savings and mortgages).

[Sir Runcible Spoon]

Re: To the people asking what £450m will buy a bank...

Well, all I can say is that if that had been a government IT spend you could have added a 0 to the figure

[Anonymous Coward]

Re: To the people asking what £450m will buy a bank...

Sure, but wouldn't most have that have gone in profit margin to SAP? :P

[Anonymous Coward]

Brilliant insightful article, but....

Call me a cynic, but I just don't see it changing the behaviour of the banks or any corporations chasing revenue and cutting costs.... More like this from the Reg please....

[dabar]

HSBC - suffering a meltdown at the moment - 21-6-13

Just tried to pay in at my local HSBC to be told by one of the staff that HSBC is currently having a global outage on all systems including card transactions in shops.

[Mad Mike]

Re: HSBC - suffering a meltdown at the moment - 21-6-13

By earlier prediction in these comments of it'll happen to them all in the end seems to be coming true quicker than I thought!!!!

[Anonymous Coward]

Re: HSBC - suffering a meltdown at the moment - 21-6-13

Astonishingly prescient Mike .... I noticed that as well :).

[Anonymous Coward]

OVERSIGHT vs. Regulation....

What this case illustrates is that outsourcing, language barriers, cost cutting, and assumptions all conspired together to create the mother of all fuckups! The fact that corporations are always playing bungee chord management games in cost cutting and over-efficiency, means that assumptions will always be made. I laugh when I read that more regulation may be needed... Ya think?!

What's needed is constant OVERSIGHT. Not more regulation. Regulation is just another form of form-filling, bureaucracy and INTERPRETATION of rules. Look at Dodd-Frank for Christ sake. What we need are experts being regularly sent in to assess the true-grit reality of the situation before a crisis occurs...

"If regulation is mandated then don’t expect a quick fix. Unlike, say, the nuclear or airline industries, where accidents have led to investigations that have produced operation and safety standards, similar standards in financial systems will be difficult because of a fundamental refusal to share information".. Fine. Then send in experts who won't make the details public but will whip complacency into shape!

But please don't write more rules that will just lead to a fine! Its not enough of a compliance incentive. If the situation if found to be dire, then you need to threaten to remove their banking license.... But this is only workable if we break up these banking monoliths into lots of smaller banks. As an ex-derivatives HFT guy I cannot understand why they are not doing this.... Lobbying is the only answer!

[John Smith 19]

Nothing actually *wrong* with mainframe

So why replace it?

IIRC Assembler had a big part to play supporting ATM's on OS/370, but that decades ago due to response speed and number of them. Hard to believe it's still a key tool in the MF developers arsenal.

Back in the day Cybermation built an awesome mainframe scheduler.

It was the mother of all TSR's. The devs (patching a live mainframe OS) must have been quite special.

Sadly CA bought them and the rest is history.

[Stevie]

Bah!

Yes, let us replace all the old technology.

We'll get the expense whch can be written off (it has been an industry maxim since forever that IT departments are a money sink and a Tax Loss). Shareholders=happy.

We'll be able to employ lots of programmers. Jobs=popular.

We'll experience cost overruns in the millions. Vendors=ecstatic.

We'll miss deadline after deadline. CoB="Concerned".

We'll probably experience several false starts as we discover the hard way what we forget to spell out in the spec or what the Universities forgot to teach *this* crop of IT Graduates. CoB=fired (problem solved, at cost of "New Broom" policies that invalidate man years of work - and before you scream and leap, what woman would be stupid enough to get ensnared in this can of worms?)

I can see no downside to the plan.

On the other hand, we *could* just develop a migration path to a more cross-connectible solution over time. in which new development was built using newer ideas.

Yes, I *know* we all said "yes" when given those "are you planning to migrate from mainframe to client-server architecture in the next five years" questionnaires at every f---ing IT confab run in the 90s, but we did that to qualify for the free tote bag of goodies.

No-one in their right mind would seriously jeopardise their career by actually *doing* that. The vultures would start circling as soon as you ran the first project meeting and from that moment on your days would be numbered as every pair of eyeballs in the place would be vetting your every move just waiting for the main chance to crap all over you and take your job.

Rewrite everything in "C"? Are you barking mad? In what world does *that* make sense? All the expense of rewriting your financial software, but in a language intended for crafting device drivers. Remind me what your degree subject was.

Change is bad. Ask the guy who was put in charge of the Y2K project so his failures would be public enough to fire him should he continue to agitate with the board for unpopular tech reforms.

[Alan Esworthy]

Stick to what you're good at, Gavin

I don't think you have the background to write intelligently about the mainframe world, Gavin, and looking down the list of your last dozen or two articles suggests that you might have been mis-assigned to this story.

For one thing, you emphasize the replacement of the IBM mainframe as RBS's solution to the problems causing last year's outage but never even hint at what models are being replaced and what levels of operating system are involved. For another, IBM's parallel sysplex environment is specifically designed to make upgrades on the fly fairly straightforward (as straightforward as such things can be, I grant you), and parallel sysplex has been around for more than 30 years and as a result works rather well.

The root of the problem, as has been pointed out by several other commenters, was a failed CA-7 upgrade. This job management package has also been around for more than 30 years and is well understood. True, CA has done its usual half-fast job after acquiring it from Uccel (formerly University Computing Corp.) but the problem still boils down to the wrong people responsible for the wrong tasks at several levels of tech and management. If you have an incompetent team, taking away their old hand tools and giving them shiny new power tools just means they will do more damage more quickly.

[Escape Velocity]

And yet...

executive types still ask questions like, "why do we spend money on IT when they don't directly make money themselves?", and "why do we spend money on systems monitoring?".

The thick will only get thicker. Bad IT policy is a result of the profit-this-week-to-hell-the-rest mentality sprung upon companies by investors blinded by their own impulsive, myopic and short-term self-interests . Yes, I do believe its the investors who are the root cause, bad-children actors in all this mess.

[Tomato42]

>500 IT jobs that have been outsourced to suppliers in India.

you know, because IT guys in India have this huge experience running IBM mainframes...

[Anonymous Coward]

I heard they all had 50 years each, on their CVs.

[Tristram Shandy]

Two CEO Viewpoints

There are 2 ways of looking at an IT department:-

1) Our IT Systems work perfectly and without problems because we have many highly skilled and experienced staff.

2) our IT Systems work perfectly and without problems, so why do we have all these expensive IT staff.

[John Smith 19]

Odd but when they multiply No of staff used x No of yrs to develop banks discover

It costs a f**k of a lot of money to duplicate the functions and then test the software, to confirm you have.

You cannot overestimate how far a bank will go to avoid having to re-implement a system, especially a system that has been running reliably for decades.

BTW what's tended to happen was support systems were added over the years. Firstly as applications on the mainframe, then (typically) on DEC VMS boxes, then on to various flavours of Unix, then on Windows (and Linux) servers, with rising levels of virtualisation as that technology has improved.

Would reverse engineering this "architecture" to re-factor the system so some modules were brought "closer" together be a good idea? Probably.

Is it going to happen. Probably not.

[Mark 65]

Knight Capital

It wasn't an algorithm problem, it is believed that the market making testing software got released into production with the market making software. It then did its job and tested the automated market making software - across the entire market, not just KC's.

http://www.nanex.net/aqck2/3525.html

[Anonymous Coward]

Applying Some Brutally Honest German Rationality Here

A) Shoddy Journalism: RBS apparently bought new mainframe hardware (certainly not just one for 450 million). All that we know points to that they just bought instruction-level and Operating-System-level compatible IBM mainframes, which will happily run code from the 1960s. Essentially, the latest model of S/360. Disregard the IBM marketing bullshit "zOS". These are the latest S/360s running MVS (aka bullshit "zOS") and VM.

B) If they actually decided to completely rewrite their operational software, this would cost something like 5000 million pounds and take 10 years if they would ever finish that project. So, they will simply run their existing software on newer, faster hardware. Still S/360 compatible and it will happily run all the old assembler programs and all the programs for which source code has been lost. Which certainly is something like 5% of all source code. Don't bullshit me with "this is a bank, they have proper source control". They have NOT.

C) All we know points to the M.B.A. idiots (and other management "talent") to have purged themselves of their skilled IT/mainframe/operations specialists in a mad drive to "cut costs, improve profits and get a fat bonus for top management". In other words, my dear Englishmen and Scotsmen, the Banksters have shafted you thoroughly by eliminating well-paid, demanding, high-quality jobs. The "icing on the financial cake" so to speak.

D) The awe of the financial sector displayed in this article is just ridiculous. Stop Believing In The Money Religion. The financial industry is rife with crooks and they will crook up your entire country, the entire economy if you don't grow a healthy scepticism towards them. If you continue to let them lie unpunished, then you can look forward to the rule of Ricardo Cromwell, Prime Master in 2020.

[Everthe Optimist]

Think I'll put my savings under the mattress.

Almost all "big" banks run their core systems on IBM mainframes. They do it because they work, they are reliable, and they are cheap! Don't believe the $450m rubbish, maybe this included the new building for their new DR centre? The hardware and system software is simple, standard IBM stuff with a handfull of add-ons from other vendors like CA-7 in this RBS example. The system software provides the database management systems and the transaction management systems. This is all standard stuff and not banking specific. A typical bank has a handful of machines clustered together in a thing they call a sysplex. They can pull boxes into or out of the sysplex for maintenance or upgrade whenever they like, and almost all of these machines are less than 2 or 3 years old. The sysplex behaves as a single system, and most of these have run 24hrs per day for years without outage. Then they will have another sysplex in some other location ready to handle a disaster at the main site.

The "system" might be standard stuff, but the application software is another story. These systems have grown incrementally for half a century. Surrounding the online transactional systems that drive the ATMs or your smartphone app there will be thousands and thousands of interconnecting batch jobs which have been patched and tweaked over the years every time some new regulation comes along to tweak the banking rules, or everytime some bright marketing person comes up with another gimmick to help sell some bank service. Every bank take-over or merger escalates the complexity. This network of jobs is what CA-7 tries to manage, and what "broke" when CA-7 itself got broken, apparently by a "pilot error".

The RBS meltdown was spectacular but as they say "...you aint seen nuthing yet". Most of these same "big banks" have been steadily hollowing out their core support skills. Either by outsourcing or offshoring or just simple downsizing. The systems are so darned reliable that they normally just tick along trouble free, it is only when something goes badly wrong that those "experts" really earn their keep. And it is not the "Wall-Street-Way" to pay people who are not immediately vitally needed ... right now. So the experts are disappearing. Next time a bank has a technology meltdown who will they call? They might be hoping to rely on 3rd party expertise, but even the likes of IBM are shedding people so fast that finding someone who understands how bank XYZ's systems actually work will be an ever bigger challenge. Think I'll put my savings under the mattress.

[Anonymous Coward]

Architects with a God Complex v Darwinian IT Evolution

Here we have an example of why Outsourcing / Offshoring doesn't work and how errors are compounded by refusal to wake up and reverse bad decisions.

A working lifetime of skills don't get transferred in a time-boxed handover, only a list of procedures and instructions. Eventually (sooner that you might think) those instructions are undermined by rapid turnover of staff in the offshored / outsourced environment which is compounded by the minimal maintenance / upgrade investment of the Outsourcer who typically inherit ageing kit and who will eventually hand back vintage Unix / Mainframe / Wintel boxes that are now virtually unmaintainable, limited remaining ISV and HW Support and with limited/complex/expensive upgrade options.

The Business Brains @ RBS made a terrible gaff and made short-term savings and have put their business at unnecessary risk, now rather than admit their mistakes, they appear to be planning to rip out a perfectly working system (that they admittedly no longer fully understand) that has grown over decades - you cannot re-design systems this complicated over night.

Last I heard of someone doing something that complex was the big beardy guy upstairs. AFAIK Architects / Business Managers are not omnipresent God-like beings, just a bunch of hairless monkeys who ought to wake up and recognise that complex systems evolve over a long,long time. Some bits don't work and will die off or be adapted or updated and there's a bunch of redundant code that probably isn't used any more.

[IT pro]

What is the root cause

I have read the article and do see what is the root cause?. For an upgrade to south, and not able to recover in a short time span, means a number of things, bad planning, wrong resources being forced to take on the upgrade, and not understanding the product and the environment.. and lastly where was the fall back plan.. Should of not been approved to perform such a change. Probably to much senior pressure.

Replacing the system, whether this is hardware and software is more an executive management escape from finding what the real cause was. I doubt very much it had anything to do with the platform (software or Hardware) but like others have pointed out, lack of real experience with the environment in which they are working on..

If the same philosophy is used by Senior/Exec Management years from now the same will occur.. Whether this be Mainframe, Windows, Linux..

This occurs far to much all sectors within large organizations, and they go and spend millions to appease share-holders and the board, that something is being done..

But again many Exec Management only stay around 3-5years and move on... Exec management 101

[Anonymous Coward]

RBS uptime

even internet banking has a maximum 97.2% uptime and that is before the other fiascos...

below is every single day!

From the RBS website:

System maintenance - 02:00am to 02:40am

Please be aware that our systems update every morning between 02:00am and 02:40am. During this time you will be unable to log into your Digital Banking