Lloyds Group probes server crash behind ATM, cash card outage

Lloyds Banking Group is investigating the cause of a Hewlett-Packard server failure it blames for taking down thousands of its ATMs and crippling cash cards at the weekend. The crash saw thousands of customers unable to withdraw money from their accounts or make payments using debit cards on Sunday afternoon. The outage left …

[Anonymous Coward]

I don't know anything about bank setups but wouldn't they use failovers?

If not, why not?

[Velv]

Never mind Banking, clearly you know nothing at all about IT.

Yes, they have and use failovers. But it depends on the nature of the fault before you can invoke a failover.

Hardware generally yes, no problem, automatic failover can be almost instant, even geographically dispersed.

Software, middleware, transaction processing - maybe not - it just depends what has failed and if its possible to move the multiple transaction threads or if you need to stall the processing and move it in a controlled manual manner.

[Anonymous Coward]

CEO's tweet implied hardware

The CEO's tweet clearly implied that one specific HP server had failed. As software is not specific to any single computer, that clearly implies it was a hardware fault. If not, HP has grounds for complaint - if not legal redress.

I found it extremely odd that he saw fit to name a supplier in that way.

[Solmyr ibn Wali Barad]

Re: CEO's tweet implied hardware

Yes, it's odd. Maybe they've decided to burn the bridges.

[Captain Scarlet]

Failovers dont always work

You can have the odd issue where the machine being mirrored is responding to heart beats but not actually working anymore, hardware problems can be a nightmare, if the storage system for instants starts responding sluggishly it may not flag a failure anywhere even though it can no longer complete all requests thrown at it before they timeout.

[OpenIndiana]

Re: Failovers dont always work

@Cpt Scarlet

Thanks for the explanation

[Anonymous Coward]

No only do they use failovers, but it's likely that the machine which failed was a tandem, they're more reliable than mainframe, but the only thing you can be sure of with a system is that the system will fail.

[Chris Miller]

Tandem

"it's likely that the machine which failed was a tandem" - if by that you mean HP NonStop (formerly Tandem), they're effectively immune to single points of hardware failure (if correctly configured - which I imagine/hope a bank's systems would be). Dual hardware failures can be the result of (a) extreme bad luck - second component fails before the first failed component can be repaired; or (b) human error - engineer sent to replace failed component pulls the good one by mistake (should never happen, but I've seen it done).

If a dual failure does occur, it can cause significant problems as the software tends to be designed on the assumption that it can never happen.

[Anonymous Coward]

Re: Tandem

"On the assumption it can never happen."

Yes, that's wise...

PS, I want to see some sci-fi/fiction where the characters have infinite "back up plans" and redundant equipment. Like Mission Impossible, cross Star Trek and some McGiver for... well, backup.

[Chris Miller]

@ TechnicalBen

You can code elaborate software recovery routines that will be useful once every 50 years (and potentially reduce the stability of the software), or accept a very rare outage as a result of dual hardware failure. Your choice.

[James R Grinter]

Re: Tandem

When I wrote some code for "Non-stop" Tandems (TAL, 20 years ago or so, as part of an industrial placement), you had to write the code to synchronise across two CPUs yourself. I think the only thing running like that on those production systems was the command shell. I'm sure it sounded great to whoever got the original sales pitch, but the reality was a long way away.

[JoshOvki]

Cash

Time to start carrying cash again, or having bank accounts with two separate banks.

[Anonymous Coward]

Re. multiple bank accounts

Anyone with any nouse should already have accounts with several banking providers, to take advantage of the various offers freebies and incentives they all dish out. Me, I've got 3 bank accounts, and 5 credit cards with 3 different providers.

[TRT]

"@dangerfield_gem Gemma, no truth in this. The issue was caused by a HP Server failing here in the UK. PDP"

Is he blaming DEC as well?

[Adrian 4]

Compaq bought DEC. And HP bought Compaq. So, yes, it's DEC's fault. Nothing whatsoever to do with Lloyd's management, cost savings, hacked together back office systems etc.

[Peter2]

Note that he says that "the HP server failed in the UK" when the assertation was that had they have had a competent and experianced IT support team in the UK they wouldn't have the issue.

Is it just because i'm a hardened cynic that I notice Sir Humprhy-esque evasion in answers like this and assume that he means that the lack of experianced staff (and BCM/DR tests) was the issue that caused this problem...?

[TRT]

I was going to post something about him not getting any brown sauce on his sausage sandwich this morning, but then I remembered what the HP on HP sauce stands for and it suddenly made sense. failure is the usual mode for HP and IT.

[Anonymous Coward]

No announcement of the problem

Servers going down is one thing, (there but for the grace of $deity...), but why were there no annoucements of the problem? Nothing on their web site, and their official Twitter feed stopped just at the same time as the incident started and didn't resume until this morning.

At least the all other unfortunates posting to Twitter and Facebook reassured me that it wasn't just my account that might have been rifled of its balance, but it was a very unpleasant half hour between having my card declined and finding out what had happened.

[Anonymous Coward]

Re: No announcement of the problem

Maybe they don't have 24/7/365 twitter writers, maybe it was their day off and they were using Facebook instead, or maybe Google+, or even a blog, heck maybe they shared some photo's in Pinterest whilst syncing it with their Flickr accounts ...who knows.

Still glad after 30 minutes your entire would stopped collapsing and you could return to you instant gratification.

[ADJB]

PDP?

I notice that Mr Pester ends his tweet with PDP. Are these his initials or the technology the system runs on ?.

[Chris King]

Re: PDP?

They're his initials (middle name David) and he signs off all his tweets that way.

[Anonymous Coward]

Single point of Failure

Memo to bosses: Time to pay more peanuts to your monkeys.

[Anonymous Coward]

HP Servers?

Surely a bank would know that it's cheaper to buy your kit outright than doing a Hire-Purchase agreement???

Thanks, my coat's the rental one over there ->

[Anonymous Coward]

Re: HP Servers?

While it may be cheaper, it's the banks we are talking about here.

They would (read as "did") sell the office, building, chairs and hardware to have a nice large sum of cash to declare to the share holders, along with the reduced costs of having to replace and repair those things.

Never mind in the long run the price rises in rental could eat up all that temporary cash. If it looks good on paper, and for a short enough time to get the market buying your shares, well, you can always parachute out before the reality hits the ground.

[Fatman]

Re: HP Servers?

... well, you can always parachute out before the reality hits the ground. FAN!!!

FTFY

[Anonymous Coward]

Re: HP Servers?

> Thanks, my coat's the rental one over there ->

Is it a Rumbelows' shop coat?

[Steve Davies 3]

Re: HP Servers?

No it was the one from 'Grace Bros'

[jonathanb]

Re: HP Servers?

They could be doing an HP agreement with Blackhorse Finance to shift profits elsewhere in the group.

[Anonymous Coward]

ATM Network?

It wasn't the ATM Network that was down, was it?

I thought all Lloyds Banking Group debit card transactions were not working - i.e. in shops too and regardless of whether it was a LBG owned ATM

[TRT]

Re: ATM Network?

It was the TSB. Or Titsup Savings Bank.

[Anonymous Coward]

Re: ATM Network?

That's not correct, the outage affected all LBG banks (including TSB). It was just that TSB's CEO Paul Pester chose to comment.

[dervheid]

Dear Banks

Sort your fucking IT acts out. It's the 21st Century FFS.

(I also see too much "clearly you know nothing at all about IT" and comments of that nature.

So. Fuck. What.)

I think that customers have a right to demand far more robustness from Bank (and other vital institutions) IT systems than there currently is. Stop paying your already grossly overpaid executives obscene amounts of money, bolster your infrastructure (including security) and treat your customers with more respect.

[Pen-y-gors]

Re: Dear Banks

I think that customers have a right to demand far more robustness from Bank (and other vital institutions) IT systems than there currently is.

I don't entirely agree - a large proportion of personal customers in the UK don't actually pay for a current account (okay, charges for overdrafts etc) but basic banking is free. Once this was funded by the fact that they didn't pay interest on current accounts, but with interest rates at zero they don't have any income to pay for the services. It's a bit like Google - you can't really complain or have expectations about something you're getting for nowt.

But yes, on the whole bankers are overpaid parasitic scum who should be condemned to twenty years chained to an oar in the bottom level of the slave galleys.

[Anonymous Coward]

Re: Dear Banks@ Pen-y-gors

"I don't entirely agree - a large proportion of personal customers in the UK don't actually pay for a current account (okay, charges for overdrafts etc) but basic banking is free. Once this was funded by the fact that they didn't pay interest on current accounts, but with interest rates at zero they don't have any income to pay for the services."

Let's stop for a moment and consider why interest rates are so low. Would it be the fault of the self same banks bleating they've no money, whose reckless lending meant the state had to bail the thieving scum out in the first place, accompanied by QE and near zero interest rates?

If they have to provide "free" current accounts at a loss for a few years, simply because their collective greed, incompetence and dishonesty wrecked the whole system, then I say tough luck. If they don't want to do retail banking, then let them close or sell their high street branches.

[Tony S]

Re: Dear Banks@ Pen-y-gors

"why interest rates are so low"

It's also worth noting that the interest rates paid to the account holders are low; but the banks themselves use that money to fund loans. Often at 10 times what they pay to the account holders.

Some of that money also goes into the stock market in various forms. Value of investments can go down as well as up, but even so, many of them are making 20 or 30 times what they pay the account holder. (I saw one indication where a bank was making more per day than they were paying out per year)

As for the "free" banking accounts; they reserve the right to make charges for certain things. Some of these will only be levied if you make a mistake, but even so, just one of these charges could mean that your "free" account is costing you more than you are earning in interest on the money in your account.

For what it's worth, my bank keep pestering me to move to their "premier" account. I'd end up paying more per month than my savings would earn in interest over 5 years.

[Aitor 1]

Re: Dear Banks@ Pen-y-gors

Completely false.

they have your money, and use it as collateral to get 5x 10x that amount in the interbak-funds market and lend it at market price + premium.

So yes, just by having money in your account, you are paying them.

Plus, if you get your salary by bank transfer, they get paid, plus credit/debit card transactions, plus you getting money out of a different network ATM, plus non free money transfers (YMMV), plus the extraordinary expensive currency rates they apply...

So yes, you ARE paying them.

[Lamont Cranston]

Re: "don't actually pay for a current account"

Really? I (and all the other customers) give the bank all my money, and let them do what they like with it, so long as they hold enough in reserve that I can take it back out again. Interest rates might well be in the dirt, but I'm sure that they've found some way of turning a profit from my money.

[Anonymous Coward]

@ Pen-y-gors

"Once this was funded by the fact that they didn't pay interest on current accounts, but with interest rates at zero they don't have any income to pay for the services."

Please think this through. Interest rates are at zero for money that WE lend to BANKS. Have you checked out mortgage or other loan interest rates for money BANKS lend to US lately? Not to mention the far higher rates they can get for lending to actual productive ventures such as industry and commerce.

Even at a very low rate of interest, banks could make plenty of money to run their businesses out of the interest they can get on the vast amounts of money deposited with them. Don't forget, that's a substantial fraction of all the money in the UK.

[Anonymous Coward]

Credit cards ?

I always carry my main (Nationwide) debit card. However I also have a Barclaycard (which is almost exclusive reserved for expenses-type purchases). In the event my debit card fails, I can always cover it with the Barclaycard.

And then get medieval on the Nationwides ass ....

[Anonymous Coward]

Re: Credit cards ?

Same here, I may always carry a Debit Card for withdrawing cash, but I do all my purchases by Credit Card if possible, and I carry multiple cards with me from different banks... so I have multiple ways to pay if I have to, and of course I carry cash just incase the whole thing goes tits up or I want to purchase from small vendor.

[Nick 6]

Re: Credit cards ?

Ironically, Nationwide is one of the few with a core banking platform designed and built this decade. Almost all the rest are derived from the last century. This doesn't make them immune from failure of course, nor from the challenges of managing all the interconnected systems, but at least they don't have to treat the whole lot as the IT banking equivalent of Buckaroo.

[Anonymous Coward]

End of the world

Some ATMs not working for a few hours on a rainy sunday !>= End of world

[Why Not?]

Re: End of the world

Hopefully it was for the PHB bean counter who recommended the cuts and has been asked to clear his/her desk.

we can but hope.

can we expect it to finally come out that the support is done in 'Bongo Bongo land'?

[Anonymous Coward]

Re: End of the world

Not sure the support is done in Bongo Bongo Land, but HBoS outsourced much of their support to HCL in India in about 2007. Many of the in house staff were made redundant a few years later when the LTSB thing happened.

A/C cos I still have friends there.

[Securitymoose]

Re: End of the world and backups

No, the bean counter won't cop it - the man responsible for cuts and outsourcing at RBS is still hanging on, despite all his colleagues doing the decent thing and falling on their swords. All the BC has to do is ignore all decency, carry on like nothing's wrong and plan what he's going to spend his next bonus cheque on.

Regarding the fact that Lloyds had no backup, well they do have - every other bank in the country - you can use most other cash machines for free, pay with cash, credit cards etc., so not sure why everyone is whittling on about it, unless they are hoping for a handout as well.

[Anonymous Coward]

Re: End of the world and backups

... so not sure why everyone is whittling on about it...

Because they were made to feel like idiots in front of strangers. I'd have thought that was obvious.

Cash is only any good for paying for your weekly shopping/tank full of petrol if you have any on you.

Credit cards are only any good if you have one, believe it or not, some of us don't have one, because we don't feel the need to spend money we don't have.

It's an immediacy problem, that's the thing with epos systems, you've already got the goods.

When you don't know you'll need lots of cash in your pocket to pay for things, you aren't generally carrying a lot of cash.

So you're left standing at the front of a line of complete strangers, in front of another complete stranger, looking like an idiot who can't pay, and who is now going to cause all of those strangers to wait whilst you fuck around trying to find another way of paying.

Making people feel stupid will (rightfully IMO) always get you complained about.

[Anonymous Coward]

Re: End of the world

Some ATMs not working for a few hours on a rainy sunday !>= End of world

It's not the ATMs which are the problem, it's the epos systems.

People standing in front of a cashpoint being told to fuck off, might not be pleased, but they're not embarrassed enough to go shouting about it in public. Stand them in front of a checkout girl, and refuse them access to their money, and they're going to shout very loudly, and very publicly about how you're an incompetent wanker, who has ruined their life.

[Anonymous Coward]

Re: End of the world

"It's not the ATMs which are the problem, it's the epos systems."

Exactly. Standing at the counter of a petrol station with your car full of petrol and finding that your bank is refusing to let you have any of your money to pay for it is a very unpleasant experience.

Gas station attendants basically treat you like the lowest form of life on earth in this situation and make you fill out some very evil sounding documents before they'll allow you to leave - even offering to leave the vehicle and its keys with them while you go source the required dosh isn't always accepted(*).

(* Happened to me once a long time ago, when I found after filling my bike with gas that my wallet wasn't in my pocket. Only lived just down the road, but they wouldn't let me walk home to get it)

[Anonymous Coward]

Re: End of the world

even offering to leave the vehicle and its keys with them while you go source the required dosh isn't always accepted(*).

Car could be stolen even with keys. Same goes for any iPhone or whatever you offer. Most places in the UK though will just check your drivers license to see who you are and copy the info down and copy your reg plate and let you go.

Also you made the mistake of riding a motorbike which automatically means your invisible to 95% of road users, a legitimate target to another 2% and only seen as a nice guy to the remaining 3%. I've given people pillion rides back to there bike when they have turned up on foot (we took fuel +£5 deposit on petrol can) while on shift so not all people are asses.

[Anonymous Coward]

Re: End of the world

Oddly enough as an ex- retail manager and having worked in IT at financials this sounds exactly like what happens at both ends of the outage.

[Anonymous Coward]

Re: End of the world

Unfortunately, it doesn't matter how embarrassed or angry or inconvenienced the customer is. In theory, he or she can take their business to another bank that performs better. In the real world, there is no such bank. Instead of competing to provide better service for less money - as theory dictates - they have realised it's much more profitable (and less work) to standardize on a uniformly low level of service. If the customer doesn't like it, he (or she) can do the other thing.

In theory, too, cartels are illegal. In practice, there is no conceivable way of preventing them. (Even if most of our rulers weren't in their pay).

[Anonymous Coward]

Re: End of the world

Unfortunately, it doesn't matter how embarrassed or angry or inconvenienced the customer is

I suspect it matters to the C level people, when they're personally being insulted for their incompetent operations. If for no other reason than potential future employers being able to read all about what an incompetent fuckwit they are.

[Anonymous Coward]

Worth knowing a bit of consumer law

If you should ever find yourself without the means to pay for fuel, having filled your car :

1) Explain situation to cashier

2) Offer to pay as soon as reasonably possible (say go an tap a friend/relative for the money, or get a chequebook)

3) Show proof of identity (driving licence, usually)

at this point it becomes "a civil matter sir", and plod will not be interested.

If any attempt is made to prevent you leaving it will be assault, and false imprisonment.

This happened to a friend of mine who did exactly that. The station was manned by one person, so he couldn't stop him leaving, but he did call the police. By the time the police contacted my friend, he had returned and paid. The police were extremely unhappy at being lied to by the cashier who told them my friend had "driven off". As my friend commented to them, weren't they surprised that the cashier had his name and address ?

[Anonymous Coward]

HP Server?

As it affected card authorisations, there is one clear candidate then.

I guess the NonStop stopped!

[sugerbear]

Re: HP Server?

Or HP Almost NonStop

[returnmyjedi]

Having revently seen the hardware that one of the big banks has chugging away behind their fancy web pages and mobile apps, I can only assume that they ran out of valves, transistors and duct tape to fix one of them.

[Zot]

It was done on purpose, because their reserves were low...

http://en.wikipedia.org/wiki/Fractional_reserve_banking

"In most legal systems, a bank deposit is not a bailment. In other words, the funds deposited are no longer the property of the customer. The funds become the property of the bank, and the customer in turn receives an asset called a deposit account (a checking or savings account). That deposit account is a liability of the bank on the bank's books and on its balance sheet. Because the bank is authorized by law to create credit up to an amount equal to a multiple of the amount of its reserves, the bank's reserves on hand to satisfy payment of deposit liabilities amount to only a fraction of the total amount which the bank is obligated to pay in satisfaction of its demand deposits.

Fractional-reserve banking ordinarily functions smoothly. Relatively few depositors demand payment at any given time, and banks maintain a buffer of reserves to cover depositors' cash withdrawals and other demands for funds. However, during a bank run or a generalized financial crisis, demands for withdrawal can exceed the bank's funding buffer, and the bank will be forced to raise additional reserves to avoid defaulting on its obligations. A bank can raise funds from additional borrowings (e.g., by borrowing in the interbank lending market or from the central bank), by selling assets, or by calling in short-term loans. If creditors are afraid that the bank is running out of reserves or is insolvent, they have an incentive to redeem their deposits as soon as possible before other depositors access the remaining reserves. Thus the fear of a bank run can actually precipitate the crisis."

And yeah, I probably watch too much Keiser Report on RT !

[Anonymous Coward]

Re: It was done on purpose, because their reserves were low...

nope, just not the case

[Dave@]

All your eggs in one basket - Strategy?

Recall when I worked at the Halifax the executives like Mark Fisher moved many systems from the more robust Halifax systems like IBM System z onto less capable Lloyds platforms like HP because that was the strategy to unify the platforms and it was cheap. Now all the eggs are in one basket including the separate TSB and that basket is looking increasingly fragile.

Add to this the culture to "outsource everything" including ownership and responsibility (was it really HP's fault?) and you have the perfect storm with no one competent in charge. Even the recently appointed head of IT David Oldfield has no qualifications or experience in IT (reminds you of the credit crunch where none of the Banks executives had any banking qualifications).

[seasider]

Re: All your eggs in one basket - Strategy?

Yep - It is not really a surprise though. The upper echelons of the banking world have amply demonstrated in recent years that they would not recognise a risk if it came past and bit them on the leg. As we all know to our collective cost.

[Anonymous Coward]

Knock on effects

I had to use my credit card to pay for groceries yesterday after my Halifax debit card was declined at the till. Just discovered that that had triggered fraud detection causing my credit card to be blocked. I can only imagine what it must be like to be stuck far from home without any way of paying for fuel/taxis/transport/accommodation.

[heyrick]

Re: Knock on effects

"Just discovered that that had triggered fraud detection causing my credit card to be blocked."

Using your credit card triggers "fraud detection" which blocks the card? The hell...?

[Aitor 1]

Tandem non stop

I don't think it is as easy as that.

I still remember one day we had a VERY similar problem ant a big spanish bank.. the Tandem stopped, but I still don't know well what happened, because we got crashes in cascade: the systems were way too coupled... It took us 16 hours for full recovery (I think it was 3-4 hours "official", but batch etc was down).. and I had to work some 26 hours straight.

A yes.. it seems that the world is going to end, at least for those involved. There was almost no outsourcing.

I still remember it.. and it was 15 years ago!!

[Anonymous Coward]

Lets get to facts not fiction

Okay, lots of B/S comments here.

Yes it was a HP NonStop (formally Tandem).

In response to the comment regarding duplicating processes on multiple CPU's - if you truly knew about the platform you'd understand about resource nodes - but like most of the last 20 years that probably passed you by.

Problems can occur on HP NonStops that can lock a CPU up (e.g. rogue TCP/IP process as seen at ABSA in the mid 2000's). Other abends can occur if poorly written software results in a deadlocked resource scenario (particularly when using ENSCRIBE). Its not an entirely foolproof platform (its only as good as the humans that designed it -or the software running on it).

To me this looks like a sequence of unfortunate events. Most Tier 1 banks in the UK use HP NonStops for POS and ATM transactions - as they were traditionally more reliable than IBM Mainframes - these days there isn't really much in it (other than the significant cost differential - IBM Mainframes are ridiculously expensive to run workloads on).

[Richard Jones 1]

Most Have Never Seen Real OLTP Systems

I have seen several real time on line transaction systems get halted through system or software failures. One relied on a core changeover part, think 'Y' switch a key part of this device failed and since it was so small and so reliable and so under stressed there was no heart beat for the IC in question. It was only used during a swap over over so its status was only ever important for those few nanoseconds. Twelve hours earlier it was on line then the system went dead. Both sides reported no errors but neither could be forced back on line. That took some creative fault finding and time to clear.

Another time the on line system 'became busy', busy is not a fault condition so it reported no fault and no change over was initiated. Unfortunately the 'busy' it was involved with caused two issues, it was an unproductive 'busy' so no work was done and two, because it was 'busy' it stopped listening to ALL inputs. A system status change had to be forced to recover service and the previously on line system continued to work away doing nothing useful until a way was found to break into its (mis)operations.

The above two examples were not banking systems and not bespoke software, but systems in use in many other places. So the reality is that even systems which have been deployed many times in different countries and with different users can still have initially irrecoverable 'breakdowns' without needing human error or a lack of human operative training to cause them.

Both of the above cases resulted in expensive system re-development though as far as I am aware the changes were only ever tested by provision of enhanced 'heartbeat' and routine maintenance and never again through a system issue.

[Anonymous Coward]

DR Test

The word from the inside is that the hardware failure occurred during a Disaster Recovery test failover.

This sounds plausible, given the Sunday afternoon timing -- these tests are normally run at a weekend. DR tests are mandated by the powers that be, so all banks will do them every six months or so. Hardware failures in the process do have a tendency to leave both halves broken...

[Anonymous Coward]

Re: DR Test

Kudos to them if they REALLY do DR.. but they should do them 02:00 AM saturday as we did.. so if something goes wrong, only a small amount of clients will be affected.

[Anonymous Coward]

> Using your credit card triggers "fraud detection" which blocks the card? The hell...?

Debit card never used for Internet transactions. Credit card only used for Internet and occasionally when e.g. hiring a car when abroad. Don't think I've ever used a credit card in a supermarket before.