back to article Black Horse Down: Lloyds Banking Group goes TITSUP*

Lloyds Bank, Halifax and Bank of Scotland’s online and mobile banking services have all gone down this morning, leaving a trail of angry customers in their wake. According to DownDetector, the major problems started as early as 9.40am on Thursday morning, with more than 2,000 reports coming in across all three banks. Hundreds …

  1. Aristotles slow and dimwitted horse

    What I guarantee you won't hear...

    What is this... the fourth or fifth TITSUP occurrence for this bunch of cowboys this year? What I guarantee you won't hear from their shoddy PR department is this...

    "This outage was caused by our strategic decision to lay off all of our experienced UK legacy technical and integration teams to save costs and improve shareholder dividends by farming this all of this out to cheap Indian university farmed resources whom have all of the theory and certificates, but none of the experience, context or communication skills."

    1. N2
      Pint

      Re: What I guarantee you won't hear...

      Pint

      one upvote isnt enough

      Lloyds is a crock of shit, they even have cross site scripting on the login page as well as leaving the keyboard live to select your memorable word.... ooops should I have said that?

      1. Anonymous Coward
        Anonymous Coward

        The shittyhorse

        As my wife refers to Lloyds after they "lost" our mortgage application several times, then tried to pursue us for breach of contract when we went elsewhere despite never having completed it or received funds.

    2. TRT Silver badge

      Re: What I guarantee you won't hear...

      I was on hold with First Direct last night. Johnny Cash isn't exactly the sort of music I would have expected to hear given the association with cowboys.

      1. Stevie

        Re: Johnny Cash

        The association I would be nervous about is the fact that Mr Cash did time ...

        1. allthecoolshortnamesweretaken

          Re: Johnny Cash

          Popular misconception. Johnny Cash never served a prison sentence. He did land in jail seven times for misdemeanors, but he stayed only one night on each stay.

        2. Doctor Syntax Silver badge

          Re: Johnny Cash

          Maybe they thought his name was appropriate for a bank.

          1. TRT Silver badge

            Re: Johnny Cash

            I would mourn a Cashless society. Johnny Bonkpay does have a ring to it, though.

    3. John70

      Re: What I guarantee you won't hear...

      Have an upvote because my job was sent over to India just recently and just waiting for the disaster stories to filter through from friends.

  2. Khaptain Silver badge

    The real reason - Please

    Why are we never given the real reasons for outages like this. It could possibly help with the avoidance of further incidents by others. Why do corporates always want to hide their failings, can't they trusted or is it just an ego thing.

    Datacentres are built around redundance; offsite locations, backup, power supplies etc so what is really happening here when entire banking systems go offline.

    1. Anonymous Coward
      Anonymous Coward

      Re: The real reason - Please

      In my recent experience hard infrastructure outages are almost guaranteed not to be the issue, short of someone physically pulling out wires it's highly unlikely to be datacentre related.

      Most outages are caused by software and bugs in integrations and transports.

      These are generally harder to find, particularly if many providers are involved all with vested interests in making the problem someone else's (even if the providers are all internal)

      That's why you never find out, relatively trivial, albeit highly inconvenient issues arise but they become political more than technical therefore not reported.

      Bah humbug...

    2. ZenaB
      Trollface

      Re: The real reason - Please

      Don't worry, I'm sure we'll get a full, detailed analysis of the situation as it happened, just like Monzo provided the other day!

    3. Anonymous Coward
      Anonymous Coward

      Re: The real reason - Please

      "Datacentres are built around redundance; offsite locations, backup, power supplies etc so what is really happening here when entire banking systems go offline."

      Pay peanuts, get monkeys...

    4. jayAyyyy

      Re: The real reason - Please

      I'd go for a massive hack - Silence:

      "At least 10 financial organisations in multiple regions including Russia, Armenia, and Malaysia"

      Are we thinking they only target non-european countries?

    5. Angry IT Monkey

      Re: The real reason - Please

      "Datacentres are built around redundance; offsite locations, backup, power supplies etc so what is really happening here when entire banking systems go offline."

      Erm... hands up anyone who knows a datacentre that charges for resilience but can't provide it.

      Prefix that with "Good" and I agree completely.

  3. Doctor Syntax Silver badge

    "Total Inability To Support Usual Pecuniary activity"

    That's an improvement. Not long ago they were impecunious.

    1. Anonymous Coward
      Anonymous Coward

      Totally Impecunious Tw*ts Show Usual Perfidiousness

  4. K
    Facepalm

    "outsource management of its data centres to Big Blue"

    I worked for another FTSE company who shall remain nameless (Dixons) who did exactly this. Costs more than a network engineers annual salary to get a simple change made.

    Theres a joke here somewhere..

    How many Committees, Project Managers, Auditors, Tickets, Indian call center staff, engineers etc does it take to plugin a CAT5E? I don't know, ask our IBM Account Manager.

    1. This post has been deleted by its author

      1. Anonymous Coward
        Anonymous Coward

        Re: "outsource management of its data centres to Big Blue"

        Thats unfortunately the way of the world - I used to work for the guy who signed the IBM contract, he's still in the job!

      2. Keith Langmead

        Re: "outsource management of its data centres to Big Blue"

        Firing someone just for being crap is very difficult, so if they don't do anything incompetent enough to warrant dismissal then the next best option is to palm them off elsewhere. Give them a glowing reference, help them get a massive payrise, and they become someone else's problem!

        1. Mark York 3 Silver badge
          Coat

          Re: "outsource management of its data centres to Big Blue"

          @Keith.

          It also worked for Gordon Brittas (Stated somewhere with in episode 1 along with their older son(s) who disappeared never to be heard of again).

      3. Anonymous Coward
        Anonymous Coward

        Re: "outsource management of its data centres to Big Blue"

        "who implemented such a draconian process that people installing equipment in data centres had to order Cat5 cables at the right length, rather than make them up on site."

        Nothing wrong with that if you want high grade certified cables. You normally keep an extensive stock of different lengths on site though...

      4. Doctor Syntax Silver badge

        Re: "outsource management of its data centres to Big Blue"

        "The senior management never learn, or they are in another world. How people with a previous reputation of messing up still keep on getting top positions"

        Two possible, and not mutually exclusive, explanations:

        1. They move on on the basis of the projected savings of what they did in the last place.

        2. People value people who look like themselves so wankers in management appoint more wankers instead of workers.

  5. Anonymous Coward
    Anonymous Coward

    Jungle Drums say

    Remaining A/C, for reasons. Was just on the phone to someone in the security community, who mentioned they're hearing it's a breach.

    Unconfirmed, obviously, but this was said to me before I saw El Reg's article. Indeed, I checked here to see whether the rumour had any substance

  6. Anonymous Coward
    Anonymous Coward

    Black Horse Down

    How many years, if not decades have you been waiting to use that?

    1. JimboSmith Silver badge

      Re: Black Horse Down

      I don't know but it was worth it.

      Just wish I wasn't so stuck with Lloyds Banking group for my banking. I had diversified, doing things with Halifax and then they went and bought that.

  7. Anonymous South African Coward Bronze badge

    Now there are things you can outsource and there are things which you cannot outsource.

    Aircon and facilities management (electricity/diesel/UPS etc) can be safely outsourced since those technicians don't always need to be available 24x7.

    However, critical systems, such as databases, NAS units, core routers etc, all of these are critical assets and will need 24x7x365 babysitting, because you don't know what will happen.

    And when the brown stuff hits the big windmill, you are entirely dependent on your outsourced IT team, who will only be able to respond in 3 to 5 hours (or worse).

    Worse still, they will have to drop other clients in order to attend to your brownware period. First they will send their appy or trainees, which will take a shufty at your stuff, brown themselves, and call for more experienced persons to come out.

    WHEREAS if you had your own onsite IT team, they will be able to respond promptly (save for those members who are on leave). And even then they will know who to contact for assistance. And if you have been paying them good salaries and are on good terms with your own IT team, most team members on holiday may be prepared to go the extra mile and try to assist remotely where possible. (But on the other hand, if you're stingy with salaries and bonuses etc, then the members who are on vacation will give you a big GFY ~ which you deserve ~ and leave you to handle the fallout best you can). Also, you have the final call on quality - you can select the best persons for your job, and know what they can (and cannot) do - and have more control over your own IT.

    After all, most beancounters does not realize that "outsourcing" is the same as "timesharing" - the sharing of resources such as IT techs, IT sysadmins and IT gurus... and there's absolutely no control over who the contractor will call in to try and bodge your systems even more. Also, another thing to consider is if an contractor have an emergency to deal with, and all of their IT geniuses are attending to that emergency, and you come along with your own emergency, then you'll have to wait for the first emergency to be cleared first. Or am I wrong here?

    1. Anonymous Coward
      Anonymous Coward

      Aircon and facilities management (electricity/diesel/UPS etc) can be safely outsourced since those technicians don't always need to be available 24x7.

      Experience suggests that UPS and standby are often a problem when you really need them, and aircon always needs to work if you've got a datacentre full of heat to dump. Outsourcing makes sense because the suppliers technicians will have more experience than a captive team, but you do actually need them on call 24/7.

      1. TheVogon

        "Experience suggests that UPS and standby are often a problem when you really need them"

        Experience for me suggests that normal practice is n+1 resilience and regular testing!

        1. Norman Nescio Silver badge

          >> Experience suggests that UPS and standby are often a problem when you really need them

          > Experience for me suggests that normal practice is n+1 resilience and regular testing!

          Experience for me is that that should be normal practice, but sadly, frequently isn't.

      2. Anonymous South African Coward Bronze badge

        I stand corrected, thanks :)

    2. Anonymous Coward
      Anonymous Coward

      What also happens when your company gives Wipro and Tata and the likes your remote support or IT management/control is that they present a set of engineers that give the impression of competency to the offshoring company while the deal is being worked out but as soon as the ink has dried on the contract, those guys get moved to the next contract being negotiated and then hothoused "university" educated newbies get the job of running your servers etc.

      Even if they don't swap out your competent IT dudes/dudettes, as soon as they get trained on new products he/she is straight to the job market to find another job/company and a raise.

      12 months after my company got wipro doing support, only about 20% of the original guys were still working for Wipro, the rest were wipro in house 'trained' and they were fúcking useless. How did we find this out?, some of our managers (pissed off with increasing wait times and poor outcomes) and a few of the people sent out there to train them in the first place went out there and saw this for themselves.

      I have been told that the big 5 offshoring companies are cracking down on staff jumping ship looking for raises, but my experience shows a different story.

      Finally, some of our in house applications are being supported by indian callcenter types now, but the people who used to do that were the people who wrote the software, who have been given the boot. So yeah, no updates for months and months, no matter how big the bug. Including security holes. Popcorn is waiting.

    3. a_yank_lurker

      @ASAC - The problem with dumb sourcing is it done to 'save money' in the budget for something like salary. But what is overlooked, is when the brown storm hits, you will almost certainly be hit with some very significant charges that escalate depending on scope and speed to solve. These charges probably wipe out the salary savings but they are in a different line item.

      There was an old TV ad over here for automotive oil filters with the tag line "Pay me now or pay me later". The point of the ad was spending money upfront is often much cheaper than spending money later. Same principle applies here, hire quality locals as internal staff will cost money now but the crisis will be less costly, if it occurs.

    4. Phil O'Sophical Silver badge

      After all, most beancounters does not realize that "outsourcing" is the same as "timesharing" - the sharing of resources such as IT techs, IT sysadmins and IT gurus.

      Or as we call it today "The Cloud".

  8. Anonymous Coward
    Anonymous Coward

    There may be trouble ahead

    This is what scares me about the elites plan to make us go cashless in the future. These outages would place an even bigger impact on us. At least we have cash we can still spend regardless of the incompetent management of the banks.

    1. Dan 55 Silver badge

      Re: There may be trouble ahead

      Well, you are allowed to choose a bank which isn't HBODGE.

      1. Ken Hagan Gold badge

        Re: There may be trouble ahead

        "Well, you are allowed to choose a bank which isn't HBODGE."

        Are you? Please identify the bank that hasn't outsourced any of its core competencies.

        1. Doctor Syntax Silver badge

          Re: There may be trouble ahead

          "Please identify the bank that hasn't outsourced any of its core competencies."

          It's usually only the IT core competency that gets outsourced. They hang onto the other half of the business, the casino.

        2. Dan 55 Silver badge

          Re: There may be trouble ahead

          There aren't any, but if you choose to bank with Lloyds, Bank of Scotland, Halifax, or Natwest then you know that it's going to go down more than [insert favourite end to saying here].

          Past performance may not be indicative of future results, but in this case they probably are.

        3. EnviableOne

          Re: There may be trouble ahead

          Got to go with a building society then

          Newcastle Insource everything and even do other ppls

    2. JimboSmith Silver badge

      Re: There may be trouble ahead

      That was Allied Dunbar (who used "There May Be Trouble Ahead" in their adverts) not Lloyds but I agree entirely with the rest of your post. I ended up paying a cab driver with a $20 bill because I'd only just arrived back in the country the previous day. The PDQ (card machine) in his cab was working fine but he couldn't get a connection to process the transaction. I also couldn't get a signal on any of my phones. When I asked what he normally did in these situations he said they drive to a cashpoint (™Lloyds Banking Group) Hole in the Wall (™ Barclays) etc. As I didn't have time for all that we worked out the exchange rate and decided that the fare was about $20 including a small tip.

      1. Anonymous Coward
        Anonymous Coward

        Re: There may be trouble ahead

        >Allied Dunbar

        Who we used to call "Allied Crowbar" for their ability to remove money from your wallet without offering any perceptible benefit.

  9. EddieD

    Salutory warning

    It shows how delicate the electronic finance infrastructure can be, and why the dream of a cashless society is probably best tempered (for the time being) with a healthy dose of scepticsm and a wad of twenties in your wallet.

    1. Ken Moorhouse Silver badge

      Re: It shows how delicate the ... finance infrastructure can be

      I had a heated discussion with my bank today (not one of the shower mentioned). Renewal Debit Card did not arrive despite being posted (they told me) two weeks prior, so I rang to tell them so. They are sending me a new one. In the meantime they've seen fit to cancel not only the one lost in the post, but my current one too! It seems that their system can not differentiate. Muppets.

  10. Bob the Skutter

    Was using lloyds banking app fine this morning and Halifax one has just let me in ok

  11. Anonymous Coward
    Anonymous Coward

    You get what you give and in this case the the birds are coming home to roost. This is not a fun thing for some people as they can't pay bills etc. I've already had someone elderly (who is totally non technical) call me asking "if the internet is broken cos the bank doesn't work on my phone".

  12. Anonymous Coward
    Anonymous Coward

    Loving the comments about "you get what you pay for" when no one commenting knows what the problem is/was (or if you do, then tells us you do instead of speculating).

    I generally agree outsourcing/offshoring creates more problems than it fixes, but lets stop jumping to conclusions and beat the bean counters around the head with facts, not just noise.

    1. Salestard

      Well that's kinda relying on banking megacorp doing a Monzo and actually coming clean about what the problem was/is, rather than just chucking some offshore bods under the bus.

      1. NinjasFTW

        Except you know it wont be offshore bods thrown under the bus because that would mean that they have to admit that offshoring causes issues.

        Its more likely to be blamed on one of the few remaining guys as a justification to move the rest offshore.

        Never let a good crisis go to waste and all that

  13. chivo243 Silver badge

    Speaking with two mouths

    "We are aware that "some"* of our customers are experiencing problems logging in to internet and mobile banking.

    I love the downplaying of issues by the PR spinners. I remember being told to tell the truth, and the whole truth ;-}

    *"some" of the customers are just the ones that tried today!

  14. Keith Langmead

    PR responses

    "Many PR departments up and down the country these days prefer to delay responses until they've fixed the cause of the outage."

    Hopefully because PR don't have the actual facts available to them, and those who do are busy finding the issue and fixing it rather than briefing others, and have no real way to know how long it'll take to find the cause until they've found it! Fix first, analyse causes later.

  15. Anonymous Coward
    Anonymous Coward

    disgruntled customers took to anger echo chamber Twitter

    in this day and age, if you're not on facebook or twitter, you can anger echo chamber down your toilet to make you feel better. Oh where are the angers of yester year when I was able to waste £££ trying to anger echo chamber my frustration down a BT handset, only to hear, after 46 min of "stayin' alive" by beegees, that special, affectionate sound of line getting - suddenly - disconnected...

  16. JimmyPage Silver badge
    Holmes

    On the day the BoE base rate goes up ?

    I wonder if they are running a piece of software somewhere that is less than 10 years old, and not been tested against a BoE rate rise ?

    1. f4ff5e1881
      Joke

      Re: On the day the BoE base rate goes up ?

      I think you might be on to something there. I hear that Lloyds’ RM Nimbus has always had a blind spot with percentages…

      1. Peter Danckwerts

        Re: On the day the BoE base rate goes up ?

        When I last visited a manager at Lloyds Bank, which wasn't that long ago, his computer was still on Windows XP. They probably use one to run their websites – or perhaps they are using a Sinclair QL. I can't imagine them buying anything as classy as a Nimbus.

  17. Anonymous Coward
    Anonymous Coward

    re - Anonymous South African Coward

    Employees of HBOS/LBG tuped across to IBM (Sept) and afaik they still perform the same roles, so the resources are on-site as they were before it was outsourced to IBM (with application development and some infrastructure support to WIPRO before that).

    As has already been said the most likely cause is user related caused by change but LBG will never admit to truly what caused the issue and will likely blame it on some faulty process.

    A/C - previous employee

  18. Anonymous Coward
    Anonymous Coward

    Makes a change

    It's normally RBS group having the IT problems

  19. Anonymous Coward
    Anonymous Coward

    TheBorg

    Well well .... outsource and it all goes tits up !!

    Jacqueline Guichelaar - CIO of infrastructure and technology services, has a lot to answer for :)

    She did this at Deutsche Bank and shafted all their employees as they were TUPE'd across to HP/HPE/DXC .... then she promptly buggered off and joined Lloyds before it had time to start to smell

    1. Alistair
      Windows

      Re: TheBorg

      Apparently Jerry Brace is her mentor. She'll be off to Asia or such next.

  20. Paul Herber Silver badge

    We apologise for any inconvenience

    Oh, hello Fenchurch, I didn't see you there.

  21. psychonaut

    compensation

    "We don't have a time frame at the moment, sorry. ^BJ"

    that tweet was very odd.

    we dont have a time frame at the monment, sorry, please have a blow job?

    reminds me of the national friendly bank

    wanky wanky wank

  22. Selden

    ATMs running XT?

    In the United States, an alarming percentage of ATMs are still running Windows XT. Is the situation any better in the UK?

    1. hplasm
      Coat

      Re: ATMs running XT?

      Not much better - but at least XT is 5 releases newer than XP...

    2. Ken Moorhouse Silver badge

      Re: ATMs running XT?

      https://en.wikipedia.org/wiki/IBM_Personal_Computer_XT

      Are you sure about that?

  23. greenwood-IT

    Logically I'm more interested in the common link between these banks - they all have outages at the same time???

    Are they all outsorced to the same partner, are they all in the same data centre, do they all run the same codebase? Perhaps even someone has attached the UK banking industry?

    As many say, we will never get to hear the truth - it would make someone high up look bad and ruin their future earning potential (in an ideal/dream world).

    In the 80s when I asked why senior managers had IBM PCs and lower grades had Zenith I was told "nobody ever got sacked for buying an IBM" - I'm guessing this rule still applies for cloud based services.

  24. DBAosaurus
    FAIL

    Horto Osario's latest shag

    The rot at LBG set in long before the IBM outsource. I can never forget or forgive them for announcing one set of redundancies and layoffs in 2015 as 'a strategic replatforming to better align with business towers'.

    There will be a special pit in the seventh circle of hell reserved for anyone who can think, talk or communicate like that.

  25. unwarranted triumphalism

    Works for me

    Don't see what all the toys are being thrown out of prams about. Time you had a nap before going back to pretending to be adults?

  26. Bill_Sticker

    Ah, upgrade time, eh?

  27. Gruumpyoldman
    Unhappy

    ATM Owned by RBS Still out after 5 days.

    For the last 5 days the ATM at York Parade Tonbridge has been out of order guess who owns it RBS via Ulster bank and of cause the Postoffice Counters. They told me I could get my money out via the card reader at the counter so joined the long line as normal only one person available to serve everybody.

    Guess what they don't except Tesco Debit Cards or even Credit cards so wasted almost 20 minutes.

    In the end had catch bus to High street about 15 minute wait then 15 minute journey, get my money and travel all the way back again.

    So l wasted most of the morning just because nobody can be bothered to fix the machine. What really winds me up is that at least once a week this ATM does not work.

    In a way I hope it's not repaired so that RBS does not get any revenue from it.

    But it is the only ATM in the small shopping area and lots of people always need to use it.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like