back to article Black Wednesday: DXC hosting services wonky for almost 8 hours after core switches go rogue

The dust is settling at DXC Technology following an outage at its Wynyard colocation site in England yesterday that hosts services for a bunch of FTSE 50 financial services clients, insiders have told us. Primary hosting is done at DXC Wynyard with disaster recovery provided by the DXC Doxford data centre. Before lunch at …

  1. m0rt

    "El Reg asked DXC about the outage but the company declined to comment. ®"

    So, is this what is meant by the 'Apple effect'? :D

    1. Anonymous Coward
      Anonymous Coward

      Driving Xtreme Cuts. Lots of people there still cared, were doing their best. Senior management are one trick nags.

  2. Anonymous Coward
    Anonymous Coward

    Perhaps the cause can be found here

    https://www.theregister.co.uk/2019/05/24/dxc_fy2019_q4_results/

  3. Miss Lincolnshire

    Did.....

    ....the local peripatetic driveway tarmacking community nick the copper link that goes over the A19 again?

    1. David 132 Silver badge

      Re: Did.....

      No idea who you're referring to. Now, don't make me laugh, my side's been giving me gyp. Oh! to be young again.

    2. tip pc Silver badge

      Re: Did.....

      Do you need to translate for the over seas community to understand what you mean?

    3. Notovoda

      Re: Did.....

      Yes, they were digging tarmac up when I passed yesterday. Pick axes and pneumatic drills were in use.

      Low tech beats high tech.

  4. Anonymous Coward
    Anonymous Coward

    TITSUP...

    Totally Inexperienced Technicions Succumb Under Pressure

  5. Doctor Syntax Silver badge

    Perhaps they should outsource it.

    1. David 132 Silver badge
      Devil

      Don't worry, all their in-house, in-country, experienced staff who are at advanced points in their careers will leap onto it with enthusiasm, fired up by their sheer love of working for a company that values them so very much.

      1. neilo

        Fun fact: Us experienced ones who have got the hell out of this rotting carcass have noticed something interesting: the hand-off / knowledge transfer is not happening, or if it is happening it's rushed an incomplete.

        When I gave notice, I was left alone to finish up what work I could, with a rough customer hand-off dome for one client on the last day.

        When a co-worker (a solution architect) gave notice, she was left alone. Then in the last few days it was realized she had 35 current client projects and then all hell broke loose. The hand-offs were rushed and incomplete again.

        There is a real "head in the sand" approach going on here... and it's accelerating the clients that are leaving because they are not being looked after!

        1. Doctor Syntax Silver badge

          "the clients that are leaving because they are not being looked after!"

          How dare they? Clients expecting something for their fees? How can DXC run a business if clients keep doing that?

        2. tip pc Silver badge

          Bean counters don’t value the revenue attributed to happy customers. From what I’ve seen unhappy customers spend more trying to fix problems their outsourcer insist is not their fault when the customers insourced staff would have easily fixed stuff for vastly less cost as they know the systems and have a vested interest in sorting stuff.

  6. James Anderson Silver badge

    Do not call anything Backup!

    Leaving aside the obvious management foo bar of increasing this quarters profits by sacking your most experienced staff, and, the pervasive incompetence which results from publicly announcing you hate your staff.

    Even for a well run site having a 'Backup' anything is asking for trouble.

    As soon as something is declared as the backup it gets ignored, testing is skimped, fixes are missed and switch over procedures go stale.

    What to do -- run without backups?

    No you call one site 'Red', ' South', and the other site 'Blue', 'North' or some similar name of equal semantic value.

    The during the time of your lowest load (say second Sunday in the month) you switch from one site to the other -- every month!

    There are numerous advantages to doing it this way.

    -- Your staff no how to switch over 'cos they do it every month.

    -- procedures are kept up to date and regularly streamlined.

    -- Small faults and glitches are found as they happen, not the tsunami of bugs which have accumulated since the last switch-over two years ago.

    The other advantage is it simplifies maintenance and upgrades.

    You apply changes to the dormant site and they take effect on switch over.

    If all is well you can then apply the changes to the newly dormant site, if, not

    you switch back and try again next month.

  7. Jay Lenovo
    Flame

    A Brighter Future

    DXC: "Ok, we'll prove those El Reg naysayers wrong! ...starting tomorrow.. or maybe the next day. Got a new guy in charge of that now, I forget the name."

    1. Chris King

      Re: A Brighter Future

      Bungee Boss ?

  8. fredesmite2

    Remember. Cloud Computing ...

    Is nothing but running your crap on someone else's computers.

  9. IGotOut Silver badge

    Maybe they should move it to the cloud

    And leave it to those fresh faced graduates to fix it.

    What do you mean by "real world experience"? Surely everything they use is the latest kit, all from the same manufacturer, all on the same firmware levels and with fantastically written documentation.

  10. Anonymous Coward
    Coat

    Danger!

    See, they're right to ban Huawei after all. Look what could happen if some evil commie government could disrupt our core switches!

    <small>Um, whose switches was it?</small>

    1. Anonymous Coward
      Anonymous Coward

      Re: Danger!

      Hint on the switches, it was an IRF problem ;).

      Its ok "many pairs of eyes" will be watching during the change window to restore resilience again next weekend.

      I'm still amused/saddened we got more info from here first than we did internally on the issue!

  11. Anonymous Coward
    Anonymous Coward

    I always get confused between DXC and DIC (https://en.wikipedia.org/wiki/Disseminated_intravascular_coagulation)

    Which one is it that causes acute pain followed by death?

    1. Steve K Silver badge

      Trick question

      Trick question - it's both?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020