back to article Southwest Airlines blames IT breakdown for stranding holiday travelers

Winter storms and staff shortages were only the tipping point that sent Southwest Airlines IT infrastructure over the edge, leaving thousands still stranded across the US, chief operating officer Andrew Watterson has explained. In a call with employees, Watterson blamed the extended delays and cancellations on outdated …

  1. Kevin McMurtrie Silver badge

    Failed while failing

    Southwest received $7.2 billion of public subsidies to help them get their act together. Maybe it's time to look for criminal misuse of funds? Throwing money at contractors and cloud providers, if that was even done, doesn't count as money properly spent.

  2. david 12 Silver badge

    Blame the Computer

    After running closer and closer to the wall, eventually we realized that we just didn't have enough crews and planes to run scheduled operations.

    So we had to abandon our scheduled operations, and run a skeleton operation while we did crew off-time and aircraft maintenance.

    Then, since our scheduling system wasn't able to schedule 5 flights into 6 flights, we blamed the software.

    SouthWest has a specific problem: a larger-than-normal fraction of their flights are loops, rather than out-and-back flights. With an out-and-back flight, when one airport is closed, and you cancel one "out" flight, the corresponding "back" flight is also cancelled, and you still have a plane and crew in position. With a loop schedule, when one leg is cancelled, the entire loop is cancelled, and the plane and crew isn't in position for off-time and maintenance. They also have a general problem: there isn't any fat in their operations, and they don't have enough spare crews to cover crew down-time and aircraft positioning when the schedule is badly disrupted. Blaming this on the scheduling software is disingenuous.

    I did a simple optimization problem for cutting shapes out of sheet stainless-steel. I understand that there is no closed-form solution for allocating the position of shapes withing a shape. Sometimes you wind up with off-cuts where the total area of offcut is larger than the next small piece you want to cut. But your offcuts are thin jagged pieces, not the circle or square you need next. But at some point, a "better optimisation system" isn't what you want. What you need is more sheets of stainless steel.

    1. Yet Another Anonymous coward Silver badge

      Re: Blame the Computer

      It's the trade off, you want cheap flights the airline can't have a backup aircraft and crew fro each flight.

      1. Malcolm Weir Silver badge

        Re: Blame the Computer

        Ah, but Southwest doesn't really offer cheap flights. They market their operation as if they do, but with the exception of a few loss-leaders (the $49 fares they'll advertise every so often), the actual prices charged are generally well within the realm of what the other can carriers charge.

        There are several "gotchas" to that claim, of course: WN doesn't charge for the first two piece of hold baggage, so that's a savings of $120 on a round-trip compared to DL/UA/AA, but only if you would have paid for checked baggage. And WN also doesn't allocate seats, so if you want something approximating a specific seat choice, you'll probably have to pay for the "Early Bird" auto-check-in option, which adds $30 round-trip. WN loves to operate from "atypical" airports (Love Field in Dallas, Midway in Chicago, etc) which can be a mixed bag: fewer options if things go wrong, typically fewer hotels and rental cars, but often easier to access (at least by car).

        Southwest is not an operation like Ryanair (they don't actively loathe their customers), but a lot of their low-cost-model comes from operating a point-to-point network, which often avoids having to route passengers in the wrong direction. And a lot of their current fragility comes from the same thing...

        1. Yet Another Anonymous coward Silver badge

          Re: Blame the Computer

          I assume in a capitalist country you have a range of many excellent premium airlines who all offer exemplary service for the right price (or you're stuck with United)

        2. abetancort

          Re: Blame the Computer

          Southwest was the inspiration for Ryan Air operation model but Ryan Air failed to take notice of Southwest's customer service.

          1. Anonymous Coward
            Anonymous Coward

            Re: Blame the Computer

            Nope. Ryanair (spit!) did know about Southwest's customer service. That cheapskate gobshite bollox O'Leary decided Ryainair (spit!) wouldn't do that. Lower overheads for his buses with wings meant juicier profit margins and therefore more money for himself as a major shareholder.

    2. Merrill

      Re: Blame the Computer

      I wonder whether Southwest, with their point-to-point route structure, depends more on the spare capacity of the other hub and spoke airlines to move their crews from home to base and base to flight? In the widespread storm conditions, the other airlines have no spare seats for Southwest crews traveling on standby.

      1. jake Silver badge

        Re: Blame the Computer

        Can't fly out on another airline when everybody's grounded.

        Normally, there are other aircraft in the loop being made ready to fly when a crew is rested and ready.

        Why am I reminded of an old Nixon for President bumper sticker?

    3. anothercynic Silver badge

      Re: Blame the Computer

      This. What makes Southwest exceptionally efficient in normal conditions makes them very vulnerable when the weather completely makes a hash of everything.

      JetBlue had a similar problem a few years ago (in 2014 IIRC), and they learned from that and changed how they did things whilst still trying to stick to their model. Sounds like Southwest will need to wake up and smell the coffee and do something similar, or face the possibility of things going south again the next time continental blizzard conditions hit (and given weather changes, it's going to happen more often).

      1. An_Old_Dog Silver badge

        Re: Blame the Computer

        Southwest was ultimately forced to schedule crews to flights manually. A process Watterson described as an "extraordinarily difficult" and "tedious, long process."

        And that's a management failure right there: the failure to have effective alternate (non-computer) methods.

  3. A Non e-mouse Silver badge

    [Southwest] also acknowledged the need to upgrade the airline's infrastructure to avoid future IT breakdowns

    Management always put off infrastructure upgrades until it affects their bonus when the infrastructure fails.

    1. Anonymous Coward
      Anonymous Coward

      My bet is that any upgrades have been determined by management rather than the IT department (if they still have one)

      1. ecofeco Silver badge

        You would win that bet every time.

    2. abetancort

      They are upgrading their Excel from 2000 to 365.

    3. chivo243 Silver badge
      FAIL

      "We need to double down on our already existing plans to upgrade systems for these extreme circumstances so that we never again face what's happening now,"

      B-U-L-L-S-H-I-T BINGO!

  4. Anonymous Coward
    Anonymous Coward

    Cloud

    Flight management systems running on Azure and AWS. What could possibly go wrong...

    1. Anonymous Coward
      Anonymous Coward

      Re: Cloud

      You’ll find it’s more about the GDS than Azure.

    2. Gene Cash Silver badge

      Re: Cloud

      It seems to be working for 'em... better than the self-inflicted hellhole Southwest is in.

    3. Anonymous Coward
      Anonymous Coward

      Re: Cloud

      Yeah, at least the article points out which airlines to watch out for over the next bit of time. AWS migration is one of those things that does tend to go wobbly, but hearing about that ground up rebuild on M$ scares me more. Big bang upgrades have a poor track record at that scale. I wish them the best of luck, but I will be booking on an alternate carrier during their transition...

  5. Anonymous Coward
    Anonymous Coward

    Very simple explanation...former CEO Gary Kelly

    Long time very happy SWA customer here. Going back almost 40 years.

    Herb Kelleher built the best airline in the world bar none in the decades he ran SouthWest. It was a genuine pleasure to fly SWA and everyone who worked for SWA not only seemed to love their job but made you feel really welcome. Only British Caledonian in my experience even started to come close in such a positive cabin crew / ground crew experience.

    Then Herb retired and soon after this bean counter named Gary Kelly took over. Think of him as another Willie Walsh. But without the charm (yes, that bad). And very quickly stories started appearing of how Kelly and his people were treating the employees in an increasingly shoddy way. As long as Herb was still alive Kelly was under some restraint in how badly he treated the staff but once Herb died Kelly was soon treating the staff as badly as almost all the other big carriers.

    Kelly and his people payed themselves huge salaries / bonuses then spent billions on "stock buybacks". Which tells you everything you need to know about his priorities. Then he bailed last year and handed the big stinking mess on the point of collapse over to a guy named Bob Jordan who looks like a genuine good guy but who was the designated fall guy for the Kelly years which destroyed SWA.

    The current and previous SWA collapses have a lot in common with the never ending BA debacles. But the best analogy is the way GEC which had been built up by Weinstock over decades into a huge success and was totally destroyed in only a few years by his successor, George Simpson. Another bean counter. Built up in 40 years, gone in 4.

    The people I feel sorriest for are the employees of SWA. To see the company they dedicated their lives to, and for which they worked so hard for, utterly destroyed by some greedy nasty worthless bean counter. Next time I have to book a West Coast flight it will be with SWA. As an act of loyalty to such a fabulous workforce who have made what is usually a miserable experience, flying US domestic, an always pleasant experience. No matter how bad a flying experience it proves to be it will be still better than flying United, AA etc.

    Since that first day in the mid 1980's when I handed over my $35 for a ticket to LA and got a supermarket checkout style receipt and a very flimsy boarding-card and I thought, well this is going to be different. And it was, it a very good way. For the next four decades.

    1. ecofeco Silver badge

      Re: Very simple explanation...former CEO Gary Kelly

      Excellent explanation.

      Thanks. I wondered what happened to SWA. I was pretty sure it was management change, as that is usually the case.

      And lo and behold, you've proven it was.

      1. Yet Another Anonymous coward Silver badge

        Re: Very simple explanation...former CEO Gary Kelly

        And competition from both majors consolidating and offering cheap connections with their affiliates and other budget airlines copying the playbook.

        You can't just say, if only they didn't care about money but continued to pay well and offer cheaper fares they would be great!

        1. Richard 12 Silver badge

          Re: Very simple explanation...former CEO Gary Kelly

          Stock buybacks are the massive red flag.

          If they've got the cash to do that, then they've got the cash to invest in the business.

          Stock buybacks achieve only two things - increasing the share price, and giving a select few shareholders a large windfall.

          They are simply a way to take cash out of a business.

    2. Malcolm Weir Silver badge

      Re: Very simple explanation...former CEO Gary Kelly

      Kinda, but you've overlooked the two mergers in the 2000s: bits of AirTran, and all of AirTran f/k/a ValuJet. Neither of those had had the benefit of Herb's culture-building, so you got an influx of people unused to "the Southwest way", which diluted the perceived value of the upbeat culture.

      The critical problem for any airline is that they can't put a price on genuine service, because some other airline will come along and undercut them (hi, Michael O'Leary). And by service, I mean a lot more than bone china for your mixed nuts! Things like seat availability, schedule convenience, minimized lay-over times, flexible ticketing, and so on all count.... but absolutely nothing counts more than how the organization responds when things go wrong.

      Which brings me to observe that, no, flying Southwest will NOT be better than flying DL/UA/AA. Sometimes it might be, but as this week's news shows, it can often be much, much worse.

      1. Gene Cash Silver badge

        Re: Very simple explanation...former CEO Gary Kelly

        they can't put a price on genuine service

        Bingo. I've never heard it better stated.

        I have several friends^Wacquaintances that buy the cheapest shit, then moan incessantly that it's the cheapest shit and now broken the next day. These are the people that buy Ryanair tickets and then bitch they have to pay for luggage and their knees are tucked up under their chin.

        If I had one wish, I'd wish all these people would have instant heart attacks. They're ruining the world.

        1. Anonymous Coward
          Anonymous Coward

          Re: Very simple explanation...former CEO Gary Kelly

          A little harsh, but I think I can get behind your solution.

      2. Anonymous Coward
        Anonymous Coward

        Re: Very simple explanation...former CEO Gary Kelly..the mergers and competition

        Yeah I know about the problems with the mergers. But it mostly worked out. Now given the history of mergers of US regional carriers over the last 40 years and just how low that sets the bar for "success" the SWA mergers were a resounding success. I know people who are still bitter at losing their local carrier 30 years ago. And rightly so. Come to think of it, just look what BA did to BCal. And I still miss BMI. Nice people. Bloody Germans..

        For the bottom feeder passengers in the US, the people who used to take Greyhound, there is Spirit Air and Frontier. That was never SWA's target market. The competition from the legacies in the cheap and cheerful end of the market is very spotty. Usually just special deals. SWA only has one real competitor, Jet Blue. Who are quite good. Flew them a bunch of times on code shares. Good experience.

        But the legacy airline, the majors, with the exception of Delta, have all the utter nastiness of RyanAir with the added bonus of the employees who hate their jobs. If you want a pretty pleasant flying experience its basically SWA, Delta and JetBlue. Continental were also pretty good until United destroyed them too

        When it comes to reproducing the experience of flying on Soviet era Aeroflot then United cannot be beat.Yes, there is an airline even more unpleasant than RyanAir. Except in this case it is not intentional.

        When I buy my own domestic tickets it going to be SWA. Or Alaska. Although I still remember the Flight 261 accident. And the why. So always a last resort. When other people buy the tickets or its a code share I get to sample the delights of all the other carriers. Jet Blue and Delta. Good. All the rest, brace yourself for a terrible experience.

        As for this week. The storm was predicted two weeks out. It was Christmas. And there were already serious staffing issues in all parts of the industry. Three very good reasons not to fly anywhere in that 10 day window. If its too far to drive, dont go. For me anything under 850 miles is driving distance. Door to door 12 - 14 relaxed hours of travel beats door-to-door 8 or 9 hours of not very relaxed travel. Plus no cost of a rental and none of that utterly pathetic TSA security theater BS.

        So its basically drive or SWA. Or Delta or JetBlue as a backup..

        1. vcragain

          Re: Very simple explanation...former CEO Gary Kelly..the mergers and competition

          If you take into account the amount of aggravation & the time getting thru all the procedural stuff at airports it is never quicker to fly anywhere unless the drive time is more than 8-10 hours or so, or an overseas flight of course with little choice ! I've been on several work trips where I seriously regretted not getting in my car to get to that meeting !

          1. Michael Wojcik Silver badge

            Re: Very simple explanation...former CEO Gary Kelly..the mergers and competition

            I drove from the Rockies to the Atlantic this summer, and back, rather than fly. And I'd make that choice again.

        2. Commswonk

          Re: Very simple explanation...former CEO Gary Kelly..the mergers and competition

          SWA only has one real competitor, Jet Blue. Who are quite good.

          IMHO this JetBlue pilot is extremely good:

          https://www.youtube.com/watch?v=epKrA8KjYvg

          1. Anonymous Coward
            Anonymous Coward

            Re: Very simple explanation...former CEO Gary Kelly..the mergers and competition..Jet Blue

            That was one hell of a landing. Fantastic job by the pilot and co-pilot.

            The other thing I noticed was what a great job the KTLA chopper pilot and cameraman did in tracking the aircraft all the way down. They were flying in some very busy airspace around LAX. I think it was that chopper crew or the guy from KCBS who did the jaw dropping job of live coverage a few years ago of CalFire helicopters doing close in water drops on the ridge-line of Sepulveda Pass, in the dark, before sunrise, with high winds, during very serious wildfires. I've driven that pass on the San Diego Freeway many times over the years and the crosswinds can be brutal. At ground level. So up on the ridgeline during a wildfire.. Truly awe inspiring airmanship. And because of the bravery of the CalFire pilots they broke the back of that part of the fire front so stopped a repeat of the early 1990's fire in that area. Which burned all the way to PCH on the coast.

            One of the good things about flying in the US is that so many of the pilots are still ex-military. Became on the rare occasion something goes wrong having a guy up front who has usually dealt with everything that flying can throw at you (not to mention incoming ordnance) gives everyone on board that extra edge to making it safely to the ground.

            1. Commswonk

              Re: Very simple explanation...former CEO Gary Kelly..the mergers and competition..Jet Blue

              That was one hell of a landing. Fantastic job by the pilot and co-pilot.

              If you liked that then you'll like this... even if it is a bit off - topic for this thread.

              https://www.youtube.com/watch?v=9KhZwsYtNDE

    3. Michael Wojcik Silver badge

      Re: Very simple explanation...former CEO Gary Kelly

      I only few Southwest a few times (four, I think), but I did not like it at all. And that was well before the Kelly era – he took over in 2008, according to Wikipedia.

      I've never understood some people's fondness for Southwest. When I used it, the ground crews were friendly but did not enforce boarding priority, which was great for people who abused the system and lousy for those who tried to follow the rules. And the lack of seat assignments meant it was then difficult to sit with my companion for the flight.

      There were a number of rows with one person (always a man, shockingly) claiming "someone" had already taken the adjacent seats. (These were full flights; I don't know what they were hoping to gain, unless they thought some attractive woman would be forced to share their company for the flight. That doesn't make me any more inclined to fly Southwest.)

      Free checked baggage ought to have relieved the idiotic fighting over overhead-compartment space, but if memory serves it didn't, particularly. (I always check my bag, as I hate to squabble over space. It just makes the experience that much worse.)

      I haven't flown for a few years and don't miss it a bit, but back when I flew frequently, I'd always prefer a crew that enforced the rules over a cheerful one that let assholes get away with bad behavior.

      1. Anonymous Coward
        Anonymous Coward

        Re: Very simple explanation...former CEO Gary Kelly..different priorities

        I've probably flown more than 2 dozen flights on SWA over the last 40 years. And flow about the same number of airlines. Maybe more. Most of my flying is long haul international. Never the slightest problem with SWA. All my flights were happy flights. But all were West Coast.

        I'd guess you dont fly much as I never ever check in baggage unless I am doing a long trans Atlantic hop and I need to move stuff. I will go to great lengths to never check in baggage. Even once took on an Air France check-in woman with a bad attitude in CDG in Paris who claimed my legal carry on size bag was "too big". Won that one. By being French. Any heavy use flyers will know exactly why. One bag packing and all valuable on ones person at all times. And dont pack anything in checked in baggage that will cause serious problems when lost. Of the five check-ins I have done in the last 20 years. One was lost for a few days and a second almost lost (for a long time) except I was watching the luggage tagging very closely at check in time

        I'd also guess you probably dont do too much long haul. To me a 4 or 5 hour domestic is a short flight. If you have been doing the West Coast to UK run even longer than flying SouthWest. Although the 12 / 14 plus hours non stop gets old after a few decades so now I always stop over on the East Coast. For a quick coffee and to stretch my legs. On the domestic leg is when I get to try out the various US carriers. In the last decade United and AA were never again. Neither were that bad 20 or 30 years ago. JetBlue and Delta were very good. And the rest ..bleh..

        Now if you want the worst flight ever (even worse than RyanAir) then it does not get any worse than 13 hours on a Pan Am 747 London to San Francisco a few months before Pan Am went bankrupt. Would have been 1990. The cabin crew were angry and surely. Smoking was still legal on flights and it looked like a smokers convention was on board. The aircraft was falling apart. As were the seats. And the pilot decided to take a short cut over Canada to save fuel. Right into a very large thunderstorm system. And an hour or two of roller-coaster quality turbulence. In penance the pilot made the absolute smoothest and gentlest landing at SFO I have ever experienced. Just floated down.

        So SWA - your mileage may vary. And that very much depends on just what you are comparing it with. Me I will always have RyanAir. Although a friend tells me Spirit Air are even worse.

  6. Anonymous Coward
    Anonymous Coward

    Doesn’t take Sherlock Holmes….

    … to figure out which GDS provider they use.

    1. Anonymous Coward
      Anonymous Coward

      Re: Doesn’t take Sherlock Holmes….

      Exactly - I made this point earlier. There’s a reason most other airlines go elsewhere - that’s not to say they’re perfect either. Too much legacy crap for anyone to untangle or replace it would seem.

  7. Yet Another Anonymous coward Silver badge

    Some FAA flexibility ?

    So it's fine for your crew to have spent 24hours commuting to get to this flight. But if they go over flight hours while sitting on the plane waiting 2hours for deicing you all have to go back to the gate, unload and wait for another 'fresh' crew to spend 24 hours battling through snow storms to get there.

    I wonder if there is some scope for emergency rules in bad weather, if they can be written so they don't become the normal working conditions

    1. Bubba Von Braun

      Re: Some FAA flexibility ?

      Those rules are written after decades of experience.

      Most (read All Airlines) would love to re-define duty hours, as the risk posed by tired crew is seen a cost of doing business.

      Problem is the impact is measured in lives.

      BvB

    2. IceC0ld

      Re: Some FAA flexibility ?

      only the military can 'bend the rules' dependant on bad weather, and then it is usually only used in times of combat

      the rules have been carefully put together to give the best outcome for the given circumstances, and, as stated below, mistakes in air rules are measured in lives :o(

    3. Malcolm Weir Silver badge

      Re: Some FAA flexibility ?

      No, there is zero scope for flexibility in bad weather. Bad weather is why the rules are written the way they are. Air Florida Flight 90 and Tenerife are the poster children for this: if the weather had been good, nothing would have gone seriously wrong... it wasn't, so over 650 people died!

      There are, of course, options that Southwest doesn't use because it costs money. The obvious one is operating empty flights (with only crew members on board). These would only require the pilots flying the aircraft to be "current", but the FA's and off-duty pilots in the back can be fast asleep for all the FAA cares. While drastic, this would allow the schedulers to start from a much cleaner sheet, and would clear up the mess much faster. (To be fair, it seems that the magnitude of the mess was not immediately obvious, so the time for such drastic measures might have already passed).

      1. Yet Another Anonymous coward Silver badge

        Re: Some FAA flexibility ?

        I thought Tenerife was the textbook for why rigid rules are bad = we have to take off in the next 15 minutes or we will be out of hours and stranded 10,000km from home without a replacement crew - so there is a push to believe you heard the takeoff clearance.

        Yes working hours affect safety but there are some ridiculous rules, you spent 24+ hours travelling constantly to get form your home to your home base, that doesn't count as working hours. You sit in the cockpit with the engines off at the stand doing paperwork, does that count? You sit in the de-icing queue for 2 hours with the engines off, does that count?

        1. Malcolm Weir Silver badge

          Re: Some FAA flexibility ?

          On Tenerife, no, or at least not anything like directly.

          The issue with the KLM pilots' duty time restrictions had nothing to do with the flight that they were operating, but with the _next_ flight!

          KL4805 was operating AMS-LPA but got diverted to TNF because LPA was closed. Once LPA re-opened, they had to fly TNF-LPA to get there, offload the passengers, pick up new ones and head home LPA-AMS. The "risk" was that the crew might be out of hours for the LPA-AMS flight, which is why the captain refueled in TNF, which delayed them and blocked the Pan Am aircraft from departing. The Clipper crew had just flown in from JFK, and were quite cranky that the KLM aircraft was blocking them in and preventing them from getting to their beds, but as they did nothing wrong, that has no bearing on the events that followed.

          The key problem was that the Dutch captain "believed he sat on the right seat to God" (as a Dutch pilot described it). So rather than let the company dispatchers figure it out, he decided to refuel in TNF, even though he had plenty to get to LPA, and he tried to do the job of dispatch. What he had been told was that as long as he could leave LPA by 19:00, they'd be fine, but to call from LPA for the official word on that. The accident occurred at 17:05, so you can see the "urgency" isn't what you suggest!

          Had Capt van Zanten opted NOT to refuel in TNF, both the Clipper and his aircraft would have been moving nearly an hour earlier, the Clipper crew would have been an hour less cranky and may have challenged the ATC instructions ("take the third on left", which would have been a very difficult hairpin), the weather would have been better, etc.

          But ultimately the accident happened because van Zanten didn't follow the rules... he began his takeoff without clearance.

        2. Malcolm Weir Silver badge

          Re: Some FAA flexibility ?

          On your second point, airline crew have incredible flexibility on where they live versus where their work base is. But that's a personal choice, so if you "spend 24+ hours travelling", that's on you.

          The job requires that you show up rested. If you can't do that, then perhaps you ought to move house?

    4. Anonymous Coward
      Anonymous Coward

      Re: Some FAA flexibility ?

      I took a late flight back from a overseas holiday into atrocious weather in the UK, the flight was delayed, the baggage was delayed, had to deal with a car completely iced up in the car park, and then very tiring drive back home in the early hours. I was supposed to be flying first thing the next day, but all weather reports indicated it would be cancelled, so rocked up to the briefing confident I would be heading straight back home to catch up on sleep - except there was a break in the weather and we had a go.

      It only stayed clear for an hour before a wall of snow just before Coventry hsalted our progress, but unfortunately my super keen number 2 on nav/comms quickly re-planned a route to the west around Birmingham. That went fine until we found ourselves trapped on all sides by low level snowfall and impenetrable icing conditions for the type I was flying. Spent an hour trapped in that small bubble of clear conditions, the only field inside was Derby which was closed for tree cutting. While circling aimlessly ATC were asking if we were still in contact with the ground (i.e. still in visible flying conditions) and offering a radar vector to East Midlands which we couldn't take.

      Eventually we spotted a clearing along the M1 and made a dash for it, hung a left in to Sheffield (days before it closed for good) only to meet with horrendous wind-sheer when we got down to the height of the buildings alongside the runway. As I aborted and did a go around, I was the most frightened I'd ever been in the air, as I knew I was far too tired to be doing this, and was imagining the text of the fatal accident report that would be my epitaph. By some miracle on the next attempt the wind dropped, and I got her down. I was never more glad to be on the ground, and did actually kiss the tarmac.

      After a dropping off the freeloader in the rear who had no headset and was oblivious to all the dramas of the flight, had a couple of hours stop over, before the trip back where I swapped roles and handled nav and comms on the way back. Despite ominous looking conditions it was mainly clear except one more wall of black snow cloud somewhere in Shropshire, but a second miracle resulted in a 500ft high archway of light snow opening up just in front of us, it was only about a mile long with clear conditions on the other side, so we dived through only gathering a dusting of ice, and made it back for an uneventful landing, followed by a debrief which left out quite a few of the hairier details until a long time later.

      I wasn't bound by any commuting hours limits on that flight, it was my purely my own stupidity which could have resulted in 3 lives lost, so I'm glad there are inflexible rules to prevent commercial pilots flying when tired where hundreds of lives are at stake. You never know when you are going to meet conditions which are going to push you right to your limits, and you don't want those limits reduced by fatigue.

  8. jake Silver badge

    Outdated scheduling software?

    Shirley the blame is on whoever allowed (and still allows) that particular software to be used, right?

    It's a poor craftsman that blames his tools & all that.

    I remember flight scheduling working quite nicely in the '60s, so how old IS that software, anyway?

    1. Bubba Von Braun

      Re: Outdated scheduling software?

      Root cause is likely under investment in the scheduling system. By way of an example when SouthWest were expand to international routes their systems simply were not capable of handling the expanded requirements.

      Issue here is a system that works in normal situations, so its built for the 99% of the time what has occurred in the past week, is exceptional circumstances.

      Like anything run it hard at redline levels and when it breaks it goes in a huge way.

      As to their communications/crisis management now that needs a whole lot of work!!

      BvB

      1. Malcolm Weir Silver badge

        Re: Outdated scheduling software?

        And when you had a solid, shared corporate culture, crisis management works better. But under the old Southwest culture, the station managers and CEOs would have rolled up their sleeves and loaded bags / answered phones / etc...

      2. Yet Another Anonymous coward Silver badge

        Re: Outdated scheduling software?

        Or if the computer says that you have an aircraft at X, the crew at Y and the alternate airport at Z is closed with weather = then you blame the software for the cancellation

      3. jake Silver badge

        Re: Outdated scheduling software?

        The problem is that their scheduling includes multi-point loops, as opposed to most other airlines, which use use hubs. Means that the other airlines usually have an aircrew available for any given aircraft that is ready to fly, but these clowns have a miss-match between available aircraft and crews, and when many aircraft are grounded due to weather it compounds.

    2. Bubba Von Braun

      Re: Outdated scheduling software?

      And flight volumes were 1/10th of what they are today. Crew/Aircraft scheduling systems make 3D Chess look simple.

      BvB

      1. jake Silver badge

        Re: Outdated scheduling software?

        "And flight volumes were 1/10th of what they are today."

        In the same time, computers have gone from thousands of instructions per second (IBM's System/360 Model 30 could do 34,500 IPS in 1964) to many billions of IPS in modern multi-core mainframes. Memory, storage and I/O have more or less kept up. Between them, they should cover the change in flight volume more than adequately.

        "Crew/Aircraft scheduling systems make 3D Chess look simple."

        The basic algorithm for passenger aircraft hasn't changed appreciably since it became an issue in the '50s... and 3-D chess doesn't really exist as a thing (Star Trek's famous "tri-dimensional chess" was just a prop, with no rules to go along with it).

        1. Arthur Daily

          Re: Outdated scheduling software?

          Bad seasonal weather problems for airlines has been solved for decades. There is no excuse. There is a process called Operations Research or OR. Apparently given billions, SW failed to employ a decent analyst. You can bet there was no cost cutting on the revenue maximization engine that is responsible for jacking up fares. The only possible, just excuse is that the airports changed their software, and the inputs not available to the scheduling software. On the plus side, each worker is GPS tracked, hours worked and available known, if they have a mobile.

        2. Phil O'Sophical Silver badge

          Re: Outdated scheduling software?

          Star Trek's famous "tri-dimensional chess" was just a prop, with no rules to go along with it

          I'll bet that somebody, somewhere, has written rules. Probably translated them into Klingon as well.

          1. jake Silver badge

            Re: Outdated scheduling software?

            Yes to both. I've even heard filk. Hard-core Trekkies are nothing if not thorough.

            No, I'm not ... but I know quite a few. Came with the territory.

        3. Anonymous Coward
          Anonymous Coward

          Not quite.

          Star Trek's had smaller floating platforms, Dragon Chess (Rules initially published in TSR's Dragon Magazine #100, 1985) uses three stacked boards and extended pieces and moves, played non professionally for a quarter century now. There are also published rules for numerous other 3d chess variants. People have been playing in the modern era for at least 50 years in some form or another. Maybe no pro completions and no big paychecks, but that is hardly the benchmark for either board games in general, or the complexity of game theory.

          I will also point out the scheduling problem HAS changed since the 1950's, mostly due to the lack of a known universal solution to that class of scheduling problems. So as the scale and complexity of the international airline industry expanded, the old code will no longer cut it. Also, as others point out, this particular incident was more than a software failure, the Airline was running without enough crew to provide the needed resilience to handle the impact of disruptions, and decided to shutdown operations and "reset" it's flight schedule, which may have been it's only choice once the initial cancellations started the dominoes falling.

          The big trap you stepped in on the scheduling problem is that it isn't a friendly linear system, and the possible solutions are resource bound, not CPU bound. Part of of this scheduling crash was that after a certain number of failures, there were no ideal solutions possible with the resources that were available. Only prior planning, along with extra crews and probably aircraft could have allowed them to maintain flights during an extensive disruption. All of AWS couldn't compute an alternate to that. All they can do is try to get closer to the most efficient possible solution, and that is a solution of 40,000+ airfields, thousands of pilots, grounds crews, supply chain issues, local, national, and international law, currency markets, and a plethora of other variables. So 3d chess is for lightweights by comparison, and this isn't the 50's by a long shot.

      2. DoctorPaul

        Re: Outdated scheduling software?

        Absolutely.

        When I did my PhD in expert systems in the 80s, recalculating flight schedules was one of the classic problems that we were trying to crack.

        1. John Brown (no body) Silver badge

          Re: Outdated scheduling software?

          Have you done it yet? I know an airline that might be interested :-)

  9. Gene Cash Silver badge

    "promised to investigate whether Southwest is complying with its customer service plan"

    Wait, that's not an instant "hell no!" ??

  10. PhilipN Silver badge

    Time for IT from this century

    Does that mean it is better?

    Or to put it another way - you have 2 lists: Crew and Planes, yes?

    Matching them is the typically routine almost clerical operation which lends itself to computerisation, and has been so handled for decades.

    Or to put it yet another way, I do not believe their explanation.

    1. wub

      Re: Time for IT from this century

      "IT from this century"

      Two things:

      When I visit my local car dealership's repair department, the contact folks are running Win10/11 with a terminal emulator on the screen and what looks like 70's or maybe 80's mainframe software. Rather than focusing on which century, how about software written by someone who understood the problem to be solved? And when you've been bashing at a system for decades, most of the obvious bugs have found and squashed.

      Also, there was a nice article in Scientific American back in the 80's (I think) about a day when severe weather over the central US disrupted airlines. The authors explained how the scheduling fell over and what had to be done to stand it back up again.

      At that time, there were >three< different interlocking schedules: aircraft, cockpit crew and cabin crew and each of them moved in their own distinct cycles. I suspect this is still true today - a real 3-ring circus.

      1. Malcolm Weir Silver badge

        Re: Time for IT from this century

        On the three ring circus: exactly. aircraft have to be positioned for routine (and eventually major) maintenance, and for whatever reasons cabin and cockpit crew tend to be rostered individually rather than as a unit (Southwest operates some aircraft with less than 150 seats = 3 cabin crew, and some with more than that = 4, and I suppose the pilots take PTO and training individually, so any long-term effort to keep them together for more than a few flights will fail).

  11. Anonymous Coward
    Anonymous Coward

    It wasn't so much a breakdown of the IT systems as the incapablity of the IT systems in the first place. The system assumes that the crews are in the proper location for a flight and without a crew tracking system, everything got manually done.

  12. Pirate Dave Silver badge
    Pirate

    "In a call with employees, Watterson blamed the extended delays and cancellations on outdated scheduling software, "

    So, no two ways about it, the blame lies with manglement's reluctance to spend the money to update software and systems. Where have I heard that before? Or, perhaps the better question, iwhere HAVEN'T I heard that before? Seems to be a common theme these days. Because cheaping-out on updates/upgrades never, ever comes back to bite you in the ass...

    1. Arthur Daily

      Business rules matter

      Rather than port and reuse business rules, there is major major no-no's where the lowest bidder for a new system gets to redevelop the lot, and get this, NOT held accountable for not implementing critical rules that were in the old system that worked: aka less is better. Get that early delivery bonus. Over-promise, under deliver. Porting and Conversion is a dirty word.

  13. Russell Chapman Esq.
    Unhappy

    Such a pity

    When Southwest was started by Kelleher and King, it was an amazing company. So sad to see it now rotting because of bad management :(

  14. John Brown (no body) Silver badge
    Facepalm

    Who, me?

    "chief operating officer Andrew Watterson...blamed the extended delays and cancellations on outdated scheduling software,"

    Who, if not the COO, would be the final sign off on keeping the systems up to date?

    See title.

    1. Richard 12 Silver badge

      Re: Who, me?

      I believe the current COO and CEO have both been in post for less than a year.

      Blame lands squarely on the outgoing beancounter pair, who know the cost of everything and the value of nothing.

  15. This post has been deleted by its author

  16. MachDiamond Silver badge

    Does anybody know....

    In some of the "news" stories I've read, the whole of Southwest's computer system was overloaded. There seems to be some indication that they've loaded everything into one big program rather than segmenting operations and allocating server resources based on needs. Passengers banging away on the SW web site wind up preventing crew from being able to access the crew system and get notices about reassignments and new schedules. The airline will know where aircraft are, but might not be able to get notices from pilots or ground crew about repair or maintenance issues so parts and mechanics can't be dispatched which takes that aircraft out of service.

    There are a lot of moving parts in an airlines operations, but if their computer systems are a giant single point of failure, it will cost them huge amounts of money to silence the screaming.

    I'm also going to point fingers at the passengers. Too many are popping off at SW staff who are already overwhelmed and probably can't do anything for them anyway. The best thing you might be able to accomplish is getting your name on a list for compensation and in a queue for the next available seat when operations resume. There is also the fact they took the trip when they should have seen from weather reports that the likelihood of serious delays was almost a guarantee. The best thing to do would have been to cancel even if that meant eating the cost of the ticket(s). Being hung up in some random airport with everybody else in a similar situation and hotel rooms likely costing as much as renting a luxury flat in Tahiti more than outweighs the lost ticket money. A friend of mine who lived in LA was put down in Chicago on 9/11. He kept his head and went straight to the car rental booth and hired a nice Cadillac to drive home. He didn't join the mob at the airline counter and "demand his rights". It was pretty obvious that there weren't going to be any flights for at least a few days. Stop, think and solve the problem.

    1. jake Silver badge

      Re: Does anybody know....

      "Stop, think and solve the problem."

      The GreatUnwashed? Think? Really?

      What are you smoking? I'd like to avoid it.

      1. MachDiamond Silver badge

        Re: Does anybody know....

        "The GreatUnwashed? Think? Really?

        What are you smoking? I'd like to avoid it."

        I'm writing for the audience here, not on TikTok.

    2. Michael Wojcik Silver badge

      Re: Does anybody know....

      I'm also going to point fingers at the passengers.

      Well, yeah. Always. In the US, at least, a great many air passengers seem to have a mighty sense of entitlement. Flights where I didn't witness some abysmal childish, bullying behavior from other passengers are definitely in the minority.

      I've seen plenty go out of their way to find ground crew to bully. I flew into O'Hare one evening in a snowstorm, and all the outgoing flights were (quite sensibly) canceled. Fine, weather happens; this sort of thing is tough on a lot of people (traveling with kids on a tight budget, say), but I was traveling for business so the agency would get me a room if there was one nearby to be had and I'd fly out the next day. I went to Customer Service to book a flight. There was a row of computer kiosks sitting idle, three or four representatives at the desk, and a queue of maybe a hundred people waiting to talk to them, with more arriving all the time.

      So I went to one of the kiosks and had my ticket and boarding pass for the morning flight in about 30 seconds. No one else used one of the machines while I was there. That's not a usability problem – the UI is simple and straightforward, it's likely all those passengers routinely used ATMs, and many of them had probably checked in with a machine in the first place. But they all needed someone to complain to, apparently, and they were willing to stand in line for the privilege.

      A friend of mine who lived in LA was put down in Chicago on 9/11.

      Yup, similar story here. I landed in Philladelphia on the 10th, and was living in Lincoln, Nebraska at the time. Stayed for a few days until things calmed down a bit (and to spend time with some co-workers who were there from overseas and definitely wouldn't be going home for a while), then rented a car and drove back. Not the only time I've had to switch modes when a flight didn't work out.

  17. Anonymous Coward
    Anonymous Coward

    Cheap ass infrastructure?

    Not the best look for an airline.

  18. xyz123 Silver badge

    I'm surely its "purely co-incidental" that the airline knows it can offer "alternate flights" which are unfortunately MUCH more expensive.

    Yes, they'll add the cost of your original flight as a discount, but you can either a) wait in a hotel until jan 2nd and miss New Year with friends and family or b) fork over an extra $600 for the immediate flight.

    Airlines across the world have been pulling this "holiday delay" scam for years. Hell the UK's airlines did it at the end of July 2022, knowing 100% there were families going away for the summer and that any delays would eat into the parents PTO.

    1. Michael Wojcik Silver badge

      wait in a hotel until jan 2nd and miss New Year with friends and family

      Honestly, not sure I see the downside to avoiding New Year's Eve stupidity in a nice hotel room. But that's just me.

  19. xyz123 Silver badge

    Fun fact: the system ALL airlines use has to be the "lowest common denominator" as they have to use whatever the oldest, least used airport uses for compatibility.

    ALL airlines use a system called WINBABS. it's a purely text-based greenscreen dos program shoehorned into an emulator window.

    it has such LOW resource requirements, that a single modern PC could handle every single customer across hundreds of airports repeatedly and simultaneously.

    tests revealed a pretty basic setup with just a few bog-standard servers could handle the entire US daily flight rota.

    But airlines keep saying "we're overloaded" as a scam KNOWING 100% people will pay extra to get on the next flight.

    A certain British Airline did this last summer, cancelling thousands of flights due to "IT Issues", knowing people have limited holiday from work, and wanting to get to their holiday destination as quickly as possible and will pay extra for the next-quickest flight.

    Here's a link to an image of WINBABS, and you can see how simplistic and ancient the system is (unchanged since 1980 except it runs inside an emulator window now):

    https://imgur.com/a/EjXRdcf

  20. G.Y.

    zynep tufecki

    wrote a NYT op-ed on this; seems to be a lot of technical debt -- the staff even mentioned it during contract negotiations

    https://www.nytimes.com/2022/12/31/opinion/southwest-airlines-computers.html

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like