Airbus: We were hours from pausing production in Spain

Airbus is overhauling its datacenter contingency plans after a ten-hour power outage across Spain and Portugal in April nearly forced a complete production shutdown. The blackout – one of the worst in Iberian history – was triggered by voltage surges that overwhelmed the electrical grid. It knocked out traffic lights, forced …

[AnAnonymousCanuck]

An Aviation Company Runs Out of Gas?

Maybe they should have had one of their pilots dip the tank once/twice a year.

YMMV, maybe not if the tank is empty.

AAC

[Snake]

Re: An Aviation Company Runs Out of Gas?

I would think that this teaches companies to plan mission-critical data centers with diesel fuel *and* access to [natural] gas feeds so that they can have both types of generators available. Even with large fuel supplies, unless you treat diesel fuel it can have problems with long-term stability *and* diesel generators do not like sitting long-term without use (why they have programmed exercise cycles); natural gas-fueled generators are much more tolerant of long-term standby.

[Anonymous Coward]

Re: An Aviation Company Runs Out of Gas?

"long-term stability"? Stuff starts growing in it.

[Lazlo Woodbine]

Re: An Aviation Company Runs Out of Gas?

Can confirm this.

Back at the turn of the millenium, I was working in a large retailer. We didn't really bother too much about the odd hour with no power back then, but one year, the shopping centre was having an extensive refit which would involve completely shutting down occasionally, including a whole bank holiday weekend. This would be a problem for the freezer department, which could only stay below -18°c for about 12 hours without power.

We decided to look at the generator on the roof, a big old Cummins diesel. Dipping the tank found it to be empty.

No problem, we'd get a barrel delivered and pump it up to the roof using the pump in the loading bay provided for that purpose.

Nope, the pump was dead, because nobody could remember it ever being used.

We rolled the barrel into the goods lift, but this only took us to the level below the roof, so we had to very carefully, and slowly, walk the barrel up a flight of stairs to reach the generator, where we used a hand pump to fill the tank.

Then we tried to start the generator, and it wouldn''t.

Luckily, my dad is a truck mechanic, so he came out and soon got it running.

We tested the generator properly by shutting the power off, it worked fine.

The next day the shopping centre people told us they wouldn't need to shut the power off.

At least we now had a working back-up generator...

[Sam not the Viking]

Re: An Aviation Company Runs Out of Gas?

As part of some projects we organise standby-power generation for important sites which need very high availability.

Although the generators are run on a schedule, the main fuel tank is often inadequately monitored, because, you know, instrumentation costs money. And have you seen the price of fuel these days?

So keeping the tank full (or 80% full) is often neglected in favour of 'cost-savings'. It isn't the bean-counters who get the blame when things don't work, which is far more often than it should be.

[Anonymous Coward]

Re: An Aviation Company Runs Out of Gas?

When I worked at Marconi, the generators kicked in, everything was fine. The tank was full (don't remember how many 1000L, but it occupied the space under a car park for maybe 100 cars). Suddenly the generators spluttered and stopped. Reason? Forgot to connect the outlet of the auxiliary tank, so the generator ran out when the primary tank ran dry.

Now to Airbus; while the local stuff stayed up the whole time, but the new shinny Google Workspace died horribly (who's link, I don't know, but we still had connectivity to other Internet services). SAP is likely being moved off-site next; which is much worse than losing email and calendar. Anon for reasons.

At home the UPS kept us up, but the fibre link died, and the wireless link went from good to worse than 75kps.

[DS999]

Re: An Aviation Company Runs Out of Gas?

But the downside with natural gas is that if the supply is not available (which may happen during a longer power outage) then your generators can't run. You'd have to store propane on site to insure a supply, but that's got its own set of issues.

[Evaluator]

Re: An Aviation Company Runs Out of Gas?

Thats right. Pretty much all the cloud data centers have contracts for fuel delivery that set priority for those data centers. But you have to negotiate that in advance.

[DS999]

Re: An Aviation Company Runs Out of Gas?

A fuel delivery contract isn't worth the paper (or electrons) it is printed on during a widespread outage, because the fuel delivery chain is vulnerable to not only the outage but perhaps whatever event may have caused it (i.e. earthquake, terrorism, hackers) You might later get whatever credits your contract specifies, but that doesn't do you any good when the fuel isn't delivered on schedule before you run out.

If you want to keep operating during major grid events like the 2003 US eastern blackout you need sufficient fuel (or in the future perhaps battery storage will be the most cost effective alternative) on site to last for a day or however long you want to be protected. You can't assume fuel deliveries will arrive as scheduled before you run out.

As mentioned, that's a problem for diesel because it goes "bad" so you need a way to refresh the supplies. If the area around your site has a lot of truck traffic maybe you could operate a truck stop as a side business (I'm half serious...this is probably a feasible option for the type of mega AI data centers being built now) If there's an outage you put a "closed" sign on the truck stop (or just let the lack of lights say that for you) and use everything in those underground tanks to keep the datacenter humming along.

[The Organ Grinder's Monkey]

Re: An Aviation Company Runs Out of Gas?

"perhaps battery storage"

Did the DeLorean with its "Mr Fusion" device have vehicle-to-grid capability?

(If not, is that a failure of back-to-the-future proofing?)

[wolfetone]

Re: An Aviation Company Runs Out of Gas?

Diesel, actually.

[David Hicklin]

Re: An Aviation Company Runs Out of Gas?

> Diesel, actually.

It the Americanization of the website where the Yanks call liquid fuels like petrol gas

Nearly backfired on them in North Africa in WW2 when the Germans learned that the Americans were stocking up on large quantities of Gas and they started to think chemical warfare, thankfully someone realised that the yanks were referring to Petrol.

[Anonymous Coward]

Re: An Aviation Company Runs Out of Gas?

It will have been the accountants observing that as the fuel stored for the emergency generators had not been needed for over 6 months then they should sell it back to the supplier and remove the dead weight from the accounts.

[Yet Another Anonymous coward]

Re: An Aviation Company Runs Out of Gas?

> they should sell it back to the supplier and remove the dead weight from the accounts.

Or sell it to a subsidiary and then lease it back from them - on the block chain

[Anonymous Coward]

Re: An Aviation Company Runs Out of Gas?

>It will have been the accountants observing that as the fuel stored for the emergency generators had not been needed for over 6 months then they should sell it back to the supplier and remove the dead >weight from the accounts.

Many moons ago, in my company, I saw that writing on the wall. Our energy networks are extremely stable and after having no outage for 5 years they were getting antsy.

I did a proposal to enroll our generators in the control energy market (secondary and tertiary). Now these generator are earning a stipend, by just being there and earning money when the grid operator activates them. The diesel fuel is not getting old, the monthly power on tests can be skipped, when they were used in anger. We even got a deal with the grid operator, that we get their activation and monitoring data as proof for our regulator about working backup power.

Over all they are paying for their maintenance costs and wear (reserves for replacement) plus change by helping the network operator. But it was a fight. Facilities were responsible for the generators and they were quite opposed to this new fangled thing. It helped, that the bean counters had read the proposal and were quite happy to turn a financial burden into an asset.

[Evaluator]

Re: An Aviation Company Runs Out of Gas?

Interesting, Airbus has close relationships with the airlines so they could source fuel from them in a pinch. Seeing a Jet A truck on the highway from the airport to the Airbus DC in Spain would be interesting. You can run diesel generators on Jet A, no problem but need to add some 2-cycle oil for lubrication.

[sal II]

What about connectivity

"Airbus is now ensuring adequate generator fuel reserves and priority supply contracts across its facilities in Spain, France, Germany, and the UK. The goal is to maintain operations for several days during a "massive blackout" without relying on external fuel deliveries."

They will also need to make sure that whoever provides their connectivity has similar provisions. Otherwise it will be a bit futile.

[Roland6]

Re: What about connectivity

The other piece of planning is to ensure during those “couple of days” they have the means to effect a full controlled shutdown.

Before getting the solar battery system. My UPS ran for 10~15 minutes, sufficient to ride out voltage fluctuations but totally insufficient for a major power outage. My thinking was if the power was out for more than 5 minutes it was going to be out for several hours and thus I had bigger problems, like no Internet/land line and mobile after circa 1 hour.

[sal II]

Re: What about connectivity

yes but looks like their goal is to continue production, not just give them more time for controlled shut down

[Roland6]

Re: What about connectivity

Whilst the goal is to continue production, at some point a degree of realism is going have to enter the picture. Ie. What level of disruption and dependency are you prepared to live with.

In the recent case in Spain, a question would be around provision for the workforce and their continued ability to work beyond the end of the current day.

[Mike_R]

Re: What about connectivity

From my experience, high-voltage outages take something like twenty minutes to automatically recover.

More than that, and it's probably something more serious, probably running into hours.

[Anonymous Coward]

Re: What about connectivity

That's what Starlink is for.

[ICL1900-G3]

Stsrlink...

'cos Musk us totally stable and reliable. Always.

[Roland6]

Flawed business continuity plan…

>” The company scrambled to order more diesel, but fuel suppliers were overwhelmed with calls from other affected customers.”

Saw this with Covid where DR/BC plans assumed:

a) only the single business was experiencing an outage

b) stuff (eg. SIMs, phones, laptops etc.) could readily be obtained from suppliers (because of assumption ‘a’).

[Korev]

Re: Flawed business continuity plan…

There was a fire at somewhere I used to work, the emergency plans mostly worked really well and the important people setup camp in the DR room in a different building. The only big problem was that people had gone from BlackBerrys and Nokias to iPhones and you can imagine which cables were in the said room. The big bosses were using their phones a lot for obvious reasons, so towards the end of the day getting hold of iPhone cables suddenly became one of the most important tasks.

They borrowed mine and never gave it back, the rude ****ers!

[An_Old_Dog]

Re: Flawed business continuity plan…

They borrowed mine and never gave it back, the rude ****ers!

For executives mentally-flying at 30,000 feet up ("big-picture thinkers") things like returning your cord are just "little details" which they quickly forget/don't care about.

That is one reason I never do "little favors" for such people.

[A. Coatsworth]

Re: Flawed business continuity plan…

As Roy Trenneman succintly put it, £20 pound doesn't even register to him as an amount of money! He won't remember it!

[retiredFool]

Re: Flawed business continuity plan…

Old story, is it worth Bill Gates time to pick up a "benjamin"(100 dollar bill) he dropped. Nope.

[tip pc]

Re: Flawed business continuity plan…

Nokias & Blalckberrys would have been good for at least 2 days of constant calls, mails & texts..

was always amazed at how long my blackberry lasted despite the constant alert texts, emails and conference calls ii sued it for,

my iphone barely lasts a day even when i'm at home and barely touch it.

[Anonymous Coward]

Re: Flawed business continuity plan…

Strange - my iPhone 16Pro usually goes on my nightstand to get charged to 80% overnight - but, unless it's had some heavy use, it's rarely down to 50% by nightfall. After a year, its battery health reports 215 cycles and 100% maximum capacity. But then, I mainly use it for calls and texts, not as a portable game m/c or doomscroller. It gets used for directions when I'm driving to a new place, or a long distance (the latter usually to warn of delays ahead) - it then sits on charge. Even so, I still keep a battery in my EDC - just in case - BSS (Boy Scout Syndrome)!

[FirstTangoInParis]

Re: Flawed business continuity plan…

> Nokias & Blalckberrys would have been good for at least 2 days of constant calls, mails & texts..

The original Nokia’s would last 2 weeks on standby when fitted with the big battery. Those were the days.

[T. F. M. Reader]

Re: Flawed business continuity plan…

@Korev: They borrowed mine and never gave it back, the rude ****ers!

One of the many great dialogues from the movie "A Bronx Tale":

Calogero: He owes me 20 dollars. It's been two weeks now, and every time he sees me he keeps dodging me. He's becoming a real pain in the ass. Should I crack him one, or what?

Sonny: Sometimes hurting somebody ain't the answer. First of all, is he a good friend of yours?

Calogero: No, I don't even like him.

Sonny: You don't even like him. There's your answer right there. Look at it this way: It costs you 20 dollars to get rid of him... He's out of your life for 20 dollars. You got off cheap. Forget him.

[Lazlo Woodbine]

Re: Flawed business continuity plan…

Yep, supplies were a major issue during Covid.

I was working in a decently run boarding school at the time, and through contacts in China, we knew what was coming, so had started to make plans.

We ordered laptops for all staff that didn't have one, webcams for desktop computers, additional bandwidth, everything we could think of.

The problems started when webcams started disappearing, presumably so staff could zoom family from home. Our usual supplier had run out, our back-up supplier had run out and Amazon had more than doubled the price. We eventually managed to order a box of 24 direct from Logitec in Switzerland.

The same with laptops, when the new ones ran out, and jealous owners of older laptops wanted one the same as a colleague, we found HP had put prices up by over 50%.

[David Hicklin]

Re: Flawed business continuity plan…

Most Natural Disaster plans assume that only one part of the planet is affected at any one time and that there will be someone , somewhere who can send help, relief or assistance.

It all goes pear shaped when the problem is a global one - at which point we are all screwed and the outlook will be very grim indeed.

[An_Old_Dog]

BOFH-Excuse

https://www.euronews.com/next/2025/04/28/what-could-have-caused-the-major-power-outage-in-spain-and-portugal-experts-weigh-in

"These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European networks," Portuguese energy company REN reportedly told the BBC on Monday afternoon.

REN later refuted these claims in the Portuguese media.

https://www.livescience.com/planet-earth/energy/what-is-induced-atmospheric-vibration-and-did-it-really-cause-power-outages-across-spain-and-portugal

"The blackout that hit the entire territory of mainland Portugal today is the result of a significant voltage fluctuation in the Spanish grid at a time when Portugal was importing energy from Spain," REN said in a statement on Monday afternoon. "With this fluctuation, the control and protection systems of the Portuguese power plants, as expected in a situation with this configuration, shut down, causing the blackout."

The "induced atmospheric vibrations" explanation sounds like a line out of the BOFH's excuse list.

[MiguelC]

Re: BOFH-Excuse

Just this morning I posted a review of the disinformation spread about that event on another story's forum, you can check it at https://edmo.eu/publications/disinformation-about-the-blackout-went-around-the-world-impersonated-media-rare-atmospheric-phenomenon-and-russian-networks/

[Yet Another Anonymous coward]

Re: BOFH-Excuse

Can't access from here but there is a great technical overview about it on youtube by a British engineer

The initial problem was a single line going out because of heat/weather. The issue was most of the grid was solar+wind with no inertia from turbines. The turbines that they did have were nuclear and auto disconnected from the grid 'for safety' and they didn't have any spare stabilization loads.

I learned there are entire un-powered power stations driven from the grid feeding power back to the grid just to provide rotational inertia and so frequency stability

Spain/Portugal also has very poor links to the rest of Europe for historical reasons (cough Franco)

[Abbas]

Re: BOFH-Excuse

And will continue, because the so-called gree attitude of succesive socialist governments perceive French electricity as tainted because of the nuclear origin of a good deal of french electricity. As always, politics above common sense and technical logic.

[mbpaz]

Re: BOFH-Excuse

Franco? It's been some time since that and high voltage lines don't last that long.

Until the 70s, Spanish political isolationism was a reason for poor links between Spain and France. Recently it's the French who are reluctant to build those links - solar and wind power from Spain are cheaper than largely nuclear electricity production from France. Better connectivity would mean higher profits for Spanish renewables and big problems for the French nuclear industry. Even more if/when high voltage links are built under the Strait of Gibraltar - there's quite a lot of *very* sunny terrain in Morocco.

[Yet Another Anonymous coward]

Re: BOFH-Excuse

How are we going to get British electricity from Gibraltar to the UK?

And will it be duty free ?

[david 12]

Re: BOFH-Excuse

significant voltage fluctuation in the Spanish grid is the same as anomalous oscillations in the very high voltage lines

That's not refutation: it's confirmation.

[Anonymous Coward]

Re: BOFH-Excuse

Here's a preliminary report into the blackout published in October. The final report will be published when the investigation is complete.

[Anonymous Coward]

Wasn't Just a Power Outage

I remember the outage because my wife and I arrived in Madeira (on holiday) that day and neither of our phones had coverage when we landed - my first check being to see if our luggage had arrived with us, or was still back at our departure airport! Nor was their internet access via the hotel wifi that evening. We discovered the reason the following morning, at breakfast. The island's power was uninterrupted (local generation) but communications went via the mainland...

[IGotOut]

It's alright having enough fuel..

....but if you don't use / swap out that fuel, it's going to go bad. Ask any classic car owner or restorer.

I'd expect them to fire up those generators for a least a few hours every 6 months, if not every 3.

[Anonymous Coward]

Re: It's alright having enough fuel..

Monthly testing is a minimum.

Have multiple generators capable of maintaining the required load.

[Paul Crawford]

Re: It's alright having enough fuel..

Have multiple generators capable of maintaining the required load.

For long-term backup this is a big factor, as typically engines have service intervals of 200 to 500 hours, which is 8-21 days. Longer than most grid-level outages, but conceivable if you are arse-end-of-nowhere and a major event causes repair work to be prioritised everywhere else. Then you might have fuel coming but need to swap generator and service the previous one, etc, as you go.

[Paul Crawford]

Re: It's alright having enough fuel..

We are using HVO for that reason. Yes, it costs something like 20-30% more than white diesel but its stated storage life is around 10 years. Good if you have a big tank of stuff you hope never to need.

Added advantage for us is that HVO is (relatively) non-toxic and biodegradable, so any spill is much less of an environmental nightmare and expense to clear up.

[ComicalEngineer]

Re: It's alright having enough fuel..

If you're not using the fuel regularly then Diesel in particular, can get green slime formation which will block the fuel filters plus condensation in the tank. This is well known and can easily be prevented by the installation of a fuel polishing unit. These circulate a small amount of fuel through a filter. I recently did some work at a large UK hospital which has 2x100,000 litre diesel tanks for their standby generators. Their fuel polishing units operate 24/7.

Most competent companies with standby generators will test them monthly.

The Airbus issue wasn't generating capacity but that their tanks were only sized for a short period of loss of mains power. Thus their power hungry data centre plus manufacturing equipment was simply running out of fuel before mains power was restored.

[Apocalypso - a cheery end to the world]

Paying a premium for fuel

> This may involve paying a premium for the fuel to power backup generators, yet pales into insignificance compared to the potential cost of halting a production line.

And the response from the bean-counters: "Challenge accepted!"

You just know that the priority supplies at a premium price aren't going to survive a couple of year's worth of "budget analysis". "Why do we need so much fuel?" "Why do we have to run the generators for so long during the weekly tests". "Why do we have to test them weekly?"

Icon: my local bean-counter

[David 132]

Re: Paying a premium for fuel

Shades of what seems to happen in the UK with regularity every winter, wherein the cry goes up from hard-pressed council tax payers: “why has the profligate council spent vast amounts of money on snowploughs, which never get used because we never get snow?”

…followed inevitably a winter or two later, by “why did the short-sighted council sell off its snowploughs, there’s been heavy snowfall and all the roads are impassable…”

[collinsl]

Re: Paying a premium for fuel

"What we should be doing is renting our snowploughs when we need them!"

Turns out that every council then would rent from a contractor who rents them from another contractor down a massive chain until you arrive at a guy called Dave who drives a tractor with a shovel on the front who will "get to you when he can" as he's "busy clearing the M4"

[deadlockvictim]

Is Spain collapsing?

I get the impression that Spain is collapsing, slowly albeit, but collapsing nonethless.

Here in Switzerland the number of Spanish-speaking people in the twenties are risen markedly in the last decade.

I hear Spanish spoken as a matter of course on the train here. Is there a flight of educated young people from Spain? A brain-drain, so to speak?

Whenever I go to Spain (roughly every 2 years), the ground seems as dry as ever.

News of drought makes the headlines every year or so, often in Catalonia.

And Catalonia: the ticking time-bomb.

Reports of the Spanish health system indicate that it is over-extended but no politician dare fix it.

Madrid, Barcelona & Valencia, in common with many European cites (hello Dublin!) have priced all but their wealthiest citizens out them.

As an example from February 2022, a cup of coffee in the centre of Valencia costs €4.50 (the same as in Zürich) while 3 stops out on the train, the same cup of coffee costs €1.80.

Climate change threatens to wreak havoc on the important food producing areas in Andalucia.

And now this.

Am I over-egging the pudding?

I suppose that one could over-dramatise the situation in any country with a few select examples.

[cd]

Re: Is Spain collapsing?

Wharton is the root of all evil.

[Anonymous Coward]

Re: Is Spain collapsing?

Are you sure? Spanish-speaking doesn't mean Spanish.

August 2025: Spain’s economy keeps growing — why is the country doing so well?

[Anonymous Coward]

Re: Is Spain collapsing?

>Spanish-speaking doesn't mean Spanish.

But Switzerland isn't letting anyone from S. America into the country

Unless they are just popping back to check their vaults

[Anonymous Coward]

Re: Is Spain collapsing?

People of Spanish descent (parents, grandparents) can get Spanish nationality after two years residency in Spain. People of Italian descent (was up to great grandparents but recently cut back to grandparents) can get nationality by descent.

[el Jaimito]

Ten hours? You lucky Spanish city dwellers

I'm in a deep rural area in Northern Spain and for us it was 18 hours, not 10. Fortunately because of said location, I have a small generator and was able eventually to get lights on around the house and the fridge running: not before my wife fell in the dark and hurt herself, though. My attitude to home battery packs has subsequently changed, but I am yet to invest. As I'm on Starlink (deplore the bloke, service is good though) I was also able to get CGNAT Internet - as all backbone routers, 4G and 5G were down everywhere, that was the only outside link I had. Makes yer think...

[Anonymous Coward]

Re: Ten hours? You lucky Spanish city dwellers

Two and a half hours for me so I guess I was lucky to in one of the areas where they took in electricity from France and used it to start up the rest of the country. As for that deplorable bloke, consider Conéctate35. Has real customer service, unlike Starlink.