OVH flames scorched cloud customers with pledge to build data centre fire simulation lab

French cloud company OVH has pledged to create a lab to model the effects of data centre fires – less than two weeks after one of its data centres was destroyed by fire. News of the lab came in a video update from OVH founder and chair Octave Klaba, who advised that some services in the company’s ruined Strasbourg data centre …

[Stuart Moore]

20x7

... is that because of a 4 hour lunch break?

[seven of five]

Re: 20x7

France, so: Yes, ideally. Two at least.

[Phil O'Sophical]

Re: 20x7

It's France, so 20 days a month, 7 months a year. The rest is vacation time.

[A.P. Veening]

Re: 20x7

It's France, so 20 days a month, 7 months a year. The rest is vacation time.

I am missing strike time in your calculation, it is France after all.

[seven of five]

Re: 20x7

Did not want to spoil the surprise when the first barricade (on fire) comes into view :)

[Muppet Boss]

Re: 20x7

That's 20 days / month x 7 hours / day. Because emergency.

Should get back to normal 7 days / month x 20 hours / week soon. Vive la France!

[sean.fr]

Racist remark

Would you tolerate these remarks if the word french was replaced with black/asian? ..

A Cantona fan.

[Pascal Monett]

Re: Racist remark

Not racist. Jealous. ;)

And if you can't take a gentle ribbing every now and then, chill.

[A.P. Veening]

Re: Racist remark

It isn't a racist remark as French isn't a race. It however is a culturalist remark and like often, the truth seems to hurt.

[sean.fr]

j just a cultural stereotype - so thats OK ?

The slur is that the French are lazy.

They are no worse than the USA or Germany.

Can we agree that if 35 person hours in France produces about as much as 40 person hours in the UK, that the French are doing something right? They are working to similar rules.

So the question comes down to can the French really produce in 35 hours about much as a Brit in 40.

Fortunately this is actually something the UK government has looked at.

International comparisons of UK productivity

(ICP), final estimates: 2016 page 3

[UK]

lower than that of Italy by 10.5%, with the gap widening from 9.6% in 2015

lower than that of the US by 22.6%, with the gap narrowing from 23.1% in 2015

lower than that of France by 22.8%, with the gap widening from 22.2% in 2015

lower than that of Germany by 26.2%, with the gap narrowing from 26.8% in 2015

If the authors are American - productivity per hour about the same as at home. A lot better than Canada and the UK.

[Anonymous Coward]

Re: j just a cultural stereotype - so thats OK ?

After 30 years living in France I can quite categorically say that anyone who thinks the French are actually more productive than the US or UK is smoking something very interesting.

The figures are in part skewed by the large amount of invisible work in France, people working "au noir" where they don't declare their work to avoid paying tax, which boosts the apparent GDP. Traditionally Britain has tended to clamp down much more harshly on that. France sticks to its legal 35-hour week, and ignores the people working 40+ hours & getting paid under the table for the extra.

[julian.smith]

Answering the wrong question

A couple of years ago I was travelling in rural France.

I just missed the pharmacy closing for lunch - I thought, lazy bastards.

Then it dawned on me - who's got this quality of life thing right and who's got it wrong?

My conclusion: the Frogs have got it right - there's more to life than work.

Being just another alienated mouse on the 24/7 capitalist treadmill is not my idea of living

YMMV

[Jusme]

SBG1 data centre status in doubt again

If you look at the site layout, it would be awkward for SBG1 to remain while SBG2 is demo'd and rebuilt. As SBG1 looks half toast anyway, moving any remaining kit elsewhere seems sensible, even it it's into more containers north of SBG4. Of course they may still call that "SBG1"...

https://lafibre.info/images/ovh/202103_ovh_strasbourg_plan.jpg

https://lafibre.info/images/ovh/202103_incendie_ovh_strasbourg_3.jpg

[Anonymous Coward]

"Of course they may still call that "SBG1"..."

Or maybe BBQ1...

[Warm Braw]

A lab to model the effects of data centre fires

It seems to me that they already had one of those, but perhaps in retrospect it wasn't necessary to run live workloads on it for the model to be valid.

[Muppet Boss]

Re: A lab to model the effects of data centre fires

Was it, "we just created a new lab or two to model the effects of data centre fires"?

[TheWeetabix]

Re: A lab to model the effects of data centre fires

That depends entirely on their attitude towards Chaos Engineering.

[Lon24]

The backup is on top of the cabinet ...

My company discovered that's not a good idea after a data centre was wiped by the IRA back in the bad old days. Hilariously the reserve hardware was located in Belfast!

Backing up SBG2 to SBG3 is almost as bad. Do people never learn? If it hadn't been a fire it could have been a network or power setup that took out the whole Strasbourg complex. These days geographical separation of data should be more than 100 yards - Oops metres. Preferably in a different country.

The point is the easy bit having been wiped out in SBG2 is getting hardware running elsewhere - a VPS can be up and running in minutes and a few mouse clicks from a wide choice of providers. But if you have to wait WEEKS before you can get your hands on and start loading the backup - well what can one say? And if a customer finds the (presumably paid for) backup has gone up in smoke too then any sympathy one may have for OVH being the victim of a possible third party UPS error follows it up the chimney.

The French IT lawyer industry must be licking their lips - provided they don't host their billing system with OVH.

[Pascal Monett]

Re: Preferably in a different country

You don't need a different country, a different department would be largely enough.

Unless your data is hosted in Paris, in which case you need to choose your department a bit more carefully.

[pardo_bsso]

Re: The backup is on top of the cabinet ...

> These days geographical separation of data should be more than 100 yards - Oops metres. Preferably in a different country.

So let's build datacenters across the border and we can comply with both easily.

[IGotOut]

Re: The backup is on top of the cabinet ...

When I worked in IT, we had customers saying back up / DR sites should be at least 10 miles apart.

In fact, we set up a site 25 miles away in the end, to rule out shared comms links.

[ckm5]

Re: The backup is on top of the cabinet ...

Depends on the risk assessment. If you are in a place with mass-scale events like earthquakes, then 25 miles is too close. At a previous employer, our US West Coast systems are backed up by systems in the middle of the country.

IMHE this is realatively easy & cheap to do with fully-cloud native applications & infrastructure. In our case, we had a complete replicate of our infrastructure on "cold standby" replicated for less than $500/mo with continous data backups to a distributed infrastructure and daily full snapshots. And, yes, we tested it 2x/year - took 15 min to spin up the replica.

[A.P. Veening]

Re: The backup is on top of the cabinet ...

Depends on the risk assessment. If you are in a place with mass-scale events like earthquakes, then 25 miles is too close. At a previous employer, our US West Coast systems are backed up by systems in the middle of the country.

I hope you are aware everything east of San Andreas will eventually drop in the Atlantic ;)

[The Basis of everything is...]

Re: The backup is on top of the cabinet ...

Most of my customers are quite happy for key systems to be parked in separate rooms of the same building or complex. It's cheap and thankfully events like this are incredibly rare.

Some are a bit more cautious and have a bit more money to spare so say they need to be at least a few miles apart to protect against local issues, but within c. 80km of each other 'cos it does take time for network packets to get through a stack of kit and down the cables and the microseconds soon add up.

A few expect DR systems to be a least 300km away although I'm not quite sure what they're trying to protect against as I repeatedly point out that an event that can affect sites 299km apart is likely to be of global significance and do they really expect my ops teams to be staying at their posts to deal with it or running home to their families for a last goodbye? Admittedly one was in California so having something on the other side of the Rockies probably makes more sense for them.

On the other hand, I've seen dual-redundant servers, comms and storage not only running in the same rack, but all running off a single (and worryingly warm) power strip plugged into one 13A wall socket. I did not like being in that room...

[Xalran]

Re: The backup is on top of the cabinet ...

You have no idea what can be found in French datacenters...

I have the backup remote connection equipments ( used to do support for $TELCO ) sitting on top of the main remote connection equipments. ( and under yet another remote equipment ) in a a specific room, in a given rack.

And I have another one ( again for a $TELCO ) where the equipment sits next to each other ( in two racks, in the same row, in the same Exchange Building converted to datacenter )

[katrinab]

Re: The backup is on top of the cabinet ...

100 metres west from that site does take you to a different country (Germany), so, preferably a bit further than that.

[Neil Barnes]

the company will open-source its water-cooling technology

Presumably so that the community can improve it to either prevent the fire starting, or to put it out once it does?

[Eclectic Man]

Re: the company will open-source its water-cooling technology

Ahem, I believe that means something called "rain", they just intend to leave the roof off the replacement building.

I'll get my coat, it's the one with the umbrella in the pocket.

[A.P. Veening]

French cloud company OVH has pledged to create a lab to model the effects of data centre fires

Let me guess, they will run it in the cloud

.

.

.

of smoke

[Anonymous Coward]

fire propagation

“I want to test how the fire is going in different rooms,” he said. “I want to find how to extinguish the fire in different rooms and share with the rest of the industry.”

Sure, mate, but no need for a lab, I can educate you:

- fire goes up, which is accelerated if you have abundant oxygen intake and also non fire-contained levels in your DC .... erm, stack of containers. Especially true with vertical cable trays that are not fire sealed

- fire normally eats everything above the source and also horizontally, if the previous condition is met. This is especially true when there is absolutely fuck all automated fire suppression system (Innergen is the norm and will extinguish a major fire in mere seconds) of any kind

- having 2 poor sods, alerted by the fire sensors, come on zone (well, containers) with probably outdated portable fire extinguishers, doesn't cut it for a major fire, at all. When your curtains catch fire at home, it does the job, though.

- a stack of containers is NOT fit for DC purpose and multiple lines of 20 kV. It's fit for purpose for carrying idle goods on ships.

I trust the lab will probably be a funny way of proving the above.

PS: Klaba, there are other nice jobs than DC management. You should really try those and leave this for people that know better.

[IGotOut]

Re: fire propagation

Double sets of doors..

Aircon off.

Systems down

Fire suppresents on.

There's a good start.

[Anonymous Coward]

Re: fire propagation

Also no wooden floor.

News from https://www.europe1.fr/economie/apres-lincendie-la-bataille-dovh-pour-redemarrer-ses-datacenters-de-strasbourg-4033013

[The Basis of everything is...]

Re: fire propagation

True story:

Building insurer: That building is a fire risk. You must fit a sprinkler system.

Equipment insurer: You have a sprinkler system in that room. You must fit a hood over the electrical equipment to stop it getting wet should the sprinklers activate.

[Anonymous Coward]

Re: fire propagation

Actually, some countries (Singapore, 10 years ago, come to mind) have rendered *mandatory* the usage of water sprinklers as sole anti-fire measure.

Yes, even in DCs. I've seen that in this country. You really don't want those to trigger if you're still inside.

[Pascal Monett]

"server fires and electrical fires need different dousing tech"

I'm sure he is much more competent than me, but could someone explain to me how a server fire is different from an electrical fire ?

Is it that an electrical fire is just the power cables and their insulation, whereas a server fire is the electricity plus the toxic plastics and metals that are melting ?

Does that really make such a difference ?

[The Basis of everything is...]

Re: "server fires and electrical fires need different dousing tech"

If you've ever tried to solder anything you'll soon discover that burning electrical insulation can be pretty nasty stuff.

Or maybe that's just me.

[Anonymous Coward]

Re: "server fires and electrical fires need different dousing tech"

Normally, IT kit can set up a fire (short cut, other, like in this case) but barely contribute to it at all after that.

What will propagate the fire is mostly plastic of all cables, of which there are tons. Plus anything built with wood, like apparently was the case at OVH (you read this correctly, one building had its floor in wood).

This combined with fancy flows of pulsed fresh air all over the place and also inter-levels of the building, where ... cables are running, will ensure multi-level fire.

EDIT: there is no difference between an electrical fire and any other fire. They start from different causes but then, after ignition, it's all the same.

[Anonymous Coward]

Re: "server fires and electrical fires need different dousing tech"

"how a server fire is different from an electrical fire ?"

His terminology wasn't great, since a server fire is likely an electrical fire. If you have a server on fire you have a small amount of combustible material and the added concern of something like a 230V 15A? electrical circuit pumping in added energy. If the power equipment for a large datacenter has a fire, you likely have more combustibles, plus you have higher voltage and insanely higher fault currents available.

[Sparkus]

Re: "server fires and electrical fires need different dousing tech"

It all depends on the recover/restart philosophy.

If the intent/need/promise is to recover kit and recommission it, then yes, you want different fire control chemicals and procedures than a simple 'flood the place and hope for the best' approach.

If the approach is to just put the damned fire out and who cares about the aftermath, well, a couple of hundred thousand deca-liters of water will do the job.

[m0th3r]

Re: "server fires and electrical fires need different dousing tech"

An electrical fire is only electrical while there is electricity present in the burning materials. As soon as you turn off electrical supply, it becomes a “standard” fire.

The only rule is you don’t put water on an electrical fire, as you can create electrocution risks, and also cause further shorts elsewhere that start new fires. That’s why we have inert gases for DCs, CO2 extinguishers, etc.

[Sparkus]

OVH may want to retire the name SBG1

in the spirit of airlines that retire flight numbers the meet unfortunate endings.......

[cd]

Re: OVH may want to retire the name SBG1

They should mutate their own name to OVAH.

[m0th3r]

This was all predictable...

I’ve now written two posts about this on Medium. These DCs had wood in the structure, had no suppression other than handheld extinguishers, were built like a giant chimney with no control over ventilation that would allow to close air intake during a fire... the list goes on.

Bottom line is, you get what you pay for. You have to think hard why is it you can get servers for a few quid a month, instead of paying big bucks for metal in a proper DC...

The fire lab is a useless idea, fire behavior has been simulated to death by suppression system vendors and fire departments/labs.

[A.P. Veening]

Re: This was all predictable...

The fire lab is a useless idea, fire behavior has been simulated to death by suppression system vendors and fire departments/labs.

Wrong, this will be the only way to convince the owner how incredibly stupid he was. The expression "Penny wise and pound foolish" comes to mind.