LHC knocked out by ANOTHER power failure

The Large Hadron Collider - most puissant particle-punisher ever assembled by the human race - has suffered another major power failure, knocking not only the atomsmasher itself but even its associated websites offline. The machine remains unserviceable at present. However its crucial cryogenics seem to have been unaffected, and …

[TRT]

Paradox?

Has anyone been watching Paradox? Another BBC epic WIN that is a FAIL that will go the way of Crime Traveller.

I only mention it because it might have been more plausible to have the trigger event be the LHC, or maybe it's an interaction between the LHC and solar flares? Should have set it in Geneva.

[Steve X]

Welll....

Would be nice to think it was some sort of negative feedback from the future (hey, would that cure global warming too?) but if I were a betting man I'd put my 0.02CHF on the cause being related to Monday's snowfalls, first of the winter... Sorry.

[alain williams]

How complicated is it ?

It would be nice to know how many components there are in the LHC, then compare that to a car/aircraft/... Then how many of these are non-standard (ie not well tried & tested). This might help people to understand why there are failures.

[Anonymous Coward]

Except that...

Except that the failure this time was in a simple power line. We're not talking about a failure of some esoteric component hand crafted from dew picked unobtanium and rolled upon the thighs of virgin geeks, but a simple power line that are common the world over.

You would think that with something which is allegedly of such imporatance to mankind they would have built in some more resilience. However it doesn't surprise me that a bunch of geeks with their minds on higher things (ahem) might not have considered something so mundane as power supply redundancy.

[Les Matthew]

Just the ticket

to have power automatically restored when you don't know what has gone wrong.

[Neil 6]

They did...

....the diesel kicked in. Generators Watson, generators.

[Anonymous Coward]

Virgin geeks? Ten a penny!

Rolled upon the thighs of virgin geeks? Is that meant to be special? I think to create a truly special component you'd have to use a rarely spotted non-virgin geek...

[Dave Murray]

Learn to read

"However it doesn't surprise me that a bunch of geeks with their minds on higher things (ahem) might not have considered something so mundane as power supply redundancy"

Except they did consider it. The article clearly states that diesel generators kicked in and kept the refrigeration circuits running. That's power supply redundancy making sure the important parts are kept running.

[Anonymous Coward]

Unobtanium

'rolled upon the thighs of virgin geeks'

So not that rare then?

[Annihilator]

Why is a title required to reply

"It would be nice to know how many components there are in the LHC"

I believe there are 3 components, but I lost count at 2, so could be wrong

[James Hughes 1]

@How complicated is it...

Very. I believe it is the largest machine ever made, and has more components than any other machine ever made, although trying to find the figures is daunting. I think the previous most complicated machine built was the Space shuttle, and that doesn't have a very good success rate either.

Lets say...it has 10000 times the number of components of a car. That might be about right, but probably an underestimate. They are almost all custom made, so being tried and tested for the first time. Many also have to withstand temps nr absolute zero, which cars only have to endure when on the A1 near Newcastle.

Of course, there is also the software to consider - mostly custom made and untested.

That should give some idea of why there are failures.

[deshepherd]

the real reason?

Scientists took all the webservers etc off-line while they worked out how to explain to the world that the LHC computer analysis had just reported:

"Yes, I can decice whether the Higgs-Boson exists .... but its going to take me seven million years"

[jules 4]

Apparently

A small Blue telephone box was rumoured to have been seen in the vicinity !

[Paul Crawford]

All OK now?

Time to don the tinfoil hat and wonder if the 'blackout' was just the comms links and web site going down, or if some multi-tentacled pan-dimensional being slipped through and gave the staff an eye-watering probing they dare not remember, before slinking off to leave its inseminated stooges to front the worlds' news.

Of course, snow fall is another explanation...

Paris - eye-watering probing of course...

[The Original Ash]

If Ghostbusters taught me anything...

It's that you shouldn't cross the streams. Ever!

I would have thought these techno-boffs would know that.

[rpjs]

If the LHC is going to kill us all...

...I will die happy knowing that there is still one other person in the English-speaking word who knows the correct meaning of "presently". Top hole Lewis!

[Goatan]

No baguette visible this time

Thanks to El regs ingenius photographers its possible to see the real reason the power failed. If you look closely they have captured the responsible party on camera.

Thants right, its no black pipe but a ninja in disguise.

Who knows why these mysterious warriors want the LHC out of commision, maybe they know something we don't about the LHC. Maybe they were hired by an unknown evil doer who is trying to create their own doomsday machine.

One things for certain, with ninja's working to sabotage the LHC you can rest asured that their will be more problems to come.

[Will Shaw]

Speak for yourself.....

I lost count of the ninjas I couldn't see in that photo. Sneaky!!!

[Anonymous Coward]

But...

"We ourselves find it hard not to suspect the involvement of some pan-dimensional police force, seeking to prevent humanity acquiring parallel-universe portal capability before we're ready to use it responsibly."

Most of mankind isn't capable of using a baguette responsibly.

Paris 'cos I'd butter her baguette. :)

[Andy Enderby 1]

it's not avian bread bombers.....

.....you need to worry about it, it's the head crabs.

[pixel]

power consumptio

and we're not even allowed to run an 100 watt light bulb anymore.

[Steven Jones]

@Anonymous Coward 11:05

I'm sure that resilience can be put into the main electrical supply lines, but at what costs, and what benefit? If you are a major service provider, and millions of people can't go about there normal business is your service fails (as would happen if CERN was eBay, or a telco, or a bank or the like) then it could well be cost justified to go to the expense of putting in such standby services. However, CERN is not one of these, and a delay of a day or two in discovering the Higgs Boson isn't going to be earth shattering (of course the unlikely accidental creation of a black hole just might, but that's a different matter - I'll leave that to the alarmists).

In this case, you put the required resilience in to stop damage occuring and reduce the time required to restart, not spend tens of millions of pounds on duplicated main power feeds to avoid the loss of a couple of days of research...

[lukewarmdog]

Is that

A crowbar I can in the works?

Dunno why the computers are connected to the same power supply, if something breaks how is anyone supposed to check the website to find out?

Altho.. I guess if the website is down you know the power went out..

Still.. isn't it about time they took it offsite?

[Anonymous Coward]

found original pic

that pic was doctored here is the original http://alturl.com/bafv

[Anonymous Coward]

I would copy your URL.....

..... and take a look but El Reg has decided not to allow me to copy text off the comment pages and I'm too lazy to bother to open another IE (yep, that's right, I work for a multinational on XP & IE 6 FFS) and type it in.

Fail for El Reg not letting me copy URL's off the comment page, not the original post

[Jamie Jones]

Mr.

No, fail YOU for blaming El Reg for your buggy software.

And fail YOU for being so paranoid you think El Reg is intentionally trying to stop you

[Alexander 3]

eh?

I was able to copy that without issue. Could your company have restricted context menus and the Ctrl-C combination in IE? You know, in case they wanted to do something dodgy...

[Anonymous Coward]

Ohh come on redun-dunce-ies

Its not always needed to have 'redundancy'. You lot dont get out much, but if you did, you would notice that your care only has one steering wheel - why no redundancy? Because its not necessary and if it were to fail two things may happen 1) you crash and risk death 2) you spot it in the MOT and prevent catastrophe. Or you could have a second steering rack/wheel .... Increasing the cost and complexity. Unnecessary!

The same goes for power to the LHC, if it loses power, the critical systems are looked after by other systems and the main LHC goes quiet ... as it has. The saving of the critical systems only is the best financially viable choice - little damage and we tax payers dont pay for pointless redundancy.

[Anonymous Coward]

I dont know how to say this but...

there used to be a shiny metal conductor now its a charred and burnt out [black hole]

Question is does the black hole collapse/pop out of existance or is still there?!?!?

[Andus McCoatover]

Funny...

..how I was thinking about Prince Philip's erroneous quote earlier today. Apt.

When he was inspecting a Scottish factory, he remarked that a piece of electrical kit "looked like it'd been installed by an Indian*".

Now, he's been proved correct. Should've used the "outsourced" word as well.

*For other readers less versed, he meant "looked like it'd been installed by a Cowboy" - being a euphamism for shoddy labourers in UK.

[QUOTE] "You've tried the cowboys now try the indians", Indian builder's van

[Sam Liddicott]

Am I the only one...

Am I the only one who did a double-take as I scanned the article and read this portion:

...the Collider had disappeared. Presently much of it returned, and with it came...

It could be the plot from Doom; as oxygen pervades the atmosphere...

Sam

[Marvin the Martian]

Snowfall, schmowfall.

The cause is probably birdies again. Websites being offline too points to cables being cut: by woodpeckers, I surmise.

[The Indomitable Gall]

Power going out can only mean one thing...

When the power goes out at a scientific facility it's because an experiment has gone wrong and overloaded the system, leading to the creation or summoning of a potentially deadly enemy.

Why yes, I was brought up in the 80s. How did you know?

[Anonymous Coward]

It's the Higgs Boson

it doesn't want to be discovered and is destroying the LHC so that it's not discovered.

[Ken Hagan]

Power requirements

LEP, the previous big ring at CERN, gobbled so much power that the French only let them use it in good weather. Are those asking for "redundancy" really asking that CERN have a spare copy of the French national grid in the car-park somewhere that they can fire up at short notice?

[Neil 13]

All those components and all that complexity.

And yet it's still more reliable than my Sky+ box.

[MinionZero]

Its kind of funny ...

how it keeps failing, because each time, they think they know what they are doing, then it fails again. Each time its proving they don't know exactly what they are doing, (even though they think and say they do), because they keep getting tripped over by unforeseen failure points.

Yet they still think they can claim its 100% safe to keep pushing towards ever higher energies. The possibility they could be tripped over by unforeseen failures and oversights in their theories is something they arrogantly hit out at, yet time and time again we see them getting tripped over by oversights in their thinking.

So each failure is almost as if, around the world, millions of voices suddenly cried out in confusion and were suddenly heard saying, doh!. Thus the human race lurches once more towards trying to understand the universe. :)

(So much for the nobility and majesty of pure thought. Its more like, Doh!, doh!, doh!, errmm.. doh!, huh? oh doh!).

[Abremms]

Einstien'd!

"If we knew what we were doing, it wouldn't be called research, would it?"

-Albert Einstein

[Anonymous Coward]

'Birdy bread-bomber from the future'?

No - I reckon it's the birdy bread-bomber from the PAST!

That's right - the particle-bothering boffins have succeeded in making a Groundhog Day machine and now they are doomed to come to work each day, watch while a bird thwarts their plans with a baguette and then troop off home again to repeat the cycle the next day.

Until they cop off with a Hollywood actress.

Which will be never.

I bet the problem is caused by the circular nature of the beam. Time is disappearing up its own arse.

[Stevie]

Tsk!

This thing breaks down more often that the Holodeck on the Enterprise "D".

Let's hope it never become as lethal to the crew.

[Vladimir Plouzhnikov]

Is Luis Sancho OK?

I see Mr Sancho is conspicuous by the absence of his comments in this thread...

I hope he has not tried to stop the hellish Machine by heroically shorting the 18kV transmission line with his own body?

[Alexander 3]

Looks like the Daily Mail!

That headline. Seems a little tabloid-esque, don't you think? Using capitalistion to rouse us all into a mob of luddites who will demand the shutdown of the LHC project as it's just ANOTHER example of an EU quango that's mis-managing everything disastrously until all the directors can retire on fat cat pensions...

Or is A.N.O.T.H.E.R. a shady terrorist organisation bent on disrupting vital science by the Centre for Effective Ruffian Neutralisation?

[Doug Glass]

@Learn to read

No, that's not redundancy, that's fail safe. Redundancy provides for no interruption in service should a single system fail. Such as with a nuclear power station. They have redundancy and in fact what is termed "defense in depth" with many systems having three channels available for continued full operation. One channel may fail safely but the plant remains online and continues to provide power to the grid.

A "Fail Safe" allows the unit to safely fail continued operation with minimal damage, which certainly happened here, with no thought to continued full operation.

[Doug Glass]

@It's the Higgs Boson

Exactly. From the past via temporal Ancestral Command.

[Richard Porter]

Pan-dimensional police force?

"We ourselves find it hard not to suspect the involvement of some pan-dimensional police force, seeking to prevent humanity acquiring parallel-universe portal capability before we're ready to use it responsibly."

Nothing so sinister - they've just let the smoke out again.

[Bod]

Diesel Generator... is that Vin Diesel?

Calling Gordon Freeman!

[Kevin Reader]

If I remember my Harry Harrison correctly...

Since the discussion of the LHC always degenerates into a poor TV SciFi plot I feel I should stir some good Science Fiction into the mix.

In the Stainless Steel Rat's various adventures the pan-dimensional police are originally revealed to be "The Time Corps" who are "So secret that there aren't even rumours about them!". The well read will recall that deGriz himself works for the Special Corps who are "So secret that there are only rumours about them".

When they hatch a clever scheme to dispose of some dodgy aliens using a Time Portal (see LHC relevant) they are prevented by the "Morality Corps" who are "So secret that the Time Corps don't know about them". Clearly one of *these* organisations it at work because of the LHC's dimension warping potential.

Although its pleasing that the LHC suffered no major damage and the backup power worked for the cryogenics one wonders if the device was actually running and the beam & magnetic pipes were shutdown. After all wouldn't you want to be the one to shout across the control room "I'm losing containment" because that's a Warp Core Breach involving anti-matter (obligatory Trek reference) and the LHC is as close to that as we are likely to get.

Paris - ofcourse - as she knows all about losing containment... and the sudden loss of energy in her immediate vicinity.

[J. Cook]

Looks like a burnt insulator, short or arcing?

But at least it wasn't a resonance cascade failure.

Mine's the Mk. IV HEV with the crowbar.

[Eeyore3061]

Non-Scheduled Stand-By Power Test Successful ... well, that's one way of looking at it

From the comment that the Diesels had cut in as expected in the article by the controlers. This is a good thing as I've found that 'acceptance tests' and reactions to real world flying monkey wrenches ... don't always match up. ;)

[DI_Wyman]

Sarnie?

Anyone seen my cheese and pickle sarnie?

[Eeyore3061]

@ 'Rolled on the thighs of Virgin Geeks'

It's a ref to an old marketing ploy for some of the better/most expensive Cuban Cigars from the '20's -'60's. Yes boys and girls, sex sells.

[amanfromMars 1]

When Particle Physicists Party, the Cosmic Radiation is Free

"Lets say...it has 10000 times the number of components of a car. That might be about right, but probably an underestimate. They are almost all custom made, so being tried and tested for the first time. Many also have to withstand temps nr absolute zero, which cars only have to endure when on the A1 near Newcastle.

Of course, there is also the software to consider - mostly custom made and untested." ..... James Hughes 1 Posted Wednesday 2nd December 2009 14:14 GMT

And they're smashing nuclear sized particles at each other at practically twice the power of light speed just to what happens and opens/explodes and disappears, and think its a good idea in order to imagine what a Big Bang and Evolving Thinkers Created with Natural Assets and Global Resources over Time is Space and in Control of Words Instructing Imagination and Mentoring Resolve. What is there to possibly go wrong? :-)

The Secrets of the Past are always Revealed in the Future and how you Create the Future is surely the same as how the Past was Created Before* [which is a lot simpler to imagine than thinking about, .... In the Beginning, there was nothing and then .......... Magic Order and Tales to Tell and Follow for Historical Record of Recent Memories/Personal Tales Told as News forming Views for Support and Reality....... and you would not believe the number of businesses milking that particular and peculiar cow for its cream stream.

* The Present is a Myth for it never exists, being always fated to be Immediately Past for the Future to appear with its Gifts of Shared Noble Thoughts .... Alien Memes to Competitive Destructive Human Beings and Ignorant Business Machines

[invasion13]

did you notice the hour it occured?

0 1 2 3 = 01H23 jpr computer bug ?

[Anonymous Coward]

51 comments and I get to a page with only 1

What is this paging of comments all about? Having read the article, I clicked on the comments link and get taken to page 2 containing a reply to a comment I haven't read.

Dumb, stupid, pointless, idiotic and a complete waste of time.

We have to scroll anyway when there are more than just a couple of comments, so just leave them all on one page.

Who came up with the daft idea of linking to the last page of the comments anyway?

[Dave 30]

Reddit

Rather than spend time making a comment system that allows replies but doesn't use a tree view to group them, why not use an existing good one e.g. reddit.

Admittedly it doesn't have the post icons though...

[William3RD]

Use Porcelain Insulators

Moulded insulators have verticle seams that can get contaminated and form a path of reduced surface insulation strength. Once a path of reduced surface insulation strength is formed, even a slight peak in line voltage can begin short arcs that can elongate along a moulded seam. (Little arcs form ozone, which is a gas that provides a medium to conduct relativly high voltage.) The short arcs growing longer furnish a path that can result in flashover, which burns the surface just as is shown in the picture associated with this article.

[Anonymous Coward]

and virgin media are off the air too

I know which i care about more

[Anonymous Coward]

Voltage always gets a bad rap

"...colourful speculation is to be anticipated." Such prescience . Too bad that it can't be turned to predicting the next cock-up.

And, give current its due. Without current, those 18kVs wouldn't pose as much threat as a well-rubbed kitten in a dry house.

And Paris, because she's an expert on cock-ups.

[wv9e]

Who is responsible...

I am surprised that the obvious escapes so many... Dr.Who is surely saving earth and fooling with

power couplings is the safest way.. not wanting to injure any humans you know...

[elderlybloke]

Wots the fuss about?

The power cable supplying the LHC failed.

Our power supply fails occasionally, but it doesn't get into the international news.

Do seem to enjoy the problems of the LHC even when they are beyond its control.

[TeeCee]

Who was it?

Who forgot to put 50p in the meter then?

[amanfromMars 1]

It is an Excluding Mistake to Not Imagine that Cyber Security is ...... at least, Unusual*.

""We ourselves find it hard not to suspect the involvement of some pan-dimensional police force, seeking to prevent humanity acquiring parallel-universe portal capability before we're ready to use it responsibly." Most of mankind isn't capable of using a baguette responsibly" .... Anonymous Coward Posted Wednesday 2nd December 2009 11:36 GMT

That had me tittering at the absurd spread of capability available to Intelligence for Use and Abuse, AC. :-)

As for "before we're ready to use it responsibly" you might like to add ... "and control it absolutely to market it exclusively for crazy fortunes as Sole AIgent." Don't be holding your breath on that progression knocking down your door though, unless you Really have Virtual Talents and Talons to Offer, in always doing things Right and Proper Fit for Future Purpose rather than Deferring and Demurring to Present Comfortably XSSive Supply Convenience to Maintain the Status Quo Establishment of a Moribund Progress and Suppressive State Architecture ....... SurVeiled InfraStructure/Spooked Nation.

And re .... "If I remember my Harry Harrison correctly..." ... Kevin Reader Posted Wednesday 2nd December 2009 16:51 GMT .... If Fact follow Fiction in a Real Live Parallel, What and Who are then in Control of the Future? And how many of them are there, and where do they Hang Out and Chill and Plan Impossible Viral Surprises and Crashing Digital Delights .... Operating Systems Meltdown by Domino Cascade because of Fundamental Failure of Imitation and Reconstituted Key Foundation Stones.

Renegades and Rogues doing their Pirate Best in Seas of Mass Mediocrity and Zero Integrity ....... for a Turkey Shoot in Favour and Furtherance of Enterprising Buccaneers.

*Irregular and Unconventional.

[Blue eyed boy]

No need to consider time-travellers protecting their past

Just consider the many-worlds hypothesis.

On any timeline where LHC *works*, earth is instantly destroyed. So we're only here to ponder the question "What the f happened this time?" if something has indeed gone wrong. Again. And again. And again.

[Fractured Cell]

My Goodness!

"I never thought I'd see a Resonance Cascade, let alone create one!"

[ThomasF]

Luis Sancho come back all is forgiven

Where is our only hope when you need him.

http://www.unificationtheory.com/God.html

Its Pyramid selling big time but Luis is the man to do it and stop Cern in its tracks.

OK he failed first time around but him and Walter Sanchez at least tried to stop Cern from

turning the world to "A Grey Aldi Xmas Gooey paste"

http://cosmiclog.msnbc.msn.com/archive/2008/09/26/1457536.aspx

Luis get a hair job and go on the X-Factor or even Cern-needs-Talent

Spread the word - Cern is going down!

[luis sancho]

PERSPECTIVES...

I see some 'Ad hominem fans' missing my comments (-;. Well for a change i will add some quotes from more relevant people:

The Human-perspective:

‘We are a thin layer of mush on a rock speck

in a corner of the Universe. Now, departing

from those facts, we can talk about man.’

Schopenhauer, father of modern Philosophy.

‘I am Kali, God of death’

Oppenheimer, father of The Atomic Bomb

'And nothing can we call our own but death

and that small model of the barren Earth,

which serves as paste and cover to our bones...'

Shakespeare, father of modern Literature

‘From a legal perspective the Large Hadron Collider is the biggest genocide in human history.

With a mere probability of 10% of risk, an unlikely case, which is the original estimate of the company,

it means a genocide of 10% x 6.6 billion=66 million statistical dead

by the mere fact of having the LHC running’

Mark Leggett, expert risk.

The Theoretical Perspective:

'In the 1920s, Bose and Einstein predicted a dense state of matter in which many particles at very low temperatures share the same spatial location, a prediction which was verified experimentally in the 1990s. The simplest mathematical model of such a Bose–Einstein condensate uses a non-linear interaction between the many particles in the condensate. The non-linearity poses a great challenge to both physicists and mathematicians, leading often to unpredictable behavior. For example, strong attractive interaction among quark condensates may lead to collapse, infinite density, and model breakdown, the properties expected in the classic treatment of black holes.'

Wikipedia, Quark Condensates.

The Industrial /Ethic/Political Perspective:

'This case is of concern to all the people of America, not only to physicists. It should be addressed by the Congress'

Hellen Gillmor 'Sancho et Al vs. CERN'

‘Technological civilization is programmed by the principle that something ought to be done because it is technologically possible. If it is possible to build nuclear weapons, they must be built, even If they might destroy us all. Once this principle is accepted, humanist Values (something has to be done because it is needed by man) are Dethroned and technological development becomes the foundation of ethics'.

Eric Fromm, father of political psychology