Computer error triggers mass rocket launch

Americans love their fireworks on Independence Day, but it is possible to have too much of a good thing. That's what spectators got on July 4 in San Diego, California, when an errant computer triggered every rocket in the city's annual display to launch at once. The pyrotechnics were meant to last 18 minutes. Instead, the …

[Phil Endecott]

Last time...

Last time I read something like this, the twits had entered a whole load of timing data thinking the format as hh:mm, when it was actually mm:ss. Hence a display 60X faster than expected. Seems about the same speedup as this incident.

[Anonymous Coward]

Re: Last time...

Timing for these is usually hh:mm:ss:ff, driven by SMPTE LTC timecode from the audio source.

However, even if everything is correct in the pyro sequencer, if somebody hits "fast forward" on the timecode generator while the system is armed...

Anon because by the grace of God go I. Big public building openings? Oh yeah. Not nervous. At all. Everything's ready, it'll go fine.

[Anonymous Coward]

I like a good bang!

[JeffyPooh]

Re: Last time...

That doesn't make any sense. If the time format was assumed to be HH:mm then the plan would have a temporal resolution of one minute. That would make for a rather boring show. So it's a non-starter as a possible explanation.

[Anonymous Coward]

Re: Last time...

I heard from a friend of a friend of a mate of the guy who worked at the Oban event.

It was not an HH:mm against mm:ss confusion as that would have been extremely obvious. It was a mm:ss against ss:ff mixup.

This is also one of the reasons why most timecoded pyro operators want at least 20 seconds of good timecode before the first 'fire' to give them some time for an "Ohshit" moment and pull the key!

Anon to protect the red faces.

[James Micallef]

Re: Last time...

I like the way the company statement tries to take cover behind public ignorance of computers by saying "it wasn't human error, it was the program that malfunctioned", when really the malfunctioning program is because it was badly programmed by a human.

[James O'Brien]

????

Some sort of computer virus caused this? I smell bullshit from someone who needed something to blame besides their own incompetence. My guess is they probably had the display to set based off the computer clock and the clock was set wrong instead of using a good old fashioned timer. And even if they were using a timer script they probably dropped a couple 00's.

[Colin Miller]

Happend in Oban for Guy Fawkes night

http://www.bbc.co.uk/news/uk-scotland-15611160

[smudge]

Re: Happend in Oban for Guy Fawkes night

Yup - that's the first thing I thought of.

Where Oban leads, San Diego follows eight months later :)

[Anonymous Coward]

Virus? I'm gonna go out on a limb and guess a problem with the keyboard->chair interface.

[ElReg!comments!Pierre]

Yes, probably

"must be a virus" is the modern age equivalent of "the dog ate it". get it for all kind of things, from data deletion to forgotten passwords.

Last time it was for the inability to select the right printer queue. Apparently a "virus" (of which no trace could be found upon inspection) caused the change of the default printer queue hence much whining about the network being broken. I know it's bollocks, they know I know it's bollocks, but they keep trying. It has become a social convention, a bit like "how do you do" and "have a nice day". Meaningless converstation lubricant.

"we have a virus" actually means "can you come and show me how to find my own arse with the help of a map, a compass and a lighthouse. Again."

[Euripides Pants]

Re: Yes, probably

"can you come and show me how to find my own arse with the help of a map, a compass and a lighthouse. Again."

Now, if you could ram the lighthouse up the clueless twit's arse....

[Old Handle]

You know what Agent Smith says.

[ElReg!comments!Pierre]

You know what Agent Smith says.

Can you feel it Mr. Anderson?

[Anonymous Coward]

Virus

Yeah that'll be it. Nothing to do with programming errors or anything else that might have been caused by them, this clearly has virus written all over it.

Idiot.

[edge_e]

Repeat after me

The computer does what it's told to by the program

If the computer does something unexpected it's usually because the programmer fucked up

[Michael H.F. Wilkinson]

Re: Repeat after me

The programmer OR the user fucked up.

(not XOR, btw)

My bet would be on the user. But, as I write programs, it would be, wouldn't it?

[LaeMing]

Re: Repeat after me

Computers do exactly what they are told. They do it faster than a human can realise they told it the wrong thing.

[bazza]

Re: Repeat after me

Forget computers, it's photocopiers that I don't get on with. It wasn't so bad ten years ago, pressing the wrong button would waste only a few hundred sides of paper. Now the damn things are so fast they spit out thousands and thousands of wasted sheets in the blink of an eye, all neatly collated and stapled.

[Aaron Em]

'Program glitch' my shiny pink arse!

The eventual "glitch" will turn out to be whatever is the west-coast equivalent of a well-lubricated "Hey, y'all, watch this!", said revelation being swiftly followed by the quiet departure of one or more former pyro technicians.

You heard it here first.

[Khaptain]

That was unique........lol

Just to be on the safe side that could have added that it was probably Anonymous that plotted this and the virus was called the Gunpowder.Plot.

Or maybe they forgot to turn on their FIREworksWALL

[Michael H.F. Wilkinson]

This is just a practice run for the RotM

Next time they will do it using ICBMs!!!

We're doomed, DOOMED!!!!

[Graham Marsden]

I bet...

... the audience were thinking "Wow! If that's how it starts, what's the rest of it going to be like?! [Long pause] Err..."

[Anonymous Coward]

Someone's selected the wrong time-code when loading the scripts in to the firing system, which is what happened at Oban.

[Anonymous Coward]

You sir read the same forum as I do

and it does seem likely doesn't it?

[Stevie]

Bah!

Everyone is being a bit harsh - clearly the "spokesman" was on the spot and not terribly computer literate, using "virus" for "glitch" from the phrasing there.

Coordinator at the optimum viewing point with a radio and human beings with manual ignition systems in the barges is the way to go, because if Mr Boss goes off-plan and says "everyone light everything at once" the bargees will say "No".

Just because you can put a computer into your event doesn't mean you should.

[ElReg!comments!Pierre]

Re: Bah!

"Coordinator at the optimum viewing point with a radio and human beings with manual ignition systems in the barges is the way to go, because if Mr Boss goes off-plan and says "everyone light everything at once" the bargees will say "No"."

If I remember correctly they make damn sure that no soul remains aboard the barges at the time of firing. They even triple-check. Too dangerous.

Plus, big modern fireworks are very precisely timed ballets, often needing the firing of a dozen devices situated on separate barges at the exact same time, for example. The time between firings is also extremely precisely controlled to ensure that a rocket blooms in the exact center of the aftermath of the previous one for example. Humans just can't react that fast/precisely, especially not over a walkie connection. The way it used to be done was by using very precisely calibrated fuses but that's more much prone to errors and mishaps than a computer and electrical ignition.

Modern fireworks are actually an almost perfect use case for a computer.

[Jon Green]

Re: Bah!

They certainly made sure that no soul remained aboard _those_ barges!

[Stevie]

Re: Bah!

"Plus, big modern fireworks are very precisely timed ballets, often needing the firing of a dozen devices situated on separate barges at the exact same time, for example. The time between firings is also extremely precisely controlled to ensure that a rocket blooms in the exact center of the aftermath of the previous one for example."

Makes you wonder how they ever managed to pull the trick off for the three hundred years or so before computers were ready to take up the slack.

Or not, as in the case of the San Diego Fireball of Unentertainment.

I imagine you'll be telling me next why the pyramids require the intervention of UFOs to get built.

[ElReg!comments!Pierre]

Re: Bah!

> Makes you wonder how they ever managed to pull the trick off for the three hundred years or so before computers were ready to take up the slack.

As I stated: very precisely calibrated fuses. Not as precise or safe as electronic system but it did the trick

[Naughtyhorse]

it's iran im telling you

It's stuxnet.

Hitting the infidel where it hurts.

Allah Akbar

told you something like this would happen

[bazza]

Re: it's iran im telling you

Iran, on Independence day? Surely not. Most likely to be the Brits don't you think?

[auburnman]

Not us, guv'nor

Us? We like a good fireworks display as much as anyone else. We also like to laugh at someone else'e ballsup a little more than we should probably admit, but it's not as much fun if you deliberately cause them to eff up.

On the subject of the "must be a virus" guy, I have a fair bit of sympathy for him as he suddenly became the centrepiece for a highly public SNAFU on a major holiday event. If it happened to me I'd be probably dropping all the FUD and chaff in my verbal arsenal to buy time for a retreat and regroup.

[edge_e]

Re: it's iran im telling you

All your rockets belong to us dear chap

[PyroMoth]

RE: Bah!

I will assume you have never stood in close proximity to a large scale show, or had to fire a show manually to rapid cues.

Just like the world of lighting, computerised firing controllers are ubiquitous. They are far safer and can give much more accurate choreography. Thousands of shows are run in the same way each year without issue.

I'm pretty sure someone just made a mistake under pressure on the busiest day of the year, and the script got messed up as a result. It will have happened to most companies one way or another. No point lying.

[Stevie]

Re: RE: Bah!

"I will assume you have never stood in close proximity to a large scale show, or had to fire a show manually to rapid cues."

Then you will assume incorrectly.

Makes you wonder what your other assumptions about this San Diego Fiasco are worth, eh?

[Phil Koenig]

Advanced WiFi control system

How much you want to bet the person who designed the control system used a $39 WiFi hotspot with encryption turned-off? Maybe it was the same intellectual that designed the retail point-of-sale system for TJX in the States...

http://www.schneier.com/blog/archives/2007/10/tjx_hack_blamed.html

[Richard 12]

Re: Advanced WiFi control system

Nobody even considers using WiFi for this.

This kind of event needs around 10ms accuracy, and WiFi cannot possibly do that as its latency varies wildly.

WiFi is great for email and surfing the web, it's ok for buffered video but it's utterly useless for anything that needs even mildly accurate timing.

[ElReg!comments!Pierre]

Re: Advanced WiFi control system

> Nobody even considers using WiFi for this.

What you meant was "Nobody _with actual knowledgẹ_ even considers using WiFi for that."

Stevie up there seriously suggested using fleshlings on the barges with manual ignition systems, walkie-takies and a central coordinator. The latency of that being in the seconds range obviously (for road safety purposes the reaction time of a human is estimated to be roughly one second; add the "stellar" clarity of radio communication to that...)

[Richard 12]

Re: Advanced WiFi control system

Actually, while humans have a long latency to unexpected events, given suitable warning and a 'click track' most of us mere fleshlings can reliably hit a cue within 50ms, and some people down to 10ms.

It's all down to the rhythm of the dance.

[Kev99]

My data, your computers? Uh-uh!

And the pundits expect me to put my proprietary & other data in the hands of someone else's computers "in the cloud"? I don't think so, Tim.

[stucs201]

Is it really that much of a screwup?

From the background cheering the audience seemed to like it.

(whether that lasted when they realised how short it was is another matter)

[Peter Fox]

What could possibly go wrong?

As clocks go 'tick' and cows go 'moo' so programmers go 'WCPGW". Did no programmer not think 'maximum rate of firing instructions is X'?

[Esskay]

Not entirely a bad thing

As any fule kno, the quicker the fireworks are over, the quicker you can get back to drinking.

[I ain't Spartacus]

Re: Not entirely a bad thing

Excuse me, but why are the two mutually exclusive?

If the bar's not serving, then that's what the Good Lord invented the hip flask for. Admittedly the fireworks displays I've been to in England tend to be New Year and November. So a warming nip of whisky doesn't go amiss.

[qt101]

USA's 'Premature Ejeculaunches'? On July 4th ;-D

They've always gotta get to the big Bang fast.

Re: Virus in our software. Pfft please! lol

Next Excuse will be: We've had specialist analysist's concluded that we've indeed been Pwn'd by a Hacker!

[amanfromMars 1]

Cogitant ergo sunt .. and an APT rallying call in these changed days of strange ways and memes?

Neil, Hi,

A word of advice, which you can if you want to, classify as insider information and a hot tip ..... don't put any money you cannot afford to lose, nor any shirt which you want to keep, on a bet that would support the article's sub-headline ....... Rise of the machines thwarted ... for now

Any latin scholars [are you an El Reg reader, Boris J?] out there who can supply the correct translation into Latin of ..... "They think therefore they are" , which of course is a play on that well known classic line ..... Cogito ergo sum.

[Paul Hovnanian]

Its only fireworks

It could have been worse.

"How about a nice game of chess, Doctor Falken?"