Easynet blames network collapse on dodgy router update

Early investigations of a nationwide collapse of the Easynet network have pointed the finger at a software update to a Cisco router. The outage began at 8.32am on Wednesday, and for some customers lasted most of the working day. "An Easynet edge router crashed due to a software bug condition triggered by an invalid update …

[Alasdair Russell]

Hmmmmmmm - do we need a translation

"an invalid update from an external source into the Easynet network"

Does this translate as an external attempt to hack a Cisco Router with a default or easily guessed password?

[Anonymous Coward]

Invalid update

"an invalid update from an external source into the Easynet network"

Could be a software update from Cisco, or possibly a bad routing table update sent from another ISP causing a routing loop.

[ElNumbre]

Who?

Who do they think they are, BT?

Good ISPs have self healing networks with routers which can failover automagically, if not seamlessly, then at least within a couple of minutes.

[Anonymous Coward]

Was the SLA breached?

Is the SLA 99.9% per day, or 99.9% overall? I would imagine overall surely

When was the last time EasyNet went down? By my count, if it had been up without outage for the past 25 days solid, the a 6 hour outage still keeps it within 99.9% uptime

If you make it only business hours, say 7 hours per day and exclude weekends, then they are still 99.9% if they have been up solid for the past 12.5 weeks

[Chris Williams]

Re: Was the SLA breached?

The availability guarantee is quarterly. I have added that to the story.

So 6+ hours down is getting on for 0.3 per cent of the quarter.

[Lee Dowling]

An edge router

A single edge router took down an ISP. Nice redundancy / monitoring / replacement policies there. I thought the point of buying very expensive Cisco hardware was that this sort of thing wouldn't affect the connectivity as a whole, rather than having to have some Cisco guy read commands to you over the phone when things go wrong?

[Ivor]

Three nines.

Doesn't 99.9% give them just over 8 hours in a year? what's the duration of their SLA?

[Anonymous Coward]

Carrier Class?

Well if you will use an enterprise class solution, dont expect it not to fall over sometimes!

[Andy Barker]

A single router?

I am always amazed when it is reported that a single router buggers up so much of an ISP's network.

Redundancy?

Failover?

[Chris Miller]

99.9 per cent uptime guarantee

Over what period? I'd expect monthly (although the publicly available material is silent on the topic), but if they sneakily measure annually you could fit in a single 8-hour service interruption and still meet the guarantee.

[Tom 15]

Eh?

Unless I'm understanding something wrong 99.9% uptime allows up to 8.76 hours of downtime per year so 6 hours doesn't break that?

[Candy]

SLA broken?

Um, not unless there were additional outages or the 99.9% uptime mentioned was inaccurate. Three 9s buys you up to 8hrs 45 minutes (and change) of outage per annum

[Kevin Gurney]

Note the word "QUARTERLY"

Easyney sells its SureStream ADSL product as an alternative to a leased line, with a 99.9 per cent uptime guarantee (quarterly).

[Andy ORourke]

You're assuming......

People read the article BEFORE they jumped in to comment that it was really bad that a single router could take down a substantial amount of an ISP's network and then go on to debate the 99.9 uptime guarantee figures :-)

[Chris 211]

Sounds like

They did a misconfiguration of the router which tried to automaticly update itself OMG why! Dont people research images before applying and apply images to routers put into standby mode. Old fashioned it maybe but if your going to update a critical router its best to have an 'engineer' involved not a script.

[Candy]

Oopsie.

Mea culpa...

[Fuzzysteve]

redundancy's all fine and good

What makes big networks break, is when their routing tables get hosed by the bad updates being spewed by a badly configured/misupdated router.

No redundancy can stop that from happening. Though it shouldn't take 6 hours to fix.

[Matt 80]

We are still down....

but we are getting a new FREE router in the morning !! Will it make any difference ? I doubt it.

[Anonymous Coward]

external ROUTING update, which is something real routers do all the time!

there was a major event around that time which impacted several makes of routers (apparently from multiple vendors) causing their BGP sessions to go haywire, however it was quickly suppressed at the source (well, likely at their upstream provider) and the only ones who seemed to have noticed were those with alerts set up, i suspect in easynets case this was probably a trigger event that exposed an unrelated problem with their network, which caused the extended outage

however their comments on the outage do seem to imply a single router with no automatic failover which is rather worrying...

more details: http://www.merit.edu/mail.archives/nanog/msg14487.html

[sw1sstopher]

these aren't the redundancy figures you're looking for...

maybe if they hadn't made so many engineers redundant this would have happened...

[Matt 80]

Guess what ?

On checking this morning at my place of work, we still don't have a link to the outside world - I really hope the new router which is on it's way works......