London's stock exchange crashes again

The London Stock Exchange has suffered yet another systems crash, leaving brokers high and dry since 9.30 this morning. The Exchange last went down in September 2008 and took almost the entire day to get back online. That outage, on one of the Exchange's busiest days, was the day after the $200bn bailout of US housing giants …

[Dirk Koopman]

.net?

I thought the LSE had bought a linux system + the Indian software house that wrote it? And that it was fast and super and wonderful and would never crash again?

[arreg]

They haven't moved from .net to Linux yet...

The LSE are going to move over to a Linux/Solaris platform (see tinyurl.com/yauqcdk), but it takes more than a few days/weeks to transfer a system of that size over to a new platform and adequately test it. I'd give it a year before it's fully up & running, more if the LSE start asking for design changes during the implementation phase :-)

[Rob]

Bears and woods

"design changes during the implementation phase" I think that's a given ;)

[Wilko]

It's run on .NET?

dear lord!

[amanfromMars 1]

LSE .... Surfing AIGravitational Wave.....EMPathetically

Strange Shenanigans in Deed, indeed. AIdDisturbance and ReAlignment of Viral Virtual Forces, No Doubt.

EMPath et ic ally .... XXXXTra Terrestrials Intelligence Community Ally.

[Paul_Murphy]

So does that mean.

That the overpaid winkers have to give some of their stupidendous bonuses back or something?

ttfn

[TeeCee]

Unreliable?

"...despite a .NET upgrade overseen by Accenture."

I can see two likely candidate reasons for it being a pile of poo in that statement alone.

[NB]

serves them right

for using a shitty platform. They should seriously consider getting some linux goodness ;)

[Anonymous Coward]

Some sort of bungled change maybe...

Hmm , hope the support in India didn't get their time-zones mixed up and thought it was end of trading .

Appears to be a holiday in the US also I note.......

How I miss days like this...sigh

[Bug]

Hmm...

Are they by any chance using WIndows, or employ Seimens to do their IT (or both)?

[Anonymous Coward]

It's worse than that

They're using Accenture. Remember, Siemens still has some credible engineering on its back, Accenture is draining as fast as it can its talent pool and replacing it with young unexperienced chaps in Philippines or India.

[Anonymous Coward]

Re: Seimens

Seimens IT services are not based upon credible engineering at all. Wait until the sory of this week's massive Seimens-induced network fail in the BBC gets out, that's all I am saying..

[Anonymous Coward]

Last outage was just two months ago

http://www.ft.com/cms/s/0/02ccdcb0-cd73-11de-8162-00144feabdc0.html?nclick_check=1

[Anonymous Coward]

"despite a .NET upgrade overseen by Accenture"

Hahaha hahahaha hahahahaha hahahaha <gaaaaaaaasp> ha haha hahahaha hahahahahaha hahaha.

Stop the internet, nothing can ever beat the humour of this statement.

[Anonymous Coward]

Right.. and?

"The TradElect platform on which trading depends has a flakey history despite a .NET upgrade overseen by Accenture."

Why use the word "despite" as if it was a good thing that Accenture upgraded a .NET application. Two wrongs don't make a right...

[Anonymous Coward]

Looking in the wrong place

From what I've seen of the LSE's IT operations, the problem probably lies between the desks and the chairs of the phools running the show. AC for obvious reasons.

[Anonymous Coward]

You know what it is, don't you

Turn off the Large Hadron Collider NOW!!!!!!1!!

[Ocular Sinister]

So how many times...

How many times has the "five 9s" MS/.Net platform failed so far? Three times? Or are there more that I'm not aware of? And that in the space of, what, 3 years? Considering MS/Accenture were treating this as a flagship installation to show the world how great their respective platforms are, this is a pretty poor show.

[Steve Cooper]

Normal service will resume shortly....

14:00 according to http://www.londonstockexchange.com/global/incident/incident.htm

Would love to know details on this - looks like a comms issue to me, not a core trade platform problem. Do they use BT Radianz for their connections?

[Sir Runcible Spoon]

@AC 13:46

surely you mean a chair to keyboard interface error?

[Stephen Channell]

... and nobody mentioned Java!

It’s kinda comforting to an old C++ developer to know that there are still applications that really do need careful management of the heap without a garbage collector.. back in the days of yore there was this robot at Imperial College that could play table-tennis.. except it missed every tenth shot because of the Pascal garbage collector.. real-time applications need real-time solutions.

..and Java.. nobody mentioned Java.. come-on SOMEBODY must think Java would have been better..

[Anonymous Coward]

But but but

Pascal doesn't have a garbage collector. At least old school Turbo Pascal didn't. We had to "dispose" of our memory. Perhaps it wasn't a Pascal table tennis player ?

Paris, cos she's WIrth it. Coat.

[Anonymous Coward]

We're from Anderson Consulting, and we're here to help you

There are no software or hardware problems: they are all 'soft engineering' problems arising in bad management decisions and dysfunctional administration. As in, contracting out your core competence to Accenture and sidelining everyone who offered a negative opinion: that is to say, everyone who knew enough to be able to sort out the specific 'software' problem that they've got today.

Doubtless the problem will be resolved to the satisfaction of everyone who matters by a corporate rebranding exercise.

Anonymous comment, for obvious reasons...

[Anonymous Coward]

@DEAD4EVER 14:47

Breathe! You're turning blue.

[Anonymous Coward]

Lol

Some guy I know was working on this upgrade a year or 2 back. He was the smuggest git i have ever met.

He also told me that a computer science degree was the only way to go if I wanted to get into the industry.

I never asked him about the details, but I should have.

[Anonymous Coward]

No surprise

Well WTF do they expect if they try to run a mission-critical application on a Microsoft platform? Honestly!

[Steve Cooper]

Windows

I reckon you'll find many a mission critical system runs on Windows... We run ours on a mix of Windows, Solaris and RHEL just to keep us busy :)

[Anonymous Coward]

The LSE isn't alone

I work for a global market data provider. These outages are not as rare as you think. I can count 10 failures globally on particular bourses for today so far.

These are very complex systems and sometimes even a single customer can take many months to migrate. Anyone that thinks you can move platforms in a matter of months is having a laugh. This migration will take years before it is signed off.

As for .NET hmmmmm..... we are about to purchase a new mainframe for this type of work. 99% of the systems that hang off our exisitng mainframe are Solaris or Suse. Windows is suited to the desktop but is nowhere to be found in the data collection side of things here. Even *nix systems are not failureproof although most of the issues are still human error. Going back on the previous large failure by the LSE the volumes for that day were insane compared to a normal trading day. The level of investment needed for a normal days trade would make your bedroom PC setup look like a VIC-20 in comparison.

[Mark 7]

Thought they got rid of Windows and put on Linux instead?

See above

Yes, I'm one of those annoying chaps who writes the entire email in the Subject!

[Anonymous Coward]

Really?

"The Exchange restarted continuous trading at 14.00. A spokeswoman was uable to tell us what caused the failure."

The poor excuse for development code on the .Net framework? The utter incompetance and sheer bloody mindedness of the management who refused to listen to reason when the tenders for the platform where put out yonks ago?

[amanfromMars 1]

Sharks in the Gene Pool

"You know what it is, don't you ..... Turn off the Large Hadron Collider NOW!!!!!!1!!" ..... Anonymous Coward Posted Thursday 26th November 2009 13:58 GMT

"We're gonna need a bigger boat!" ... :-)

[Mike Bishop]

@DEAD4EVER

Isn't it illegal for Microsoft marketing to employ 5-year-olds?

[Pandy06269]

@AC Re: Lol

"He also told me that a computer science degree was the only way to go if I wanted to get into the industry."

I see why he was smug - that statement alone is utter cr*p. I stuck at university doing a CS degree for 6 weeks before I got bored and wanted to do some real stuff - I left qualification-less and am now a developer responsible for the systems keeping 400 shops up and trading.

The only mission-critical failure we've had in 3 years was for 5 hours when a supplier's network route failed. The failure was caused by human error - some idiot decided to add an old router to their live network as a redundant gateway, but the router still had some old routes on it causing all our traffic to them to get redirected to a black-hole.

3 of the 5 hours downtime was them saying "there's no problem here; nothing's changed." As soon as we got our CEO involved, they'd accepted, identified and fixed the problem.

Badgers - because they can understand routes better than afore-mentioned supplier.

[Anonymous Coward]

smug ... shmug

... demonstrating that a CS qualification is not a requirement for smugliness.

[microspud]

A bit harsh!

As a DOT NET application developer myself, I think you are all being a bit hard on the platform. As far as we know the problems could be down to hardware issues with the servers or networking equipment.

A well written DOT NET application can be just as reliable as anything running on linux.

[vordan]

It's not the hardware

... they are changing, it's the software. Specifically, the .Net platform and applications.

.Net can be reliable for up to four or five users, I guess...

[Bob Gateaux]

vordan are child

see subject

[Pandy06269]

Re: Harsh .NET

I agree - to a point.

We have about 30 .NET applications deployed but the one we have the most trouble with (and I wish I had time to rewrite it into UNIX) is an app that processes in the region of 20GB worth of data in 24 hours - about 16GB comes in between 3 and 6am each day.

It constantly falls over with out of memory errors even though the server has 3.5GB free at the time. What I read up was that .NET will try and allocate some memory that's the maximum size in a single chunk - so a 2GB file will not fit into the server's RAM if there isn't a continuous 2GB chunk of RAM free.

If MS exposed the memory-allocation functionality, you could allocate non-continuous blocks of smaller sizes to fit it in - like I can (and have done in the past) on Linux/UNIX.

But I do agree, 2.7ms to process a transaction is a pretty darned good achievement - even if it isn't good enough for the LSE.

[Alex 3]

Besides...

.NET runs on Linux. Whilst you may find that situation/platform comination a joke - well... look at J2E etc...

[Seventh of 7th]

Swine Flew.

If the LSE is 21% owned by the Dubai Government and a Dubai state corporation announces that it can't pay it's interest, does that mean it's less of a software/hardware problem and more of a sleight of hand problem?

I wonder if they get charged £30 for a letter telling them that they've defaulted on their £48 billion debt repayments.

[amanfromMars 1]

Re Swine Flew ...... and One's over the Cuckoos' Nests too

"If the LSE is 21% owned by the Dubai Government and a Dubai state corporation announces that it can't pay it's interest, does that mean it's less of a software/hardware problem and more of a sleight of hand problem?" .... Seventh of 7th Posted Friday 27th November 2009 02:29 GMT

Seventh of 7th,

Do the other 79% of LSE have a Solution to Offer Dubai to head off Foreclosure and IP Ruin and an Unnecessary Gratuitous Malevolent Humiliation? And if not, why not? Is Intelligence Missing?

[Marvin O'Gravel Balloon Face]

maybe...

Maybe it was just a bad day on the stock market (what with Dubai going bust an' all) and someone decided to pull out a network cable till it all calmed down.