Virgin Media taps Microsoft in lengthy email outage Virgin Media customers have been suffering email outages for several days, prompting the firm to call in Microsoft engineers to help with an urgent upgrade. A mysterious configuration problem was identified on one of VM's eight email server clusters last Wednesday. Microsft engineers have struggled to identify the cause, …
My wife has been having trouble connecting to the email servers but we are NTL customers with @ntlworld accounts.
Either a conisidence that she has had problems since they started to have problems or they are not sure of the extent of the problem (the latter I suspect).
Still it is nice to see they have spent good money putting a cover on the Metro this morning to tell everyone on 4meg BB they they are now on 10meg BB rather that using the money to improve service.
Microsoft Exchange always seems to have given problems to Blueyonder, now Virgin, which is why I never use their email service. Personally I think Sky went down the right route by outsourcing it all to a company like Google - although the way the migration was handled obviously was pants.
Virgin should do a deal with Google / Hotmail / Yahoo or someone to run their email and concentrate on running the core network.
Go re-read the article
The current system isn't a MS Exchange, they are migrating to one.
The article is dosnt mention anything about there current setup.
And M$ have been called in to perform the migration as they cant get the current setup to work. I do love the fanboys who just read MS and jump to go on a bashing spree
/golfclap
Exchange is more than just an SMTP mail transport agent (MTA, as that is what SMTP defines). It's also a POP3 and IMAP server.
I'm not sure whether exim on it's own would do the job, and it is doubtful that it would scale to handle the large number of users that VM serves. You would probably need something like Dovecot, or the pop3 and imap features of Postfix.
I won't be on them much longer - moving to Hull (work) and the soddin monopoloy (even though they get arsey when you tell them it constitutes one) of Kingston Communications. I rely on gmail anyway.
Paris cos they say they aren't obstructing competition in the area, but by use of strange hardware and high wholesale price they are.
To all the MS fanboys jumping to defend them... why exactly?
Ok the article doesn't say in detail what they're migrating "from" but the comment that: "We have been working with our platform supplier Microsoft" and that they are now migrating to a "Microsoft Exchange 2003 platform" suggests that they are currently already on some form of Microsoft Platform to deliver their service (Exchange 2000 perhaps?) and are flailing about rebooting the systems, reinstalling software and now upgrading to Exchange 2003 (probably for free) to try and see if thats ready for servicing an ISP scale mail system. Anyone in the know care to dish the dirt?
"The current system isn't a MS Exchange, they are migrating to one.
The article is dosnt mention anything about there current setup."
If current system ($cs) is null (undefined), then I suppose $cs != exchange would be true. Outside of programming though, if the linux fanboys cannot make an assumption about the current system being exchange, then you cannot make an assumption about it not being exchange.
What the article does state is that they are accelerating the migration to exchange 2003. Maybe *their* problem is that they are moving to a 5 year old piece of software.
"Microsft engineers have struggled to identify the cause, forcing several reboots"
I can't believe that a company the size of Virgin Media is using Microsoft email server software.
I am a VM customer (ex Telewest) but fortunatly use my own e-mail server and as such have not been affected by the outages.
Anyway back to the point.
hmmm Linux./Unix/Solaris/BSD .... you know the operating systems that run the majority of the entire internet..... where you can stop/restart/start services and update non-kernel components without having to reboot.
Windows on the other hand :-
<MCSE Dave> Hey, Bob, this dang server still aint working right
<MCSE Bob> Have you tried rebooting it?
<MCSE Dave> Yeah 3 times already, but I have Vista at home, so I know that it's usually the 4th reboot that fixes it ... hangong a sec <CLICK> <CLACK>
<MCSE Bob> Hmmm that didn't fix it either. What else can we try?
<MCSE Dave> I'm just going to see if there are anymore updates we can install from Windows Update..... I don't think we have updated Media Player to the latest version yet.
<MCSE Bob> Yeah great, cos Windows update will force the computer to do AT LEAST 1 more reboot..... that should fix it.
<MSCE Dave> I hope so, we need to get this sorted and move onto the next job, we have a guy on the fith floor, who can't save his best scores on Freecell.
<MSCE Bob> Sounds like a nightmare, but nothing a couple of reboots won't cure.
Exim is *just* an SMTP server. It doesn't provide POP3 or IMAP (but that's what there are POP3 and IMAP servers for: it's better to be able to play one song from start to finish with no mistakes, than any number of cool riffs). It scales extremely well (just make sure you have plenty of RAM).
It also has a meaningful configuration file syntax. There really isn't much not to like about it -- unless you were educated at Oxford!
"Go re-read the article"
I did ..
"The current system isn't a MS Exchange, they are migrating to one"
If it isn't exchange then way did they ask MS engineers to identify the cause? Like why not ask the original suppliers?
'A mysterious configuration problem was identified on one of VM's eight email server clusters last Wednesday. Microsft engineers have struggled to identify the cause, forcing several reboots'
"The article is dosnt mention anything about there current setup"
It does, it says MS engineers are trying to identify the cause of the outages.
"I do love the fanboys who just read MS and jump to go on a bashing spree
I do love it when a ms.apologist refers to everyone else as a fanboy .. :)
Having just upgraded my phone & digital tv package to include broadband and v+ (which is nicer than Sky+) their email system is not very impressive. There doesn't appear to be any way to filter spam other than blocking the senders address! It's not a big problem as I already have more email accounts than I know what to do with. But, it does tarnish what is actually not a bad service - on the evidence of the last 10 days anyway.
I am one of their ex-blueyonder customers and the e-mail system is a mixture of Exchange and open source. There was a similar massive outage a few years back when they were entirely Exchange based, a few open source bits appeared to take some of the strain.
This is part of a header from an e-mail today:
Received: from smtp-in3.blueyonder.co.uk ([172.23.146.14]) by cluster2 with Microsoft SMTPSVC(5.0.2195.6713); Mon, 18 Feb 2008 15:44:14 +0000
Received: from eback01.blueyonder.co.uk ([195.188.53.212]) by smtp-in3.blueyonder.co.uk with Microsoft SMTPSVC(5.0.2195.6713); Mon, 18 Feb 2008 15:44:14 +0000
Received: from [172.23.170.141] (helo=anti-virus02-08) by eback01.blueyonder.co.uk with smtp (Exim 4.52) id 1JR8AA-0001DU-2U for --------@blueyonder.co.uk; Mon, 18 Feb 2008 15:44:14 +0000
Received: from [38.100.169.113] (helo=mta601.email.xbox.com) by exim13.blueyonder.co.uk with smtp (Exim 4.68)
We asked VirgTeleEuroBellWestMedia (as they are in this area, Eurobell box, Telewest endpoints, virgmedia branding) to move our cable service on the 27th February (We thought we'd give them a couple of days to mess it up a little before we actually moved). They said "Yes, that's no problem". They then went on to clarify. "We can disconnect you on the 27th, however we won't be able to connect you at the new place until the 29th". DOH!.
Still with how email and the service generally is right now with VM, we won't really notice a couple of days down time. It's not like it doesn't happen at least once a month as 'normal service'.
Icon, cos penguins are cute.
The lack of understanding around how enterprises work on here is astounding!
Also I know that the MS bods involved in something like this are not going to e idiots from the street! I have worked on large projects where senior MS engineers have been involved and most of these guys have a great understanding of the code behind the product and how it should work.
Before jumping to conclusions, you should perhaps take this into account. I'm no MS lover, but I keep an open mind.
That link in the article to the support ticket says what system it is - the affected cluster is Exchange 2000. Also they're not rebooting it - they're failing over between nodes, so it's individual servers that are going down, not the whole shebang. I wouldn't fancy migrating while it was in that state though.
Anyway, speaking of Exchange not being suitable for ISPs to run an email server - anyone know what Hotmail runs on?
As a long term customer I'm 99% sure they've been using Exchange for a very long time. At least since the days MS was a major shareholder in the old TW, at least for a short while.
I've got the mail headers to prove it and to the untrained eye it looks like they use Exim for mail processing but stick our mail boxes of Ex2k servers.
I've had intermittent and recurring problems accessing my Blueyonder mailboxes for a couple of months now. Because I use Linux and Thunderbird VM's technical support has said "sorry son, you're on your own because we don't know anything about Thunderbird. We know about Outlook." even though the issue appears to be with the mail servers; my TB is the latest version and is set up as per the recommended guidance.
Sometimes it connects, sometimes not. Of course, it was set up back when Telewest's support were based in this country and a very helpful chap in Liverpool confirmed I'd got everything right. Now you mention Linux to VM support and you might as well be talking Latin to them.
In the end I moved all my e-mail stuff to Googlemail. Problem solved.
Bottom line - from this user's perspective, VM failed. From the home user's perspective they just want reliable working e-mail. Something I didn't get so moved elsewhere. VM seem to lack a certain focus here.
Part of Virgin (The bit that used to be BlueYonder) use what used to be MCIS - Microsoft Commercial Internet Services - this was a service provider targetted package which provided SMTP, POP, IMAP, Authentication and NNTP in a single package.
I was involved in the first rollout of this pile of wotsit in the UK at Cable and Wireless back in 1998 - Microsoft ended up having to do 2 custom drops of SQL Server just to support the authentication needs of 50k customers.
Roll forward several years - Blueyonder are using MCIS for their email platform. MCIS doesn't have a very good reputation, but they blunder onwards with up to 20% mail loss as they go, but it sort of works.
Microsoft in time decide that MCIS isn't a good brand and rebrand it - it's now "Microsoft Hosted Exchange" which is effectively a tweaked version of Exchange itself scaled for ManyManyLots customers (BT Connect2Business had 1.1million users on it - don't know if they still have).
BT rolled Hosted Exchange 2003 a few years ago - by the sound of things Virgin are probably still on HE2000 and are forcing an upgrade.
Exim will scale to this size of user base, but it's only an SMTP server - BT Internet, prior to merging its mail services with Yahoo! was running Exim on its mail farm for 4 million users pushing something like 20 million mails a day. You still need a POP/IMAP server (BT used a customised version of Cyrus as I recall - it's been a while) and an authentication server, but LDAP servers are easy to get.
Thing is - if you want a *seriously* large messaging system you're really looking at what used to be Software.com's Openwave platform - it was designed to sit at the middle of a mobile phone network with tens of millions of mailboxes and punt messages whether they be SMS or SMTP.
Of course, the alternative is to craft your own - ask Yahoo!, Google et. al.
Instead of speculating....
[root@email ~]# telnet mailin.blueyonder.co.uk 25
Trying 195.188.53.99...
Connected to mailin.blueyonder.co.uk.
Escape character is '^]'.
220 exim8.blueyonder.co.uk ESMTP Exim 4.68 Mon, 18 Feb 2008 17:12:02 +0000
quit
221 exim8.blueyonder.co.uk closing connection
Connection closed by foreign host.
And :-
[root@email ~]# telnet smtpin.ntlworld.co.uk 25
Trying 81.103.221.10...
Connected to smtpin.ntlworld.co.uk.
Escape character is '^]'.
220 ESMTP server ready
214-This SMTP server is a part of the InterMail E-mail system. For
214- information about InterMail, please see http://www.openwave.com
They obviously have a mixture of technologies as any large ISP would.
"Anyway, speaking of Exchange not being suitable for ISPs to run an email server - anyone know what Hotmail runs on?"
Well... there was a big fanfare about them migrating off FreeBSD.... but when the Reg scooped that they'd only migrated the front end servers to IIS and the backend was still all FreeBSD it went all quiet. Perhaps they did migrate it all in the end, but I can't find a decent article about it anywhere. There's a tech-ed piece from 2000 describing the migration (http://technet.microsoft.com/en-us/library/bb496985.aspx)... but that predates the 2001 Register story about it just being the web front end that was moved (http://www.theregister.co.uk/2001/12/12/microsoft_hotmail_still_runs/).
So no idea what it's on. MS does however try to eat its own dogfood as much as possible (or just buy out someone elses) so I'd be surprised if they weren't using MS software by now.
Now you've hit a raw nerve. I have been a Virgin.Net customer for many years, and in general, have found the service fairly reliable (although this could be because I am not a cable customer, just ADSL), but their support STINKS.
I have a bare wire ADSL install to my own DSL router, to a Smoothwall firewall and then to a mixed wired and wireless network of Linux, MS Windows, Macs and even an AIX system (yes, this is at my home). I use a mixture of Webmail (for convienience) and fetchmail to pick up my mail from their mail server.
If I mention any one of these components in a support call, they just turn off and don't even bother to understand the questions.
I'm currently having capacity problems (I'm not getting even 10% of my paid-for 8Mb/s bandwidth), but am currently at the "have you turned off your computer and router and turned them back on" stage.
I took the rash action of sending them traceroute and ping timings to each of the hops, to show where the most likely bottleneck was, but I think I must have blown the recipients poor little mind, because I never heard anything back! They also took a very dismissive attitude when I complained that my allways-on link was being dropped several times a day, which screwed up my dynamic DNS entries. I now have to manually force my entry several times a day just so I have it working for some of the day.
One of these days, I will get so sick I will change, but I can mostly work without their help, so just get by.
My wife says that I should leave now. My coat is the one with all the network cable and micro-filters filling the pockets.
Oh dear, I'm really surprised they use Exchange for customers e-mail accounts. I would have thought they would have chosen a more robust Open Source solution, might have been fixed quicker too than relying on Microsoft.
I'm glad I host my e-mail with a decent hosting company.
Rob
P.S.
I wonder if they have to buy an Exchange CAL for each customer who has an e-mail account with them?
Not necessarily, they could be mail relays and no, ISP's don't use a mixture of technologies for the same thing, at least not when the whole thing is designed from the ground up. Thats what happens when you cobble several crap companies together into one big crap company through mergers and acquisitions.
Linux is an OS.
However, you make a damn good point; even a bog-standard Linux distribution will come complete with Apache2, Squid and various other server-specific items such as a SMTP mail transport agent and Pop3 and IMAP services.
I'm not really sure how the EU would view this....
The penguin, because they're from the Southern Hemisphere.
Incoming SMTP goes via the EXIM front servers, to the spam filtering servers, to the Exchange SMTP servers (mix of 2000 and 2003 at the moment)
To the person who can't work out how to configure the anti-spam - it's off the options part of the webmail system.
This is the only reason I personally use squirrelmail, the availability of plugins to control extras like spam settings
"...Sky went down the right route by outsourcing it all to a company like Google - although the way the migration was handled obviously was pants....."
You really have no idea. No, really! None whatsoever. How could you? You wasn't there. But don't let that stop you being an expert in such matters.
Have you ever had to mass migrate over 1.2 million email accounts from one platform to another within 24 hours, one that is not even under your direct control and influence? Would you even know what tools (thats a laugh) that Google supplied Sky to use? Or would supply any other ISP or corporation trying to do the same? Do you even know the logistics of running an ISP's email platform, let alone migrating one?
Like any migration of this sort the Sky one was not without problems. But the fact that they successfully managed over 90% of accounts in the first run is a testament to how well it was achieved. The remaining 10% were comprised of what could loosely be described as "problem accounts", ones that would need significant manual intervention.
Not connected with Sky or Google in any way, but the information to how it was done is out there for those that want to know rather than just cast opinionated slurs.
IT? icon for the benefit of those that wouldn't know IT if it landed on their foot.
[/RANT]
>I'm not really sure how the EU would view this....
it's not bundled INTO the OS. If you don't want it, don't install it. Or install a different web server. Windows Server OS's all come with IIS - should we complain about bundling that then - although you have to opt to install it? Your average distro comes with several options and a default setupthat works - usually Apache - cos it's secure, OS and just works, used to be Sendmail and fetchmail - your own secure, reliable email server inbuilt - IF you wanted it. If not, don't use it. so please DON'T compare it to Microsoft bungling - sorry - bundling IE and Media Player into Windows. It's NOT the same thing at all.
I'm with F9 (plus.net) and they use Squirrel. Very little spam (now they've fixed it), simple interface for webmail, reliiable, just works.
If you don't like Hull, don't come, I'm sure we could survive without your presence & KC / Karoo isn't a monopoly there are plenty of BT business lines in Hull and if you want a home line I'm sure BT will install one for you. As to other ISPs Kingston Communications operates a policy of selling their lines at market price not at a subsidised price as laid down by Parliament, a la BT, if other ISPs want o come in all they have to do is pay the market price for a line. Alternately they can go for true open market competition and supply their own exchanges and lines, then see what they charge. I live near Hull and have a choice of BT or KC and I have opted for KC because unlike the uniformed idiots who don't understand what free competition is I choose the better overall service for my needs. When idiots like you pay my pensioner mother the excess charges that she has to pay on her phone line and calls to subsidise your internet usage then I'll listen to you. We have a term in this area, pillock, go see.
"As to other ISPs Kingston Communications operates a policy of selling their lines at market price not at a subsidised price as laid down by Parliament, a la BT, if other ISPs want o come in all they have to do is pay the market price for a line."
Presumably, by "market price" you mean "a price set high enough so that no-one else can compete with them" - after all, that's what's best for their profits. Last I heard, they even charged much more for broadband than their own out-of area ISP, let alone the competition.
"Alternately they can go for true open market competition and supply their own exchanges and lines, then see what they charge."
Aside from the fact that is probably totally unviable due to cost, I doubt any company would be allowed to do it in the first place. It's not exactly trivial to get permission to dig up the roads everywhere to lay cables.
I'm a long-standing Telewest/Virgin Media customer, and the service has just been going down and down for years. Telewest themselves were useless at the end. It all started, for me at least, around the time of the NTL merger. Probably no surprises there for anyone who was on NTHell before...
I even remember the days of Telewest using OWA for webmail, and all MS-hating, Linux-loving nonsense aside, OWA used to work perfectly quickly and perfectly well. I was working away at the time, and used it every day with no fuss whatsoever. These days, whenever I try to use webmail from work, it logs in fine, but then it claims to time out almost instantly whenever I try to open an email, or delete something, or move emails into folders... I'm not a fan, put it that way.
If the new MS Exchange software goes some way to fixing these issues and speeding up the download of email to my Outlook client at home, I'm all for it.
Maybe we could wait and see what the service is like before we all start complaining?
Oh... and it is of course also possible that the reason they have MS engineers in is because a support contract with MS is cheaper than having their own technical staff dealing with it, isn't it? Especially if it's bundled in with the MS software?
Goggles, to protect me from the enevitable flaming for suggesting we wait and see.
Nope, just because KC won't bend over like BT doesn't mean they are setting the rate at an uncompetitive level, that is illegal. Read your 2nd comment again, realise what you have written, then talk about being competitive. I don't hear these comments about areas where NTL/Telewest/VM have control through their cable supplies. People who preach about open markets then find a reason for their favourite product being subsidised make me sick. I am not prepared to pay higher phone / BB charges just so some other company can come in and cream off users.
As to the unrealistic claim, I became a KC subscriber because they expanded & invested in cables etc to my village so that they could sell me their product in fair competition to BT. Until your favourite BB ISP is willing to do the same they have no relevant comment to make.
I can't believe some of the overwhelming crap being spouted off here, you all seem to think Exchange server is some sort of Office app. No doubt if VM's SQL servers had of gone down you'd be shouting how it should be MySQL. Well no it shouldn't, MS Exchange host some massive outfits and this one is no different and the comments about having a Linux solution that would get fixed sooner is the biggest joke of the lot.
NTL users' email woes don't have any connection with this Blueyonder outage. There's no connection between those systems. The only parts of Virgin Media that are joined up (even all this time after the NTL:Telewest hookup) are billing and complaints, which seem to be their main growth areas...
Including BT Business, it's very good. Though I wonder why they are using the 2003 version when 2007 has been around for years.
I am no big user of MSoft (Mac and Ubantu and MS MacOffice 2008) also I question why you would want to dedicate all that kit and server room space to such a server hungry mail system.
But, at least VM knows who to call when they have a problem. Unlike the weakly supported Linux platforms, MSoft has a lot of competent engineers to call on; I wonder if Cambridge University could send around a few undergraduates to get VMs mail systems up and running.
I strongly doubt many others could put half a dozen engineers on site within a day of a major failure.
VM is a business, not a technological theocracy, and needs must when the devil drives.
It's not just the email that has gone down the toilet - coincidently all the problems seem to have started or gotten progressively worse since Virgin took over, but even generally browsing and network speed is now diabolical!
I've been a business user of TW broadband for 7 years and this is easily the worst service/availability I had during this time.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
