ISP panicked by MS Patch Tuesday ISP group 186K has blamed potential congestion on its network this week on bandwidth being swallowed by Microsoft's monthly security patch update. Redmond unleashed six critical fixes on Tuesday. 186K's engineers first suspected a new virus, according to the firm's status page, but later on Wednesday morning said it was …
I have absolutely no sympathy for these ISP's - they're the ones who created the market for cheap broadband and can now no longer deliver to customers expectations.
Serves them f*cking right!
I've stuck with my ISP who has always charged a premium, but for that I get an ADSL2+ connection to a good network which can easily maintain a throughput of 17Mbps.
So, rather than offering a quality service at a maintainable price, they oversell their capacity and make promises they can't keep!
You offer to provide an Internet service, and then your customers go and spoil it all by using it to *download files* - who'd'a thunk it?
Honestly, if you can't cope with a few MB of MS updates, you're really going to struggle when we're all streaming HDTV to our iPlayers ...
<rant>
Perhaps if ISP's can't afford to handle the bandwidth requirements incurred by the use of new media-centric applications and über-rich web content, they should step aside and let the big players use the revenue in more constructive ways. I mean, how pathetic, crying because Microsoft's updates are too big!
Perhaps this new trend of "big bandwidth-eating service arrives, ISP complains, nobody listens, bigger bandwidth-hungry services come along anyway" is a good thing, there are so many small-time ISPs (especially through LLU) these days that if some of the smaller ones were to pack up shop, the inevitable shift of their customer-base to companies who actually own their own infrastructure (like BT or Virgin Media) might give those larger providers enough extra revenue to stop complaining about perfectly resonable services such as the BBC iPlayer (alright, maybe we can strike the 'perfectly reasonable' bit) and do something about their infrastructure; sorry OFCOM, but it's only going to get worse with Silverlight, HDView, ad infinitum...
Sure, OK, it's a free market, competition is healthy, but in the ISP world surely it's better to have a few really large companies who therefore have larger customer-bases and can then spend more money on core infrastructure and service provision, than hundreds of little unbundled providers each with a small percentage of the overall national subscriber-base, thus spreading the total national income from Internet services so thinly that nobody can afford to do anything worthwhile!
Alright, so maybe BT and Virgin are not good examples of this point, what with BT Vision and Virgin's Teleport TV and all, but hey, if we must have competition, let's do it with the services, not the infrastructure!
</rant>
So they moan about all the f**king spam and viruses congesting the networks, then moan about the patches to fix them.
Microsoft don't need to trickle them out. Timezones and habits do that.
This week I have not logged onto one of my home pc's. Thefore it will not get it's patch until the weekend when I will log on.
Next they'll moan that we spend to long on the phone to India when the service falls over, clogging up the "service" desks.
We saw about 550Mb/s in use on our network on Monday as a result of the same thing (reported here): http://community.plus.net/forum/index.php?topic=803.msg7261#msg7261
Now, we aren't complaining, but as a result of the traffic prioritisation on our network our customers found their download capacity (P2P and Usenet are treated with less priority on our network than this traffic) would have been given up to support the patch download.
550Mb/s is around £50K a months worth of bandwidth based on the current BT IPStream pricing. There ain't exactly a lot of margin on Broadband to play with at the moment, and I'm interested in whether you guys commenting would really rather we shoved an extra 25p a month onto all our customers in order to support Microsoft patch downloads? It may be an option, but I'm afraid I'm a believer in affordable broadband rather than the Internet being something only the well paid can afford!
Ian
PlusNet Product Team
and in (parts of) S Korea the (domestic) consumer gets 100Mbs to the premises. The fault in the UK lies not with the smaller ISPs, but with the chronic under-investment in the national infrastructure, 20 quid a month for 2Mbs at 50:1 contention over a twisted pair of 68 year old bits of string soaked in poodle piss - miserable...
So, cutting out the smaller ISPs won't cure the problem
<python> Poodle piss? - you don't know how lucky you are. When I were a lad we lived two and a half miles from the nearest DSLAM and had to piss ont' telephone cables ourselves at 4 o'clock int' morning if we wanted to get more than 56kbs </python>
"I'm interested in whether you guys commenting would really rather we shoved an extra 25p a month onto all our customers in order to support Microsoft patch downloads?"
Give us a break, Ian! If PlusNet were doing anything to stop the distribution of malware to *and from* PlusNet customers, you'r bandwidth requirements could be cut, possibly as much as 80%
But instead, you want to charge each customer an exta 25p monthly so you don't have to get off your lzay arse and do something constructive.
Well, that *is*, after all, the PlusNet historical example.
"Make windows update use a bittorrent-like protocol to reduce traffic! "
Are you suggesting that we download critical OS patches through the same machines that are a signifficant part of the problems the patches address?
Not only that, but the same amount of data would have to be transfered regardless of the transfer protocol. Using a "bittorrent-like" utility might make it cheaper for M$, but everyone else (ISP and users included) will suffer even more.
Next time, before you make such a genious suggestion, count to 10...
For starters, I've been with Plusnet and they SUCKED - lippy, arsey, know it all jobsworth technicians who havent an inkling about how to deal with customers.
Secondly - why oh why dont you all just stop winging (on both sides) and cajole your relevant ISP into installing a WSUS server?
http://technet.microsoft.com/en-us/wsus/default.aspx
The ISP can then provide a small download to customers that sets up the relevant group policy settings needed (after accepting relevant legal agreements). Lo and behold, MS Updates as provided by your ISP, hence cutting down internet pipe traffic no end.
Even if some DSL lines still get saturated, then you'll at least cut down on internet throughput for those on different DSL exchanges that are not saturated.
Better for PR for the ISP too.
If torrents work correctly, you should end up downloading the patches from the machines that are closest to you on your ISP's network, so probably your neighbours. This way the traffic does not have to go through all the pipes of the ISP, just as few as needed. As for infected machines, what do they matter? Everything that is downloaded gets checksummed anyway.
Now maybe torrents don't work as well as they could... Then that clearly is something to investigate.
"Are you suggesting that we download critical OS patches through the same machines that are a signifficant part of the problems the patches address?"
A valuable Mutually Beneficial Servering of Delinquent Action... which might have them confronting the reasons for Security Patching and whatever it is they are Patching against........ which is usely Operating System Use.
Of course, they could just implement transparent proxying so at least the data is coming from within the ISP's infrastructure (providing their central pipes are big enough), but then you have users up in arms about not liking that.
You can't please any of the people, any of the time. However, you can impress girls with the size of your pipe.
I'm always bewildered by the lack of ISP-centered update servers. Almost all the ingredients are there:
- OSes that already support central patching via company-wide update servers (Mac OS X recent-ish, most Linux, Windows XP/Vista).
- Strong crypto and signing to ensure a hostile ISP can't attack user machines
- Strong crypto and signing to ensure machines can get verification of update manifests & update notifications from a central source (so hostile ISP update servers can't hide/delay patches and exploit vulnerable machines)
Unfortunately, the aforementioned OSes are designed to get patches as part of larger central management facilities from trusted servers. They provide no mechanism for discovery and use of untrusted patch mirrors - even though such a method would not be hard (no harder than, say, automatic proxy discovery) and could be made quite safe.
With Apple and Linux machines both facing tens or hundreds of megabytes per patch, and the wide deployment of Windows, you'd think they'd be wondering about this. Networks like Akamai help .... but not as much as pushing patch hosting onto the ISP as a bandwidth saving would.
I'll refrain from commenting back on the post about the MS bash (e.g. the well over 100 patches I needed to download on a Fedora install a couple months back) - whoops, I just did.
Anyway: ISP's have no excuse. It's like BT complaining that customers are phoning up to donate to Children in Need or Red Nose Day. Freephone numbers (thus flat rate - e.g covered in the line rental) and 100,000's of calls all at a short time. There's no difference.
YOU ARE PAID TO PROVIDE A CONNECTION TO DATA NETWORKS!!! HOW CAN YOU COMPLAIN WHEN PEOPLE USE THEM TO DOWNLOAD!
And as someone above mentioned, it's not all at once. Time Zones play a big part, not everybody has Vista and a number of Vista machines are behind private networks that use WSUS for updates - plus some people turn their machines off when not in use, or just have notifcation rather than download/install options turned on.
How is this any different than when a Service Pack is launched (XP SP2 anyone!)? Didn't hear any ISP's complaining then.
And actually Ian:
"as a result of the traffic prioritisation on our network our customers found their download capacity (P2P and Usenet are treated with less priority on our network than this traffic) would have been given up to support the patch download"
If you ditch the fucking stupid 'traffic prioritisation' (spying on MY data traffic then sorting in order that YOU think is best) then it wouldn't be an issue would it - don't try and make it sound like the customers are being fucked over through patching and downloads when it's your shite network infrastructure.
Instead of charging us more money, why not reduce costs? Decent spam filters at the cloud level (e.g. MessageLabs) would cut traffic no end. And then again the same could be applied to viruses. Shove some high end FortiGate ITM or Cisco kit in that scans for viruses on the fly - the same as you do when you monitor my traffic and deem it low priority regardless how much I pay you.
Bottom line is that cheap ISP's have got so cheap, that they forgot to add on a decent margin and predict that data use will INCREASE as the future unfolds. That's call a business plan and pricing strategies - suggest your overpaid fat-cat's look them up before making stupid comments that we are downloading too much data.
Fucking morons.
In 2007, nobody gets Internet access. It's all about the Web, baby.
See, although people often think they're synonymous, they're not, and those of us who think we're supposedly paying our ISPs for Internet access tend to get disappointed quite regularly.
Unfortunately, the same scenario is often mirrored in the mobile world (especially at the cheaper end of the market), where anything non-port 80 is regularly embargoed, and the Web traffic is non-transparently proxied to the hilt, despite being billed as “Internet access”.
I may have missed something, but from what I have seen its only the big ISP's that are bitching (most of whom have a poor reputation for throttling or hidden/low caps) My own ISP isnt well known, but regularly invests in extra capacity to match new sign ups, (3 new pipes this year) the result is a network which maintains a reasonable speed, and thats the problem, the big names trade on their brand name.. not the performance of their network.
Yes I have a cap but its on a generous allowance, yes they have some management in place - "as required" to maintain latency for gamers, no its not 47p per month, but we have reached the stage where the greed for customers has led some foolish ISP's into a corner of their own making, they have such low pricing that they cant maintain a decent service, Im paying slightly less than I did for my original 512k unlimited package and Im getting around 5 megs.
When will ISP's stop kidding people that they can have up to 8 meg unlimited broadband * for pennies a month? That wont happen while people are held to ransom by BTw's overpriced centrals, ancient copper wiring and Marketing led "major" ISP's who promise the earth and deliver only some poor grade dust.
* (Full Up Pipes apply)
Trickling out patches over time would be fine, because Microsoft only releases patches once a month anyway. Presumably some of those patches are finished before the deadline, so just encrypt them and allow them to be downloaded. You can then just release they decryption key on Patch Tuesday.
First of all: plusnet p**s off. coming on here and posting officially stating you "haven't complained" as if it will give you some credibility... well it won't! no one cares about the margins this or the IPStream that - simply provide the service or pack up and get out of the game. It's quite frankly about time several ISP's did this.
Secondly: WSUS. Stu & Craig hit the nail on the head. I look after hundreds of PC's in dozens of orgainsations all running WSUS 3 and with the same ISP. Because of WSUS the ISP only has to get the updates a few dozen times instead of a few hundredm - but if they had their own WSUS server then it would be only *once*. It is not hard to configure an xp/vista PC to look elsewhere for updates; just run a .reg file. And what is wrong with locally caching/proxying these files anyway? J0k.
If you follow auto update, microsoft DOES slowly pass data to your computer if you elect to have auto-update. It litereally is in a low download state until its complete; generally going hyper near the tuesday. (Run a packet sniffer, I know I did...I have to test WSUS regularly...sigh)
Would the folks posting about caching, p2p, etc as "solutions" please either STFU or get a clue about how DSL is actually priced and sold in the UK?
The problem facing most UK DSL ISPs is the (very very) expensive bit between the ISP and the DSL customer.
If a DSL customer wants the data, it has to go through that expensive bit (largely provided by BT, except in the case of the LLU ISPs).
It doesn't matter whether the data is Patch Tuesday or IPtv, or Linux ISO, or if the data is cached at the ISP, or if it comes from other customers at the same or different ISPs. The dominant cost of getting it to most UK ISP's customers is ALWAYS the BTw-provided (Ofcom-price-regulated?) bit between ISP and customer.
The cost of carrying data between ISPs, or the rest of the "core internet", is trivial compared with the cost of getting data to and from UK DSL customers.
For more facts on this subject have a read of Page 3 of this IPtv-focused Register article from February: http://www.theregister.co.uk/2007/02/10/iptv_vod_joost/page3.html
Steven Hewitt wrote
"Anyway: ISP's have no excuse. It's like BT complaining that customers are phoning up to donate to Children in Need or Red Nose Day. Freephone numbers (thus flat rate - e.g covered in the line rental) and 100,000's of calls all at a short time. There's no difference."
Absolutely wrong. When you phone an 0800 number the call is paid for by the recipient of the call. So if you phone the RAC on their 0800 number because you have broken down then the duration of that call is paid for by the RAC. So if you made 100,000 of calls then the company running the 0800 number pays 100,000 times.
This is where it would behove the government to get involved in supporting "Broadband Britain" with more than just words
Recently it was stated it would cost upto 10 billion pounds to get fibre to 90% of uk households.
I am sure that millions of Brits would feel that having fibre to the door (and cheap, fast, reliable and high speed broadband) would be a much greater benefit to them now and for the next 10 years than the 2 new aircraft carriers they blew nearly that much money on. Get the fibre laid like Stockholm did and then rent it out to a consortium of ISPs to install the equipment, maintain and run it.
Heck a simple longer term solution would be if they at least be specified in planning regulations that all new development has fibre to the door included as standard even if the equipment is not installed at teh exchange end that way the most expensive part to retrofit (getting the fibre underground and to each of the houses) is already done when they finally do realise that 20th century copper is not upto the job of 21st century communication.
So whats better for the people of Britain and the country as a whole? A) fibre to the door or B) more firepower to be the US of As whipping boy
I was a Metronet customer - Plusnet - took them over - the service level sank
The customer service was nearly non-existent - Web form based or HOURS on Non-Geographic phone numbers.
They were taken over by BT - no rebranding so little publicity
Service was still CR*P - so I asked for my Migration Code and after a number of phone calls and support ticket exchanges left in May for a LLU - Not Virgin (which is available to me)
[The Billing system is still emailing me & trying to bill me for the current month - 3 months later!!!]
MS patches download in an instant. - BBC IPlayer and 4OD files download at about 2/3rds Real-Time
All the ISPs need to do is invest sensibly in Backhaul Pipes and try to get out of the BT Local loop Cartel
Well yeah - We had a pretty painful (Some might say gaffe prone) period of time. If anyone does have an outstanding issue or something I can still help with do feel free to email me - iwild@plus.net. I can only hope we're past that stage now - Clue certainly seems to have returned to the building.
Anyway, I am a bit surprised that so many reg readers would have such a misunderstanding of both the situation discussed here and of how Broadband delivery in the UK actually works though. It's not about Internet capacity (Cost £10 per Mbps), it's about what we have to pay to BT to traverse the data across their network. That costs at least £200 per Mbps, and as each customer is on an ATM tunnel there is no where that we could aggregate the traffic to reduce the costs at all.
Now, you can call us money grabbing if you like, but if it were your business how would you solve this problem?
There is also some more discussion here:
http://community.plus.net/comms/2007/08/17/broadband-your-way-blueprint/
Ian
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
