Is this a new form of network attack?
Looks like Skype is the first service in history to DDoS itself... It reminds me of the Judean People's Front Crack Suicide Squad!
Skype has blamed last week's prolonged outage on the effects of Microsoft's Patch Tuesday. The latest security update from Microsoft required a system reboot. The effect of so many machines rebooting and subsequently trying to log onto the Skype VoIP network triggered system instability and a prolonged outage of almost two days …
"Microsoft's publishes its monthly patches on the second Tuesday of every month. Many clients will automatically download the updates on Wednesday and apply them on the next system reboot, which could be Thursday morning."
Yes, but I'd wager most home PCs are set to Automatic Update (the default since XP SP1?) - which automatically downloads and installs the patches, there and then. And then begins nagging the user to reboot every 10 minutes, until they do so. I'm sure most Windows-using readers have seen, cursed and hit that "reboot later" button plenty of times.
Only on a scheduled update is the installation delayed (typically, this might be used with an AD scenario and controlled via Group Policy, for larger/corporate-type installations). And if you have downloaded updates, you will be prompted to install them when you go to shut the system down (if you're shutting down, are you likely to restart it just to see if the patches worked?).
I would therefore think the majority that download them get them installed practically instantly.
Lastly:
"Many will apply patches before then but are unlikely to do so en masse at the same time."
There's no difference between Skype coming online because we woke up and just started Windows, or Skype coming online because we just restarted Windows because of a patch.
And anyway, this isn't the first month MS have released patches on first Tuesday. They have been doing that for ages now !
If Skype has a bug, then fine - but I'm not convinced the AU theory holds plenty of water...
Given all the online talk recently about the spooky, near clandestine, links between Microsoft and the BBC I find it very interesting that the BBC news version of this story doesn't mention Microsoft or Patch tuesday at all.
See: http://news.bbc.co.uk/1/hi/technology/6954675.stm
While I obviously don't have access to Skype's monitoring systems, so don't know for sure, I'd be very surprised if their explanation were true.
For one thing. as stated in the article, a lot of Microsoft patches require a reboot, and Microsoft have had patch tuesday for a couple of years.
For another, a lot of people don't leave their PCs running/logged on overnight. AFAIK, SkyPe doesn't run if a PC is sitting there not logged in, so I'd be surprised if their network loads the next morning were much higher than normal.
>Yes, but I'd wager most home PCs are set to Automatic Update
But most PCs are corporate, so are likely to have their patches applied at the same time thanks to a site or company wide policy.
I think the note from skype is only half the story - yes, the multiple simultaneous log-in attempts must have meant that nodes were overloaded, which made for a slow "start-up", but there have been many patch tuesdays and none of these have caused this supposedly "4-year-old" problem.
Explain also the release of a new client on Friday? If this was a central algorithm issue why would this be needed?
The issue has to be the content of the patch in question - why skype are holding back on admitting this is interesting to say the least. You'd think it would be in their interest to blame Microsoft - especially with their "Live" competing products now available...
Over the years Microsoft has got better with the testing of its patches but any large organisation should NOT be applying all the MS patches on day 1. I know that it wasn't the patch itself that caused the outage but it was the way the patches were deployed. WSUS (2 & 3) are not a brilliant tools but they do let you test and stagger the deployment of the patches that are released.
>But most PCs are corporate, so are likely to have their patches applied
>at the same time thanks to a site or company wide policy.
Yes, but probably some weeks later. Also - how many companies with a patching policy will be letting their staff use Skype? Corporate PCs are a red herring.
That said, so is the Patch Tuesday idea. Most normal people turn their PCs off overnight anyway.
"Given all the online talk recently about the spooky, near clandestine, links between Microsoft and the BBC I find it very interesting that the BBC news version of this story doesn't mention Microsoft or Patch tuesday at all."
I wouldn't read too much into that, because:
a) Skype's explanation (of which both the Register and BBC articles are essentially rewrites) doesn't mention Microsoft either (see http://heartbeat.skype.com/2007/08/what_happened_on_august_16.html) ; and
b) the BBC article does in fact mention that the restarts were caused by a Microsoft security update.
">Yes, but I'd wager most home PCs are set to Automatic Update
But most PCs are corporate, so are likely to have their patches applied at the same time thanks to a site or company wide policy."
Yes, most corporate PCs are updated - as you say - at the same time via policy. For that same reason, most corporate PCs use a local patch distribution mechanism such as SMS or WSUS. Otherwise, you get thousands of PCs all going out via the company link and downloading the patch. And what organisation lets their clients just run out and install the moment a patch is released anyway ? "Change control" is the order of the day in any well managed organisation (christ, I sound like my boss..)
Corporate patches tend to be rolled out more conservatively to clients and servers. Patches frequently cause problems of their own and just plain can't be trusted in a global push to the corporate environment. This is why we employ methods such as WSUS and SMS, so that we can target patches to test groups first, and then roll out gradually.
For this reason, I'd expect that we can largely discount corporate PC users from the Skype traffic group. Plus, and fwiw, we don't allow Skype within our domain anyway. I wouldn't expect we're alone in that.
"Given all the online talk recently about the spooky, near clandestine, links between Microsoft and the BBC I find it very interesting that the BBC news version of this story doesn't mention Microsoft or Patch tuesday at all."
Actually it does. This is how misinformation starts, through misinformed comments linked, tagged, pinged and trackbacked across the web....
"I wouldn't read too much into that, because:
a) Skype's explanation (of which both the Register and BBC articles are essentially rewrites) doesn't mention Microsoft either (see http://heartbeat.skype.com/2007/08/what_happened_on_august_16.html) ; and
b) the BBC article does in fact mention that the restarts were caused by a Microsoft security update."
a) Yes, agreed.
b) Actually it didn't say Microsoft originally but they updated it at 12:08 today to add a "from Microsoft" to the end of that paragraph.
I wonder if the Beeb reads Register comments?
Microsoft has that horrid, arrogant nature -- either you accept automatic updates and reboot "when WE say so, or you do them manually yourself which means they probably don't get done.
Neither is acceptable. A third choice is urgently needed -- install the patch, but put up a less-intrusive "reboot when convenient" message and leave it entirely to the user. No nagging in the meantime, and certainly no rebooting if the user doesn't manage to say no soon enough.
The ten minute timeout (followed by a reboot without the user's permission) doesn't help in any way if the user has gone to lunch or is in a meeting at the time. Just because the user isn't watching the screen doesn't mean that the machine isn't doing essential tasks that shouldn't be interrupted.
Kinda fails to take in timezones doesn't it? AIUI, the auto-reboot for PCs left on is 3am local time, if the machine is left on, which would mean the "mass login" would be staggered.
Similarly, for folk who switch their PCs off overnight, again, they would all login at their usual time - whatever that is - a nearly random statistic.
Corporate PCs? All of the ones I know patch from a central SMS server, not from the Microsoft vaults (it's a standard setting in most group policies) and again, the ones I know, tend to let the patches install on a Friday night, so that IT depts have a short time to fix things if they go wrong.
There may have been some connection, but frankly, it's hard to see how, unless you're a committed MS-basher.
Ah. (looks at URL bar) KK....
I've worked on systems that are meant to "self heal", that is to say ride out resource shortages, outages, and the like. They're notoriously difficult to test, because the only way to test 'em properly is to crash your whole production network and then see if it recovers. I think Skype could be forgiven for not wanting to run such a destructive test case.
I wonder if it mattered whether the *user* machines restarted, or if it only mattered that skype's own network of server machines all rebooted at once.
Just keeping a small production system available 24/7 can be sweaty work. I've had my turn in the barrel. I'm glad it wasn't me at skype desparately trying to figure it all out and get a million customers back online before they all defected to competitor X.
"Microsoft has that horrid, arrogant nature -- either you accept automatic updates and reboot "when WE say so, or you do them manually yourself which means they probably don't get done.
Neither is acceptable. A third choice is urgently needed -- install the patch, but put up a less-intrusive "reboot when convenient" message and leave it entirely to the user. No nagging in the meantime, and certainly no rebooting if the user doesn't manage to say no soon enough."
Get your network admins to change this via Group Policy - it's an option in there to either change the default time from 10 minutes to whatever, or not nag the user at all.
For the home user, the simplest way to kill that message is to wait until your patch is installed - and then do Start, Run, "net stop wuauserv". Once that service stops running, so do the nags.
This is probably a combination of Skype's own updates and Microsoft's updates. Skype probably had changed their algorithms in the past month, and had inadvertently introduced a bug which DOSed their system when Microsoft caused too many PCs to reboot. There are certain scales of testing that can't effectively be done inside a company. Think: what company has 2000000 test clients? (answer: none!) The most test client machines I have worked with has been 200, and that number would be inadequate to forsee the Skype login bug.
> Most normal people turn their PCs off overnight anyway.
A random guess! (or if you have numbers, let's see them)
I'm certainly too lazy to shut down every evening, then boot up and restart all my applications, position windows, etc. the next morning. And I'm convinced most people are as lazy as me.
I can't agree that an automatic reboot without user confirmation is acceptable in any circumstances. Microsoft are just arrogant in forcing this on users. As a very occasional reluctant Windows user I am amazed that the general public will put up with this.
This particular Tuesday was worse than usual, my wife's PC was hung on the way down, and I had a frantic call from my brother telling me that all the PCs in his office had rebooted, and worse still, Outlook Express had lost 4 months of emails. Guess who's getting the job of building a robust mirrored office server (Linux/Samba/IMAP) with a backup solution?
I used to work for a UK company with a US parent. It was policy for ALL pc's to be left on at night so that patches could be sent down by the US admins. Change control was done in the US. The size of the US parent meant they had advance notice of the patches from M$ and would tailor them to suit the apps we used. Server patching was manual and via SMS. Given that we had something like 1.3 million users worldwide, it was easy to create a problem with a bad patch. Although not the same, a virus attack from elsewhere in the world meant 40 people manually patching every office machine in one night - about 2000 machines. Repat that throughout a global network and there might be some truth behind Skype's statement.
We didn't allow Skype but it's on the record that US Robotics do. I wonder how many machines they have on line at any one time...
A third choice is urgently needed -- install the patch, but put up a less-intrusive "reboot when convenient" message and leave it entirely to the user. No nagging in the meantime, and certainly no rebooting if the user doesn't manage to say no soon enough.
There ios a third option. You can set it to install the updates when you hit the shut down button
The so-called Protect America Act, which passed both the House and Senate by wide margins just before Congress went on its August recess, allows the government to intercept the phone calls and e-mails of people in the United States who communicate with people overseas, and for the first time, allows the government to intercept communications between foreigners which are merely routed through the United States, as well as conversations of Americans traveling abroad.
...
To enable compliance with the new U.S. laws, which also include that the service providers such as Skype are not allowed to report these activities and are to be immune from prosecution claims for example for violation of the U.S. constitutional or legal rights to privacy, it would be necessary to ensure that the Skype super-nodes are upgraded with software modifications to ensure more centralised routing and easier access to monitoring.
...
Now, those agencies are free to order services like Skype, cell phone companies and arguably even search engines to comply with secret spy orders to create back doors in domestic communication networks for the nation's spooks.
...
Copied from:
http://mathaba.net/0_index.shtml?x=561193
"Just because the user isn't watching the screen doesn't mean that the machine isn't doing essential tasks that shouldn't be interrupted."
Oh, be serious! Even Microsoft knows that no one does anything essential using MS Windows. That's why it's OK to reboot spontaneously whenever there's an application crash (or are you unaware of the default settings for that?).
If you're doing something essential, you should be using a real OS, not the Xbox Game Console OS.
I dunno, the compnay where I work (it's a technology firm designing applications for mobile phones) uses Skype on a regular basis as a communications and collaboration tool - I guess the like the whole range of functions it supplies. I wouldn't be surprised if other companies do similar things.
Amazing how the MS bashers are really quick to start thumping given even the slightest opening. "Oh, MS is arrogant, wahhh" Have a good cry.
"...a previously unseen software bug within the network resource allocation algorithm..."
Read it again for good measure. This is Skype's bug, not Microsoft's. Lay the blame appropriately onto Skype for trying to pass the buck. But then again, "M$" bashing is cooler, isn't it?
I can accept that a bunch of machines all rebooted in the corporate arena, but surely if they were corporate machines they would boot to the login screen and halt until a user logged in? I guess Skype doesn't start until that's happened (if they used it at all), so rules out the multiple logins at the same time...
That said, I don't remember my PC forcing me to reboot - it just nagged me. I could click the "remind me later" option to my hearts content.
Wouldn't a mass Skype client update have the same effect anyway?
<< Also - how many companies with a patching policy will be letting their staff use Skype? Corporate PCs are a red herring.>>
Well, I might buy the 'patch Tuesday' idea. But...
At the place I work, a Very Large Telecomms Manufacturer(tm) we _can_ use Skype, and our patches start automatically, then force reboot.
The "But", of course is that we have to log in again manually. As we work flexitime, don't see us all rebooting/logging into Skype at the same time.
Well, Skype should have written their software, which runs on the Windows operating system, to NOT connect automatically.
Or perhaps they should have created more stable server software that had a first-line login server that would only process so many logins and defer other logins for a short while.
No sympathy for Skype here, I'm afraid.