Call of Duty: Modern Warfare fragged our business VOIP: US ISP blames outage on smash-hit video game rush US ISP Windstream says a video game update is to blame for an outage affecting its business VOIP service earlier this week. In a notice to customers, seen by The Register, Windstream explained that mass downloads for the latest Call of Duty title throttled its network capacity on January 22. This flood of traffic meant some …
Windstream is a legacy telephone company so if there is one thing they should get right, it's voice. They're the "provider of last resort" for most of their customers. Except for small areas with a fiber buildout they offer 3-6 mbs DSL service. You can get into the low 20s if you're close enough.
My business was eligible for 3 meg service from Windstream while cable offered 200 meg for the same price or even less. That was an easy choice.
Games like Fortnite... or GTA for that matter, have equally large user bases, with large patches. You don't see Valve complaining to... Akamai?
You are looking at 20+ million users downloading 100GB games ALL AT ONCE eventually.
Netflix... hello...
It isn`t unexpected, just unplanned.
Should be the acronym for this effect.
Essentially the fix is simple: distribute the load so if more than n games consoles all ask for a patch at the same time, it gets downloaded once then cached.
Even if this means other folks have to host the file fragments similar to iOS and W10 updates.
They term it delivery optimization, it's configurable by the user and described in https://support.microsoft.com/en-za/help/4468254/windows-update-delivery-optimization-faq
"Windows Update Delivery Optimization works by letting you get Windows updates and Microsoft Store apps from sources in addition to Microsoft, like other PCs on your local network, or PCs on the Internet that are downloading the same files. Delivery Optimization also sends updates and apps from your PC to other PCs on your local network or PCs on the Internet, based on your settings. Sharing this data between PCs helps reduce the Internet bandwidth that’s needed to keep more than one device up to date or can make downloads more successful if you have a limited or unreliable Internet connection."
Note: "PCs on the Internet" is under one of the options.
Basic qos model
Tag all voice packets as dscp EF
Apply bandwidth policing to give EF packets priority and minimum bandwidth
Enable qos on all uplinks and make sure peers are using same marking policy
Locate traffic from update servers and tag as dscp 0, set traffic to 10% of bandwidth and drop if it exceeds this.
Anyone who has managed a corporate network with voip should understand this
I'm surprised as well that they are not either using the standard IP ToS (Type of Service) flags that VOIP services use for QoS purposes already, or at least making sure their own VOIP traffic get some kind of priority.
I suppose ISPs could be stripping these flags off as they pass traffic through, if VOIP traffic was not flagged as it came into Windstream's network I could see their equipment not prioritizing VOIP over other traffic.
Seems oddly specific blaming Call of Duty, when it's far from the only game with huge patches, and far from the only massive game that launched late last year where patches were being delivered almost daily. Red Dead Redemption II on the PC being one other major source of bugs.
But, the size of the patch doesn't directly affect bandwidth, which it sounds like they lacked. The thing that affects bandwidth is the number of users attempting to download the patch. Larger size only prolongs the effect..
The FT has a recent article on "Broadband can cope with surge in home work, says BT". It's interesting to note in there the analysis of a recent peak in traffic on the BT network, perhaps some unfortunate coincidences in timing:
"In an interview with the Financial Times, he [Howard Watson] said that BT did experience a record amount of traffic on its network on Tuesday night last week, when an update to the online video game Red Dead Redemption 2 and the release of Call of Duty: Warzone — a battle royale-style online game — coincided with Champions League football matches being played, including Tottenham Hotspur’s defeat to RB Leipzig."
I need to read up a bit more on the practical reality for end-to-end IP QoS. It's rather tragic that ultra low bandwidth but loss/jitter sensitive services like voice can be affected.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
